[13:09:37] XioNoX: given this is a one-off test should I allocate a VIP address on netbox for the katran tests? [13:10:15] vgutierrez: sure, doesn't hurt [13:10:31] I can help if needed [13:10:39] 10Traffic, 10Performance-Team, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10MatthewVernon) I think the difficulty relates to the sheer number of thumbnails: ` x=0 for i in $(swift list --prefix wikipedia-c... [13:11:25] XioNoX: https://netbox.wikimedia.org/ipam/prefixes/93/ right there? [13:12:14] are you going to use it with BGP or any routing protocol? [13:12:24] nope [13:12:40] cool, then that works, yeah [13:19:37] XioNoX: stupid question.. adding it manually I don't see the option to set VRF to Global, should I be using a cookbook instead? [13:19:58] vgutierrez: no VRF is global vrf [13:20:56] I'm not sure I understood that correctly.. https://netbox.wikimedia.org/ipam/prefixes/93/ip-addresses/?page=2 shows VRF as Global for all the VIP addresses in 10.2.0.0/24 [13:21:02] *10.2.2.0 [13:22:18] vgutierrez: the UI is not clear, but if you don't specify a VRF when creating an IP it's by default in Global [13:22:25] oh gotcha [13:22:25] so you can ignore the VRF field [13:23:05] https://netbox.wikimedia.org/ipam/ip-addresses/13774/ [13:30:31] vgutierrez: whoa [13:30:34] exciting!! [14:05:37] vgutierrez: would you be willing to enable another dance with the gateway-check script today? We previously tested routing proton and it was lacking etag support - in future we're gonna prefer last-modified in lieu but for now the team think it's good to go https://gerrit.wikimedia.org/r/c/operations/puppet/+/941440 [14:06:02] "but for now" meaning? [14:06:15] etag support, last-modified, none of those two? [14:06:22] etag support for the time being [14:06:33] it's been added since we last tried [14:07:56] hnowlan: let me pull an espresso first [14:07:59] my brain is screaming [14:08:16] sounds like a good idea [14:12:21] back :) [14:12:42] switching context from debugging why a kernel refuses to reply a perfectly valid SYN packet to this hurts a little bit [14:12:48] (spoiler: disable rp_filter) [14:15:30] hnowlan: I just noticed .zero.wikipedia.org on the SNI list of rest-gateway.discovery.mnet [14:16:11] hnowlan: not a big deal but... [14:16:13] T187716 [14:16:14] T187716: Sunset Wikipedia Zero - https://phabricator.wikimedia.org/T187716 [14:16:55] vgutierrez: d'oh. I was copy-pasting from restbase [14:18:06] +1ed the change, I'd suggest disabling puppet on A:cp first [14:18:21] just to check that we are happy with the new endpoints [14:18:27] not because it should be a big deal for ATS itself [14:19:11] sounds good, I'll follow the same path as last time [14:28:57] vgutierrez: change is live on cp2037 - looks okay from a client point of view but I'm seeing `CacheResultCode:ERR_CLIENT_READ_ERROR` in atslog-backend [14:30:09] https://www.irccloud.com/pastebin/ytEjZr1w/ [14:30:19] caching works as expected I'd say [14:30:30] phew [14:31:19] etag is quite different [14:31:32] from 1164821411/e13cff30-1fca-11ee-b29b-691eec2ce164 to "1690372100|en.wikipedia.org-Tornado-a4-desktop" [14:31:52] but as I've said before, it's opaque so I don't care :) [14:32:20] yeah - long story short in restbase we have access to stuff like revision etc, but in proton it's a bit simpler so we just make do [14:32:32] but yeah, as long as it works :) [14:33:07] I'm happy to let'er rip if you are [14:33:12] yep [14:33:20] great, thanks! [15:33:44] 10Traffic, 10Performance-Team, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10MatthewVernon) [back-of-the-envelope suggests a few days, so don't expect immediate answers!] [15:44:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-tools: Add network devices fingerprints to known_hosts - https://phabricator.wikimedia.org/T327643 (10ayounsi) [15:49:15] hnowlan: BTW, what's the evolution of the services behind api-gateway and rest-gateway? [15:49:53] hnowlan: we should expect multiple services behind the same port with different paths or each service is gonna get their own TCP port? [16:01:24] vgutierrez: every service will be on the same port with different paths - but not all services will be on the same gateway [16:02:02] rest-gateway is only for restbase-migrating services. the vision is that rest-gateway will Eventually* be gotten rid of and services can respond directly [16:16:17] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Upgrade new codfw switches to Juniper recommended - https://phabricator.wikimedia.org/T341670 (10cmooney) [16:16:25] 10netops, 10Infrastructure-Foundations, 10SRE: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10cmooney) [16:40:45] 10Traffic, 10Performance-Team, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Krinkle) >>! In T211661#9044465, @MatthewVernon wrote: > […] Out of interest, I went looking at how many of these we served on 24... [21:57:40] 10Traffic, 10AS-Report, 10Performance-Team, 10SRE: Send peering requests to AS with the worst TTFB - https://phabricator.wikimedia.org/T219486 (10Krinkle) [21:58:30] 10Traffic, 10AS-Report, 10Performance-Team, 10SRE: Send peering requests to AS with the worst TTFB - https://phabricator.wikimedia.org/T219486 (10Krinkle)