[00:33:04] swfrench-wmf: yeah that's fine, thanks for checking. we will take care of the pybal and liberica restarts, please let usk now when you start here. thanks [00:34:36] sukhe: great, thank you very much. I'll aim to follow up here during my morning tomorrow once we're ready to get started. [00:35:24] ... aaaand thank you for the reviews <3 [01:12:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12161475 (10Papaul) [08:06:54] 10netops, 06Infrastructure-Foundations, 06SRE: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749#12161827 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8089257c-9548-4d05-a03b-bdb3c1224c47) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their servi... [09:10:59] 10netops, 06Infrastructure-Foundations, 06SRE: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749#12162043 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=5c71b24d-45b0-4c80-bf42-231dd7bab42a) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their servi... [09:29:03] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12162102 (10ayounsi) [09:29:10] 10netops, 06Infrastructure-Foundations, 06SRE: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749#12162103 (10ayounsi) [09:29:24] 10netops, 06Infrastructure-Foundations, 06SRE: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749#12162106 (10ayounsi) 05Open→03Resolved a:03ayounsi All done. [12:30:47] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12162617 (10Papaul) [12:31:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12162620 (10Papaul) [12:33:13] 06Traffic, 06Data-Persistence, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Data Lake - mediarequests - adapt HQL query to include counting mediarequests from webrequest_text - https://phabricator.wikimedia.org/T431475#12162623 (10JAllemandou) a:05Ahoelzl→03JAllemandou [12:33:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12162627 (10Papaul) First homer run is getting me this error ` ERROR:homer:Device asw1-603-eqsin.mgmt.eqsin.wmnet failed to render the template, skipping.... [13:00:44] hello traffic friends - I am back and ready to get started on depooling client traffic from etcd in codfw for T428495. let me know when would be a good time to coordinate. [13:00:46] T428495: Migrate conf* hosts away from bullseye - https://phabricator.wikimedia.org/T428495 [13:03:49] 06Traffic, 06Security-Team, 10WMF-General-or-Unknown, 07ContentSecurityPolicy, and 2 others: Add restrictive CSP to upload.wikimedia.org - https://phabricator.wikimedia.org/T117618#12162770 (10Rsilvola) [13:03:56] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, and 2 others: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12162771 (10Papaul) [13:08:55] swfrench-wmf: happy to start now [13:10:44] sukhe: great, thank you very much. I'll give a heads-up in -sre momentarily and then get started with the DNS side of things. [13:11:54] thanks, I can take care of the pybal and liberica restarts [13:13:50] thank you. once the SRV record change is live, I'll let you know which libericas are still connected to the codfw hosts. [13:14:04] that should be eqsin and ulsfo I think? [13:14:41] yup! I just meant that frequently it's not *all* of them, in case I could save you the work of hitting each one :) [13:14:58] ah true [13:15:14] swfrench-wmf: no worries, I am riding high on the caffeine for now, can restart all :P [13:15:22] :) [13:36:35] sukhe: DNS change is done and confd restarts are in progress. you should be good to proceed with the PyBal and liberica restarts whenever you'd like. let me know if you'd like me to merge [0] and trigger puppet on the codfw lvs hosts. [13:36:35] [0] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1314076 [13:37:51] swfrench-wmf: I can take care of that, thanks [13:37:53] on it [13:43:29] amazing, thank you [13:44:21] in case it's useful, https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Move_PyBals now has an example command for the liberica restarts [13:44:37] whoops, I mean https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Move_other_clients [13:45:00] wooo nice [13:47:03] moving on to the libericas [13:51:59] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431751#12163043 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a2dbde41-3fc7-4120-8a34-b7696ca56502) set by ayounsi@cumin1003 for 1:00:00 on 3 host(s) and their servi... [14:01:19] swfrench-wmf: all done. thanks for all the patches! [14:01:32] sukhe: amazing, thank you very much! [14:03:39] oh, one thing I noticed while checking for lingering connections: it looks like some of the IPs associated with eqsin and ulsfo LVS hosts don't have PTR records (e.g., 10.128.1.18). just wanted to mention that in case it's not expected [14:12:11] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431751#12163147 (10ayounsi) 05Open→03Resolved a:03ayounsi Upgraded. [14:13:04] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12163158 (10ayounsi) [14:18:33] swfrench-wmf: thanks, will share more. meeting [15:14:52] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 5 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#12163523 (10Krinkle) [16:37:09] swfrench-wmf: yeah that's interesting thanks. for codfw it is somewhat "expected" T431324 but not for eqsin and ulsfo where we have moved to Liberica [16:37:10] T431324: Create PTR records for codfw LVS interfaces - https://phabricator.wikimedia.org/T431324 [16:37:16] I will file a task [16:37:25] (after checking what all are we missing) [16:58:00] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#12164226 (10RobH) Replacement work scheduled for Tuesday, August 4th @ 00:00 GMT (08:00 Singapore). I'll depool and power down this single host when the engineer arrives on-site and re-pool it once t... [17:01:26] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#12164263 (10ssingh) >>! In T414411#12164226, @RobH wrote: > Replacement CPU work scheduled for Tuesday, August 4th @ 00:00 GMT (08:00 Singapore). I'll depool and power down this single host when the... [17:02:11] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#12164276 (10ssingh) Additionally, feel free to leave the repooling to us so that we can check everything is OK on the software side of things. You can verify the hardware for us. Thanks! [17:43:52] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#12164492 (10RobH) >>! In T414411#12164276, @ssingh wrote: > Additionally, feel free to leave the repooling to us so that we can check everything is OK on the software side of things. You can verify th... [19:06:36] 06Traffic, 13Patch-For-Review, 07Sustainability (Incident Followup): Experiment with single backend CDN nodes - https://phabricator.wikimedia.org/T288106#12164728 (10BCornwall) Here's the results of 7 days of a text node (cp3073): [[ https://grafana-rw.wikimedia.org/goto/dfthz76mjrrpcc?orgId=default | Netwo... [19:28:49] swfrench-wmf, sukhe: https://gerrit.wikimedia.org/r/c/operations/dns/+/1318769 [19:29:24] some oversight when working on the upgrade to the per rack vlans [19:30:02] ah, great! thank you :) dropping into a meeting, but will take a look [19:34:32] ah thank you XioNoX! [19:34:54] looking [19:54:40] FIRING: [2x] VarnishHighThreadCount: cp3069:0's Varnish thread count nearly maxed out - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [19:59:40] FIRING: [7x] VarnishHighThreadCount: cp3069:0's Varnish thread count nearly maxed out - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:09:40] RESOLVED: [7x] VarnishHighThreadCount: cp3069:0's Varnish thread count nearly maxed out - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount