[02:02:08] 10netops, 06Traffic, 06Infrastructure-Foundations: POPs LVS : remove public vlan trunking - https://phabricator.wikimedia.org/T367732#11910691 (10Papaul) 05Open→03Resolved @ayounsi we can close this task. [02:03:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#11910693 (10Papaul) [02:06:51] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#11910694 (10Papaul) [06:17:58] 10netops, 06Infrastructure-Foundations: POPs - free up 2xQSFP ports - https://phabricator.wikimedia.org/T424611#11910890 (10ayounsi) OK, that sounds good to me! Thanks [10:32:59] 06Traffic: Wikimedia Hackathon 2026: Wikimedia's Production DNS Infrastructure and GeoDNS User Routing - https://phabricator.wikimedia.org/T423331#11911680 (10Aklapper) Thanks for participating in the Wikimedia Hackathon 2026! We hope you had a great time. * If this session took place: Please change the task st... [10:50:57] 10netops, 06Infrastructure-Foundations: POPs - free up 2xQSFP ports - https://phabricator.wikimedia.org/T424611#11911725 (10cmooney) Agh hit a bit of a hiccup with this (really should have anticipated). Take drmrs for example: ` cmooney@asw1-b12-drmrs> show configuration interfaces et-0/0/48 description "Cor... [13:16:30] 06Traffic: Wikimedia Hackathon 2026: Wikimedia's Production DNS Infrastructure and GeoDNS User Routing - https://phabricator.wikimedia.org/T423331#11912530 (10ssingh) 05Open→03Declined [13:28:29] 07HTTPS, 06Traffic: Firefox: Be careful. Something doesn’t look right (Strict Transport Security) - https://phabricator.wikimedia.org/T425120#11912578 (10ssingh) Thanks for the report @Ponor. Does this issue still persist? If yes, can you please undertake the steps in https://wikitech.wikimedia.org/wiki/Report... [13:32:07] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Revert lvs1017 Mellanox NIC to Broadcom - https://phabricator.wikimedia.org/T421421#11912612 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host lvs1017.eqiad.wmnet with OS bullseye [13:49:57] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Revert lvs1017 Mellanox NIC to Broadcom - https://phabricator.wikimedia.org/T421421#11912751 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host lvs1017.eqiad.wmnet with OS bullseye executed with errors: - lvs1017 (**FAIL*... [14:08:27] \o We (ML Team) are currently working adding another service based on gRPC. Since it can't co-habitate with HTTP/s on 443 (AFAIK), we'll probably need some LVS config changes: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1282328 Looking for a reviewer and shepherd for deployment. The change is owned by Dawid, who will be back next week, so not super urgent. [14:08:38] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Revert lvs1017 Mellanox NIC to Broadcom - https://phabricator.wikimedia.org/T421421#11912829 (10ssingh) @VRiley-WMF: As John correctly pointed out, this is booting with UEFI enabled now. Is that expected and the default for all hosts now? If that is the case, we can... [14:40:52] 07HTTPS, 06Traffic: Firefox: Be careful. Something doesn’t look right (Strict Transport Security) - https://phabricator.wikimedia.org/T425120#11913059 (10Ponor) @ssingh Thanks. This has been an issue for some 10 days, but I haven't noticed it recently. I see that FF got a little update, so it's hard to tell w... [15:43:15] 06Traffic, 06Product Safety and Integrity, 06Security-Team, 072026-user-javascript-incident, and 4 others: Deduplicate CSP between VCL and MediaWiki - https://phabricator.wikimedia.org/T420604#11913458 (10BCornwall) Sorry to necro but are we interested in deduplication of CSP in upload as well? T117618... [15:46:44] 06Traffic, 06Product Safety and Integrity, 06Security-Team, 072026-user-javascript-incident, and 4 others: Deduplicate CSP between VCL and MediaWiki - https://phabricator.wikimedia.org/T420604#11913492 (10sbassett) >>! In T420604#11913458, @BCornwall wrote: > Sorry to necro but are we interested in ded... [15:49:07] 07HTTPS, 06Traffic: Firefox: Be careful. Something doesn’t look right (Strict Transport Security) - https://phabricator.wikimedia.org/T425120#11913502 (10Ponor) 05Open→03Resolved a:03Ponor [15:58:32] klausman: yes, sorry for the late response, we are happy to help [15:58:55] when are you planning it? next week is fine, let us know a time [15:59:14] Next week sounds good, as by then Dawid will be back. [15:59:41] ok [16:00:16] (plus, with Thu being a public holiday in many places, tomorrow is basically Friday and the change is a bit heavy for that kind of lead time [16:00:50] yeah :) [16:40:31] 07HTTPS, 06Traffic: Firefox: Be careful. Something doesn’t look right (Strict Transport Security) - https://phabricator.wikimedia.org/T425120#11913885 (10ssingh) >>! In T425120#11913059, @Ponor wrote: > @ssingh > Thanks. This has been an issue for some 10 days, but I haven't noticed it recently. I see tha... [17:47:45] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Revert lvs1017 Mellanox NIC to Broadcom - https://phabricator.wikimedia.org/T421421#11914178 (10BCornwall) I took a quick look and see that @VRiley-WMF seems to have run: ` $ sudo secure-cookbook sre.hosts.provision lvs1017 --no-user --no-dhcp ` That command shoul... [17:48:39] 10netops, 06Infrastructure-Foundations: ulsfo: upgrade routers (2026) - https://phabricator.wikimedia.org/T416562#11914183 (10Papaul) @ssingh and team now that we are done with the switch refresh and everything is stable in ulsfo and after we connect the missing link between cr3 and asw1-23 We will like to sch... [17:50:58] 10netops, 06Infrastructure-Foundations: ulsfo: upgrade routers (2026) - https://phabricator.wikimedia.org/T416562#11914192 (10ssingh) >>! In T416562#11914182, @Papaul wrote: > @ssingh and team now that we are done with the switch refresh and everything is stable in ulsfo and after we connect the missing link b... [17:53:31] 10netops, 06Infrastructure-Foundations: ulsfo: upgrade routers (2026) - https://phabricator.wikimedia.org/T416562#11914213 (10Papaul) @ssingh i think it will be best to depool the site since this will be my first time doing the draining process I will like to be on the safe side. [17:54:09] 10netops, 06Infrastructure-Foundations: ulsfo: upgrade routers (2026) - https://phabricator.wikimedia.org/T416562#11914214 (10ssingh) >>! In T416562#11914213, @Papaul wrote: > @ssingh i think it will be best to depool the site since this will be my first time doing the draining process I will like to be on the... [18:05:12] 06Traffic: Reboot lvs1019 for memory self-healing - https://phabricator.wikimedia.org/T426109 (10BCornwall) 03NEW [18:05:28] 06Traffic: Reboot lvs1019 for memory self-healing - https://phabricator.wikimedia.org/T426109#11914271 (10BCornwall) p:05Triage→03Medium [18:05:50] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Revert lvs1017 Mellanox NIC to Broadcom - https://phabricator.wikimedia.org/T421421#11914273 (10Jclark-ctr) >>! In T421421#11914178, @BCornwall wrote: > I took a quick look and see that @VRiley-WMF seems to have run: > > ` > $ sudo secure-cookbook sre.hosts.provisi... [18:07:35] 06Traffic: Reboot lvs1019 for memory self-healing - https://phabricator.wikimedia.org/T426109#11914286 (10BCornwall) [18:08:29] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Revert lvs1017 Mellanox NIC to Broadcom - https://phabricator.wikimedia.org/T421421#11914303 (10BCornwall) @Jclark-ctr Might wanna update https://wikitech.wikimedia.org/wiki/UEFI_Boot#Reconfigure_the_server_to_boot_via_UEFI then :) [18:10:49] 06Traffic: Reboot lvs1019 for memory self-healing - https://phabricator.wikimedia.org/T426109#11914336 (10BCornwall) [18:54:18] 10netops, 06Infrastructure-Foundations: ulsfo: upgrade routers (2026) - https://phabricator.wikimedia.org/T416562#11914466 (10ssingh) Confirmed with @Papaul that this is actually meant for May 20, same time. [19:00:35] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE: Degraded RAID on lvs2012 - https://phabricator.wikimedia.org/T425890#11914494 (10ssingh) 05Open→03Resolved a:03ssingh Things look good and lvs2012 is happily serving traffic. Marking as resolved, thanks @Jhancock.wm! [20:15:09] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Revert lvs1017 Mellanox NIC to Broadcom - https://phabricator.wikimedia.org/T421421#11914724 (10BCornwall) [20:42:31] brett: are you around for a quick review maybe? [21:09:58] topranks: yepyep [21:10:21] brett: hey thanks dude [21:10:26] I assume it's https://gerrit.wikimedia.org/r/c/operations/dns/+/1286501 - looking now [21:10:34] yeah, one of those awful reverse dns ones [21:11:02] machine generated for the most part, and either way the CI is very robust on the repo, and errors on the includes it usually gives out [21:11:05] thanks! [21:16:09] still looking... [21:20:35] topranks: woof, painful. Thanks! [21:20:40] +1 [21:21:12] brett: thanks! [21:21:27] ah I long since scripted it all up - well the v6 ones - cos just too finnicky otherwise [21:22:22] gotcha [21:22:30] I've most of the work done on T362985 which would remove the need for it, just got to get some time to get back to it [21:22:31] T362985: Improve how we generate DNS entries from Netbox - https://phabricator.wikimedia.org/T362985 [21:37:48] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#11914980 (10RobH) Scheduled a new site visit for them to go out this Friday @ 8AM Singapore Time so my Thursday @ 4PM. [22:37:47] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Revert lvs1017 Mellanox NIC to Broadcom - https://phabricator.wikimedia.org/T421421#11915171 (10BCornwall) [22:38:00] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Revert lvs1017 Mellanox NIC to Broadcom - https://phabricator.wikimedia.org/T421421#11915172 (10BCornwall) [22:59:08] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Revert lvs1017 Mellanox NIC to Broadcom - https://phabricator.wikimedia.org/T421421#11915233 (10BCornwall) @ssingh The group of patches I've uploaded add an extra step of adding lvs1017 to the end of the list and a lower priority - much like ho...