[08:14:46] 10netops, 06Infrastructure-Foundations: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#10232660 (10cmooney) >>! In T365697#10231895, @Papaul wrote: > @ayounsi I double check again that all the old IPV6 were removed. But we have the old IPV6 that were still in cache so here ar... [08:34:31] 06Traffic, 06Commons, 10MediaWiki-File-management, 10SRE-swift-storage: Commons' file is inaccessible for some users - https://phabricator.wikimedia.org/T377202#10232734 (10MatthewVernon) It's not in general a consistent problem, no - mostly MW manages to successfully upload two copies (one to eqiad, one t... [08:54:17] 06Traffic, 06Commons, 10MediaWiki-File-management, 10SRE-swift-storage: Commons' file is inaccessible for some users - https://phabricator.wikimedia.org/T377202#10232792 (10TheDJ) Maybe the filerepo layer should actively confirm if the write happened to both servers, and kick of a retry job or something if... [13:14:08] 10netops, 06Infrastructure-Foundations: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#10233442 (10Papaul) @cmooney the only issue was Arelion sent us 2 emails saying : "We still see IPV6 neighbor solicitation for the old IPV6 addresses operating in parallel with the new" and... [13:16:37] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack pfw3 and old fasw decommission - https://phabricator.wikimedia.org/T377254#10233448 (10Papaul) [14:28:04] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr1-eqiad: disk failure - https://phabricator.wikimedia.org/T372781#10233790 (10VRiley-WMF) Just wanted to ask, if the disk is showing that it's back, should we still keep this ticket open? Or, is it safe to close? [14:51:31] hello Traffic friends - I have two non-urgent ATS-related questions: [14:51:31] 1. who would be the right person to review of ATS Lua script changes? [14:51:31] 2. when ATS establishes a TLS connection to the backend, cert verification is based on the final value of the Host header, rather than, e.g., the address we may have remapped to, correct? [14:51:31] (for #2, this is why we can remap requests to mw-web-ro.discovery.wmnet, which does not appear in the SAN list presented by mw-web, for example) [14:53:14] swfrench-wmf: I can review those [14:53:28] vgutierrez: great, thank you! [14:53:28] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: cloudgw: add support and enable IPv6 - https://phabricator.wikimedia.org/T374716#10233918 (10aborrero) 05Open→03Resolved We got it all working on 2024-10-11. [14:53:58] swfrench-wmf: ATS expects to find the Host header used by the user on the SAN list of the backend certificate [14:54:28] swfrench-wmf: so if the UA is requesting en.wikipedia.org, that needs to be a valid SAN/SNI on the backend certficate [14:56:25] that's why mw-web certificate has all the SANs included in our unified cert [14:56:47] https://www.irccloud.com/pastebin/OdMLwFDz/ [14:57:33] vgutierrez: thank you very much for confirming! this matches what I'd figured must be the case based on what I can see, but wanted to check with you folks as the experts :) [15:09:50] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: CloudVPS: IPv6 in codfw1dev - https://phabricator.wikimedia.org/T245495#10234047 (10aborrero) [15:56:04] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10234300 (10cmooney) [15:56:13] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: work out IPv6 and designate integration - https://phabricator.wikimedia.org/T374715#10234301 (10cmooney) [16:57:16] low priority patch to add an ATS mapping if anyone has time to check: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1079361 [17:58:34] 10netops, 06Infrastructure-Foundations, 06SRE: Re-IP hosts on codfw row A and B to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T354869#10235144 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host phab2002.codfw.wmnet with OS bullseye [18:56:32] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: eqiad:frack network design, installation and configuration - https://phabricator.wikimedia.org/T377381 (10cmooney) 03NEW p:05Triage→03Medium [18:56:45] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: eqiad:frack network design, installation and configuration - https://phabricator.wikimedia.org/T377381#10235421 (10cmooney) [18:56:46] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Q1:eqiad:frack network upgrade tracking task - https://phabricator.wikimedia.org/T371435#10235422 (10cmooney) [19:49:38] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10235538 (10cmooney) [19:54:23] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10235556 (10cmooney) @jclark-ctr I think existing stock was used for the 100G links between the switches in... [19:57:26] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10235573 (10cmooney) [20:02:44] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10235582 (10cmooney) [21:57:50] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10236007 (10cmooney) We should also use SFP28 modules for the 'fabric-link' between the two pfw's if possib... [23:12:30] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10236211 (10cmooney) @RobH so looking at the options after discussion I think we need to do an fs.com (or a...