[01:52:44] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 06SRE: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097#11212641 (10Krinkle) >>! In T122097#2657531, @BBlack wrote: > This has been idle a while, but it's still probably a good... [07:43:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [07:58:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [09:33:06] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia: add new switches in eqiad/codfw to monitoring and make 'active' - https://phabricator.wikimedia.org/T405558 (10cmooney) 03NEW p:05Triage→03Medium [09:33:40] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia: add new switches in eqiad/codfw to monitoring and make 'active' - https://phabricator.wikimedia.org/T405558#11213293 (10cmooney) [09:41:35] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560 (10cmooney) 03NEW [09:41:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11213333 (10cmooney) [09:44:37] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad C/D refresh: move legacy switch uplinks to Nokias and migrate Vlan GWs - https://phabricator.wikimedia.org/T405562 (10cmooney) 03NEW p:05Triage→03Medium [09:44:50] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad C/D refresh: move legacy switch uplinks to Nokias and migrate Vlan GWs - https://phabricator.wikimedia.org/T405562#11213369 (10cmooney) [09:50:06] 06Traffic, 06MediaWiki-Platform-Team, 06Reader Experience Team, 10MobileFrontend (Core PHP): Toggling desktop view doesn't toggle user back into mobile mode - https://phabricator.wikimedia.org/T403866#11213405 (10Jdlrobson-WMF) I can confirm the fix on my side! FWIW this is actually a delightful new featu... [09:52:23] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad C/D refresh: move legacy switch uplinks to Nokias and migrate Vlan GWs - https://phabricator.wikimedia.org/T405562#11213413 (10cmooney) [12:10:15] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad C/D refresh: move legacy switch uplinks to Nokias and migrate Vlan GWs - https://phabricator.wikimedia.org/T405562#11214003 (10cmooney) [12:34:28] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad C/D refresh: move legacy switch uplinks to Nokias and migrate Vlan GWs - https://phabricator.wikimedia.org/T405562#11214082 (10cmooney) @Jclark-ctr @VRiley-WMF I may have missed to check we have the cables needed for these already. We're re-using exsiting... [13:06:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [13:08:31] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 06SRE: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097#11214225 (10Tgr) >>! In T122097#11212641, @Krinkle wrote: > setting/changing a cookie is equivalent to discarding the br... [13:11:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [14:22:26] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11214627 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host durum5001.eqsin.wmnet with OS trixie [14:22:34] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 06SRE: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097#11214629 (10Krinkle) >>! In T122097#11214225, @Tgr wrote: >>>! In T122097#11212641, @Krinkle wrote: >> setting/changing... [14:23:07] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11214643 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host durum7003.magru.wmnet with OS trixie [14:33:50] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602 (10cmooney) 03NEW p:05Triage→03Medium [14:36:40] 06Traffic: Move HTTP/1.0 requests rejections at HAProxy level - https://phabricator.wikimedia.org/T365456#11214754 (10Fabfur) 05Open→03Resolved Done rejecting all HTTP_1.0 requests [14:48:17] 06Traffic: Move HTTP/1.0 requests rejections at HAProxy level - https://phabricator.wikimedia.org/T365456#11214823 (10Fabfur) 05Resolved→03In progress [14:48:48] 06Traffic: Move HTTP/1.0 requests rejections at HAProxy level - https://phabricator.wikimedia.org/T365456#11214828 (10Fabfur) This has been reverted due to issues with load balancers checks [14:49:28] 06Traffic: Move HTTP/1.0 requests rejections at HAProxy level - https://phabricator.wikimedia.org/T365456#11214829 (10Fabfur) [14:59:02] Do we strip double / from URI paths for other paths than upload? I found modules/varnish/templates/upload-frontend.inc.vcl.erb that does it but only for the upload cluster right [15:00:25] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602#11214910 (10cmooney) [15:02:22] probably not, because it's hard to get that right with escaping and the URI parts-parsing and parameters, etc... [15:02:32] upload is a more-constrained environment [15:03:14] yeah we strip away query parameters for example in upload [15:03:29] but also not that I knew this before, we don't seem to be doing it in text, no (looking at the VCL) [15:04:05] all normalization that we can legally do is a Good Thing to pursue, because it reduces cache fragmentation, etc [15:04:25] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11214932 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host durum7003.magru.wmnet with OS trixie executed with errors: - durum7003 (**FAIL**) - Downtimed on Icinga/Alertmanager... [15:04:35] but fully-generalized normalization is nearly-intractable. it's hard not to break things if you don't understand every details of every application layer's full URI/parameter space [15:05:17] the double-slash thing could be done, carefully, on text, though. I think. [15:05:35] maybe after encoding-normalization, and being very careful not to touch the params/frag part of the string [15:11:03] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609 (10cmooney) 03NEW p:05Triage→03Medium [15:11:23] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602#11214968 (10cmooney) [15:11:24] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11214967 (10cmooney) [15:35:24] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11215097 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host durum7003.magru.wmnet with OS bookworm [15:41:53] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11215146 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host durum5001.eqsin.wmnet with OS trixie completed: - durum5001 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled... [16:02:45] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618 (10Papaul) 03NEW [16:22:12] 06Traffic: ncmonitor should verify that DNSSEC is disabled in MarkMonitor - https://phabricator.wikimedia.org/T402961#11215321 (10BCornwall) [16:22:15] 06Traffic: ncmonitor should check real-world NS records - https://phabricator.wikimedia.org/T402960#11215322 (10BCornwall) [16:23:12] 06Traffic: ncmonitor: Initialize tld suffix list on resource creation - https://phabricator.wikimedia.org/T372103#11215337 (10BCornwall) 05Open→03Declined Such a small issue, not really worth keeping around unless we're actively going to service. [16:26:09] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11215354 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host durum7003.magru.wmnet with OS bookworm executed with errors: - durum7003 (**FAIL**) - Removed from Puppet and PuppetDB... [16:31:20] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11215392 (10cmooney) [16:52:04] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Traffic host migrations - https://phabricator.wikimedia.org/T405623 (10RobH) 03NEW [16:56:59] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Traffic host migrations - https://phabricator.wikimedia.org/T405623#11215549 (10RobH) @BCornwall, Congrats, since we've worked together on so many other projects previously I made the #traffic team's host migration tracking task first! As such, we ma... [16:57:46] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Traffic host migrations - https://phabricator.wikimedia.org/T405623#11215552 (10RobH) [17:04:23] 06Traffic, 10Phabricator (Upstream), 06Release-Engineering-Team (Priority Backlog 📥), 07Upstream: Use preconnect for https://phab.wmfusercontent.org CDN - https://phabricator.wikimedia.org/T367290#11215648 (10Aklapper) 05Stalled→03Resolved This issue should now be fixed on phabricator.wikimedia.org... [17:18:32] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628 (10cmooney) 03NEW p:05Triage→03Medium [17:18:53] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11215779 (10cmooney) [17:18:56] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602#11215780 (10cmooney) [17:19:41] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11215792 (10cmooney) [17:25:47] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: lvs1020: reimage to move primary IP from private1-d-eqiad to private1-d7-eqiad vlan - https://phabricator.wikimedia.org/T405630 (10cmooney) 03NEW p:05Triage→03Medium [17:26:02] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: lvs1020: reimage to move primary IP from private1-d-eqiad to private1-d7-eqiad vlan - https://phabricator.wikimedia.org/T405630#11215830 (10cmooney) [17:26:04] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602#11215831 (10cmooney) [17:27:00] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: Tidy up lvs1018 L2 link to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T405499#11215832 (10cmooney) [17:27:02] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602#11215833 (10cmooney) [17:33:08] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1020: reimage to move primary IP from private1-c-eqiad to private1-c7-eqiad vlan - https://phabricator.wikimedia.org/T405632 (10cmooney) 03NEW p:05Triage→03Medium [17:33:22] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1020: reimage to move primary IP from private1-c-eqiad to private1-c7-eqiad vlan - https://phabricator.wikimedia.org/T405632#11215881 (10cmooney) [17:33:24] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602#11215882 (10cmooney) [17:37:09] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: reimage to move primary IP from private1-c-eqiad to private1-c7-eqiad vlan - https://phabricator.wikimedia.org/T405632#11215886 (10cmooney) [17:39:08] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: lvs1020: reimage to move primary IP from private1-d-eqiad to private1-d7-eqiad vlan - https://phabricator.wikimedia.org/T405630#11215905 (10cmooney) [17:40:15] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: lvs1020: reimage to move primary IP from private1-d-eqiad to private1-d7-eqiad vlan - https://phabricator.wikimedia.org/T405630#11215913 (10cmooney) [17:41:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad: new structured cabling required for fr-tech expansion and row a/b switch refresh - https://phabricator.wikimedia.org/T402432#11215927 (10cmooney) [17:42:34] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602#11215944 (10cmooney) [17:42:37] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: Tidy up lvs1018 L2 link to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T405499#11215945 (10cmooney) [17:42:55] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602#11215946 (10cmooney) [17:42:58] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: Tidy up lvs1018 L2 link to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T405499#11215947 (10cmooney) [17:43:21] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: Remove lvs1018 L2 link to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T405499#11215950 (10cmooney) [17:45:10] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Traffic host migrations - https://phabricator.wikimedia.org/T405623#11215953 (10BCornwall) p:05Triage→03Medium [17:49:15] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Netbox: General updates for Nokia switch support - https://phabricator.wikimedia.org/T404146#11215963 (10cmooney) [17:50:01] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Netbox: General updates for Nokia switch support - https://phabricator.wikimedia.org/T404146#11215967 (10cmooney) [17:52:05] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: Use LAG interface MAC address field to store LACP system-id for MC-LAG - https://phabricator.wikimedia.org/T392056#11215973 (10cmooney) 05Open→03Declined Gonna close this one for now. Doing it in our YAML data for the occasional virtual-chassis... [17:56:34] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: Update server provision script to support Nokia switches - https://phabricator.wikimedia.org/T405637 (10cmooney) 03NEW p:05Triage→03Medium [17:57:12] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: Update server provision script to support Nokia switches - https://phabricator.wikimedia.org/T405637#11216041 (10cmooney) [17:57:16] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Netbox: General updates for Nokia switch support - https://phabricator.wikimedia.org/T404146#11216042 (10cmooney) [18:02:29] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: Create script to allow multiple host migrations from old -> new switch - https://phabricator.wikimedia.org/T405640 (10cmooney) 03NEW p:05Triage→03Medium [18:04:16] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: Create script to allow multiple host migrations from old -> new switch - https://phabricator.wikimedia.org/T405640#11216198 (10cmooney) [18:04:22] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Netbox: General updates for Nokia switch support - https://phabricator.wikimedia.org/T404146#11216197 (10cmooney) [18:04:51] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: Create script to allow multiple host migrations from old -> new switch - https://phabricator.wikimedia.org/T405640#11216214 (10cmooney) [18:04:54] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Netbox: General updates for Nokia switch support - https://phabricator.wikimedia.org/T404146#11216215 (10cmooney) [18:33:30] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Traffic host migrations - https://phabricator.wikimedia.org/T405623#11216609 (10RobH) a:03BCornwall [18:35:43] 06Traffic, 06MediaWiki-Platform-Team (Radar): Write Hadoop query for progres metric of unified mobile routing metric - https://phabricator.wikimedia.org/T405429#11216639 (10Krinkle) >>! In T405429#11208800, @Krinkle wrote: > […] > * The increase could be true, if there are lots of views on URLs that are unsafe... [20:21:45] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11217124 (10Papaul) [21:50:21] 06Traffic, 10DNS, 06SRE, 06Traffic-Icebox, and 2 others: Many misc wikis lack mobile domains - https://phabricator.wikimedia.org/T152882#11217414 (10Krinkle) [22:31:00] 06Traffic, 06Data-Engineering (Q1 FY25/26 July 1st - September 30th), 06Movement-Insights (FY25-26 H1): NEW BUG REPORT: Investigate rise in May 2025 Reader metrics - https://phabricator.wikimedia.org/T395934#11217591 (10Mayakp.wiki) Backfill is happening in T405667 [22:52:06] 06Traffic, 10DNS, 06SRE, 06Traffic-Icebox, and 2 others: Many misc wikis lack mobile domains - https://phabricator.wikimedia.org/T152882#11217641 (10Krinkle) [23:02:14] 06Traffic, 10DNS, 06MediaWiki-Platform-Team, 06SRE, and 3 others: Many misc wikis lack mobile domains - https://phabricator.wikimedia.org/T152882#11217659 (10Krinkle) I was originally going to enable unified mobile routing on login.wikimedia.org today, as part of the misc wikimedia.org batch at T403510. Ho...