[05:05:34] Z [08:52:59] 10netops, 06Infrastructure-Foundations: HE transport eqiad-eqsin packet loss - october 2026 #2 or #3 ? - https://phabricator.wikimedia.org/T442591#12483112 (10ayounsi) Sent https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1359754 for the repool. @wiki_willy issue lasted 2h50min, do you think it's wo... [09:36:12] could i ask for a blessing on this tweak? https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1359259 [09:51:36] 10netops, 06Infrastructure-Foundations: HE transport eqiad-eqsin packet loss - october 2026 #2 or #3 ? - https://phabricator.wikimedia.org/T442591#12483296 (10ayounsi) a:05ayounsi→03wiki_willy Circuit is repooled, feel free to close the task at your discretion. [10:14:03] i merged the change and it solved the hummingbird access problem (jwt related) [10:49:25] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12483588 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin2003 for host lvs5005.eqsin.wmnet with OS trixie [10:51:27] 10netops, 06Infrastructure-Foundations: Setup cr3-esams - https://phabricator.wikimedia.org/T442691 (10ayounsi) 03NEW p:05Triage→03Medium [10:55:36] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12483604 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cd2fa008-95d5-4a9d-8... [10:58:23] 10netops, 06Infrastructure-Foundations: Setup cr3-esams - https://phabricator.wikimedia.org/T442691#12483625 (10ayounsi) [11:19:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12483741 (10cmooney) [11:20:51] dpogorzelski: as mentioned yesterday, we don't own rest-gateway, you should be looking for reviews there and not here [11:49:15] FIRING: [26x] ProbeDown: Service ncredir1002:443 has failed probes (tcp_ncredir_non_canonical_redirect_12_ip4) - https://wikitech.wikimedia.org/wiki/Ncredir - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:49:30] FIRING: [44x] ProbeDown: Service ncredir1001:443 has failed probes (tcp_ncredir_non_canonical_redirect_12_ip4) - https://wikitech.wikimedia.org/wiki/Ncredir - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:58:58] ncredir is depooled in eqiad.. not a big deal [12:00:51] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12483866 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin2003 for host lvs5005.eqsin.wmnet with OS trixie completed: - lvs5005 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled... [12:02:04] RESOLVED: [152x] ProbeDown: Service ncredir1001:443 has failed probes (tcp_ncredir_non_canonical_redirect_10_ip4) - https://wikitech.wikimedia.org/wiki/Ncredir - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:41:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12484047 (10cmooney) [12:44:41] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705 (10LSobanski) 03NEW [13:14:26] 06Traffic, 10Citoid, 06Editing-team, 10RESTBase-Sunsetting, and 5 others: Switch from restbase to rest-gateway for Citoid - https://phabricator.wikimedia.org/T361576#12484234 (10Jdforrester-WMF) [13:19:17] 06Traffic, 10Citoid, 06Editing-team, 10RESTBase-Sunsetting, and 5 others: Switch from restbase to rest-gateway for Citoid - https://phabricator.wikimedia.org/T361576#12484262 (10Jdforrester-WMF) [13:22:09] 06Traffic, 10Citoid, 06Editing-team, 10RESTBase-Sunsetting, and 5 others: Switch from restbase to rest-gateway for Citoid - https://phabricator.wikimedia.org/T361576#12484285 (10Jdforrester-WMF) For the "move docs" step, we nowadays have https://www.mediawiki.org/w/index.php?api=wmf-rest%2Fv1&title=Special... [13:38:45] 06Traffic, 07Sustainability (Incident Followup): Investigate the lacking data from ATS (the hole in the 50X errors) - https://phabricator.wikimedia.org/T442711#12484368 (10taavi) [13:38:55] 06Traffic: Package, prepare the upgrade, and deploy ATS 10 in production - https://phabricator.wikimedia.org/T396611#12484371 (10dkertesz) I took an initial stab at the packaging of 10.2.0 (and later 10.2.1): https://gitlab.wikimedia.org/dkertesz/trafficserver/-/commits/trixie-wikimedia-10.2.0 The summary of th... [13:39:15] 06Traffic: Package, prepare the upgrade, and deploy ATS 10 in production - https://phabricator.wikimedia.org/T396611#12484374 (10dkertesz) a:03dkertesz [14:21:50] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12484648 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin2003 for host lvs5004.eqsin.wmnet with OS trixie [14:22:02] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12484650 (10Vgutierrez) [14:38:25] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12484785 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin2003 for host lvs5004.eqsin.wmnet with OS trixie executed with errors: - lvs5004 (**FAIL**) - Downtimed on Icinga/Alertmanager... [14:40:00] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12484787 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin2003 for host lvs5004.eqsin.wmnet with OS trixie [14:40:12] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12484789 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=bb0696e8-7843-4... [14:51:08] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12484844 (10Joe) A question to answer is: why was this done before the switchover of all services to codfw that would've happened a few hours later? Or - be... [15:18:26] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12484989 (10cmooney) [15:28:44] 14Varnish, 06Fundraising-Backlog, 10FY26-27 ST6.4 - Logged-in performance, 10MediaWiki-extensions-CentralNotice, and 4 others: Special:HideBanners is not really cacheable - https://phabricator.wikimedia.org/T256447#12485032 (10Krinkle) [15:56:47] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12485200 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin2003 for host lvs5004.eqsin.wmnet with OS trixie completed: - lvs5004 (**PASS**) - Removed from Puppet and PuppetDB if present a... [16:20:13] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12485395 (10Vgutierrez) [16:23:56] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12485418 (10cmooney) Thanks for the task @LSobanski To confirm what happened here I tried to be "smart" and deviated from my steps as set out in the task l... [16:44:50] 06Traffic: Stop using the deprecated kafka Events channel on purged - https://phabricator.wikimedia.org/T442736 (10Vgutierrez) 03NEW [17:19:08] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12485760 (10cmooney) >>! In T442705#12484844, @Joe wrote: > A question to answer is: why was this done before the switchover of all services to codfw that wo... [18:47:34] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12486195 (10cmooney) [18:47:35] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12486194 (10cmooney) [19:41:40] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12486364 (10cmooney) [19:56:54] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12486454 (10cmooney) 05Open→03Resolved All done with this one. Unfortunately interr... [23:35:31] 10netops, 06Infrastructure-Foundations: HE transport eqiad-eqsin packet loss - october 2026 #2 or #3 ? - https://phabricator.wikimedia.org/T442591#12487322 (10wiki_willy) Thanks, I'll go ahead and request the credit. Did they provide you a root cause for the issue?