[06:03:36] 10netops, 06Infrastructure-Foundations: magru HE down (oct 2026) - https://phabricator.wikimedia.org/T441591#12476057 (10ayounsi) 05Open→03Resolved That was quick ! thanks! [08:32:56] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12476421 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs4010.ulsfo.wmnet with OS trixie [09:15:31] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12476548 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs4010.ulsfo.wmnet with OS trixie completed: - lvs4010 (**PASS**) - Downtimed on Icinga/Ale... [09:17:16] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12476562 (10cmooney) [09:57:07] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12476668 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs4009.ulsfo.wmnet with OS trixie [09:59:03] if someone can please approve: [09:59:03] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1357480 [09:59:03] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1357479 [09:59:04] I would appreciate it a lot. these should only affect hummingbird (https://hummingbird.wikimedia.org/) and i'm happy to revert if they don't provide the expected outcome [10:00:07] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12476699 (10Fabfur) 05Open→03Resolved This should be ok now [10:16:58] FIRING: [2x] HaproxyKafkaExporterDown: HaproxyKafka on cp2057 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [10:17:54] 06Traffic: Use go1.27 to compile purged - https://phabricator.wikimedia.org/T442513 (10Vgutierrez) 03NEW [10:18:25] FIRING: SystemdUnitFailed: haproxy.service on cp3066:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:18:33] ^^ reboot [10:19:13] RESOLVED: [2x] HaproxyKafkaExporterDown: HaproxyKafka on cp2057 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [10:19:36] 06Traffic: Use go1.27 to compile purged - https://phabricator.wikimedia.org/T442513#12476776 (10Vgutierrez) p:05Triage→03Medium [10:20:56] dpogorzelski: we don't own https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1357479 and we aren't used to review CRs in there [10:23:25] RESOLVED: SystemdUnitFailed: haproxy.service on cp3066:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:36:35] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12476826 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs4009.ulsfo.wmnet with OS trixie completed: - lvs4009 (**PASS**) - Downtimed on Icinga/Ale... [11:22:22] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12476952 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs4008.ulsfo.wmnet with OS trixie [12:02:41] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12477244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs4008.ulsfo.wmnet with OS trixie completed: - lvs4008 (**PASS**) - Downtimed on Icinga/Ale... [12:22:25] FIRING: [2x] SystemdUnitFailed: haproxy.service on cp3069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:27:25] RESOLVED: [2x] SystemdUnitFailed: haproxy.service on cp3069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:31:24] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12477398 (10Vgutierrez) [12:50:03] vgutierrez: i submitted a followup since this was not triggered immediately after the deployment of the previous changes: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1359259 [14:01:31] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12477957 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs5006.eqsin.wmnet with OS trixie [14:04:07] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12477976 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs5006.eqsin.wmnet with OS trixie executed with errors: - lvs5006 (**FAIL**) - Downtimed on... [14:06:18] 10netops, 06Infrastructure-Foundations, 06SRE, 07SecTeam-Processed, and 2 others: Routinator: CVE-2026-72752 CVE-2026-72753 CVE-2026-72754 CVE-2026-72755 CVE-2026-72756 CVE-2026-72757 CVE-2026-72758 - https://phabricator.wikimedia.org/T442502#12477982 (10sbassett) [14:06:26] 10netops, 06Infrastructure-Foundations, 06SRE, 07SecTeam-Processed, and 2 others: Routinator: CVE-2026-72752 CVE-2026-72753 CVE-2026-72754 CVE-2026-72755 CVE-2026-72756 CVE-2026-72757 CVE-2026-72758 - https://phabricator.wikimedia.org/T442502#12477984 (10sbassett) [14:09:38] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12477992 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs5006.eqsin.wmnet with OS trixie [14:11:13] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12478007 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs5006.eqsin.wmnet with OS trixie executed with errors: - lvs5006 (**FAIL**) - Removed from... [14:20:02] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12478076 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs5006.eqsin.wmnet with OS trixie [14:32:07] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12478192 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs5006.eqsin.wmnet with OS trixie executed with errors: - lvs5006 (**FAIL**) - Removed from... [14:43:07] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12478270 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin2003 for host lvs5006.eqsin.wmnet with OS trixie [15:08:56] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12478575 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin2003 for host lvs5006.eqsin.wmnet with OS trixie executed with errors: - lvs5006 (**FAIL**) - Removed from... [15:13:20] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12478605 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin2003 for host lvs5006.eqsin.wmnet with OS trixie [15:33:37] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12478816 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin2003 for host lvs5006.eqsin.wmnet with OS trixie executed with errors: - lvs5006 (**FAIL**) - Removed from Puppet and PuppetDB i... [15:37:54] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12478821 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin2003 for host lvs5006.eqsin.wmnet with OS trixie [16:08:09] 10Domains, 06Traffic: Transfer wikipedia.pt domain to community - https://phabricator.wikimedia.org/T404913#12478996 (10BCornwall) 05Open→03Stalled [16:10:49] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12479004 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin2003 for host lvs5006.eqsin.wmnet with OS trixie executed with errors: - lvs5006 (**FAIL**) - Removed from Puppet and PuppetDB i... [16:12:47] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12479033 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin2003 for host lvs5006.eqsin.wmnet with OS trixie [16:28:21] 10netops, 06Infrastructure-Foundations: HE transport eqiad-eqsin packet loss - october 2026 #2 or #3 ? - https://phabricator.wikimedia.org/T442591 (10ayounsi) 03NEW [16:48:43] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12479581 (10ayounsi) I asked Claude to have a look at this task, as the issue was alerting on AlertManager and the problem well defined. Its investiga... [16:57:53] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12479708 (10cmooney) Are we sure the issue is DAD? If it is I don't see an issue with the sysctl changes, as you say we statically configure the addr... [17:17:30] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12480028 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin2003 for host lvs5006.eqsin.wmnet with OS trixie completed: - lvs5006 (**PASS**) - Removed from Puppet and PuppetDB if present a... [17:22:39] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12480117 (10ayounsi) Not sure, but I think that's the workaround worth testing, and if it solves that issue, worth rolling out infra wide to shave som... [17:26:42] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12480139 (10Vgutierrez) [18:04:00] 10netops, 06Infrastructure-Foundations: HE transport eqiad-eqsin packet loss - october 2026 #2 or #3 ? - https://phabricator.wikimedia.org/T442591#12480487 (10cmooney) Seems it didn't last long but still {F111022877 width=600} [18:15:41] 06Traffic: dns4003 netbox IP records require manual intervention - https://phabricator.wikimedia.org/T442193#12480529 (10BCornwall) 05Open→03Resolved a:03BCornwall Deleted 198.35.26.7/28 from it, set 198.35.26.7/27 to dns4003.wikimedia.org and it seems like it's no longer showing up in the errors. htt...