[06:39:39] 10netops, 06Infrastructure-Foundations: HE transport eqiad-eqsin packet loss - october 2026 #2 or #3 ? - https://phabricator.wikimedia.org/T442591#12487905 (10ayounsi) > We are seeing congestion between Marseille and Singapore due to 3rd party fiber circuit maintenance. They have routed eqsin-eqiad through S... [08:34:19] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12488124 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs6003.drmrs.wmnet with OS trixie [09:16:38] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12488255 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs6003.drmrs.wmnet with OS trixie completed: - lvs6003 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled... [09:47:02] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12488381 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs6002.drmrs.wmnet with OS trixie [09:49:42] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12488402 (10Vgutierrez) [10:26:16] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12488518 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs6002.drmrs.wmnet with OS trixie completed: - lvs6002 (**PASS**) - Downtimed on Icinga/Ale... [10:52:00] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12488660 (10Vgutierrez) [11:06:20] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12488721 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs6001.drmrs.wmnet with OS trixie [11:28:31] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), and 2 others: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12488800 (10jijiki) 05Open→03Resolved a:03jijiki resolving :) [11:42:50] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12488873 (10cmooney) Just to document the results here this has had a pronounced affect on the profiling of traffic on our internet ed... [11:47:24] 06Traffic: Stop using the deprecated kafka Events channel on purged - https://phabricator.wikimedia.org/T442736#12488884 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez [11:47:55] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12488886 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs6001.drmrs.wmnet with OS trixie completed: - lvs6001 (**PASS**) - Downtimed on Icinga/Alertmanager - Disabled... [11:58:28] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12488928 (10Vgutierrez) [12:03:50] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12488944 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs3010.esams.wmnet with OS trixie [12:13:01] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12489222 (10cmooney) Actually @ayounsi alerted me to a better stat to share. We actually had a saturation incident earlier today on o... [12:21:51] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12489247 (10BBlack) Super-cool outcome, nice work all! [12:30:01] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12489290 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs3010.esams.wmnet with OS trixie executed with errors: - lvs3010 (**FAIL**) - Downtimed on... [12:30:24] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12489291 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs3010.esams.wmnet with OS trixie [12:37:36] 06Traffic, 06DC-Ops, 10ops-esams: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859 (10Vgutierrez) 03NEW [12:40:05] 06Traffic, 06MediaWiki-Media-Platform-Team, 07Regression: Download button on Commons pages doesn't download - https://phabricator.wikimedia.org/T442313#12489367 (10Krinkle) [12:41:21] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12489376 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1004 for host lvs3010.esams.wmnet with OS trixie executed with errors: - lvs3010 (**FAIL**) - Removed from Puppet and PuppetDB i... [12:48:37] 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12489392 (10MLechvien-WMF) Could we attach a [[ https://wikitech.wikimedia.org/wiki/Incident_response/Incident_severities | severity ]] tag to this task? [12:50:39] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12489400 (10Fabfur) Very nice to hear this, good job! [13:16:25] FIRING: [2x] SystemdUnitFailed: haproxy.service on cp6016:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:21:25] RESOLVED: [2x] SystemdUnitFailed: haproxy.service on cp6016:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:02:59] XioNoX, topranks hi! we've lost lvs3010 due to HW issues, that's the secondary load balancer in esams. So before depooling esams we thought we could go active/active in there :) [15:03:30] the first step would be bringing high-traffic2 services (upload and upload-https) to lvs3008 [15:03:39] vgutierrez: I like that ! [15:03:50] there is nothing on the network layer that could block that, right? [15:04:09] vgutierrez: not that I can think of, BGP drives it all [15:04:25] yeah.. we actually ran esams in active/active for a while by mistake [15:05:03] ok... so i'll try to bring upload/upload-https into lvs3008 and report back :D [15:05:30] vgutierrez: cool, let me know if I can help [15:06:10] volans will enjoy this on-call shift :D [15:06:32] * volans sets the pager to 301 to vgutierrez [15:06:42] hahahah [15:06:47] sounds fair to me [15:07:32] * XioNoX geodns volans home IP to esams [15:08:31] sorry I'm a drmrser :D [15:09:14] !log pooling upload/upload-https on high-traffic1 (lvs3008) in esams - T442859 [15:09:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:09:21] T442859: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859 [15:10:13] https://www.irccloud.com/pastebin/MbEP692W/ [15:10:52] looking good on lvs3008 [15:12:12] * vgutierrez waiting for a spike on NEL.... [15:13:11] haha, I was checking the routers in drmrs, and though wtf is it routing through the core sites?! [15:13:47] 😅 [15:21:22] everything lgtm, checked v6 too [15:22:34] ok, I'm proceeding with lvs3009 [15:22:40] that should relief lvs3008 a little bit :D [15:23:11] I'm curious on how close to 50/50 it will get [15:25:35] !log pooling high-traffic1 services on high-traffic2 (lvs3009) in esams - T442859 [15:25:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:25:40] T442859: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859 [15:25:54] ! [15:26:56] lvs3009 looking good https://www.irccloud.com/pastebin/KGOrNlAf/ [15:33:47] vgutierrez: it even have a beneficial impact on the cpu usage: https://grafana.wikimedia.org/d/bfef1e1b-e005-4a5c-9e66-b8aa6f0cc6d5/load-balancers-katran?from=now-1h&to=now&timezone=utc&var-site=esams&refresh=1m [15:34:09] XioNoX: sure.. lvs3008 is handling less traffic than before [15:34:49] 367 kps / 362kps [15:34:51] not bad [15:36:25] yeah close enough from 50/50 :) [15:36:44] yup [15:37:01] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12490513 (10ssingh) p:05Triage→03High [15:46:26] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12490612 (10RobH) We also had another issue iwth a ganeti host in esams where we updated the firmware, it would clear the error, only for the error... [15:49:19] XioNoX: https://grafana-rw.wikimedia.org/goto/szdqcq?orgId=default [15:49:53] what's that? [15:49:54] XioNoX: the spike of non-syn LRU misses happens when the load balancer starts getting traffic that was handled initially by another load balancer [15:50:04] right, makes sens [15:50:18] we could enable some more deterministic routing on the routers I think [15:51:00] to be investigated when we start doing active/active for real [15:51:34] right.. our busiest PoP isn't "for real" :D [15:52:13] haha, when active/active becomes the norm and not an emergency :) [15:52:32] :) [16:09:55] socket errors on cp servers look steady, nothing crazy going on [16:30:51] 06Traffic, 06MediaWiki-Media-Platform-Team, 07Regression: Download button on Commons pages doesn't download - https://phabricator.wikimedia.org/T442313#12490939 (10Krinkle) [16:33:35] 06Traffic, 06MediaWiki-Media-Platform-Team, 07Regression: Download button on Commons pages doesn't download - https://phabricator.wikimedia.org/T442313#12490960 (10Krinkle) @Vgutierrez Thanks! Works great now. [16:48:37] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12491017 (10RobH) I need to clear the SEL to see if the error clears up properly, so uploading the system event log here for reference. FYI: it als... [16:49:08] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12491019 (10RobH) {F111250042} [16:59:16] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12491071 (10RobH) 05Open→03Resolved a:03RobH lvs3010 firmware idrac was 7.00.00.00 now 7.30.30.54 bios 1.9.2 now 1.23.1 backplane 6.9... [17:03:35] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12491113 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1004 for host lvs3010.esams.wmnet with OS trixie [17:40:40] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12491346 (10ssingh) Great work all around, folks! <3 [18:10:40] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12491542 (10ssingh) Thank you for taking care of this so quickly, @RobH. [18:15:35] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12491556 (10Vgutierrez) Thx. Host has been reimaged successfully [19:04:01] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (non-WMF native apps treated as bot-traffic) - https://phabricator.wikimedia.org/T413570#12491859 (10TheDJ) >>! In T413570#12491590, @ssingh wrote: > Thanks for sharing the error message string 0c640b1. Unfortunately, given that, it is expected... [19:05:20] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (non-WMF native apps treated as bot-traffic) - https://phabricator.wikimedia.org/T413570#12491863 (10ssingh) >>! In T413570#12491859, @TheDJ wrote: >>>! In T413570#12491590, @ssingh wrote: >> Thanks for sharing the error message string 0c640b1....