[00:10:25] FIRING: SystemdUnitFailed: acme-chief-certs-sync.service on acmechief1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:28:28] ^failing due to connection timeouts [00:35:25] RESOLVED: SystemdUnitFailed: acme-chief-certs-sync.service on acmechief1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:40:25] FIRING: SystemdUnitFailed: acme-chief-certs-sync.service on acmechief1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:45:25] RESOLVED: SystemdUnitFailed: acme-chief-certs-sync.service on acmechief1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:01:55] FIRING: [3x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [06:35:40] FIRING: [5x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [07:53:53] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12358459 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=21ede500-1983-4730-b871-1c7ca225b230) set by ayounsi@cumin1004 for 2:00:00 on 20 host(s) and their services with rea... [08:03:02] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12358473 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=728b2b9e-70ae-4304-acc5-5669decbb5dd) set by ayounsi@cumin1004 for 2:00:00 on 3 host(s) and their services with reas... [08:40:08] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12358708 (10Tgr) >>! In T425216#12351266, @BBlack wrote: > To not copy any more application-specific busi... [08:45:05] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12358769 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=3ba5e2c1-099e-4219-85d1-7c50f64c9b83) set by ayounsi@cumin1004 for 2:00:00 on 3 host(s) and their services with reas... [08:46:55] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12358779 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=5d53fa31-8467-4252-9f52-e7035b9810c9) set by ayounsi@cumin1004 for 2:00:00 on 19 host(s) and their services with rea... [09:28:07] 06Traffic: Upgrade dnsdist to v1.9.16-0+deb13u1 - https://phabricator.wikimedia.org/T439034#12358926 (10Aklapper) [09:32:17] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12358952 (10ayounsi) [09:39:57] inflatador: I'm depooling cirrussearch1120, feel free to repool it when it's back online [09:40:08] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw possibily resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12359058 (10hnowlan) >>! In T438896#12356442, @JMeybohm wrote: > Things that where... [10:36:55] FIRING: [5x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [11:15:40] FIRING: [5x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [11:16:35] 06Traffic: Dump stats-file before haproxy restart/reload - https://phabricator.wikimedia.org/T419524#12359474 (10dkertesz) This work is now being done in T343000 so I think we can close this ticket as duplicate [11:16:55] FIRING: [5x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [11:33:02] 06Traffic: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828#12359508 (10Vgutierrez) [12:10:17] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12359688 (10JMeybohm) [12:16:30] 06Traffic: Dump stats-file before haproxy restart/reload - https://phabricator.wikimedia.org/T419524#12359696 (10Fabfur) 05Open→03Declined Yes, thanks [12:58:24] 10netops, 06Infrastructure-Foundations: Restart lsw1-a2-codfw for T437984 - https://phabricator.wikimedia.org/T439134 (10ayounsi) 03NEW [12:59:03] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12359905 (10ayounsi) [12:59:04] 10netops, 06Infrastructure-Foundations: Restart lsw1-a2-codfw for T437984 - https://phabricator.wikimedia.org/T439134#12359906 (10ayounsi) [13:08:34] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12359950 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir4004.ulsfo.wmnet with OS trixie [13:24:26] 10netops, 06Infrastructure-Foundations: Juniper hostkey-algorithm deprecation - https://phabricator.wikimedia.org/T439136 (10ayounsi) 03NEW p:05Triage→03High [13:35:08] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12360080 (10Krinkle) >>! In T425216#12351266, @BBlack wrote: > I was under the impression (implicitly, du... [13:58:44] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Juniper hostkey-algorithm deprecation - https://phabricator.wikimedia.org/T439136#12360238 (10ayounsi) ` bast6003:~$ ssh -o HostKeyAlgorithms=ssh-dss -o PubkeyAcceptedAlgorithms=ssh-dss cr1-drmrs.wikimedia.org Unable to negotiate with 2a02:ec80:600:... [14:00:35] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12360247 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir4004.ulsfo.wmnet with OS trixie completed: - ncredir4004 (**WARN**) - Downtimed on Icin... [14:05:40] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Juniper hostkey-algorithm deprecation - https://phabricator.wikimedia.org/T439136#12360291 (10ayounsi) [14:08:30] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12360302 (10CDobbins) [14:09:14] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12360304 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir3005.esams.wmnet with OS trixie [14:37:15] hello traffic, could we get some hands in _security to coordinate restoring codfw dns servers please? [14:40:42] might not be the best person for that, but I'm here [15:00:02] 06Traffic, 10observability, 06SRE, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12360510 (10dkertesz) [15:02:01] 06Traffic, 10observability, 06SRE, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12360520 (10dkertesz) I have merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1343039 which introduces support for persistent stats in haproxy; to test that... [15:03:56] 06Traffic, 10observability, 06SRE, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12360525 (10dkertesz) [15:16:55] FIRING: [3x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [15:18:28] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12360574 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir3005.esams.wmnet with OS trixie completed: - ncredir3005 (**WARN**) - Downtimed on Icin... [15:52:57] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12360809 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir1001.eqiad.wmnet with OS trixie [16:30:11] 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work, 07Kubernetes: Calico IPv4 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12361009 (10BTullis) 05Open→03Resolved I'm going to re... [16:33:18] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12361028 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir1001.eqiad.wmnet with OS trixie completed: - ncredir1001 (**WARN**) - Downtimed on Icin... [17:08:04] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw possibily resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12361158 (10Ladsgroup) >>! In T438896#12356442, @JMeybohm wrote: > - Huge number o... [17:21:38] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12361214 (10CDobbins) [17:22:33] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12361228 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie [18:04:53] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12361328 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie executed with errors: - ncredir5004 (**FAIL**) - Remove... [18:10:08] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12361336 (10Verdy_p) Still not solved, and signaled with more details on https://commons.wikimedia.org/wi... [18:25:45] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12361389 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie [18:46:17] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover, 13Patch-For-Review: Investigate high saturation in codfw possibily resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896#12361427 (10jasmine_) [19:08:09] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12361540 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie executed with errors: - ncredir5004 (**FAIL**) - Remove... [19:13:56] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12361556 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie [19:14:52] 10netops, 06Infrastructure-Foundations: Junos 25 and gNMI metrics - https://phabricator.wikimedia.org/T439166 (10ayounsi) 03NEW p:05Triage→03High [19:16:55] FIRING: [3x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [19:56:14] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12361721 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie executed with errors: - ncredir5004 (**FAIL**) - Remove... [23:16:55] FIRING: [3x] PybalBackendDown: Pybal backend cirrussearch1120:0 is down (search-https_9243) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown