[00:06:24] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:05:42] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12372284 (10jasmine_) >>! In T438896#12371812, @Ladsgroup wrote: > Thank you for the investigation Chris! <3 I updated the title to reflect the... [01:15:27] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12372301 (10jasmine_) [04:06:24] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:56:24] RESOLVED: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:14:03] 06Traffic, 06SRE: Prometheus Ferm MSS export does not work for IPv6 - https://phabricator.wikimedia.org/T433672#12372837 (10cmooney) 05Open→03Resolved a:03cmooney Thanks @CDobbins for getting to the bottom of this! [08:25:01] 10netops, 06Infrastructure-Foundations: Junos: ae speed not reported corectly through gNMI - https://phabricator.wikimedia.org/T439202#12372876 (10cmooney) No objection to us working with Juniper to fix this, or even the workround with the description / rewrite. But I think for our monitoring we should be ign... [08:45:31] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12372945 (10Clement_Goubert) [08:55:50] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Juniper hostkey-algorithm deprecation - https://phabricator.wikimedia.org/T439136#12372989 (10ayounsi) 05Open→03Stalled p:05High→03Low Marking it as stalled until we have upgraded all the devices. [08:56:20] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Juniper hostkey-algorithm deprecation - https://phabricator.wikimedia.org/T439136#12372992 (10ayounsi) [08:56:21] 10netops, 06Traffic, 06Infrastructure-Foundations: 2026 Junos upgrade - https://phabricator.wikimedia.org/T416444#12372993 (10ayounsi) [09:08:01] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Junos: ae speed not reported corectly through gNMI - https://phabricator.wikimedia.org/T439202#12373029 (10ayounsi) a:03ayounsi Great point, I sent https://gerrit.wikimedia.org/r/c/operations/alerts/+/1345934 for that. Transit/Peering interfaces... [09:26:53] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Junos: ae speed not reported corectly through gNMI - https://phabricator.wikimedia.org/T439202#12373119 (10ayounsi) 05Open→03Resolved Deployed! [10:20:30] 06Traffic, 06MediaWiki-API-Platform-Team: Purged fails to parse purge URLs for /api/rest_v1/page/mobile-html - https://phabricator.wikimedia.org/T439533 (10Vgutierrez) 03NEW [10:22:51] 06Traffic, 06SRE, 13Patch-For-Review: Migrated busted container image to Trixie - https://phabricator.wikimedia.org/T437442#12373489 (10dkertesz) I have verified with @Joe and [[ https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/fc9d7b7af374288a0070781f68014fd8647940... [10:23:06] 06Traffic, 06SRE, 13Patch-For-Review: Migrated busted container image to Trixie - https://phabricator.wikimedia.org/T437442#12373502 (10dkertesz) a:03dkertesz [10:23:22] 06Traffic, 13Patch-For-Review: Migrated busted container image to Trixie - https://phabricator.wikimedia.org/T437442#12373504 (10dkertesz) 05Open→03In progress [10:38:42] 06Traffic, 13Patch-For-Review: Migrated busted container image to Trixie - https://phabricator.wikimedia.org/T437442#12373558 (10MoritzMuehlenhoff) Please also remove the image from the registry. This can be done with docker-registryctl on build2004: ` sudo -i docker-registryctl delete-tags docker-registry.di... [10:40:17] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535 (10cmooney) 03NEW p:05Triage→03Medium [10:41:26] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535#12373591 (10cmooney) [10:41:30] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12373592 (10cmooney) [10:46:29] 10netops, 06Infrastructure-Foundations, 06SRE: Support channelized / breakout interfaces on SR-Linux - https://phabricator.wikimedia.org/T439537 (10cmooney) 03NEW p:05Triage→03Medium [10:47:20] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535#12373630 (10cmooney) [10:47:22] 10netops, 06Infrastructure-Foundations, 06SRE: Support channelized / breakout interfaces on SR-Linux - https://phabricator.wikimedia.org/T439537#12373629 (10cmooney) [11:42:05] 06Traffic: Migrated busted container image to Trixie - https://phabricator.wikimedia.org/T437442#12373877 (10dkertesz) Thanks for the tip! I have deleted the following images: - `docker-registry.discovery.wmnet/busted-bullseye:0.1.1-20260830` - `docker-registry.discovery.wmnet/busted-bullseye:0.1.1-20260920` -... [11:49:54] 06Traffic, 06MediaWiki-API-Platform-Team, 10Page Content Service: Purged fails to parse purge URLs for /api/rest_v1/page/mobile-html - https://phabricator.wikimedia.org/T439533#12373890 (10Jgiannelos) [14:08:59] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, and 3 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12374600 (10Krinkle) LGTM now, for some definition of "good". 169 URLs when purging 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12374687 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie [15:36:06] FIRING: [44x] ProbeDown: Service ncredir5004:443 has failed probes (tcp_ncredir_non_canonical_redirect_11_ip4) - https://wikitech.wikimedia.org/wiki/Ncredir - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:41:06] RESOLVED: [78x] ProbeDown: Service ncredir5004:443 has failed probes (tcp_ncredir_non_canonical_redirect_10_ip4) - https://wikitech.wikimedia.org/wiki/Ncredir - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:41:25] FIRING: [2x] SystemdUnitFailed: rsyslog.service on ncredir5004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:46:25] RESOLVED: [2x] SystemdUnitFailed: rsyslog.service on ncredir5004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:47:25] FIRING: [2x] SystemdUnitFailed: rsyslog.service on ncredir5004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:22:25] FIRING: [3x] SystemdUnitFailed: wmf_auto_restart_varnish-frontend-hospital.service on cp5028:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:22:47] brett: ^ cookbook? [16:32:25] FIRING: [3x] SystemdUnitFailed: wmf_auto_restart_varnish-frontend-hospital.service on cp5028:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:37:25] FIRING: [3x] SystemdUnitFailed: wmf_auto_restart_varnish-frontend-hospital.service on cp5028:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:45:49] yeah, weird [17:02:25] FIRING: [3x] SystemdUnitFailed: wmf_auto_restart_varnish-frontend-hospital.service on cp5028:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:03:48] FIRING: PuppetFailure: Puppet has failed on ncredir5004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:03:49] er, they're not failed.... [17:04:03] the hospital service is healthy, check must be outdated [17:47:25] RESOLVED: [2x] SystemdUnitFailed: rsyslog.service on ncredir5004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:48:48] RESOLVED: PuppetFailure: Puppet has failed on ncredir5004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:51:25] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12376064 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir5004.eqsin.wmnet with OS trixie completed: - ncredir5004 (**WARN**) - Removed from Pupp... [18:58:22] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12376431 (10CDobbins) [19:43:32] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12376645 (10jasmine_) [19:44:36] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12376649 (10jasmine_) [19:45:15] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12376653 (10jasmine_) [23:22:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#12377400 (10Papaul)