[00:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [00:56:21] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12585629 (10Eevans) I used the query workbench in logstash to run... `lang=sql SELECT `@timestamp`, wiki, message FROM logstash-* WHERE `wiki` = 'enwiki' AND `chan... [02:37:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:14:50] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 22d 15h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [04:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [06:37:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:14:50] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 22d 11h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [08:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [09:03:14] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586274 (10ops-monitoring-bot) Host dns7001* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [09:38:40] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586369 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [09:45:15] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586373 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [09:56:29] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586406 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [09:57:53] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586408 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [10:02:22] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586416 (10ayounsi) Test is successful. # rebooted twice doh1002 as a control. Both time BFD stayed down. # Created a file `/etc/sysctl.d/99-tes... [10:37:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:44:25] 10CAS-SSO, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-11-20 - 2026-12-11): Enable the OAuth device authorization grant in CAS and register a wmf-s3-login client - https://phabricator.wikimedia.org/T435602#12586730 (10Gehel) [11:00:36] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12586867 (10ops-monitoring-bot) Host doh1002* rebooted by ayounsi@cumin1004 with reason: Troubleshooting BFD [11:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 22d 7h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [11:17:25] FIRING: [3x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [14:03:18] 10netops, 10Ganeti, 06Infrastructure-Foundations: Invalid private v6 IP on netflow on routed Ganeti - https://phabricator.wikimedia.org/T445660 (10ayounsi) 03NEW [14:38:20] 10netops, 10Ganeti, 06Infrastructure-Foundations: Invalid private v6 IP on netflow on routed Ganeti - https://phabricator.wikimedia.org/T445660#12587895 (10ayounsi) I manually fixed netflow4003 [14:39:16] 10netops, 06Infrastructure-Foundations: Race condition setting up IPv6 address - https://phabricator.wikimedia.org/T445667 (10ayounsi) 03NEW p:05Triage→03Medium [15:01:01] 10netops, 06Infrastructure-Foundations: IPv6: use stable link-local gateways (fe80::1) - https://phabricator.wikimedia.org/T445676 (10ayounsi) 03NEW [15:05:38] 10netops, 06Infrastructure-Foundations, 06SRE: QFX5120 reporting 0 for neighbor bgp prefix counters on 23.4R2-S8.7 - https://phabricator.wikimedia.org/T433490#12588126 (10cmooney) So annoying: ` Hi Cathel, I have been working with our engineering teams and was informed that this feature is not fully suppo... [15:08:28] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12588133 (10ayounsi) Unfortunately it's more complicated than expected... Setting up that configuration option would trigger more of the issue docume... [15:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 22d 3h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [15:17:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:27:56] 10netops, 06Infrastructure-Foundations: Race condition setting up IPv6 address - https://phabricator.wikimedia.org/T445667#12588225 (10cmooney) > The quickest fix forward is to modify /etc/network/interfaces and replace up ip addr add with up ip addr replace which would not fail if the IP is already present.... [15:43:09] 10netops, 06Infrastructure-Foundations: IPv6: use stable link-local gateways (fe80::1) - https://phabricator.wikimedia.org/T445676#12588348 (10cmooney) > There are 2 options for a stable IP, either use the switch side interface IP, for example 2620:0:861:13d::1, which would be similar to how it's done with v4,... [16:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [16:42:57] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12588598 (10cmooney) 05Resolved→03Open p:05Medium→03Low Nokia support actually came back, and seem to have found a sequence of operations that can get the switch in... [19:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 21d 23h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [19:17:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:06:00] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [20:32:25] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:42:25] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:58:55] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12589668 (10Eevans) Looking at the most recent requests that //actually// had missing results (rows count of 0)... `lang=shell-session $ egrep "^0 " counts.txt |he... [21:08:40] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12589799 (10Eevans) Following up on my original comment: >>! In T444597#12548317, @Eevans wrote: > Querying Cassandra directly: Technically the example provided i... [21:09:10] 10CAS-SSO, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Enable the OAuth device authorization grant in CAS and register a wmf-s3-login client - https://phabricator.wikimedia.org/T435602#12589805 (10Gehel) [21:23:16] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12589888 (10Eevans) ## TL;DR Of the 1503 logging events captured starting Nov 1, all but 225 seem to be erroneous. Meaning, the error reports no suggestions when... [23:13:14] 10netops, 06Infrastructure-Foundations, 10Observability-Logging: ~5k/logs/sec from netdev - https://phabricator.wikimedia.org/T412143#12590372 (10colewhite) >>! In T412143#12502580, @ayounsi wrote: > We still need to upgrade more switches. Do you have an updated list, so we can compare to our upgrade plans a... [23:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 21d 19h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry