[00:29:40] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [03:11:41] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:11:41] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:29:40] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [08:29:40] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [10:41:46] 10Packaging, 10Wikidata, 10Wikidata-Query-Service: Migrate WDQS to Java 11 - https://phabricator.wikimedia.org/T316103#11622247 (10trueg) 05Open→03Resolved a:03trueg The WDQS parts as maintained in https://gerrit.wikimedia.org/r/q/project:wikidata/query/rdf are now jdk17-compatible. In fact the rdf... [13:03:27] 10netops, 06Infrastructure-Foundations: magru: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416442#11622614 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a08d08c2-5950-4fed-a93a-c3c2670c6e3e) set by ayounsi@cumin1003 for 2:00:00 on 2 host(s) and their services wi... [13:11:25] 10netops, 06Infrastructure-Foundations: magru: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416442#11622638 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7f778211-928f-4b3a-b850-c48eecb6889b) set by ayounsi@cumin1003 for 2:00:00 on 2 host(s) and their services wi... [13:13:58] 10netops, 06Infrastructure-Foundations: magru: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416442#11622661 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=99c5c739-235a-4c04-8c28-d8b0ef056232) set by ayounsi@cumin1003 for 2:00:00 on 40 host(s) and their services w... [13:52:48] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11622821 (10Jclark-ctr) 05Open→03Resolved [14:08:42] 10netops, 06Infrastructure-Foundations: magru: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416442#11622925 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=349bbdfb-74d0-421c-9a4c-07e586c71db9) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their services wi... [14:12:57] 10netops, 06Infrastructure-Foundations: magru: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416442#11622966 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=709d9f76-c776-4510-a7e0-1c4545cf4710) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their services wi... [14:19:25] FIRING: SystemdUnitFailed: netbox_ganeti_magru03_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:34:25] RESOLVED: SystemdUnitFailed: netbox_ganeti_magru03_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:45:05] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 06SRE, and 2 others: Replace Spamassassin with Rspam for VRTS on Postfix - https://phabricator.wikimedia.org/T402260#11623148 (10ABran-WMF) 05Open→03Stalled [15:08:33] 10netops, 06Infrastructure-Foundations: magru: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416442#11623304 (10ayounsi) [15:08:43] 10netops, 06Infrastructure-Foundations: magru: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416442#11623307 (10ayounsi) 05Open→03Resolved a:03ayounsi [15:45:25] i'm trying to log in to labtesthorizon.wm.o (via cloudidp-dev.wm.o) but cas just keeps sending me back to the login page, any ideas why? [15:45:55] in the cas logs on cloudidp2001-dev I see things like "Invalid key for dir with A128CBC-HS256, expected a 256 bit key but a 384 bit key was provided" and "DecryptionException" (yes, with no extra detail on that one) [18:37:13] 10Mail, 06Infrastructure-Foundations, 06Trust-and-Safety: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost - https://phabricator.wikimedia.org/T338032#11624691 (10RoySmith) I'll let the person involved identify themselves if they wish, but I just heard of another incident today where... [19:01:54] 10Mail, 06Infrastructure-Foundations, 06Trust-and-Safety: Some emails to emergency@wikimedia.org are getting lost - https://phabricator.wikimedia.org/T338032#11624753 (10A_smart_kitten)