[02:05:25] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [03:06:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 28d 15h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [06:05:25] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [07:06:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 28d 11h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [08:17:17] 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email: Cannot receive wiki mail from other editors anymore - https://phabricator.wikimedia.org/T443114#12558684 (10Clarettt) Hello again, Normally on the left sidebar there is a tool "Email this user". When I was blocked that option has been disappeared, fo... [08:28:54] 10Packaging, 10AbuseFilter, 06Infrastructure-Foundations, 06Product Safety and Integrity: Bundled AbuseFilter wothout wikimedia/equivset - https://phabricator.wikimedia.org/T444899 (10alex-mashin) 03NEW [10:05:25] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:49:11] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [11:06:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 28d 7h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [14:05:25] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:28:33] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12559448 (10ssingh) FWIW, we saw this again today withe the reboots: 09:25:40 <+jinxer-wm> FIRING: [18x] BFDdown: BFD session down between cr1-codfw... [14:49:11] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [15:06:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 28d 3h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [18:05:25] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:49:11] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [19:06:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 27d 23h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [22:05:25] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:49:11] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [23:06:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 27d 19h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry