[00:05:56] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [00:42:25] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 21d 15h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [04:05:56] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [04:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 21d 11h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [07:38:58] 10netops, 06Infrastructure-Foundations: Race condition setting up IPv6 address - https://phabricator.wikimedia.org/T445667#12590803 (10ayounsi) [08:05:56] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [08:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 21d 7h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [12:05:56] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [12:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 21d 3h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [15:16:41] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [16:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 20d 23h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [19:20:56] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [20:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 20d 19h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [23:20:56] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire