[00:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 20d 15h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [03:20:56] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [04:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 20d 11h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [07:20:56] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [08:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:22:01] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592118 (10Peachey88) [10:25:25] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592121 (10tstarling) I paged SRE. [10:27:26] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592123 (10tstarling) {F116770103,width=800} [10:46:20] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592157 (10Hason-LEK-SIN) If it's limited to ESEAP, is it because servers in Singapore are down? Are there servers in Japan? (Which Metawiki page de... [10:49:02] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592166 (10Blake) Hi folks, we now expect this to be mitigated, please let us know if that isn't the case. [10:49:51] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592168 (10Novem_Linguae) >>! In T445768#12592157, @Hason-LEK-SIN wrote: > If it's limited to ESEAP, is it because servers in Singapore are down? Ar... [11:05:24] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: wiki pages taking up to a minute to load intermittently - https://phabricator.wikimedia.org/T445768#12592186 (10tstarling) 05Open→03Resolved a:03ayounsi >>! In T445768#12592168, @Novem_Linguae wrote: > I doubt an entire server or data cent... [11:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 20d 7h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [11:20:56] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [12:42:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:14:51] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 20d 3h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [15:20:56] FIRING: [4x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire