[00:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [02:07:40] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:37:25] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:14:50] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 23d 15h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [04:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [06:37:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:14:50] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 23d 11h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [08:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [08:52:30] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Network: gnmic monitoring issues and improvements - https://phabricator.wikimedia.org/T441780#12580470 (10ayounsi) Looks like they found something ! Not sure how that integrates in the big picture, but worth looking into. > After carefully... [10:36:01] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12580936 (10ayounsi) Is that an ongoing issue? I noticed a bit of discarded packets in the past, most likely due to burst saturation (the server only have a 1G NIC)... [10:37:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:12:26] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12581185 (10Michael) >>! In T444597#12580936, @ayounsi wrote: > Is that an ongoing issue? I noticed a bit of discarded packets in the past, most likely due to burst... [11:14:50] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 23d 7h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [12:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [13:43:39] 10SRE-tools, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review: Make standard cookbooks compatible with hosts not in Icinga - https://phabricator.wikimedia.org/T445087#12582298 (10taavi) a:03taavi [14:37:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:10:51] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Network: gnmic monitoring issues and improvements - https://phabricator.wikimedia.org/T441780#12582804 (10ayounsi) > At this time, I cannot confirm whether a fix is planned for the 24.4R2-Sx branch or whether the recommended Junos release f... [15:14:50] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 23d 3h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [16:05:31] 10netops, 10Cassandra, 06Infrastructure-Foundations: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597#12583390 (10Eevans) >>! In T444597#12580936, @ayounsi wrote: > Is that an ongoing issue? I noticed a bit of discarded packets in the past, most likely due to burst... [16:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [17:21:20] 10CFSSL-PKI, 10Beta-Cluster-Infrastructure, 06Infrastructure-Foundations: profile::pki::client does not hash symlinks and does not populate ca-certificates.crt - https://phabricator.wikimedia.org/T436562#12583864 (10dr0ptp4kt) @Southparkfan I am terribly sorry, I missed the request to review and merge the pa... [18:37:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:14:50] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 22d 23h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [19:26:48] 10netops, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 06Infrastructure-Foundations, and 2 others: Slow connections from Cloud VPS to cdn.cypress.io make CI builds take up to 50 minutes - https://phabricator.wikimedia.org/T445584 (10pwangai) 03NEW [20:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [20:32:14] 10netops, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 06Infrastructure-Foundations, and 2 others: Slow connections from Cloud VPS to cdn.cypress.io make CI builds take up to 50 minutes - https://phabricator.wikimedia.org/T445584#12584573 (10pwangai) 05Open→03Invalid CI seems to be healing... [20:38:31] 10Mail, 06English-Arbitration-Committee, 06Infrastructure-Foundations: Email forwarding from arbcom-en(at)wikipedia.org to arbcom-en(at)wikimedia.org - https://phabricator.wikimedia.org/T417410#12584590 (10jhathaway) 05Declined→03In progress @Keegan thanks for raising this issue again. On reflection I ma... [21:13:44] 10Mail, 06English-Arbitration-Committee, 06Infrastructure-Foundations: Email forwarding from arbcom-en(at)wikipedia.org to arbcom-en(at)wikimedia.org - https://phabricator.wikimedia.org/T417410#12584734 (10HouseBlaster) 05In progress→03Resolved We have confirmed that email forwarding works. Thank you... [21:49:20] 10CFSSL-PKI, 10Beta-Cluster-Infrastructure, 06Infrastructure-Foundations: profile::pki::client does not hash symlinks and does not populate ca-certificates.crt - https://phabricator.wikimedia.org/T436562#12584898 (10Southparkfan) >>! In T436562#12583863, @dr0ptp4kt wrote: > @Southparkfan I am terribly so... [21:49:23] 10CFSSL-PKI, 10Beta-Cluster-Infrastructure, 06Infrastructure-Foundations: profile::pki::client does not hash symlinks and does not populate ca-certificates.crt - https://phabricator.wikimedia.org/T436562#12584899 (10Southparkfan) 05Open→03Resolved a:03Southparkfan [22:37:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:14:50] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 22d 19h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry