[00:07:06] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [00:08:24] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1189606 [00:08:24] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1189606 (owner: 10TrainBranchBot) [00:23:42] RESOLVED: [2x] ProbeDown: Service wdqs1020:443 has failed probes (http_wdqs_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1020:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:30:29] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1189606 (owner: 10TrainBranchBot) [00:36:41] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-dse_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:44:00] FIRING: [2x] OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [01:00:43] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [01:12:40] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 11m 56s) [01:21:44] PROBLEM - mailman archives on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [01:21:44] PROBLEM - mailman list info on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [01:26:34] RECOVERY - mailman archives on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 54827 bytes in 0.074 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [01:26:34] RECOVERY - mailman list info on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 9234 bytes in 0.183 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [01:43:25] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:02:48] FIRING: PuppetFailure: Puppet has failed on wdqs2016:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [02:04:00] FIRING: [10x] SystemdUnitFailed: prometheus_ferm_mss.service on wdqs1018:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:29:12] FIRING: SystemdUnitFailed: prometheus_amd_rocm_stats.service on ml-serve1013:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:03:11] FIRING: ProbeDown: Service phab1004:443 has failed probes (http_phabricator_wikimedia_org_collab_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#phab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:05:49] FIRING: [3x] ProbeDown: Service phab1004:443 has failed probes (http_phabricator_wikimedia_org_collab_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#phab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:13:11] RESOLVED: ProbeDown: Service phab1004:443 has failed probes (http_phabricator_wikimedia_org_collab_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#phab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:15:49] RESOLVED: [3x] ProbeDown: Service phab1004:443 has failed probes (http_phabricator_wikimedia_org_collab_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#phab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:16:44] PROBLEM - mailman archives on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [03:16:44] PROBLEM - mailman list info on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [03:21:34] RECOVERY - mailman archives on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 54827 bytes in 0.077 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [03:21:34] RECOVERY - mailman list info on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 9234 bytes in 0.188 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [03:44:00] FIRING: CertAlmostExpired: Certificate for service data-gateway-staging:30443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#data-gateway-staging:30443 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [03:44:39] !log ryankemper@cumin2002 START - Cookbook sre.hosts.reimage for host wdqs2017.codfw.wmnet with OS bullseye [03:44:58] !log ryankemper@cumin2002 START - Cookbook sre.hosts.move-vlan for host wdqs2017 [03:45:14] !log ryankemper@cumin2002 START - Cookbook sre.dns.netbox [03:50:57] ryankemper@cumin2002 reimage (PID 4127754) is awaiting input [03:55:56] !log ryankemper@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2017 - ryankemper@cumin2002" [03:56:02] !log ryankemper@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2017 - ryankemper@cumin2002" [03:56:02] !log ryankemper@cumin2002 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [03:56:03] !log ryankemper@cumin2002 START - Cookbook sre.dns.wipe-cache wdqs2017.codfw.wmnet 154.32.192.10.in-addr.arpa 4.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [03:56:07] !log ryankemper@cumin2002 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2017.codfw.wmnet 154.32.192.10.in-addr.arpa 4.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [03:56:08] !log ryankemper@cumin2002 START - Cookbook sre.network.configure-switch-interfaces for host wdqs2017 [03:56:15] !log ryankemper@cumin2002 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2017 [03:56:15] !log ryankemper@cumin2002 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2017 [04:29:10] FIRING: BFDdown: BFD session down between cr2-magru and fe80::ee38:73ff:fee8:9c58 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [04:34:10] RESOLVED: BFDdown: BFD session down between cr2-magru and fe80::ee38:73ff:fee8:9c58 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [04:36:26] (03PS1) 10Cwhite: Revert "logstash: drop 99% eventgate-analytics-external logs" [puppet] - 10https://gerrit.wikimedia.org/r/1189625 [04:36:41] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-dse_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:44:00] FIRING: [2x] OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [05:00:50] PROBLEM - Backup freshness on backup1014 is CRITICAL: All failures: 1 (install1005), Fresh: 141 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [05:03:07] (03PS1) 10Ilias Sarantopoulos: ml-services: deploy ml services that use event streams [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189626 (https://phabricator.wikimedia.org/T403664) [05:06:37] (03CR) 10Ilias Sarantopoulos: [C:03+2] ml-services: deploy ml services that use event streams [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189626 (https://phabricator.wikimedia.org/T403664) (owner: 10Ilias Sarantopoulos) [05:08:19] (03Merged) 10jenkins-bot: ml-services: deploy ml services that use event streams [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189626 (https://phabricator.wikimedia.org/T403664) (owner: 10Ilias Sarantopoulos) [05:09:00] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:09:14] !log isaranto@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [05:09:21] !log isaranto@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' . [05:11:34] !log isaranto@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . [05:12:10] !log isaranto@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [05:12:54] !log isaranto@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [05:13:11] !log isaranto@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [05:17:05] !log ryankemper@cumin2002 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2017.codfw.wmnet with OS bullseye [05:34:00] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:41:44] PROBLEM - mailman archives on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [05:41:44] PROBLEM - mailman list info on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [05:43:40] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:46:34] RECOVERY - mailman archives on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 54827 bytes in 0.071 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [05:46:34] RECOVERY - mailman list info on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 9234 bytes in 0.178 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250919T0600) [06:00:48] RECOVERY - Backup freshness on backup1014 is OK: Fresh: 142 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [06:03:03] FIRING: PuppetFailure: Puppet has failed on wdqs2016:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [06:04:00] FIRING: [8x] SystemdUnitFailed: prometheus_ferm_mss.service on wdqs1018:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:24:12] (03CR) 10Muehlenhoff: [C:03+2] Re-re-add maps2011 as maps master [puppet] - 10https://gerrit.wikimedia.org/r/1189517 (https://phabricator.wikimedia.org/T381565) (owner: 10Muehlenhoff) [06:29:12] FIRING: SystemdUnitFailed: prometheus_amd_rocm_stats.service on ml-serve1013:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:36:46] (03CR) 10Elukey: redfish: improve log_entries for idrac 10 (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/1189518 (https://phabricator.wikimedia.org/T392851) (owner: 10Elukey) [06:47:27] !log trigger full planet import for maps codfw/bookworm T381565 [06:47:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:47:32] T381565: Move maps servers to Bookworm - https://phabricator.wikimedia.org/T381565 [06:54:01] FIRING: [9x] SystemdUnitFailed: prometheus_ferm_mss.service on wdqs1018:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:56:15] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1187384 (owner: 10Majavah) [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250919T0700) [07:02:51] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-eqiad:xe-3/2/2 (Transport: cr2-codfw:xe-0/1/1:1 (Lumen, 442550293) {#5249}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [07:07:51] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:xe-0/1/1:1 (Transport: cr2-eqiad:xe-3/2/2 (Lumen, 442550293) {#12253_12334-2}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [07:18:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [07:23:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [07:25:12] (03PS1) 10Stevemunene: dse-k8s: Deploy network policies for istio setup in codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189634 (https://phabricator.wikimedia.org/T404433) [07:28:20] (03CR) 10Stevemunene: [V:03+2 C:03+2] Add a dummy Ceph user keys for the cephcsi plugin to use [labs/private] - 10https://gerrit.wikimedia.org/r/1189133 (https://phabricator.wikimedia.org/T404576) (owner: 10Stevemunene) [07:32:11] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 149912840 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [07:32:51] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:xe-0/1/1:1 (Transport: cr2-eqiad:xe-3/2/2 (Lumen, 442550293) {#12253_12334-2}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [07:33:11] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 37104 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [07:37:51] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:xe-0/1/1:1 (Transport: cr2-eqiad:xe-3/2/2 (Lumen, 442550293) {#12253_12334-2}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [07:42:12] (03CR) 10Federico Ceratto: "Bookworm ships facter (4.3.0-2) while Trixie has facter (4.10.0-1)." [puppet] - 10https://gerrit.wikimedia.org/r/1180999 (owner: 10Dzahn) [07:43:25] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:44:00] FIRING: CertAlmostExpired: Certificate for service data-gateway-staging:30443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#data-gateway-staging:30443 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:13:28] (03PS1) 10Brouberol: Fix linting errors [software/spicerack] - 10https://gerrit.wikimedia.org/r/1189764 [08:13:28] (03PS1) 10Brouberol: Fix test_flush_markers_on_clusters [software/spicerack] - 10https://gerrit.wikimedia.org/r/1189765 [08:13:28] (03PS1) 10Brouberol: Pass the timeout to the underlying http client [software/spicerack] - 10https://gerrit.wikimedia.org/r/1189766 [08:14:25] (03PS15) 10Slyngshede: P:puppetserver::volatile Include XCheeseScore private repo [puppet] - 10https://gerrit.wikimedia.org/r/1188770 (https://phabricator.wikimedia.org/T404688) [08:14:46] (03CR) 10Brouberol: [C:03+1] dse-k8s: Deploy network policies for istio setup in codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189634 (https://phabricator.wikimedia.org/T404433) (owner: 10Stevemunene) [08:15:56] (03CR) 10Stevemunene: [C:03+2] dse-k8s: Deploy network policies for istio setup in codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189634 (https://phabricator.wikimedia.org/T404433) (owner: 10Stevemunene) [08:22:10] (03CR) 10CI reject: [V:04-1] Fix linting errors [software/spicerack] - 10https://gerrit.wikimedia.org/r/1189764 (owner: 10Brouberol) [08:22:16] (03CR) 10CI reject: [V:04-1] Fix test_flush_markers_on_clusters [software/spicerack] - 10https://gerrit.wikimedia.org/r/1189765 (owner: 10Brouberol) [08:22:17] (03CR) 10CI reject: [V:04-1] Pass the timeout to the underlying http client [software/spicerack] - 10https://gerrit.wikimedia.org/r/1189766 (owner: 10Brouberol) [08:23:22] (03Merged) 10jenkins-bot: dse-k8s: Deploy network policies for istio setup in codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189634 (https://phabricator.wikimedia.org/T404433) (owner: 10Stevemunene) [08:32:18] !log stevemunene@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [08:32:31] (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/6993/co" [puppet] - 10https://gerrit.wikimedia.org/r/1188770 (https://phabricator.wikimedia.org/T404688) (owner: 10Slyngshede) [08:33:43] !log stevemunene@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [08:34:34] (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/6994/co" [puppet] - 10https://gerrit.wikimedia.org/r/1188770 (https://phabricator.wikimedia.org/T404688) (owner: 10Slyngshede) [08:35:24] (03CR) 10Majavah: [C:03+2] offboard-user: Add acl*wmcs-team to privileged groups [puppet] - 10https://gerrit.wikimedia.org/r/1187384 (owner: 10Majavah) [08:35:28] !log stevemunene@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/echoserver: apply [08:35:43] !log stevemunene@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/echoserver: apply [08:36:41] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-dse_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:39:54] (03CR) 10Majavah: [C:03+2] O:extdist: Drop buster support [puppet] - 10https://gerrit.wikimedia.org/r/1189475 (owner: 10Majavah) [08:40:22] (03PS16) 10Slyngshede: P:puppetserver::volatile Include XCheeseScore private repo [puppet] - 10https://gerrit.wikimedia.org/r/1188770 (https://phabricator.wikimedia.org/T404688) [08:41:56] (03CR) 10Majavah: [C:03+2] O:labs::lvm::mnt: Remove unused role [puppet] - 10https://gerrit.wikimedia.org/r/1189478 (owner: 10Majavah) [08:43:16] (03PS1) 10Stevemunene: Register ingress CNAME record for the echoserver-dse-k8s-eqiad service [dns] - 10https://gerrit.wikimedia.org/r/1189795 (https://phabricator.wikimedia.org/T404433) [08:43:16] (03CR) 10CI reject: [V:04-1] P:puppetserver::volatile Include XCheeseScore private repo [puppet] - 10https://gerrit.wikimedia.org/r/1188770 (https://phabricator.wikimedia.org/T404688) (owner: 10Slyngshede) [08:44:00] FIRING: [2x] OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [08:45:42] (03PS2) 10Stevemunene: Register ingress CNAME record for the echoserver-dse-k8s-codfw service [dns] - 10https://gerrit.wikimedia.org/r/1189795 (https://phabricator.wikimedia.org/T404433) [08:50:36] (03PS6) 10Stevemunene: dse-k8s:Enable CSI and the Ceph CSI plugin on dse-k8s-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1188754 (https://phabricator.wikimedia.org/T404576) [08:55:02] (03PS1) 10Slyngshede: Add cdn_private_git_token dummy [labs/private] - 10https://gerrit.wikimedia.org/r/1189798 [08:56:02] (03PS2) 10Slyngshede: Add cdn_private_git_token dummy [labs/private] - 10https://gerrit.wikimedia.org/r/1189798 [09:03:59] (03CR) 10Slyngshede: [V:03+2 C:03+2] Add cdn_private_git_token dummy [labs/private] - 10https://gerrit.wikimedia.org/r/1189798 (owner: 10Slyngshede) [09:05:30] (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/6996/co" [puppet] - 10https://gerrit.wikimedia.org/r/1188770 (https://phabricator.wikimedia.org/T404688) (owner: 10Slyngshede) [09:06:49] !log klausman@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [09:08:11] !log klausman@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [09:08:54] !log btullis@cumin1003 START - Cookbook sre.hosts.decommission for hosts an-backup-datanode[1001-1007].eqiad.wmnet [09:18:11] btullis@cumin1003 decommission (PID 3226295) is awaiting input [09:18:49] (03PS1) 10Majavah: P:toolforge::k8s: Remove absent resources [puppet] - 10https://gerrit.wikimedia.org/r/1189800 [09:18:50] (03PS1) 10Majavah: P:toolforge::k8s::haproxy: Add a HTTPS listener [puppet] - 10https://gerrit.wikimedia.org/r/1189801 (https://phabricator.wikimedia.org/T283948) [09:18:51] (03PS1) 10Majavah: P:toolforge::proxy: Set backend to new HAProxy HTTPS service [puppet] - 10https://gerrit.wikimedia.org/r/1189802 (https://phabricator.wikimedia.org/T283948) [09:23:13] (03PS2) 10Majavah: P:toolforge::k8s::haproxy: Add a HTTPS listener [puppet] - 10https://gerrit.wikimedia.org/r/1189801 (https://phabricator.wikimedia.org/T283948) [09:23:13] (03PS2) 10Majavah: P:toolforge::proxy: Set backend to new HAProxy HTTPS service [puppet] - 10https://gerrit.wikimedia.org/r/1189802 (https://phabricator.wikimedia.org/T283948) [09:25:19] (03CR) 10Btullis: [C:03+1] "Nice. Thanks. It will be interesting to see if this works on k8s 1.31 without any changes." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1188754 (https://phabricator.wikimedia.org/T404576) (owner: 10Stevemunene) [09:27:24] (03CR) 10Btullis: [C:03+1] "LGTM" [dns] - 10https://gerrit.wikimedia.org/r/1189795 (https://phabricator.wikimedia.org/T404433) (owner: 10Stevemunene) [09:28:05] !log installing distro-info-data updates on Bullseye [09:28:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:29:51] (03CR) 10Marco Fossati: ReaderExperiments' ImageBrowsing stream configuration (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1187413 (https://phabricator.wikimedia.org/T403259) (owner: 10Marco Fossati) [09:30:13] (03PS4) 10Btullis: Remove the manifests for the absented product_analytics jobs [puppet] - 10https://gerrit.wikimedia.org/r/1189204 (https://phabricator.wikimedia.org/T404639) [09:30:14] (03PS1) 10Elukey: Add support for AMD MI300X GPUs (kernel and firmwares) [puppet] - 10https://gerrit.wikimedia.org/r/1189807 (https://phabricator.wikimedia.org/T403697) [09:30:27] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1189204 (https://phabricator.wikimedia.org/T404639) (owner: 10Btullis) [09:30:33] (03PS3) 10Majavah: P:toolforge::proxy: Set backend to new HAProxy HTTPS service [puppet] - 10https://gerrit.wikimedia.org/r/1189802 (https://phabricator.wikimedia.org/T283948) [09:31:30] (03PS2) 10Elukey: Add support for AMD MI300X GPUs (kernel and firmwares) [puppet] - 10https://gerrit.wikimedia.org/r/1189807 (https://phabricator.wikimedia.org/T403697) [09:33:16] (03CR) 10Elukey: [V:03+1] "PCC SUCCESS (CORE_DIFF 1 DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1189807 (https://phabricator.wikimedia.org/T403697) (owner: 10Elukey) [09:33:56] (03CR) 10Elukey: Add support for AMD MI300X GPUs (kernel and firmwares) [puppet] - 10https://gerrit.wikimedia.org/r/1189807 (https://phabricator.wikimedia.org/T403697) (owner: 10Elukey) [09:34:17] (03Abandoned) 10Majavah: Read running tools from grid-webservices tool [software/tools-manifest] - 10https://gerrit.wikimedia.org/r/703189 (https://phabricator.wikimedia.org/T284564) (owner: 10Majavah) [09:36:56] !log btullis@cumin1003 START - Cookbook sre.dns.netbox [09:37:42] PROBLEM - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin1002 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [09:40:07] (03CR) 10Stevemunene: [C:03+2] dse-k8s:Enable CSI and the Ceph CSI plugin on dse-k8s-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1188754 (https://phabricator.wikimedia.org/T404576) (owner: 10Stevemunene) [09:40:35] !log btullis@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-backup-datanode[1001-1007].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [09:41:14] !log btullis@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-backup-datanode[1001-1007].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [09:41:14] !log btullis@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:41:15] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-backup-datanode[1001-1007].eqiad.wmnet [09:41:26] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-dse_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:41:47] !log jmm@cumin2002 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1043.eqiad.wmnet [09:43:27] !log jmm@cumin2002 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet [09:44:20] !log jmm@cumin2002 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1043.eqiad.wmnet [09:47:54] (03Merged) 10jenkins-bot: dse-k8s:Enable CSI and the Ceph CSI plugin on dse-k8s-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1188754 (https://phabricator.wikimedia.org/T404576) (owner: 10Stevemunene) [09:49:32] (03PS1) 10Elukey: profile::amd_gpu: add support for amd-smi from ROCm 6.4.3 [puppet] - 10https://gerrit.wikimedia.org/r/1189815 (https://phabricator.wikimedia.org/T403697) [09:49:34] (03PS1) 10Elukey: Enable ROCm 6.4.3 amd-smi on ml-serve{1012,1013} [puppet] - 10https://gerrit.wikimedia.org/r/1189816 (https://phabricator.wikimedia.org/T403697) [09:50:09] (03CR) 10CI reject: [V:04-1] profile::amd_gpu: add support for amd-smi from ROCm 6.4.3 [puppet] - 10https://gerrit.wikimedia.org/r/1189815 (https://phabricator.wikimedia.org/T403697) (owner: 10Elukey) [09:51:29] 06SRE, 06Traffic, 10MediaWiki-Platform-Team (Radar): Have CDN edge set the `X-Request-Id` header for incoming external requests - https://phabricator.wikimedia.org/T221976#11196722 (10GGoncalves-WMF) Thanks Valentín, I'm thinking probenet can be a useful signal, but our current focus is to experiment with sp... [09:51:33] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST certificaterequests) on k8s-mlstaging@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s-mlstaging&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [09:52:01] (03PS2) 10Elukey: profile::amd_gpu: add support for amd-smi from ROCm 6.4.3 [puppet] - 10https://gerrit.wikimedia.org/r/1189815 (https://phabricator.wikimedia.org/T403697) [09:52:01] (03PS2) 10Elukey: Enable ROCm 6.4.3 amd-smi on ml-serve{1012,1013} [puppet] - 10https://gerrit.wikimedia.org/r/1189816 (https://phabricator.wikimedia.org/T403697) [09:52:17] (03PS1) 10Brouberol: opensearch-operator: allow egress traffic from the operator to the k8s api [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189817 (https://phabricator.wikimedia.org/T404906) [09:56:08] (03PS3) 10Elukey: profile::amd_gpu: add support for amd-smi from ROCm 6.4.3 [puppet] - 10https://gerrit.wikimedia.org/r/1189815 (https://phabricator.wikimedia.org/T403697) [09:56:08] (03PS3) 10Elukey: Enable ROCm 6.4.3 amd-smi on ml-serve{1012,1013} [puppet] - 10https://gerrit.wikimedia.org/r/1189816 (https://phabricator.wikimedia.org/T403697) [09:56:33] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST certificaterequests) on k8s-mlstaging@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s-mlstaging&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [09:56:37] (03CR) 10Filippo Giunchedi: [C:03+1] P:toolforge::k8s: Remove absent resources [puppet] - 10https://gerrit.wikimedia.org/r/1189800 (owner: 10Majavah) [09:57:15] (03PS4) 10Elukey: profile::amd_gpu: add support for amd-smi from ROCm 6.4.3 [puppet] - 10https://gerrit.wikimedia.org/r/1189815 (https://phabricator.wikimedia.org/T403697) [09:57:15] (03PS4) 10Elukey: Enable ROCm 6.4.3 amd-smi on ml-serve{1012,1013} [puppet] - 10https://gerrit.wikimedia.org/r/1189816 (https://phabricator.wikimedia.org/T403697) [09:57:34] (03PS2) 10Brouberol: opensearch-operator: allow egress traffic from the operator to the k8s api [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189817 (https://phabricator.wikimedia.org/T404906) [09:57:34] (03CR) 10Majavah: [C:03+2] P:toolforge::k8s: Remove absent resources [puppet] - 10https://gerrit.wikimedia.org/r/1189800 (owner: 10Majavah) [09:59:06] (03CR) 10Filippo Giunchedi: [C:03+1] "Nice!" [puppet] - 10https://gerrit.wikimedia.org/r/1189801 (https://phabricator.wikimedia.org/T283948) (owner: 10Majavah) [10:00:13] (03CR) 10Majavah: [C:03+2] P:toolforge::k8s::haproxy: Add a HTTPS listener [puppet] - 10https://gerrit.wikimedia.org/r/1189801 (https://phabricator.wikimedia.org/T283948) (owner: 10Majavah) [10:00:27] FIRING: [2x] ProbeDown: Service wdqs2009:443 has failed probes (http_wdqs_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs2009:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:02:13] !log klausman@deploy1003 helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [10:03:03] FIRING: PuppetFailure: Puppet has failed on wdqs2016:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:03:19] !log klausman@deploy1003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [10:07:11] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1189802 (https://phabricator.wikimedia.org/T283948) (owner: 10Majavah) [10:08:22] (03PS1) 10DCausse: Reapply "cirrus-streaming-updater: test flink bookworm base image" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189820 [10:09:45] !log jmm@cumin2002 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet [10:10:27] RESOLVED: [2x] ProbeDown: Service wdqs2009:443 has failed probes (http_wdqs_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs2009:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:11:25] !log klausman@deploy1003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [10:11:43] (03CR) 10DCausse: [C:03+2] Reapply "cirrus-streaming-updater: test flink bookworm base image" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189820 (owner: 10DCausse) [10:11:53] !log klausman@deploy1003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [10:12:06] !log jmm@cumin2002 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2033.codfw.wmnet [10:12:18] !log stevemunene@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [10:13:19] !log stevemunene@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [10:13:28] (03Merged) 10jenkins-bot: Reapply "cirrus-streaming-updater: test flink bookworm base image" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189820 (owner: 10DCausse) [10:14:40] !log dcausse@deploy1003 helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [10:14:52] !log dcausse@deploy1003 helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [10:16:58] (03CR) 10Slyngshede: [V:03+1] "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/1188770 (https://phabricator.wikimedia.org/T404688) (owner: 10Slyngshede) [10:18:37] (03PS3) 10Brouberol: opensearch-operator: allow egress traffic from the operator to the k8s api [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189817 (https://phabricator.wikimedia.org/T404906) [10:18:37] (03PS1) 10Brouberol: opensearch-operator: install the operator via admin_ng [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189822 (https://phabricator.wikimedia.org/T404906) [10:20:54] (03CR) 10Majavah: [C:03+2] P:toolforge::proxy: Set backend to new HAProxy HTTPS service [puppet] - 10https://gerrit.wikimedia.org/r/1189802 (https://phabricator.wikimedia.org/T283948) (owner: 10Majavah) [10:26:55] (03PS1) 10DCausse: flink-operator: allow upstream config to be applied [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189823 (https://phabricator.wikimedia.org/T404340) [10:27:56] (03CR) 10Muehlenhoff: Add support for AMD MI300X GPUs (kernel and firmwares) (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1189807 (https://phabricator.wikimedia.org/T403697) (owner: 10Elukey) [10:29:12] FIRING: SystemdUnitFailed: prometheus_amd_rocm_stats.service on ml-serve1013:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:29:34] (03CR) 10Slyngshede: [V:03+1] "Adding Jesse as well, for input on the changes to the git::clone." [puppet] - 10https://gerrit.wikimedia.org/r/1188770 (https://phabricator.wikimedia.org/T404688) (owner: 10Slyngshede) [10:31:52] (03Abandoned) 10Slyngshede: C:ip_reputation_vendors::datacenter_vendors: Known datacenters [puppet] - 10https://gerrit.wikimedia.org/r/1178866 (https://phabricator.wikimedia.org/T398161) (owner: 10Slyngshede) [10:36:26] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-dse_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:37:41] RECOVERY - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin1002 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [10:41:49] (03CR) 10Elukey: Add support for AMD MI300X GPUs (kernel and firmwares) (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1189807 (https://phabricator.wikimedia.org/T403697) (owner: 10Elukey) [10:58:42] (03CR) 10Brouberol: [C:03+1] "The CI diff looks good. No configuration key was removed." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189823 (https://phabricator.wikimedia.org/T404340) (owner: 10DCausse) [10:59:00] FIRING: [9x] SystemdUnitFailed: prometheus_ferm_mss.service on wdqs1018:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250919T0700) [11:00:05] jelto, arnoldokoth, and mutante: GitLab version upgrades (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250919T1100). Please do the needful. [11:03:05] (03PS4) 10MusikAnimal: tables-catalog: add CommunityRequests tables [puppet] - 10https://gerrit.wikimedia.org/r/1188413 (https://phabricator.wikimedia.org/T403559) [11:03:17] (03CR) 10Ladsgroup: [V:03+2 C:03+2] tables-catalog: add CommunityRequests tables [puppet] - 10https://gerrit.wikimedia.org/r/1188413 (https://phabricator.wikimedia.org/T403559) (owner: 10MusikAnimal) [11:04:37] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.12 point update - https://phabricator.wikimedia.org/T403852#11196911 (10MoritzMuehlenhoff) [11:06:32] (03PS1) 10Muehlenhoff: Reset maps1011 [puppet] - 10https://gerrit.wikimedia.org/r/1189829 (https://phabricator.wikimedia.org/T381565) [11:11:33] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST certificaterequests) on k8s-mlstaging@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s-mlstaging&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [11:12:30] (03PS1) 10Majavah: P:wmcs: root-keys: Update Taavi's keys to security keys [puppet] - 10https://gerrit.wikimedia.org/r/1189830 [11:14:44] (03CR) 10Muehlenhoff: Add support for AMD MI300X GPUs (kernel and firmwares) (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1189807 (https://phabricator.wikimedia.org/T403697) (owner: 10Elukey) [11:16:33] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST certificaterequests) on k8s-mlstaging@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s-mlstaging&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [11:29:23] (03PS1) 10Bartosz Wójtowicz: ml-services: Update the articletopic model on staging. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189839 (https://phabricator.wikimedia.org/T404294) [11:29:51] (03PS3) 10Cathal Mooney: Nokia: EBGP configuration base build [homer/public] - 10https://gerrit.wikimedia.org/r/1187092 (https://phabricator.wikimedia.org/T402577) [11:31:16] (03PS1) 10Majavah: P:toolforge::k8s::haproxy: Drop old TCP listener [puppet] - 10https://gerrit.wikimedia.org/r/1189840 (https://phabricator.wikimedia.org/T405078) [11:31:18] (03PS1) 10Majavah: P:toolforge::k8s::haproxy: Handle API gateway external access [puppet] - 10https://gerrit.wikimedia.org/r/1189841 (https://phabricator.wikimedia.org/T405078) [11:32:45] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 4): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/6998/co" [puppet] - 10https://gerrit.wikimedia.org/r/1189841 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [11:33:30] (03CR) 10Cathal Mooney: Nokia: EBGP configuration base build (039 comments) [homer/public] - 10https://gerrit.wikimedia.org/r/1187092 (https://phabricator.wikimedia.org/T402577) (owner: 10Cathal Mooney) [11:34:07] (03PS2) 10Majavah: P:toolforge::k8s::haproxy: Drop old TCP listener [puppet] - 10https://gerrit.wikimedia.org/r/1189840 (https://phabricator.wikimedia.org/T283948) [11:34:09] (03PS2) 10Majavah: P:toolforge::k8s::haproxy: Handle API gateway external access [puppet] - 10https://gerrit.wikimedia.org/r/1189841 (https://phabricator.wikimedia.org/T283948) [11:37:11] (03PS2) 10Bartosz Wójtowicz: ml-services: Update the articletopic model on staging. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189839 (https://phabricator.wikimedia.org/T404294) [11:41:27] (03CR) 10Muehlenhoff: [C:03+1] "I had a look at the current state of the docs and the existing docs already document the SSH client native to Windows 10 and later: https:" [puppet] - 10https://gerrit.wikimedia.org/r/875899 (https://phabricator.wikimedia.org/T198138) (owner: 10Majavah) [11:44:00] FIRING: CertAlmostExpired: Certificate for service data-gateway-staging:30443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#data-gateway-staging:30443 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:46:39] (03CR) 10Ilias Sarantopoulos: [C:03+1] "Let's go!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189839 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [11:46:53] (03PS3) 10Elukey: Add support for AMD MI300X GPUs (kernel and firmwares) [puppet] - 10https://gerrit.wikimedia.org/r/1189807 (https://phabricator.wikimedia.org/T403697) [11:46:53] (03PS5) 10Elukey: profile::amd_gpu: add support for amd-smi from ROCm 6.4.3 [puppet] - 10https://gerrit.wikimedia.org/r/1189815 (https://phabricator.wikimedia.org/T403697) [11:46:53] (03PS5) 10Elukey: Enable ROCm 6.4.3 amd-smi on ml-serve{1012,1013} [puppet] - 10https://gerrit.wikimedia.org/r/1189816 (https://phabricator.wikimedia.org/T403697) [11:47:23] (03CR) 10CI reject: [V:04-1] Add support for AMD MI300X GPUs (kernel and firmwares) [puppet] - 10https://gerrit.wikimedia.org/r/1189807 (https://phabricator.wikimedia.org/T403697) (owner: 10Elukey) [11:47:28] (03PS2) 10Majavah: hieradata: disable agent forwarding in eqiad1 [puppet] - 10https://gerrit.wikimedia.org/r/875899 (https://phabricator.wikimedia.org/T198138) [11:48:27] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1189815 (https://phabricator.wikimedia.org/T403697) (owner: 10Elukey) [11:49:06] (03PS4) 10Elukey: Add support for AMD MI300X GPUs (kernel and firmwares) [puppet] - 10https://gerrit.wikimedia.org/r/1189807 (https://phabricator.wikimedia.org/T403697) [11:49:06] (03PS6) 10Elukey: profile::amd_gpu: add support for amd-smi from ROCm 6.4.3 [puppet] - 10https://gerrit.wikimedia.org/r/1189815 (https://phabricator.wikimedia.org/T403697) [11:49:07] (03PS6) 10Elukey: Enable ROCm 6.4.3 amd-smi on ml-serve{1012,1013} [puppet] - 10https://gerrit.wikimedia.org/r/1189816 (https://phabricator.wikimedia.org/T403697) [11:49:53] (03CR) 10Bartosz Wójtowicz: [C:03+2] "Trying out the combined articletopic model!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189839 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [11:50:12] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/875899 (https://phabricator.wikimedia.org/T198138) (owner: 10Majavah) [11:51:45] (03Merged) 10jenkins-bot: ml-services: Update the articletopic model on staging. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189839 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [11:53:29] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1189807 (https://phabricator.wikimedia.org/T403697) (owner: 10Elukey) [11:55:25] FIRING: SystemdUnitFailed: imposm.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:59:05] (03PS1) 10Bartosz Wójtowicz: ml-services: Remove the transformer pod for articletopic on staging. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189850 (https://phabricator.wikimedia.org/T404294) [12:01:18] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.12 point update - https://phabricator.wikimedia.org/T403852#11197132 (10MoritzMuehlenhoff) [12:03:10] FIRING: BFDdown: BFD session down between cr2-eqdfw and fe80::b6f9:5dff:fe30:e538 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:04:06] (03CR) 10Filippo Giunchedi: [C:03+1] "Nice cleanup, tbh I don't fully grasp all the implications/moving parts but LGTM nonetheless" [puppet] - 10https://gerrit.wikimedia.org/r/1189841 (https://phabricator.wikimedia.org/T283948) (owner: 10Majavah) [12:04:18] (03CR) 10Filippo Giunchedi: [C:03+1] P:toolforge::k8s::haproxy: Drop old TCP listener [puppet] - 10https://gerrit.wikimedia.org/r/1189840 (https://phabricator.wikimedia.org/T283948) (owner: 10Majavah) [12:05:25] RESOLVED: SystemdUnitFailed: imposm.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:06:13] (03CR) 10Majavah: [C:03+2] P:toolforge::k8s::haproxy: Drop old TCP listener [puppet] - 10https://gerrit.wikimedia.org/r/1189840 (https://phabricator.wikimedia.org/T283948) (owner: 10Majavah) [12:08:10] RESOLVED: BFDdown: BFD session down between cr2-eqdfw and fe80::b6f9:5dff:fe30:e538 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:08:56] (03CR) 10DCausse: [C:04-1] "slithly worried by the log4j changes esp that it seems to pull a RollingFileAppender and a duplicate key on `appender.console.layout.type`" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189823 (https://phabricator.wikimedia.org/T404340) (owner: 10DCausse) [12:19:17] (03PS1) 10Muehlenhoff: Remove Hiera option disable agent forwarding [puppet] - 10https://gerrit.wikimedia.org/r/1189855 (https://phabricator.wikimedia.org/T198138) [12:19:38] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1189855 (https://phabricator.wikimedia.org/T198138) (owner: 10Muehlenhoff) [12:20:06] (03PS2) 10Muehlenhoff: Remove Hiera option to disable agent forwarding [puppet] - 10https://gerrit.wikimedia.org/r/1189855 (https://phabricator.wikimedia.org/T198138) [12:23:53] (03CR) 10Ilias Sarantopoulos: ml-services: Remove the transformer pod for articletopic on staging. (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189850 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [12:23:53] (03PS1) 10Jforrester: LQT->Flow converter: Add a dryRun flag [extensions/Flow] (wmf/1.45.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1189856 [12:24:43] (03PS1) 10Jforrester: LQT->Flow converter: Add flag to ignore $wgFlowReadOnly [extensions/Flow] (wmf/1.45.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1189857 [12:24:55] (03PS1) 10Jforrester: LQT->Flow converter: Skip pages which have no threads [extensions/Flow] (wmf/1.45.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1189858 [12:25:38] (03CR) 10Ilias Sarantopoulos: ml-services: Remove the transformer pod for articletopic on staging. (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189850 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [12:26:13] (03PS2) 10Bartosz Wójtowicz: ml-services: Remove the transformer pod for articletopic on staging. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189850 (https://phabricator.wikimedia.org/T404294) [12:27:46] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1189855 (https://phabricator.wikimedia.org/T198138) (owner: 10Muehlenhoff) [12:29:14] (03CR) 10Ilias Sarantopoulos: [C:03+1] ml-services: Remove the transformer pod for articletopic on staging. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189850 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [12:33:00] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: Remove the transformer pod for articletopic on staging. (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189850 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [12:34:52] (03Merged) 10jenkins-bot: ml-services: Remove the transformer pod for articletopic on staging. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189850 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [12:36:35] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Move lvs1020 link from ssw1-f1-eqiad to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T404959#11197244 (10cmooney) [12:37:26] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' . [12:37:39] (03PS1) 10Tchanders: Increase the number of shards used for temp user name generation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189861 (https://phabricator.wikimedia.org/T404131) [12:38:53] (03CR) 10Tchanders: [C:04-2] "This must not be done until new shards have been added to the database table, since we need to initialise them with a value greater than 1" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189861 (https://phabricator.wikimedia.org/T404131) (owner: 10Tchanders) [12:39:37] (03CR) 10Dreamy Jazz: [C:04-1] "Until we have populated the values for these new shards on production so that no collisions occur per WMF slack conversations" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189861 (https://phabricator.wikimedia.org/T404131) (owner: 10Tchanders) [12:39:55] (03CR) 10Dreamy Jazz: [C:04-1] "Edit conflict 😄" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189861 (https://phabricator.wikimedia.org/T404131) (owner: 10Tchanders) [12:40:08] (03CR) 10Dreamy Jazz: Increase the number of shards used for temp user name generation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189861 (https://phabricator.wikimedia.org/T404131) (owner: 10Tchanders) [12:44:00] FIRING: [2x] OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [12:53:09] (03PS2) 10DCausse: flink-operator: allow upstream config to be applied [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189823 (https://phabricator.wikimedia.org/T404340) [12:53:23] (03Abandoned) 10DCausse: flink-session-cluster: Remove service.name from the ECS logger [deployment-charts] - 10https://gerrit.wikimedia.org/r/715532 (owner: 10DCausse) [12:57:19] (03PS3) 10DCausse: flink-operator: allow upstream config to be applied [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189823 (https://phabricator.wikimedia.org/T404340) [13:01:01] (03PS1) 10Dreamy Jazz: SI: Add a configuration flag to hide SI even if the feature is enabled [extensions/CheckUser] (wmf/1.45.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1189869 (https://phabricator.wikimedia.org/T405076) [13:01:16] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 22 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/CheckUser] (wmf/1.45.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1189869 (https://phabricator.wikimedia.org/T405076) (owner: 10Dreamy Jazz) [13:07:21] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.12 point update - https://phabricator.wikimedia.org/T403852#11197394 (10MoritzMuehlenhoff) [13:09:52] (03PS1) 10Majavah: P:toolforge::k8s::haproxy: Allow passing list of IPs for VIPs [puppet] - 10https://gerrit.wikimedia.org/r/1189870 (https://phabricator.wikimedia.org/T405078) [13:11:49] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:13:13] (03CR) 10CDanis: "overall lgtm but some nits & question" [puppet] - 10https://gerrit.wikimedia.org/r/1188770 (https://phabricator.wikimedia.org/T404688) (owner: 10Slyngshede) [13:15:30] 10ops-codfw, 06SRE, 06DC-Ops, 06Traffic: Create boot environment of Bullseye with a 6.1 kernel - https://phabricator.wikimedia.org/T405102 (10MoritzMuehlenhoff) 03NEW [13:22:10] (03PS1) 10Ilias Sarantopoulos: ml-services: update articletopic in prod and remove trasnsformer [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189871 (https://phabricator.wikimedia.org/T404294) [13:25:32] (03CR) 10Elukey: [C:03+1] Reset maps1011 [puppet] - 10https://gerrit.wikimedia.org/r/1189829 (https://phabricator.wikimedia.org/T381565) (owner: 10Muehlenhoff) [13:37:45] 10SRE-swift-storage, 06Commons, 10MediaWiki-File-management, 10Thumbor, 10UploadWizard: "Could not acquire lock" error when publishing larger files - https://phabricator.wikimedia.org/T386640#11197495 (10Ladsgroup) Random note: Let's please add some prefix to the locks: ` 127.0.0.1:6381> RANDOMKEY "0de6... [13:48:15] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 553050776 and 20 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [13:49:15] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 63144 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [13:51:06] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [13:51:13] btullis@cumin1003 decommission (PID 3256066) is awaiting input [13:56:06] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [13:57:06] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [14:02:06] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [14:03:03] FIRING: PuppetFailure: Puppet has failed on wdqs2016:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:03:25] (03CR) 10Peter Fischer: [C:03+1] "LGTM" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189823 (https://phabricator.wikimedia.org/T404340) (owner: 10DCausse) [14:05:31] !log jmm@cumin2002 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet [14:11:54] (03PS1) 10Phuedx: xLab: Fix instrument to produce valid events [extensions/WikimediaEvents] (wmf/1.45.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1189877 (https://phabricator.wikimedia.org/T404420) [14:12:18] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 22 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/WikimediaEvents] (wmf/1.45.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1189877 (https://phabricator.wikimedia.org/T404420) (owner: 10Phuedx) [14:15:01] !log btullis@cumin1003 START - Cookbook sre.dns.netbox [14:15:11] (03PS2) 10Huei Tan: xLab: Fix instrument to produce valid events [extensions/WikimediaEvents] (wmf/1.45.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1189877 (https://phabricator.wikimedia.org/T404420) (owner: 10Phuedx) [14:17:46] !log btullis@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:19:00] 10ops-codfw, 06SRE, 06DC-Ops: Inbound errors on interface cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549) {#12267}) - https://phabricator.wikimedia.org/T404472#11197628 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm blipped a week ago. no isssues detected since then. anoma... [14:20:11] !log jhancock@cumin1002 START - Cookbook sre.hosts.provision for host mc-misc2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [14:21:55] !log btullis@cumin1003 START - Cookbook sre.hosts.decommission for hosts an-worker[1079-1094].eqiad.wmnet [14:25:08] (03PS1) 10Kosta Harlan: hCaptcha: Enable on API account creation on test2wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189879 (https://phabricator.wikimedia.org/T405107) [14:25:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 22 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189879 (https://phabricator.wikimedia.org/T405107) (owner: 10Kosta Harlan) [14:29:12] FIRING: SystemdUnitFailed: prometheus_amd_rocm_stats.service on ml-serve1013:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:30:38] !log jhancock@cumin1002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc-misc2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [14:31:17] !log jhancock@cumin1002 START - Cookbook sre.hosts.reimage for host mc-misc2001.codfw.wmnet with OS bookworm [14:31:25] 10ops-codfw, 06SRE, 06DC-Ops, 06serviceops: Q1:rack/setup/install mc-misc200[12] - https://phabricator.wikimedia.org/T372800#11197648 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host mc-misc2001.codfw.wmnet with OS bookworm [14:34:27] FIRING: SystemdUnitCrashLoop: wdqs-blazegraph.service crashloop on wdqs2016:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [14:36:41] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-dse_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:38:16] (03CR) 10Scott French: "Thanks, Jasmine! This looks right to me, doubling checking against the live config." [dns] - 10https://gerrit.wikimedia.org/r/1189587 (https://phabricator.wikimedia.org/T399891) (owner: 10Jasmine) [14:39:17] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 220494584 and 13 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [14:40:19] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 91016 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [14:40:19] (03CR) 10Scott French: [C:03+1] "Thanks, Jasmine!" [dns] - 10https://gerrit.wikimedia.org/r/1189598 (https://phabricator.wikimedia.org/T399891) (owner: 10Jasmine) [14:40:38] (03CR) 10Gmodena: [C:03+1] "Good catch!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189823 (https://phabricator.wikimedia.org/T404340) (owner: 10DCausse) [14:50:19] btullis@cumin1003 decommission (PID 3259716) is awaiting input [14:51:44] PROBLEM - mailman archives on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [14:51:44] PROBLEM - mailman list info on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [14:53:41] !log jhancock@cumin1002 START - Cookbook sre.hosts.provision for host mc-misc2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [14:53:57] RESOLVED: SystemdUnitCrashLoop: wdqs-blazegraph.service crashloop on wdqs2016:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [14:55:27] 10SRE-SLO: Pyrra calculations for the Initial error budget value of calendar windows - https://phabricator.wikimedia.org/T403729#11197685 (10elukey) This is an example of what could become confusing for an SLO owner: {F66081126} The Tone Check's service SLO is related to the amount of HTTP 200 responses under... [14:56:40] RECOVERY - mailman list info on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 9235 bytes in 6.735 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [14:56:40] RECOVERY - mailman archives on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 54829 bytes in 6.821 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [14:58:27] btullis@cumin1003 decommission (PID 3259716) is awaiting input [14:58:39] (03CR) 10Cwhite: [C:03+2] Revert "logstash: drop 99% eventgate-analytics-external logs" [puppet] - 10https://gerrit.wikimedia.org/r/1189625 (owner: 10Cwhite) [14:58:56] !log jhancock@cumin1002 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc-misc2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [14:59:00] FIRING: [9x] SystemdUnitFailed: prometheus_ferm_mss.service on wdqs1018:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:01:59] !log jhancock@cumin1002 START - Cookbook sre.hosts.provision for host mc-misc2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [15:02:20] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 674880544 and 55 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [15:05:20] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 0 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [15:09:00] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:12:26] !log jhancock@cumin1002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc-misc2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [15:14:47] (03PS1) 10CDanis: docker_registry: limit layer blobs to 4.5GB [puppet] - 10https://gerrit.wikimedia.org/r/1189886 (https://phabricator.wikimedia.org/T404742) [15:20:35] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [15:20:49] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [15:25:35] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [15:25:41] (03CR) 10Alexandros Kosiaris: [C:03+1] "Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1189886 (https://phabricator.wikimedia.org/T404742) (owner: 10CDanis) [15:26:52] (03CR) 10Dzahn: "ah, good point about the facter version. thanks for adding that" [puppet] - 10https://gerrit.wikimedia.org/r/1180999 (owner: 10Dzahn) [15:27:31] (03PS1) 10Dreamy Jazz: SI: Enable on enwiki and frwiki while hiding special page [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189890 (https://phabricator.wikimedia.org/T405109) [15:27:52] (03PS2) 10Dreamy Jazz: SI: Enable on enwiki and frwiki while hiding special page [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189890 (https://phabricator.wikimedia.org/T405109) [15:28:18] (03CR) 10Elukey: docker_registry: limit layer blobs to 4.5GB (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1189886 (https://phabricator.wikimedia.org/T404742) (owner: 10CDanis) [15:29:34] FIRING: DiskSpace: Disk space serpens:9100:/ 3.618% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=serpens - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:29:50] 10SRE-SLO: Evaluate Sloth as a possible replacement for Pyrra - https://phabricator.wikimedia.org/T404171#11197772 (10elukey) Today I tried to implement the Sloth rolling window for ToneCheck's latency SLO and I came up with: ` 1 - (( sum_over_time(( ( sum(rate(istio_request_duration_milliseconds_count{app="is... [15:29:59] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns for sretest2009 - cmooney@cumin1003" [15:30:31] (03CR) 10CDanis: docker_registry: limit layer blobs to 4.5GB (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1189886 (https://phabricator.wikimedia.org/T404742) (owner: 10CDanis) [15:31:09] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns for sretest2009 - cmooney@cumin1003" [15:31:09] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:34:00] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:34:07] (03CR) 10Alexandros Kosiaris: [C:03+1] docker_registry: limit layer blobs to 4.5GB (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1189886 (https://phabricator.wikimedia.org/T404742) (owner: 10CDanis) [15:36:13] PROBLEM - mailman list info ssl expiry on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [15:36:51] btullis@cumin1003 decommission (PID 3259716) is awaiting input [15:38:32] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "sync subnet info for vlan in codfw rack e2 - cmooney@cumin1003" [15:38:58] !log cmooney@cumin1003 END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "sync subnet info for vlan in codfw rack e2 - cmooney@cumin1003" [15:39:03] RECOVERY - mailman list info ssl expiry on lists1004 is OK: OK - Certificate lists.wikimedia.org will expire on Fri 05 Dec 2025 08:25:21 PM GMT +0000. https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [15:39:29] (03CR) 10JHathaway: P:puppetserver::volatile Include XCheeseScore private repo (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1188770 (https://phabricator.wikimedia.org/T404688) (owner: 10Slyngshede) [15:42:01] (03PS1) 10Majavah: redis: Drop unused client class [puppet] - 10https://gerrit.wikimedia.org/r/1189893 [15:42:50] (03Abandoned) 10Majavah: proxylistener.py: Port to Python 3 [puppet] - 10https://gerrit.wikimedia.org/r/670937 (https://phabricator.wikimedia.org/T247364) (owner: 10CRusnov) [15:44:00] FIRING: CertAlmostExpired: Certificate for service data-gateway-staging:30443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#data-gateway-staging:30443 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:51:04] (03CR) 10Elukey: [C:03+1] docker_registry: limit layer blobs to 4.5GB (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1189886 (https://phabricator.wikimedia.org/T404742) (owner: 10CDanis) [15:52:04] !log jhancock@cumin1002 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc-misc2001.codfw.wmnet with OS bookworm [15:52:11] 10ops-codfw, 06SRE, 06DC-Ops, 06serviceops: Q1:rack/setup/install mc-misc200[12] - https://phabricator.wikimedia.org/T372800#11197842 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host mc-misc2001.codfw.wmnet with OS bookworm executed with errors: - mc-mi... [15:52:18] !log btullis@cumin1003 START - Cookbook sre.dns.netbox [15:52:56] (03PS1) 10Divyaratann Srivastava: Update swagger documentation for CertificateSigningRequestSpec and ResourceClass [software/cfssl-issuer] - 10https://gerrit.wikimedia.org/r/1189894 (https://phabricator.wikimedia.org/T201491) [15:58:03] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1189893 (owner: 10Majavah) [15:58:04] btullis@cumin1003 decommission (PID 3259716) is awaiting input [15:58:27] (03CR) 10Majavah: [C:03+2] redis: Drop unused client class [puppet] - 10https://gerrit.wikimedia.org/r/1189893 (owner: 10Majavah) [16:01:02] (03CR) 10Scott French: [C:03+1] "Thanks, Chris!" [puppet] - 10https://gerrit.wikimedia.org/r/1189886 (https://phabricator.wikimedia.org/T404742) (owner: 10CDanis) [16:01:53] (03PS1) 10Cathal Mooney: Network data: add new subnets for eqiad/codfw nokia racks [puppet] - 10https://gerrit.wikimedia.org/r/1189898 (https://phabricator.wikimedia.org/T402590) [16:03:58] !log btullis@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-worker[1079-1094].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [16:04:51] !log btullis@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-worker[1079-1094].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [16:04:51] !log btullis@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [16:04:52] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-worker[1079-1094].eqiad.wmnet [16:05:26] (03CR) 10JHathaway: [C:03+1] Network data: add new subnets for eqiad/codfw nokia racks [puppet] - 10https://gerrit.wikimedia.org/r/1189898 (https://phabricator.wikimedia.org/T402590) (owner: 10Cathal Mooney) [16:06:40] (03CR) 10Cathal Mooney: [C:03+2] Network data: add new subnets for eqiad/codfw nokia racks [puppet] - 10https://gerrit.wikimedia.org/r/1189898 (https://phabricator.wikimedia.org/T402590) (owner: 10Cathal Mooney) [16:08:58] (03CR) 10Scott French: [C:03+1] "Oh, and to elaborate on my comment about chunked blob uploads:" [puppet] - 10https://gerrit.wikimedia.org/r/1189886 (https://phabricator.wikimedia.org/T404742) (owner: 10CDanis) [16:12:49] (03CR) 10CDanis: [C:03+1] "Thanks all!" [puppet] - 10https://gerrit.wikimedia.org/r/1189886 (https://phabricator.wikimedia.org/T404742) (owner: 10CDanis) [16:13:33] (03PS4) 10Brouberol: opensearch-operator: allow egress traffic from the operator to the k8s api [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189817 (https://phabricator.wikimedia.org/T404906) [16:14:35] !log cmooney@cumin1003 START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS bookworm [16:14:42] 10ops-codfw, 06SRE, 06DC-Ops: sretest2009 test in nokia rack - https://phabricator.wikimedia.org/T404115#11197883 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1003 for host sretest2009.codfw.wmnet with OS bookworm [16:15:11] (03PS5) 10Bking: opensearch-operator: allow egress traffic from the operator to the k8s api [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189817 (https://phabricator.wikimedia.org/T404906) (owner: 10Brouberol) [16:16:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:16:16] (03PS6) 10Bking: opensearch-operator: allow egress traffic from the operator to the k8s api [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189817 (https://phabricator.wikimedia.org/T404906) (owner: 10Brouberol) [16:20:47] (03PS1) 10GergesShamon: eswiki, commonswiki, wikidata: lift IP cap for edit-a-thon [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189902 (https://phabricator.wikimedia.org/T405095) [16:21:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:21:39] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 22 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189902 (https://phabricator.wikimedia.org/T405095) (owner: 10GergesShamon) [16:21:42] (03PS1) 10Andrew Bogott: Bump horizon docker version [puppet] - 10https://gerrit.wikimedia.org/r/1189903 [16:22:20] (03CR) 10Andrew Bogott: [C:03+2] Bump horizon docker version [puppet] - 10https://gerrit.wikimedia.org/r/1189903 (owner: 10Andrew Bogott) [16:25:23] !log cmooney@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2009.codfw.wmnet with OS bookworm [16:25:30] 10ops-codfw, 06SRE, 06DC-Ops: sretest2009 test in nokia rack - https://phabricator.wikimedia.org/T404115#11197907 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1003 for host sretest2009.codfw.wmnet with OS bookworm executed with errors: - sretest2009 (**FAIL**) - Downt... [16:26:15] !log cmooney@cumin1003 START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS bookworm [16:26:23] 10ops-codfw, 06SRE, 06DC-Ops: sretest2009 test in nokia rack - https://phabricator.wikimedia.org/T404115#11197909 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1003 for host sretest2009.codfw.wmnet with OS bookworm [16:29:03] !log cmooney@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2009.codfw.wmnet with OS bookworm [16:29:11] 10ops-codfw, 06SRE, 06DC-Ops: sretest2009 test in nokia rack - https://phabricator.wikimedia.org/T404115#11197911 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1003 for host sretest2009.codfw.wmnet with OS bookworm executed with errors: - sretest2009 (**FAIL**) - Remov... [16:29:26] !log cmooney@cumin1003 START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS bookworm [16:29:34] 10ops-codfw, 06SRE, 06DC-Ops: sretest2009 test in nokia rack - https://phabricator.wikimedia.org/T404115#11197912 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1003 for host sretest2009.codfw.wmnet with OS bookworm [16:33:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:34:43] (03CR) 10Btullis: opensearch-operator: allow egress traffic from the operator to the k8s api (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189817 (https://phabricator.wikimedia.org/T404906) (owner: 10Brouberol) [16:36:44] PROBLEM - mailman archives on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [16:36:44] PROBLEM - mailman list info on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [16:37:05] (03PS1) 10Ahmon Dancy: mediawiki services: Update path to scap-created yaml file [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189905 (https://phabricator.wikimedia.org/T405110) [16:38:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:40:35] 10SRE-tools, 06Infrastructure-Foundations, 07Python3-Porting: Puppet: forbid new Python2 code - https://phabricator.wikimedia.org/T197804#11197934 (10Tacsipacsi) 05Invalid→03Declined Actually, it was a valid request at the time, it just became unnecessary since then. [16:41:40] RECOVERY - mailman archives on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 54829 bytes in 5.521 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [16:41:40] RECOVERY - mailman list info on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 9235 bytes in 5.579 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [16:44:00] FIRING: [2x] OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [16:52:08] !log jhathaway@cumin2002 START - Cookbook sre.hosts.provision for host sretest2009.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [16:54:37] !log cmooney@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2009.codfw.wmnet with OS bookworm [16:54:43] 10ops-codfw, 06SRE, 06DC-Ops: sretest2009 test in nokia rack - https://phabricator.wikimedia.org/T404115#11197953 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1003 for host sretest2009.codfw.wmnet with OS bookworm executed with errors: - sretest2009 (**FAIL**) - Remov... [16:54:52] (03CR) 10Reedy: hCaptcha: Enable on API account creation on test2wiki (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189879 (https://phabricator.wikimedia.org/T405107) (owner: 10Kosta Harlan) [16:56:00] !log jhathaway@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2009.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [16:57:25] reports that folks are unable to publish edits on enwiki [16:58:04] looks to be a replag issue [16:58:11] PROBLEM - MariaDB Replica IO: s1 #page on db2203 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2013, Errmsg: error connecting to master repl2024@db1163.eqiad.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Lost connection to server at waiting for initial communication packet, system error: 110 Connection timed out https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [16:58:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 0.7669% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:58:16] that'd be it [16:58:23] that'd do it [16:58:26] I'll depool [16:58:27] !incidents [16:58:28] 6761 (UNACKED) db2203 (paged)/MariaDB Replica IO: s1 (paged) [16:58:28] 6760 (RESOLVED) [3x] ProbeDown sre (phab1004:443 probes/custom eqiad) [16:58:36] !ack 6761 [16:58:37] 6761 (ACKED) db2203 (paged)/MariaDB Replica IO: s1 (paged) [16:58:43] oh, thanks taavi! [16:58:47] thanks taavi! [16:58:54] oh dear [16:58:57] that's the codfw master [16:59:07] ... I was just about to ask, yeah [16:59:07] 0.7669% idle [16:59:11] so I can't just depool it [16:59:18] I was about to say [16:59:28] We have circular replication running right now [17:00:09] uh I think we're actually in a quite bad outage situation if mw-api-ext is full [17:00:16] `show slave status\G` shows 'error connecting to master [...] "connection timed out"' [17:00:47] mw-api-ext in eqiad is 100% saturated [17:00:58] https://grafana.wikimedia.org/goto/BINRSfjNg?orgId=1 [17:01:41] alright, so, we're being paged for s1 codfw master, application layer badness appears to be in eqiad [17:01:46] (just recapping) [17:01:51] yep [17:02:04] I guess s1 edits on eqiad would be timing out because the codfw master can't have things written into? [17:02:12] that seems likely [17:02:13] would setting s1 read-only on the app layer help? [17:02:15] FIRING: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-ext releases routed via main (k8s) 2.5s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [17:02:45] taavi: probably [17:02:46] https://grafana.wikimedia.org/d/35WSHOjVk/application-servers-red-k8s?orgId=1&from=2025-09-19T16:02:17.226Z&to=2025-09-19T17:02:17.226Z&timezone=utc&var-site=eqiad&var-deployment=mw-api-ext&var-method=GET&var-code=200&var-handler=php&var-service=mediawiki&refresh=1m&viewPanel=panel-37 [17:02:51] I think this panel validates the theory [17:02:57] !log cmooney@cumin1003 START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS bookworm [17:03:00] should I do that? [17:03:04] 10ops-codfw, 06SRE, 06DC-Ops: sretest2009 test in nokia rack - https://phabricator.wikimedia.org/T404115#11197971 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1003 for host sretest2009.codfw.wmnet with OS bookworm [17:03:08] taavi: please go ahead [17:03:15] db1163 (eqiad s1 master) is pegged on CPU [17:03:15] FIRING: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 0.1995% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:03:25] swfrench-wmf: possibly, trying to replicate [17:03:26] PROBLEM - MariaDB Replica Lag: s1 on clouddb1013 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 605.91 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:03:28] PROBLEM - MariaDB Replica Lag: s1 on db1154 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 609.08 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:03:29] PROBLEM - MariaDB Replica Lag: s1 #page on db1206 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 609.12 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:03:32] but making s1 ro hopefully limits the badness on mw-api-ext [17:03:33] PROBLEM - MariaDB Replica Lag: s1 #page on db1169 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 613.26 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:03:35] PROBLEM - MariaDB Replica Lag: s1 #page on db1232 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 614.89 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:03:35] PROBLEM - MariaDB Replica Lag: s1 #page on db1218 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 614.94 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:03:36] PROBLEM - MariaDB Replica Lag: s1 #page on db1234 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 615.10 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:03:37] PROBLEM - MariaDB Replica Lag: s1 on dbstore1008 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 615.45 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:03:37] PROBLEM - MariaDB Replica Lag: s1 #page on db1235 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 615.47 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:03:38] Can it actually be a problem with db1163? [17:03:38] PROBLEM - MariaDB Replica Lag: s1 #page on db1251 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 615.55 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:03:41] ok let's move to #-sre [17:03:47] PROBLEM - MariaDB Replica Lag: s1 #page on db1186 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 626.11 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:03:47] looking [17:03:52] PROBLEM - MariaDB Replica Lag: s1 on clouddb1017 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 632.81 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:03:59] PROBLEM - MariaDB Replica Lag: s1 #page on db1184 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 639.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:04:00] PROBLEM - MariaDB Replica Lag: s1 #page on db1196 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 639.05 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:04:03] !log taavi@cumin1003 dbctl commit (dc=all): 'set s1 ro', diff saved to https://phabricator.wikimedia.org/P83441 and previous config saved to /var/cache/conftool/dbconfig/20250919-170402-taavi.json [17:04:03] PROBLEM - MariaDB Replica Lag: s1 #page on db1195 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 642.83 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:04:03] PROBLEM - MariaDB Replica Lag: s1 on an-redacteddb1001 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 642.89 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:04:07] PROBLEM - MariaDB Replica Lag: s1 #page on db1219 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 646.11 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:04:13] PROBLEM - MariaDB Replica Lag: s1 #page on db2203 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 652.17 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:04:14] PROBLEM - MariaDB Replica IO: s1 #page on db2203 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 1040, Errmsg: error reconnecting to master repl2024@db1163.eqiad.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Too many connections https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:05:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [17:05:27] PROBLEM - MariaDB read only s1 #page on db1163 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [17:08:15] FIRING: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 23.2% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:08:42] cmooney@cumin1003 reimage (PID 3277370) is awaiting input [17:08:51] !incidents [17:08:52] 6761 (ACKED) db2203 (paged)/MariaDB Replica IO: s1 (paged) [17:08:52] 6762 (ACKED) db1206 (paged)/MariaDB Replica Lag: s1 (paged) [17:08:52] 6763 (ACKED) db1232 (paged)/MariaDB Replica Lag: s1 (paged) [17:08:53] 6764 (ACKED) db1218 (paged)/MariaDB Replica Lag: s1 (paged) [17:08:53] 6765 (ACKED) db1235 (paged)/MariaDB Replica Lag: s1 (paged) [17:08:53] 6766 (ACKED) db1251 (paged)/MariaDB Replica Lag: s1 (paged) [17:08:53] 6767 (ACKED) db1234 (paged)/MariaDB Replica Lag: s1 (paged) [17:08:53] 6768 (ACKED) db1169 (paged)/MariaDB Replica Lag: s1 (paged) [17:08:54] 6769 (ACKED) db1186 (paged)/MariaDB Replica Lag: s1 (paged) [17:08:54] 6770 (ACKED) db1196 (paged)/MariaDB Replica Lag: s1 (paged) [17:08:54] 6771 (ACKED) db1184 (paged)/MariaDB Replica Lag: s1 (paged) [17:08:55] 6772 (ACKED) db1195 (paged)/MariaDB Replica Lag: s1 (paged) [17:09:03] !log cmooney@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2009.codfw.wmnet with OS bookworm [17:09:15] 10ops-codfw, 06SRE, 06DC-Ops: sretest2009 test in nokia rack - https://phabricator.wikimedia.org/T404115#11197992 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1003 for host sretest2009.codfw.wmnet with OS bookworm executed with errors: - sretest2009 (**FAIL**) - Remov... [17:10:15] RESOLVED: [4x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [17:10:57] (03CR) 10Kimberly Sarabia: [C:03+1] ReaderExperiments' ImageBrowsing stream configuration (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1187413 (https://phabricator.wikimedia.org/T403259) (owner: 10Marco Fossati) [17:12:24] !log jhathaway@cumin2002 START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie [17:12:42] !log jhathaway@cumin2002 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2009.codfw.wmnet with OS trixie [17:12:47] RECOVERY - MariaDB Replica Lag: s1 #page on db1186 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:12:52] RECOVERY - MariaDB Replica Lag: s1 on clouddb1017 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:12:59] RECOVERY - MariaDB Replica Lag: s1 #page on db1184 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:00] RECOVERY - MariaDB Replica Lag: s1 #page on db1196 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:03] RECOVERY - MariaDB Replica Lag: s1 #page on db1195 is OK: OK slave_sql_lag Replication lag: 0.02 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:03] RECOVERY - MariaDB Replica Lag: s1 on an-redacteddb1001 is OK: OK slave_sql_lag Replication lag: 0.12 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:07] RECOVERY - MariaDB Replica Lag: s1 #page on db1219 is OK: OK slave_sql_lag Replication lag: 0.48 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:13] RECOVERY - MariaDB Replica IO: s1 #page on db2203 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:14] RECOVERY - MariaDB Replica Lag: s1 #page on db2203 is OK: OK slave_sql_lag Replication lag: 0.47 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:15] RESOLVED: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 23.2% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:13:25] RECOVERY - MariaDB read only s1 #page on db1163 is OK: Version 10.6.22-MariaDB-log, Uptime 1484984s, read_only: False, event_scheduler: True, 81.63 QPS, connection latency: 0.025246s, query latency: 0.000578s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [17:13:26] RECOVERY - MariaDB Replica Lag: s1 on clouddb1013 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:30] RECOVERY - MariaDB Replica Lag: s1 on db1154 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:31] RECOVERY - MariaDB Replica Lag: s1 #page on db1206 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:35] RECOVERY - MariaDB Replica Lag: s1 #page on db1169 is OK: OK slave_sql_lag Replication lag: 0.08 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:36] RECOVERY - MariaDB Replica Lag: s1 #page on db1232 is OK: OK slave_sql_lag Replication lag: 0.20 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:37] RECOVERY - MariaDB Replica Lag: s1 #page on db1218 is OK: OK slave_sql_lag Replication lag: 0.22 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:37] RECOVERY - MariaDB Replica Lag: s1 on dbstore1008 is OK: OK slave_sql_lag Replication lag: 0.35 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:38] RECOVERY - MariaDB Replica Lag: s1 #page on db1234 is OK: OK slave_sql_lag Replication lag: 0.36 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:39] RECOVERY - MariaDB Replica Lag: s1 #page on db1235 is OK: OK slave_sql_lag Replication lag: 0.43 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:13:40] RECOVERY - MariaDB Replica Lag: s1 #page on db1251 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [17:16:25] !log ladsgroup@cumin1003 dbctl commit (dc=all): 'Set s1 to RW', diff saved to https://phabricator.wikimedia.org/P83443 and previous config saved to /var/cache/conftool/dbconfig/20250919-171624-ladsgroup.json [17:16:49] I set it to RW [17:17:15] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-ext releases routed via main (k8s) 1.096s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [17:19:58] PROBLEM - Check correctness of the icinga configuration on alert1002 is CRITICAL: Icinga configuration contains errors https://wikitech.wikimedia.org/wiki/Icinga [17:23:58] ^ icinga config is complaining about a nonexistent "lsw1-e2-codfw" host group for sretest2009 [17:29:00] FIRING: [2x] OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [17:34:25] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "force sync to remove sretest2009 - cmooney@cumin1003" [17:36:24] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "force sync to remove sretest2009 - cmooney@cumin1003" [17:37:05] (03CR) 10Dzahn: [C:03+2] zuul::main: let nodepool connect to zookeeper on the host machine [puppet] - 10https://gerrit.wikimedia.org/r/1189602 (https://phabricator.wikimedia.org/T401614) (owner: 10Dzahn) [17:44:23] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189913 [17:46:47] woah, many changes to networking/firewall constants on puppet run. guess a new network was just introduced [17:48:28] !log cmooney@cumin1003 START - Cookbook sre.hosts.decommission for hosts sretest2009.codfw.wmnet [17:49:51] (03CR) 10Jforrester: SI: Enable on enwiki and frwiki while hiding special page (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189890 (https://phabricator.wikimedia.org/T405109) (owner: 10Dreamy Jazz) [17:51:51] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [17:54:50] taavi: ACK. looks like topranks is removing that host now. Hope it recovers after that. [17:54:53] topranks: cc [17:56:25] sorry guys yeah I'm doing my best to sort it out here [17:56:32] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: sretest2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cmooney@cumin1003" [17:56:37] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: sretest2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cmooney@cumin1003" [17:56:37] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [17:56:38] !log cmooney@cumin1003 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts sretest2009.codfw.wmnet [17:56:40] with any luck will clear shortly [17:56:44] 10ops-codfw, 06SRE, 06DC-Ops: sretest2009 test in nokia rack - https://phabricator.wikimedia.org/T404115#11198112 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by cmooney@cumin1003 for hosts: `sretest2009.codfw.wmnet` - sretest2009.codfw.wmnet (**FAIL**) - //Host not found on Icinga, una... [17:56:49] (03CR) 10Dzahn: "Oh, right. I think it's time to get your access fixed at this point." [puppet] - 10https://gerrit.wikimedia.org/r/1188863 (https://phabricator.wikimedia.org/T404757) (owner: 10Dwisehaupt) [17:57:06] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [17:57:08] thanks topranks .ack [17:57:37] dwisehaupt: are you around for a.. deploy? [17:58:17] (03CR) 10Dzahn: [C:03+2] zuul::executor: let executor connect to zookeeper on the host machine [puppet] - 10https://gerrit.wikimedia.org/r/1189604 (https://phabricator.wikimedia.org/T403847) (owner: 10Dzahn) [17:58:53] mutante: the new networks I will leave, they are for new vlans we added and will be live in the coming weeks, I don't think they should be a problem for anything [17:59:33] my test host has to be taken offline as Icinga can't find the parent switch, and I can't add the switch to monitoring as it's a Nokia and as things stand it would try to run the Juniper checks against it [17:59:50] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [17:59:55] topranks: no problem there. it was just a little surprising to me to have a full screen of firewall changes when I reactivated puppet on a host I had deactivated the other day [18:00:28] yea, the icinga config thing is a separate topic [18:01:24] puppet will not restart it though as it did in the past..so the syntax error does not break it until someone manually restarts it [18:01:27] mutante: i am. [18:01:55] dwisehaupt: in that case.. I can merge the civicrm change.. if you can double check it after [18:02:02] can do. [18:02:08] (03CR) 10Dzahn: [C:03+2] crm: Update civicrm settings template for v6.6.1 [puppet] - 10https://gerrit.wikimedia.org/r/1188863 (https://phabricator.wikimedia.org/T404757) (owner: 10Dwisehaupt) [18:03:03] FIRING: PuppetFailure: Puppet has failed on wdqs2016:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:04:18] dwisehaupt: it's merged on the puppetserver. running puppet on crm2001 [18:07:17] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "remove sretest2009 - cmooney@cumin1003" [18:07:22] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "remove sretest2009 - cmooney@cumin1003" [18:13:27] guys still looking at this. sretest2009 remains in puppet despite the decom cookbook having been run for it [18:15:59] hmmm.. it is supposed to delete the host from puppetdb [18:16:24] and that would normally influence icinga to remove it on next puppet run [18:16:45] sounds like what can happen if the cookbook got interrupted or something [18:18:32] the decom cookbook seemed to run fine, but not find it [18:18:49] I think maybe the issue is the reimage cookbook didn't really work for it, so not all the bits are there [18:19:04] could be it..yea [18:19:20] I tried a manual "puppet node clean sretest2009.codfw.wmnet" on puppetserver1001 but that said "Could not find files to clean for sretest2009.codfw.wmnet" [18:19:37] I'll see if jesse can help when he is back [18:20:31] `puppet node clean` is for certificates, for puppetdb try `puppet node deactivate sretest2009.codfw.wmnet` [18:20:43] good point..this [18:20:46] taavi: ah ok thanks [18:20:52] I was following wikitech will try that [18:21:16] Submitted 'deactivate node' for sretest2009.codfw.wmnet with UUID 72e10ba6-879d-418e-8d53-dc6d87207c98 [18:21:21] let's see [18:21:45] try another puppet run on alert1002.. will take some time [18:22:33] then: sudo icinga -v /etc/icinga/icinga.cfg is safe and just checks the config.. if it returns no more warnings/errors then should be good [18:23:05] mutante: thanks yeah running now (about the 10th time I've done so but hopefully) [18:25:19] ok sretest2009 is gone from puppet_hosts.cfg [18:25:25] hopefully the alert recovers shortly [18:29:12] FIRING: SystemdUnitFailed: prometheus_amd_rocm_stats.service on ml-serve1013:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:29:58] RECOVERY - Check correctness of the icinga configuration on alert1002 is OK: Icinga configuration is correct https://wikitech.wikimedia.org/wiki/Icinga [18:31:53] \o/ [18:35:14] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [18:36:31] (03PS7) 10Bking: opensearch-operator: allow egress traffic from the operator to the k8s api [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189817 (https://phabricator.wikimedia.org/T404906) (owner: 10Brouberol) [18:36:41] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-dse_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:39:21] (03PS8) 10Bking: opensearch-operator: allow egress traffic from the operator to the k8s api [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189817 (https://phabricator.wikimedia.org/T404906) (owner: 10Brouberol) [18:41:10] (03CR) 10Bking: opensearch-operator: allow egress traffic from the operator to the k8s api (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1189817 (https://phabricator.wikimedia.org/T404906) (owner: 10Brouberol) [18:49:06] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [18:59:00] FIRING: [9x] SystemdUnitFailed: prometheus_ferm_mss.service on wdqs1018:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:11:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (PUT flinkdeployments) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=PUT - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [19:16:53] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (PUT flinkdeployments) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=PUT - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [19:29:49] FIRING: DiskSpace: Disk space serpens:9100:/ 2.489% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=serpens - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:44:00] FIRING: CertAlmostExpired: Certificate for service data-gateway-staging:30443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#data-gateway-staging:30443 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [20:14:34] RESOLVED: DiskSpace: Disk space serpens:9100:/ 2.443% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=serpens - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:19:06] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [20:24:31] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for ebomani. - https://phabricator.wikimedia.org/T405124 (10EBomani) 03NEW [20:34:06] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [20:34:27] FIRING: SystemdUnitCrashLoop: wdqs-blazegraph.service crashloop on wdqs2016:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [20:39:34] FIRING: DiskSpace: Disk space serpens:9100:/ 3.646% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=serpens - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:44:06] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [20:51:57] RESOLVED: SystemdUnitCrashLoop: wdqs-blazegraph.service crashloop on wdqs2016:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [21:06:38] 06SRE, 06Traffic, 10MediaWiki-Platform-Team (Radar): Have CDN edge set the `X-Request-Id` header for incoming external requests - https://phabricator.wikimedia.org/T221976#11198429 (10Ottomata) > put it in a header that JS can read client-side > send it from xLab's JS SDK with our instrument's events (I hav... [21:06:44] PROBLEM - mailman archives on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [21:06:44] PROBLEM - mailman list info on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [21:26:40] RECOVERY - mailman archives on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 54829 bytes in 5.705 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [21:26:40] RECOVERY - mailman list info on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 9235 bytes in 5.815 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [21:29:00] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - codfw - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [21:29:34] RESOLVED: DiskSpace: Disk space serpens:9100:/ 3.641% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=serpens - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [22:03:03] FIRING: PuppetFailure: Puppet has failed on wdqs2016:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [22:15:52] (03CR) 10BCornwall: [C:03+1] geo-maps: update map default to list codfw first [dns] - 10https://gerrit.wikimedia.org/r/1189598 (https://phabricator.wikimedia.org/T399891) (owner: 10Jasmine) [22:20:54] (03PS1) 10Superpes15: [enwiki] Throttle exemption for training events [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189930 (https://phabricator.wikimedia.org/T405069) [22:27:02] (03CR) 10CI reject: [V:04-1] [enwiki] Throttle exemption for training events [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189930 (https://phabricator.wikimedia.org/T405069) (owner: 10Superpes15) [22:28:40] (03PS2) 10Superpes15: [enwiki] Throttle exemption for training events [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1189930 (https://phabricator.wikimedia.org/T405069) [22:29:12] FIRING: SystemdUnitFailed: prometheus_amd_rocm_stats.service on ml-serve1013:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:35:50] 06SRE, 10SRE-Access-Requests: Requesting access to for - https://phabricator.wikimedia.org/T405129 (10TLessa-WMF) 03NEW [22:36:41] FIRING: SystemdUnitFailed: docker-reporter-kubernetes-dse_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:40:09] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for tais-lessa - https://phabricator.wikimedia.org/T405129#11198666 (10Pppery) [22:59:00] FIRING: [9x] SystemdUnitFailed: prometheus_ferm_mss.service on wdqs1018:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:38:30] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1189932 [23:38:30] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1189932 (owner: 10TrainBranchBot) [23:44:00] FIRING: CertAlmostExpired: Certificate for service data-gateway-staging:30443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#data-gateway-staging:30443 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:53:00] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1189932 (owner: 10TrainBranchBot)