[00:00:07] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [00:00:09] (03PS2) 10Ryan Kemper: airflow3: fix slow graph view for dumps DAGs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368786 (https://phabricator.wikimedia.org/T443252) [00:00:16] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [00:00:20] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [00:00:27] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [00:04:29] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:10:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [00:30:42] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:35:42] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:38:59] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1368787 [00:38:59] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1368787 (owner: 10TrainBranchBot) [00:45:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [00:45:14] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1368787 (owner: 10TrainBranchBot) [01:09:03] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1368788 [01:09:03] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1368788 (owner: 10TrainBranchBot) [01:17:23] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1368788 (owner: 10TrainBranchBot) [01:18:15] FIRING: ProbeDown: Service contint1002:1443 has failed probes (http_integration_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#contint1002:1443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:26:44] (03PS1) 10BryanDavis: wmcs: second hardware-backed ssh key for bd808 [puppet] - 10https://gerrit.wikimedia.org/r/1368790 (https://phabricator.wikimedia.org/T443002) [01:28:10] RESOLVED: ProbeDown: Service contint1002:1443 has failed probes (http_integration_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#contint1002:1443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.47% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T0200) [02:00:44] !log mwpresync@deploy2003 Started scap build-images: Publishing wmf/next image [02:08:33] !log mwpresync@deploy2003 Finished scap build-images: Publishing wmf/next image (duration: 07m 49s) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [02:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [02:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [03:15:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [03:49:31] PROBLEM - ps1-604-eqsin-infeed-load-tower-B-single-phase on ps1-604-eqsin is CRITICAL: SNMP CRITICAL - ps1-604-eqsin-infeed-load-tower-B-single-phase *-1* https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [03:49:45] PROBLEM - ps1-603-eqsin-infeed-load-tower-B-single-phase on ps1-603-eqsin is CRITICAL: SNMP CRITICAL - ps1-603-eqsin-infeed-load-tower-B-single-phase *-1* https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [03:53:38] FIRING: [2x] NetworkDeviceAlarmActive: Alarm active on cr2-eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [04:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:34:07] PROBLEM - Host mr1-eqsin.oob is DOWN: PING CRITICAL - Packet loss = 100% [05:37:45] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [05:48:46] 10ops-eqsin: Power Supply - Status - issue on cp5022:9290 - https://phabricator.wikimedia.org/T444571 (10phaultfinder) 03NEW [05:48:46] 10ops-eqsin: Power Supply - Status - issue on ganeti5004:9290 - https://phabricator.wikimedia.org/T444572 (10phaultfinder) 03NEW [05:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.49% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [05:56:23] PROBLEM - Ensure traffic_server is running for instance backend on cp5022 is CRITICAL: PROCS CRITICAL: 2 processes with args /usr/bin/traffic_server -M --httpport 3128 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [05:57:23] RECOVERY - Ensure traffic_server is running for instance backend on cp5022 is OK: PROCS OK: 1 process with args /usr/bin/traffic_server -M --httpport 3128 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [05:58:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [06:03:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [06:09:13] PROBLEM - Ensure traffic_manager is running for instance backend on cp5024 is CRITICAL: PROCS CRITICAL: 2 processes with args /usr/bin/traffic_manager --nosyslog https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [06:10:13] RECOVERY - Ensure traffic_manager is running for instance backend on cp5024 is OK: PROCS OK: 1 process with args /usr/bin/traffic_manager --nosyslog https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [06:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [06:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [06:54:08] 10ops-eqsin: eqsin power feed down Nov 2026 - https://phabricator.wikimedia.org/T444574 (10ayounsi) 03NEW p:05Triage→03High [07:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T0700) [07:00:05] marostegui, cezmunsta, and federico3: Time to do the Primary database switchover deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T0700). [07:15:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:22:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [07:23:01] RECOVERY - Host mr1-eqsin.oob is UP: PING OK - Packet loss = 0%, RTA = 218.09 ms [07:26:43] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 234.56 ms [07:27:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [07:30:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [07:35:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [07:38:44] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 12 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363463 (https://phabricator.wikimedia.org/T440376) (owner: 10Ebernhardson) [07:40:31] RECOVERY - ps1-604-eqsin-infeed-load-tower-B-single-phase on ps1-604-eqsin is OK: SNMP OK - ps1-604-eqsin-infeed-load-tower-B-single-phase 0 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [07:40:45] RECOVERY - ps1-603-eqsin-infeed-load-tower-B-single-phase on ps1-603-eqsin is OK: SNMP OK - ps1-603-eqsin-infeed-load-tower-B-single-phase 121 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [07:41:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 12 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360927 (https://phabricator.wikimedia.org/T442824) (owner: 10DCausse) [07:48:38] RESOLVED: [2x] NetworkDeviceAlarmActive: Alarm active on cr2-eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [07:49:00] (03PS1) 10Clément Goubert: cgoubert: add backup yubikey [puppet] - 10https://gerrit.wikimedia.org/r/1368928 [07:57:44] (03PS1) 10Muehlenhoff: Bump version for node images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1368929 [07:59:02] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Bump version for node images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1368929 (owner: 10Muehlenhoff) [08:00:05] Amir1, urbanecm, awight, and chlod: Time to snap out of that daydream and deploy UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T0800). [08:00:05] kostajh and dcausse: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [08:00:11] o/ [08:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:04:52] going to deploy a first config patch while waiting for Kosta [08:05:40] hi [08:05:48] Sounds good [08:05:53] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363463 (https://phabricator.wikimedia.org/T440376) (owner: 10Ebernhardson) [08:05:57] I can deploy my patch when you’re done [08:06:08] kostajh: oops bad timing [08:06:48] (03Merged) 10jenkins-bot: cirrus: Provide short cluster names [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363463 (https://phabricator.wikimedia.org/T440376) (owner: 10Ebernhardson) [08:08:08] !log dcausse@deploy2003 Started scap sync-world: Backport for [[gerrit:1363463|cirrus: Provide short cluster names (T440376)]] [08:08:12] T440376: Record query metrics against the responding cluster - https://phabricator.wikimedia.org/T440376 [08:10:20] dcausse: no worries, please ping me when you’re done [08:10:25] sure [08:10:42] !log dcausse@deploy2003 ebernhardson, dcausse: Backport for [[gerrit:1363463|cirrus: Provide short cluster names (T440376)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:13:12] !log dcausse@deploy2003 ebernhardson, dcausse: Continuing with deployment [08:19:14] !log dcausse@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363463|cirrus: Provide short cluster names (T440376)]] (duration: 11m 06s) [08:19:17] T440376: Record query metrics against the responding cluster - https://phabricator.wikimedia.org/T440376 [08:19:45] kostajh: I'm done with this patch, please feel free to go ahead [08:20:02] (03CR) 10Majavah: [C:03+2] wmcs: second hardware-backed ssh key for bd808 [puppet] - 10https://gerrit.wikimedia.org/r/1368790 (https://phabricator.wikimedia.org/T443002) (owner: 10BryanDavis) [08:20:04] dcausse: thanks [08:20:23] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368699 (https://phabricator.wikimedia.org/T444509) (owner: 10Dreamy Jazz) [08:21:09] (03Merged) 10jenkins-bot: hCaptcha: Enable on French Wikipedia for API editing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368699 (https://phabricator.wikimedia.org/T444509) (owner: 10Dreamy Jazz) [08:21:32] !log kharlan@deploy2003 Started scap sync-world: Backport for [[gerrit:1368699|hCaptcha: Enable on French Wikipedia for API editing (T444509)]] [08:21:35] T444509: Require hCaptcha for API edits on French Wikipedia - https://phabricator.wikimedia.org/T444509 [08:23:58] !log kharlan@deploy2003 dreamyjazz, kharlan: Backport for [[gerrit:1368699|hCaptcha: Enable on French Wikipedia for API editing (T444509)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:24:58] !log kharlan@deploy2003 dreamyjazz, kharlan: Continuing with deployment [08:30:59] !log kharlan@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368699|hCaptcha: Enable on French Wikipedia for API editing (T444509)]] (duration: 09m 27s) [08:31:02] T444509: Require hCaptcha for API edits on French Wikipedia - https://phabricator.wikimedia.org/T444509 [08:31:08] dcausse: over to you [08:31:14] kostajh: thanks! [08:34:38] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360927 (https://phabricator.wikimedia.org/T442824) (owner: 10DCausse) [08:35:21] (03Merged) 10jenkins-bot: search: add pool counter rules and dedicated pool for WDQS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360927 (https://phabricator.wikimedia.org/T442824) (owner: 10DCausse) [08:35:47] !log dcausse@deploy2003 Started scap sync-world: Backport for [[gerrit:1360927|search: add pool counter rules and dedicated pool for WDQS (T442824)]] [08:35:50] T442824: Create a dedicated poolcounter key for search requests federated via WDQS - https://phabricator.wikimedia.org/T442824 [08:38:12] !log dcausse@deploy2003 dcausse: Backport for [[gerrit:1360927|search: add pool counter rules and dedicated pool for WDQS (T442824)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:41:46] !log dcausse@deploy2003 dcausse: Rolling back deployment [08:42:16] !log dcausse@deploy2003 Failed scap sync-world: Backport for [[gerrit:1360927|search: add pool counter rules and dedicated pool for WDQS (T442824)]] (duration: 06m 29s) [08:42:19] T442824: Create a dedicated poolcounter key for search requests federated via WDQS - https://phabricator.wikimedia.org/T442824 [08:42:44] (03PS1) 10DCausse: Revert "search: add pool counter rules and dedicated pool for WDQS" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368940 [08:43:09] (03PS1) 10TrainBranchBot: Revert "search: add pool counter rules and dedicated pool for WDQS" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368941 [08:43:10] (03CR) 10TrainBranchBot: "dcausse@deploy2003 created a revert of this change as I9edc5bd7c68efb988c5ad8f93764c4aaf07694a9" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360927 (https://phabricator.wikimedia.org/T442824) (owner: 10DCausse) [08:43:10] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368941 (owner: 10TrainBranchBot) [08:43:33] (03Abandoned) 10DCausse: Revert "search: add pool counter rules and dedicated pool for WDQS" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368940 (owner: 10DCausse) [08:44:02] (03Merged) 10jenkins-bot: Revert "search: add pool counter rules and dedicated pool for WDQS" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368941 (owner: 10TrainBranchBot) [08:49:30] !log dcausse@deploy2003 Started scap sync-world: Backport for [[gerrit:1368941|Revert "search: add pool counter rules and dedicated pool for WDQS"]] [08:50:22] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368944 [08:51:53] !log dcausse@deploy2003 dcausse, trainbranchbot: Backport for [[gerrit:1368941|Revert "search: add pool counter rules and dedicated pool for WDQS"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:52:26] !log dcausse@deploy2003 dcausse, trainbranchbot: Continuing with deployment [08:58:27] !log dcausse@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368941|Revert "search: add pool counter rules and dedicated pool for WDQS"]] (duration: 08m 58s) [09:00:04] andre and brennen: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for MediaWiki train - Utc-0+Utc-7 Version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T0900). [09:00:06] o/ [09:00:18] (03CR) 10Hnowlan: [C:03+2] "Verified out of band - merging on claime's behalf." [puppet] - 10https://gerrit.wikimedia.org/r/1368928 (owner: 10Clément Goubert) [09:12:14] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T444025#12547604 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm stopped alerting a few days ago. [09:13:06] (03PS7) 10Kamila Součková: charts/bento: add chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337915 (https://phabricator.wikimedia.org/T371214) (owner: 10Tiziano Fogli) [09:16:58] (03PS2) 10Ilias Sarantopoulos: admin: Add FIDO SSH key for gkyziridis [puppet] - 10https://gerrit.wikimedia.org/r/1368303 (https://phabricator.wikimedia.org/T444318) (owner: 10Gkyziridis) [09:17:08] (03PS1) 10TrainBranchBot: group2 to 1.48.0-wmf.4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368964 (https://phabricator.wikimedia.org/T442222) [09:17:11] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by aklapper@deploy2003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368964 (https://phabricator.wikimedia.org/T442222) (owner: 10TrainBranchBot) [09:17:44] (03PS1) 10Marostegui: mariadb: Decommission clouddb1015 [puppet] - 10https://gerrit.wikimedia.org/r/1368965 (https://phabricator.wikimedia.org/T443768) [09:17:57] (03CR) 10CI reject: [V:04-1] admin: Add FIDO SSH key for gkyziridis [puppet] - 10https://gerrit.wikimedia.org/r/1368303 (https://phabricator.wikimedia.org/T444318) (owner: 10Gkyziridis) [09:18:22] !log marostegui@cumin1004 START - Cookbook sre.mysql.decommission [09:18:23] (03Merged) 10jenkins-bot: group2 to 1.48.0-wmf.4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368964 (https://phabricator.wikimedia.org/T442222) (owner: 10TrainBranchBot) [09:18:29] !log marostegui@cumin1004 START - Cookbook sre.hosts.decommission for hosts clouddb1015.eqiad.wmnet [09:19:06] (03PS2) 10Marostegui: mariadb: Decommission clouddb1015 [puppet] - 10https://gerrit.wikimedia.org/r/1368965 (https://phabricator.wikimedia.org/T443768) [09:19:29] (03PS3) 10Ilias Sarantopoulos: admin: Add FIDO SSH key for gkyziridis [puppet] - 10https://gerrit.wikimedia.org/r/1368303 (https://phabricator.wikimedia.org/T442758) (owner: 10Gkyziridis) [09:20:21] (03CR) 10Marostegui: [C:03+2] mariadb: Decommission clouddb1015 [puppet] - 10https://gerrit.wikimedia.org/r/1368965 (https://phabricator.wikimedia.org/T443768) (owner: 10Marostegui) [09:24:09] !log marostegui@cumin1004 START - Cookbook sre.dns.netbox [09:26:49] !log aklapper@deploy2003 rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.4 refs T442222 [09:26:53] T442222: 1.48.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T442222 [09:27:09] (03PS1) 10Marostegui: check_private_data_report: Remove clouddb1015 [puppet] - 10https://gerrit.wikimedia.org/r/1368967 (https://phabricator.wikimedia.org/T443768) [09:27:56] !log marostegui@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004" [09:28:10] !log marostegui@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004" [09:28:10] !log marostegui@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:28:11] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1015.eqiad.wmnet [09:28:14] !log marostegui@cumin1004 Removing clouddb1015 from zarcillo T443768 [09:28:16] (03PS11) 10Kamila Součková: helmfile/bento: add helmfile [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337916 (https://phabricator.wikimedia.org/T371214) (owner: 10Tiziano Fogli) [09:28:16] !log marostegui@cumin1004 END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99) [09:28:18] T443768: decommission clouddb1015.eqiad.wmnet - https://phabricator.wikimedia.org/T443768 [09:29:06] (03CR) 10Tiziano Fogli: [C:03+2] charts/bento: add chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337915 (https://phabricator.wikimedia.org/T371214) (owner: 10Tiziano Fogli) [09:30:29] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 1 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [09:30:36] (03Merged) 10jenkins-bot: charts/bento: add chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337915 (https://phabricator.wikimedia.org/T371214) (owner: 10Tiziano Fogli) [09:30:57] (03CR) 10Marostegui: [C:03+2] check_private_data_report: Remove clouddb1015 [puppet] - 10https://gerrit.wikimedia.org/r/1368967 (https://phabricator.wikimedia.org/T443768) (owner: 10Marostegui) [09:31:29] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [09:31:35] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission clouddb1015.eqiad.wmnet - https://phabricator.wikimedia.org/T443768#12547764 (10Marostegui) [09:31:44] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission clouddb1015.eqiad.wmnet - https://phabricator.wikimedia.org/T443768#12547767 (10Marostegui) This is ready for DC-Ops [09:33:02] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [09:36:57] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission clouddb1015.eqiad.wmnet - https://phabricator.wikimedia.org/T443768#12547789 (10Marostegui) a:05Marostegui→03None [09:38:21] FIRING: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 234.19497 times in the last 10 minutes due to memory usage (mw@codfw to codfw) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [09:39:31] (03PS1) 10Marostegui: aliases.yaml: Add clouddb10133 [puppet] - 10https://gerrit.wikimedia.org/r/1368971 [09:40:03] !log eevans@deploy2003 helmfile [codfw] START helmfile.d/services/data-gateway: sync [09:40:11] !log eevans@deploy2003 helmfile [codfw] DONE helmfile.d/services/data-gateway: sync [09:40:19] !log eevans@deploy2003 helmfile [eqiad] START helmfile.d/services/data-gateway: sync [09:40:32] !log eevans@deploy2003 helmfile [eqiad] DONE helmfile.d/services/data-gateway: sync [09:41:56] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply [09:41:57] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply [09:42:19] (03CR) 10CI reject: [V:04-1] aliases.yaml: Add clouddb10133 [puppet] - 10https://gerrit.wikimedia.org/r/1368971 (owner: 10Marostegui) [09:44:41] (03PS2) 10Marostegui: aliases.yaml: Add clouddb10133 [puppet] - 10https://gerrit.wikimedia.org/r/1368971 [09:46:03] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [09:46:23] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply [09:46:25] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply [09:47:07] (03CR) 10Marostegui: [C:03+2] aliases.yaml: Add clouddb10133 [puppet] - 10https://gerrit.wikimedia.org/r/1368971 (owner: 10Marostegui) [09:48:22] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply [09:48:48] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply [09:50:03] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply [09:51:12] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply [09:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.5% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [09:54:21] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply [09:55:01] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply [09:55:47] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply [09:56:14] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply [09:58:21] RESOLVED: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 31.111102880688918 times in the last 10 minutes due to memory usage (mw@codfw to codfw) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [09:58:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:03:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:03:50] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply [10:07:34] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply [10:08:43] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply [10:09:00] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply [10:11:48] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply [10:12:02] !log tappof@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply [10:17:37] (03PS1) 10Trueg: WDQSv2: Update Qlever and index on wdqs-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368974 (https://phabricator.wikimedia.org/T444329) [10:19:21] (03PS1) 10Ozge: admin: Remove non-FIDO SSH key for ozge [puppet] - 10https://gerrit.wikimedia.org/r/1368975 (https://phabricator.wikimedia.org/T442758) [10:20:13] (03CR) 10CI reject: [V:04-1] admin: Remove non-FIDO SSH key for ozge [puppet] - 10https://gerrit.wikimedia.org/r/1368975 (https://phabricator.wikimedia.org/T442758) (owner: 10Ozge) [10:20:26] (03CR) 10Gmodena: WDQSv2: Update Qlever and index on wdqs-next (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368974 (https://phabricator.wikimedia.org/T444329) (owner: 10Trueg) [10:21:13] (03CR) 10Trueg: WDQSv2: Update Qlever and index on wdqs-next (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368974 (https://phabricator.wikimedia.org/T444329) (owner: 10Trueg) [10:22:18] (03PS2) 10Ozge: admin: Remove non-FIDO SSH key for ozge [puppet] - 10https://gerrit.wikimedia.org/r/1368975 (https://phabricator.wikimedia.org/T442758) [10:23:03] 06SRE, 10SRE-Access-Requests: Requesting access to production for lucian - https://phabricator.wikimedia.org/T444593 (10lucian) 03NEW [10:23:18] (03CR) 10CI reject: [V:04-1] admin: Remove non-FIDO SSH key for ozge [puppet] - 10https://gerrit.wikimedia.org/r/1368975 (https://phabricator.wikimedia.org/T442758) (owner: 10Ozge) [10:24:22] (03CR) 10Gmodena: [C:03+1] WDQSv2: Update Qlever and index on wdqs-next (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368974 (https://phabricator.wikimedia.org/T444329) (owner: 10Trueg) [10:24:35] (03CR) 10Trueg: [C:03+2] WDQSv2: Update Qlever and index on wdqs-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368974 (https://phabricator.wikimedia.org/T444329) (owner: 10Trueg) [10:24:38] (03CR) 10Gmodena: [C:03+1] WDQSv2: Update Qlever and index on wdqs-next (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368974 (https://phabricator.wikimedia.org/T444329) (owner: 10Trueg) [10:26:29] (03Merged) 10jenkins-bot: WDQSv2: Update Qlever and index on wdqs-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368974 (https://phabricator.wikimedia.org/T444329) (owner: 10Trueg) [10:26:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [10:27:17] (03PS3) 10Ozge: admin: Remove non-FIDO SSH key for ozge [puppet] - 10https://gerrit.wikimedia.org/r/1368975 (https://phabricator.wikimedia.org/T442758) [10:28:57] (03PS3) 10Cwhite: mediawiki-common: add TTL to gauge metrics produced by statslib [deployment-charts] - 10https://gerrit.wikimedia.org/r/1305523 (https://phabricator.wikimedia.org/T394956) [10:30:33] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply [10:30:54] (03PS1) 10CWilliams: aux-k8s-services: adding pampinus [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368977 (https://phabricator.wikimedia.org/T443897) [10:30:59] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply [10:31:08] (03CR) 10CI reject: [V:04-1] aux-k8s-services: adding pampinus [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368977 (https://phabricator.wikimedia.org/T443897) (owner: 10CWilliams) [10:31:10] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply [10:31:22] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply [10:31:39] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply [10:32:08] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply [10:33:07] (03PS2) 10CWilliams: aux-k8s-services: adding pampinus [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368977 (https://phabricator.wikimedia.org/T443897) [10:34:41] (03CR) 10CI reject: [V:04-1] aux-k8s-services: adding pampinus [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368977 (https://phabricator.wikimedia.org/T443897) (owner: 10CWilliams) [10:36:15] (03PS12) 10Tiziano Fogli: helmfile/bento: add helmfile [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337916 (https://phabricator.wikimedia.org/T371214) [10:36:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [10:36:59] (03PS3) 10CWilliams: aux-k8s-services: adding pampinus [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368977 (https://phabricator.wikimedia.org/T443897) [10:38:17] (03CR) 10Muehlenhoff: [C:03+1] "Looks good, thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1368975 (https://phabricator.wikimedia.org/T442758) (owner: 10Ozge) [10:38:42] (03CR) 10Muehlenhoff: [C:03+2] admin: Remove non-FIDO SSH key for ozge [puppet] - 10https://gerrit.wikimedia.org/r/1368975 (https://phabricator.wikimedia.org/T442758) (owner: 10Ozge) [10:38:59] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [10:40:03] (03PS1) 10JHathaway: squid: allow imap.gmail.com:993 [puppet] - 10https://gerrit.wikimedia.org/r/1368983 (https://phabricator.wikimedia.org/T404891) [10:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:43:56] (03CR) 10Muehlenhoff: [C:03+1] "Looks good, thanks" [puppet] - 10https://gerrit.wikimedia.org/r/1367939 (https://phabricator.wikimedia.org/T442758) (owner: 10Aqu) [10:43:59] (03CR) 10Muehlenhoff: [C:03+2] admin: Add FIDO SSH keys for aqu [puppet] - 10https://gerrit.wikimedia.org/r/1367939 (https://phabricator.wikimedia.org/T442758) (owner: 10Aqu) [10:44:03] (03CR) 10Atsuko: [C:04-1] "I have no objections about the code, except let's merge it after `sql_xml_large` finished, therefore temporary -1." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368786 (https://phabricator.wikimedia.org/T443252) (owner: 10Ryan Kemper) [10:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [10:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [10:50:25] (03PS2) 10Dzahn: Revert "Add DNS entries for initiativeswiki" [dns] - 10https://gerrit.wikimedia.org/r/1368683 (https://phabricator.wikimedia.org/T444019) [10:50:29] (03CR) 10Tiziano Fogli: helmfile/bento: add helmfile (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1337916 (https://phabricator.wikimedia.org/T371214) (owner: 10Tiziano Fogli) [10:50:30] (03PS1) 10Muehlenhoff: Remove legacy keys for damilare, jsn, emc-wmf [puppet] - 10https://gerrit.wikimedia.org/r/1368987 (https://phabricator.wikimedia.org/T442758) [10:51:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:53:14] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy keys for damilare, jsn, emc-wmf [puppet] - 10https://gerrit.wikimedia.org/r/1368987 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [10:54:50] 06SRE, 10LDAP-Access-Requests: Grant Access to ciadmin for Peter Wangai - https://phabricator.wikimedia.org/T443855#12548215 (10AMarkossyan-WMF) Hi everyone, just in case my approval is necessary, I'm aware of this and I confirm that Peter needs access to ciadmin for the reasons started above, so I'm approving... [10:55:31] (03PS1) 10Trueg: WDQSv2: Update Qlever on prod (ns wdqs) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368989 (https://phabricator.wikimedia.org/T444329) [10:59:46] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [11:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T1100) [11:01:06] (03PS1) 10Muehlenhoff: Remove legacy key for lerickson [puppet] - 10https://gerrit.wikimedia.org/r/1368990 (https://phabricator.wikimedia.org/T442758) [11:01:49] RESOLVED: [2x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:02:02] !log trueg@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [11:02:37] (03CR) 10Trueg: [C:03+2] WDQSv2: Update Qlever on prod (ns wdqs) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368989 (https://phabricator.wikimedia.org/T444329) (owner: 10Trueg) [11:04:02] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key for lerickson [puppet] - 10https://gerrit.wikimedia.org/r/1368990 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [11:04:11] (03PS1) 10Brouberol: airflow: drop the obsolete gitlab-wikimedia external_services for gitsync [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368991 (https://phabricator.wikimedia.org/T444600) [11:04:14] (03PS1) 10Brouberol: blunderbuss: drop the obsolete gitlab-wikimedia external_services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368992 (https://phabricator.wikimedia.org/T444600) [11:04:24] (03Merged) 10jenkins-bot: WDQSv2: Update Qlever on prod (ns wdqs) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368989 (https://phabricator.wikimedia.org/T444329) (owner: 10Trueg) [11:07:40] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for fab - https://phabricator.wikimedia.org/T442665#12548283 (10MoritzMuehlenhoff) >>! In T442665#12520881, @andrea.denisse wrote: > Hi @fkaelin , once you confirm you can access using the Yubikey we can remove the old key. @fkaelin Have you been able to... [11:08:13] (03CR) 10Atsuko: [C:03+1] airflow: drop the obsolete gitlab-wikimedia external_services for gitsync [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368991 (https://phabricator.wikimedia.org/T444600) (owner: 10Brouberol) [11:12:24] (03PS1) 10Brouberol: global_config: drop the gitlab-wikimedia external services policy [puppet] - 10https://gerrit.wikimedia.org/r/1368995 (https://phabricator.wikimedia.org/T444600) [11:15:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:17:03] (03PS1) 10Dzahn: planet: remove unused theme files, templates and plugins [puppet] - 10https://gerrit.wikimedia.org/r/1368998 [11:17:19] FIRING: [4x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:17:45] (03CR) 10Nikerabbit: [C:03+1] ULS: Use the new language selector on Timeless on testwiki and mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363037 (https://phabricator.wikimedia.org/T439715) (owner: 10Huei Tan) [11:19:54] (03PS1) 10Dzahn: planet: remove unused templates, plugins and theme files [puppet] - 10https://gerrit.wikimedia.org/r/1369000 [11:32:19] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:32:56] !log trueg@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [11:34:25] jouncebot: nowandnext [11:34:25] For the next 0 hour(s) and 25 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T1100) [11:34:26] In 1 hour(s) and 25 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T1300) [11:37:19] RESOLVED: [2x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:37:30] (03PS1) 10Dreamy Jazz: Enable CheckUser AbuseFilter variables on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369005 (https://phabricator.wikimedia.org/T234155) [11:37:54] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369005 (https://phabricator.wikimedia.org/T234155) (owner: 10Dreamy Jazz) [11:38:43] (03Merged) 10jenkins-bot: Enable CheckUser AbuseFilter variables on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369005 (https://phabricator.wikimedia.org/T234155) (owner: 10Dreamy Jazz) [11:38:59] !log dreamyjazz@deploy2003 Started scap sync-world: Backport for [[gerrit:1369005|Enable CheckUser AbuseFilter variables on enwiki (T234155)]] [11:39:02] T234155: Create CheckUser-level abuse filters - https://phabricator.wikimedia.org/T234155 [11:41:07] !log dreamyjazz@deploy2003 dreamyjazz: Backport for [[gerrit:1369005|Enable CheckUser AbuseFilter variables on enwiki (T234155)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:44:11] (03PS1) 10Marostegui: sre.mysql.decommission: Capture orch "not found" [cookbooks] - 10https://gerrit.wikimedia.org/r/1369008 (https://phabricator.wikimedia.org/T444582) [11:44:40] !log dreamyjazz@deploy2003 dreamyjazz: Continuing with deployment [11:48:15] (03CR) 10Hnowlan: [C:04-1] service: move oncall to production (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345232 (https://phabricator.wikimedia.org/T439279) (owner: 10Andrea Denisse) [11:49:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12548450 (10phaultfinder) [11:49:59] 06SRE, 10LDAP-Access-Requests: Grant Access to ciadmin for Peter Wangai - https://phabricator.wikimedia.org/T443855#12548452 (10Dzahn) @pwangai You have been added to the ciadmin group. [11:50:43] !log dreamyjazz@deploy2003 Finished scap sync-world: Backport for [[gerrit:1369005|Enable CheckUser AbuseFilter variables on enwiki (T234155)]] (duration: 11m 44s) [11:50:46] T234155: Create CheckUser-level abuse filters - https://phabricator.wikimedia.org/T234155 [11:52:56] 06SRE, 10LDAP-Access-Requests: Grant Access to ciadmin for Peter Wangai - https://phabricator.wikimedia.org/T443855#12548461 (10Dzahn) 05Open→03Resolved a:03Dzahn [11:57:02] (03CR) 10Hnowlan: "Does this chart need to have IDP included within it?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344072 (https://phabricator.wikimedia.org/T438897) (owner: 10Andrea Denisse) [11:58:21] FIRING: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 966.6666666666667 times in the last 10 minutes due to memory usage (mw@codfw to codfw) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [12:00:04] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:14] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:19] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [12:00:26] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [12:00:37] !log jhancock@cumin2003 START - Cookbook sre.dns.netbox [12:01:17] (03CR) 10Klausman: [C:03+1] airflow: drop the obsolete gitlab-wikimedia external_services for gitsync [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368991 (https://phabricator.wikimedia.org/T444600) (owner: 10Brouberol) [12:01:29] (03CR) 10Klausman: [C:03+1] blunderbuss: drop the obsolete gitlab-wikimedia external_services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368992 (https://phabricator.wikimedia.org/T444600) (owner: 10Brouberol) [12:03:12] (03PS1) 10Daniel Kinzler: admin: Add Yubikey for DKinzler [puppet] - 10https://gerrit.wikimedia.org/r/1369014 (https://phabricator.wikimedia.org/T442758) [12:03:13] (03PS1) 10Daniel Kinzler: admin: Remove old SSH key for DKinzler [puppet] - 10https://gerrit.wikimedia.org/r/1369015 (https://phabricator.wikimedia.org/T442758) [12:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:04:36] !log jhancock@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding several servers to codfw - jhancock@cumin2003" [12:04:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12548509 (10phaultfinder) [12:04:41] !log jhancock@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding several servers to codfw - jhancock@cumin2003" [12:04:41] !log jhancock@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:04:47] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch2116 [12:04:57] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch2116 [12:05:07] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch2117 [12:05:25] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch2117 [12:05:43] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch2118 [12:05:52] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch2118 [12:05:59] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch2119 [12:06:17] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch2119 [12:06:24] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch2120 [12:06:38] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch2120 [12:06:47] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host ganeti2051 [12:06:56] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2051 [12:07:00] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host ganeti2052 [12:07:11] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2052 [12:07:14] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host ganeti2053 [12:07:23] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2053 [12:07:30] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host ganeti2054 [12:07:53] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2054 [12:08:02] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host ganeti2055 [12:08:12] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2055 [12:08:30] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host ganeti2056 [12:08:40] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2056 [12:08:46] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2056 [12:08:56] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2056 [12:09:00] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2057 [12:09:10] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2057 [12:09:15] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2058 [12:09:24] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2058 [12:09:29] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2059 [12:09:39] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2059 [12:09:47] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2061 [12:09:56] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2061 [12:10:03] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2062 [12:10:13] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2062 [12:10:25] (03PS1) 10Majavah: hieradata: Remove puppetdb5 site [puppet] - 10https://gerrit.wikimedia.org/r/1369032 [12:10:25] (03PS1) 10Majavah: P:puppetdb: Remove unused Puppet 7 DB compat certs [puppet] - 10https://gerrit.wikimedia.org/r/1369033 [12:10:28] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2064 [12:10:37] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2064 [12:10:47] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2065 [12:10:56] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2065 [12:11:04] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2066 [12:11:13] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2066 [12:11:20] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2067 [12:11:29] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2067 [12:11:49] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2068 [12:12:11] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2068 [12:12:25] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2069 [12:12:41] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2069 [12:12:48] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2070 [12:13:01] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2070 [12:13:25] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2071 [12:13:34] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2071 [12:13:43] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2072 [12:13:57] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2072 [12:14:05] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host mc2073 [12:14:07] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369036 [12:14:20] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2073 [12:14:29] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host pki2004 [12:14:38] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pki2004 [12:14:44] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host puppetserve2005 [12:14:57] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host puppetserve2005 [12:15:07] !log jhancock@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host puppetserve2006 [12:17:36] !log jhancock@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host puppetserve2006 [12:18:21] RESOLVED: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 396.66666666666674 times in the last 10 minutes due to memory usage (mw@codfw to codfw) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [12:20:01] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install pki2004 - https://phabricator.wikimedia.org/T436179#12548548 (10Jhancock.wm) [12:20:50] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install pki2004 - https://phabricator.wikimedia.org/T436179#12548554 (10Jhancock.wm) This server has been received and racked, but still needs to be added to preseed.yaml and site.pp before imaging. [12:21:12] !log jhancock@cumin2003 START - Cookbook sre.hosts.provision for host pki2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:21:30] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host pki2004.codfw.wmnet (push config only) [12:21:33] (03PS1) 10Dzahn: collab services: fix typos in comments and docs [puppet] - 10https://gerrit.wikimedia.org/r/1369037 [12:21:34] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host pki2004.codfw.wmnet (push config only) [12:26:27] !log bking@localhost ban cirrussearch2081 (hot shards) T444363 [12:26:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:26:30] T444363: Cirrussearch: Investigate CirrusSearchBackendMemoryIssue alerts in CODFW - https://phabricator.wikimedia.org/T444363 [12:32:00] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host pki2004.codfw.wmnet (remove config only) [12:32:05] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host pki2004.codfw.wmnet (remove config only) [12:32:20] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:33:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [12:33:24] (03PS1) 10Bartosz Wójtowicz: ml-services: move llm-qwen36-27b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369039 (https://phabricator.wikimedia.org/T444464) [12:34:59] !log jhancock@cumin2003 START - Cookbook sre.hosts.provision for host puppetserve2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:35:12] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host puppetserve2005.codfw.wmnet (push config only) [12:35:13] !log jhancock@cumin2003 START - Cookbook sre.hosts.provision for host puppetserve2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:35:18] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2005.codfw.wmnet (push config only) [12:35:28] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host puppetserve2006.codfw.wmnet (push config only) [12:35:32] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2006.codfw.wmnet (push config only) [12:35:34] !log rolling Cassandra resart, aqs1018 — T444597 [12:35:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:35:37] T444597: Missing entries for Revise Tone Structured Task - https://phabricator.wikimedia.org/T444597 [12:37:47] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12548640 (10Jhancock.wm) a:05MoritzMuehlenhoff→03Jhancock.wm [12:39:08] (03CR) 10Brouberol: [C:03+2] airflow: drop the obsolete gitlab-wikimedia external_services for gitsync [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368991 (https://phabricator.wikimedia.org/T444600) (owner: 10Brouberol) [12:40:22] (03CR) 10AikoChou: [C:03+1] ml-services: move llm-qwen36-27b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369039 (https://phabricator.wikimedia.org/T444464) (owner: 10Bartosz Wójtowicz) [12:40:46] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: move llm-qwen36-27b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369039 (https://phabricator.wikimedia.org/T444464) (owner: 10Bartosz Wójtowicz) [12:42:04] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply [12:42:21] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host puppetserve2005.codfw.wmnet (remove config only) [12:42:22] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply [12:42:23] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host puppetserve2006.codfw.wmnet (remove config only) [12:42:26] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2005.codfw.wmnet (remove config only) [12:42:30] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [12:42:32] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2006.codfw.wmnet (remove config only) [12:42:43] (03Merged) 10jenkins-bot: ml-services: move llm-qwen36-27b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369039 (https://phabricator.wikimedia.org/T444464) (owner: 10Bartosz Wójtowicz) [12:42:45] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host puppetserve2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:42:47] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host puppetserve2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:42:48] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [12:42:54] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [12:43:13] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [12:43:21] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [12:43:39] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [12:43:48] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply [12:44:06] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply [12:44:14] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply [12:44:34] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply [12:45:03] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply [12:45:06] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [12:45:11] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . [12:45:21] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply [12:45:23] !log jhancock@cumin2003 START - Cookbook sre.hosts.reimage for host puppetserve2005.codfw.wmnet with OS bookworm [12:45:31] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply [12:45:34] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host puppetserve2005.codfw.wmnet (push config only) [12:45:36] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12548701 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2003 for host puppetserve2005.codfw.wmnet with OS bookworm [12:45:38] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2005.codfw.wmnet (push config only) [12:45:49] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply [12:45:58] !log jhancock@cumin2003 START - Cookbook sre.hosts.reimage for host puppetserve2006.codfw.wmnet with OS bookworm [12:46:00] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply [12:46:05] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12548704 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2003 for host puppetserve2006.codfw.wmnet with OS bookworm [12:46:11] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host puppetserve2006.codfw.wmnet (push config only) [12:46:15] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2006.codfw.wmnet (push config only) [12:46:18] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply [12:46:52] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply [12:47:07] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply [12:47:16] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply [12:47:37] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply [12:49:37] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [12:49:45] (03CR) 10Arnaudb: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1369037 (owner: 10Dzahn) [12:49:56] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [12:50:08] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply [12:50:27] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply [12:51:19] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply [12:51:37] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply [12:53:56] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host puppetserve2005.codfw.wmnet (remove config only) [12:54:01] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2005.codfw.wmnet (remove config only) [12:54:22] !log jhancock@cumin2003 START - Cookbook sre.hosts.dhcp for host puppetserve2006.codfw.wmnet (remove config only) [12:54:27] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2006.codfw.wmnet (remove config only) [12:54:27] (03CR) 10Arnaudb: [C:03+1] "nothing surprising in the review! lgtm :-)" [puppet] - 10https://gerrit.wikimedia.org/r/1369037 (owner: 10Dzahn) [12:58:57] !log eevans@cumin1004 START - Cookbook sre.hosts.reboot-single for host aqs1018.eqiad.wmnet [12:59:47] (03PS1) 10Bartosz Wójtowicz: ml-services: move qwen38-27b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369047 (https://phabricator.wikimedia.org/T444464) [13:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T1300) [13:02:07] FIRING: [2x] ProbeDown: Service aqs1018-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:04:33] (03CR) 10Gkyziridis: [C:03+1] "LGTM!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369047 (https://phabricator.wikimedia.org/T444464) (owner: 10Bartosz Wójtowicz) [13:05:02] !log eevans@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1018.eqiad.wmnet [13:07:07] RESOLVED: [4x] ProbeDown: Service aqs1018-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:09:09] (03PS1) 10Dzahn: codesearch: keep the defaults in one place [puppet] - 10https://gerrit.wikimedia.org/r/1369049 [13:12:23] (03CR) 10Dzahn: [C:04-1] "wrong assumptions" [puppet] - 10https://gerrit.wikimedia.org/r/1369049 (owner: 10Dzahn) [13:22:12] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: move qwen38-27b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369047 (https://phabricator.wikimedia.org/T444464) (owner: 10Bartosz Wójtowicz) [13:23:53] (03Merged) 10jenkins-bot: ml-services: move qwen38-27b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369047 (https://phabricator.wikimedia.org/T444464) (owner: 10Bartosz Wójtowicz) [13:28:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:29:56] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [13:30:01] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . [13:32:38] (03PS1) 10Sbisson: CX: Skip unregistered VE plugin modules [extensions/ContentTranslation] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1369052 (https://phabricator.wikimedia.org/T444489) [13:33:17] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 12 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [extensions/ContentTranslation] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1369052 (https://phabricator.wikimedia.org/T444489) (owner: 10Sbisson) [13:37:30] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . [13:37:34] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'llmisvc' . [13:37:54] 06SRE, 10SRE-Access-Requests: Requesting access to production for lucian - https://phabricator.wikimedia.org/T444593#12548856 (10KOfori) Hello, this is approved. [13:42:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [13:47:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [13:50:27] (03PS2) 10Raymond Ndibe: webservice-runner: Add pre-flight validity checks [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1308620 (https://phabricator.wikimedia.org/T431563) [13:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.5% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [13:57:13] !log jhancock@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host puppetserve2005.codfw.wmnet with OS bookworm [13:57:19] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12548919 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2003 for host puppetserve2005.codfw.wmnet with OS bookworm execut... [13:57:44] !log jhancock@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host puppetserve2006.codfw.wmnet with OS bookworm [13:57:51] 10ops-codfw, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install (2) new puppetservers - https://phabricator.wikimedia.org/T432998#12548922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2003 for host puppetserve2006.codfw.wmnet with OS bookworm execut... [13:57:55] (03PS1) 10Santiago Faci: Test Kitchen UI: Disable deployment window validation for staging environment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369058 [14:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T1400). Please do the needful. [14:00:05] seanleong-wmde and sergi0: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [14:00:12] o/ [14:00:29] \o/ [14:00:41] (03CR) 10Phuedx: [C:03+1] "Ready whenever you are" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369058 (owner: 10Santiago Faci) [14:00:57] o/ [14:01:05] (03CR) 10Santiago Faci: [C:03+2] Test Kitchen UI: Disable deployment window validation for staging environment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369058 (owner: 10Santiago Faci) [14:01:18] Since, there's only two of us, I can start deploying my patch now? [14:01:27] oh hi Lucas! [14:01:31] seanleong-wmde: do you want to start? [14:01:40] Lucas_WMDE: sounds good! o7 [14:01:41] I assume both of you can self-service [14:01:41] I can also self-deploy my change after [14:02:48] (03Merged) 10jenkins-bot: Test Kitchen UI: Disable deployment window validation for staging environment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369058 (owner: 10Santiago Faci) [14:03:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by seanleong-wmde@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368324 (https://phabricator.wikimedia.org/T438982) (owner: 10Seanleong-wmde) [14:03:59] !log sfaci@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply [14:04:17] (03Merged) 10jenkins-bot: Switch site lookup to sitematrix lookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368324 (https://phabricator.wikimedia.org/T438982) (owner: 10Seanleong-wmde) [14:04:29] !log sfaci@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply [14:04:36] !log seanleong-wmde@deploy2003 Started scap sync-world: Backport for [[gerrit:1368324|Switch site lookup to sitematrix lookup (T438982)]] [14:04:39] T438982: Deploy new SiteMatrixLookup configuration to Wikimedia production wikis - https://phabricator.wikimedia.org/T438982 [14:06:44] !log seanleong-wmde@deploy2003 seanleong-wmde: Backport for [[gerrit:1368324|Switch site lookup to sitematrix lookup (T438982)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:09:16] !log seanleong-wmde@deploy2003 Rolling back deployment [14:09:46] !log seanleong-wmde@deploy2003 Failed scap sync-world: Backport for [[gerrit:1368324|Switch site lookup to sitematrix lookup (T438982)]] (duration: 05m 10s) [14:09:49] T438982: Deploy new SiteMatrixLookup configuration to Wikimedia production wikis - https://phabricator.wikimedia.org/T438982 [14:10:20] (03PS1) 10TrainBranchBot: Revert "Switch site lookup to sitematrix lookup" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369063 [14:10:21] (03CR) 10TrainBranchBot: "seanleong-wmde@deploy2003 created a revert of this change as I2e035f22cf9aa23f6c73ce2998300fe726790c8f" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368324 (https://phabricator.wikimedia.org/T438982) (owner: 10Seanleong-wmde) [14:10:21] (03CR) 10TrainBranchBot: [C:03+2] "Approved by seanleong-wmde@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369063 (owner: 10TrainBranchBot) [14:11:26] (03Merged) 10jenkins-bot: Revert "Switch site lookup to sitematrix lookup" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369063 (owner: 10TrainBranchBot) [14:12:23] sergi0: there's something wrong with my patch, I rollback and terminated and currently reverting the change. You can go first after this. [14:12:41] seanleong-wmde: ack, thanks for the ping! [14:14:51] sergi0: no problem, it's done now! [14:15:14] alright, I'll go with my change now [14:15:48] sergi0: o7 [14:16:12] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sgimeno@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367855 (owner: 10Sergio Gimeno) [14:17:38] (03Merged) 10jenkins-bot: GrowthExperiments: Scale early onboarding on ar/en/es/frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367855 (owner: 10Sergio Gimeno) [14:17:52] !log sgimeno@deploy2003 Started scap sync-world: Backport for [[gerrit:1367855|GrowthExperiments: Scale early onboarding on ar/en/es/frwiki]] [14:20:02] !log sgimeno@deploy2003 sgimeno: Backport for [[gerrit:1367855|GrowthExperiments: Scale early onboarding on ar/en/es/frwiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:20:31] !log sgimeno@deploy2003 sgimeno: Continuing with deployment [14:23:11] (03PS1) 10Seanleong-wmde: Config change switching site lookup to sitematrix lookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369066 (https://phabricator.wikimedia.org/T438982) [14:23:57] (03CR) 10CI reject: [V:04-1] Config change switching site lookup to sitematrix lookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369066 (https://phabricator.wikimedia.org/T438982) (owner: 10Seanleong-wmde) [14:24:47] (03Abandoned) 10Seanleong-wmde: Config change switching site lookup to sitematrix lookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369066 (https://phabricator.wikimedia.org/T438982) (owner: 10Seanleong-wmde) [14:25:27] (03PS1) 10Seanleong-wmde: Switch site lookup to sitematrix lookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369068 (https://phabricator.wikimedia.org/T438982) [14:26:34] !log sgimeno@deploy2003 Finished scap sync-world: Backport for [[gerrit:1367855|GrowthExperiments: Scale early onboarding on ar/en/es/frwiki]] (duration: 08m 41s) [14:27:01] !log UTC afternoon backport window done [14:27:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:28:08] (03PS2) 10Seanleong-wmde: Switch site lookup to sitematrix lookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369068 (https://phabricator.wikimedia.org/T438982) [14:30:19] (03CR) 10Awight: [C:03+1] Switch site lookup to sitematrix lookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369068 (https://phabricator.wikimedia.org/T438982) (owner: 10Seanleong-wmde) [14:30:28] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 12 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369068 (https://phabricator.wikimedia.org/T438982) (owner: 10Seanleong-wmde) [14:31:03] Hello, I will take a go to do backport again. [14:33:34] (03CR) 10TrainBranchBot: [C:03+2] "Approved by seanleong-wmde@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369068 (https://phabricator.wikimedia.org/T438982) (owner: 10Seanleong-wmde) [14:34:19] (03Merged) 10jenkins-bot: Switch site lookup to sitematrix lookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369068 (https://phabricator.wikimedia.org/T438982) (owner: 10Seanleong-wmde) [14:34:34] !log seanleong-wmde@deploy2003 Started scap sync-world: Backport for [[gerrit:1369068|Switch site lookup to sitematrix lookup (T438982)]] [14:34:37] T438982: Deploy new SiteMatrixLookup configuration to Wikimedia production wikis - https://phabricator.wikimedia.org/T438982 [14:36:44] !log seanleong-wmde@deploy2003 seanleong-wmde: Backport for [[gerrit:1369068|Switch site lookup to sitematrix lookup (T438982)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:37:34] (03Abandoned) 10Brouberol: deployment_server: define the postgresql-growthbook-dpe-infra kubeconfigs [puppet] - 10https://gerrit.wikimedia.org/r/1354449 (https://phabricator.wikimedia.org/T435472) (owner: 10Brouberol) [14:38:39] (03CR) 10Brouberol: [C:03+1] spark-operator: Watch the analytics and team Spark namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367770 (https://phabricator.wikimedia.org/T432322) (owner: 10Btullis) [14:40:12] seanleong-wmde: can you ping me when your done with that backport? I have another late addition as well! [14:40:30] (03PS1) 10Matthias Mullie: Move exposure logging back to JS [extensions/ReaderExperiments] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1369074 (https://phabricator.wikimedia.org/T444205) [14:40:51] !log seanleong-wmde@deploy2003 seanleong-wmde: Continuing with deployment [14:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:41:19] !log trueg@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [14:41:28] (03PS1) 10Bartosz Wójtowicz: ml-services: move llm-gpt-oss-safeguard-20b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369075 (https://phabricator.wikimedia.org/T444464) [14:42:32] !log trueg@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [14:44:25] (03CR) 10Gkyziridis: [C:03+1] ml-services: move llm-gpt-oss-safeguard-20b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369075 (https://phabricator.wikimedia.org/T444464) (owner: 10Bartosz Wójtowicz) [14:44:53] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: move llm-gpt-oss-safeguard-20b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369075 (https://phabricator.wikimedia.org/T444464) (owner: 10Bartosz Wójtowicz) [14:46:14] (03Merged) 10jenkins-bot: ml-services: move llm-gpt-oss-safeguard-20b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369075 (https://phabricator.wikimedia.org/T444464) (owner: 10Bartosz Wójtowicz) [14:46:53] !log seanleong-wmde@deploy2003 Finished scap sync-world: Backport for [[gerrit:1369068|Switch site lookup to sitematrix lookup (T438982)]] (duration: 12m 19s) [14:46:57] T438982: Deploy new SiteMatrixLookup configuration to Wikimedia production wikis - https://phabricator.wikimedia.org/T438982 [14:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [14:47:25] (03CR) 10Ladsgroup: "I talked to Valentin and he is onboard but while describing it, I realized it can be improved." [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) (owner: 10Krinkle) [14:47:47] matthiasmullie: I finished my backport, you can go ahead! Thanks! [14:48:02] seanleong-wmde: thanks! [14:48:06] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy2003 using scap backport" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1369074 (https://phabricator.wikimedia.org/T444205) (owner: 10Matthias Mullie) [14:48:46] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [14:48:51] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . [14:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [14:49:23] (03Merged) 10jenkins-bot: Move exposure logging back to JS [extensions/ReaderExperiments] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1369074 (https://phabricator.wikimedia.org/T444205) (owner: 10Matthias Mullie) [14:49:43] !log mlitn@deploy2003 Started scap sync-world: Backport for [[gerrit:1369074|Move exposure logging back to JS (T444205)]] (1.48.0-wmf.4) [14:49:46] T444205: [Semantic Search] log an exposure server-side at the conditional where $semanticEnabled is checked - https://phabricator.wikimedia.org/T444205 [14:53:26] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [14:53:51] !log mlitn@deploy2003 mlitn: Backport for [[gerrit:1369074|Move exposure logging back to JS (T444205)]] (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:54:28] !log mlitn@deploy2003 mlitn: Continuing with deployment [14:56:30] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-user for gaiarajan - https://phabricator.wikimedia.org/T444625 (10gaiarajan) 03NEW [14:57:34] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for gaiarajan - https://phabricator.wikimedia.org/T444625#12549315 (10gaiarajan) [15:00:26] !log mlitn@deploy2003 Finished scap sync-world: Backport for [[gerrit:1369074|Move exposure logging back to JS (T444205)]] (1.48.0-wmf.4) (duration: 10m 43s) [15:00:30] T444205: [Semantic Search] log an exposure server-side at the conditional where $semanticEnabled is checked - https://phabricator.wikimedia.org/T444205 [15:00:42] !log UTC afternoon backport window done [15:00:43] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for gaiarajan - https://phabricator.wikimedia.org/T444625#12549325 (10gaiarajan) [15:00:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:04:03] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [15:14:09] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [15:15:24] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:15:26] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [15:22:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [15:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T1530) [15:32:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [15:37:31] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:37:43] PROBLEM - OSPF status on cr2-codfw is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:38:39] FIRING: CoreBGPDown: Core BGP session down between cr2-codfw and cr2-drmrs (185.15.58.159) - group Confed_drmrs - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr2-codfw:9804&var-bgp_group=Confed_drmrs&var-bgp_neighbor=cr2-drmrs - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:39:10] FIRING: [2x] BFDdown: BFD session down between cr2-codfw and 185.15.58.159 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:40:25] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-drmrs:et-0/0/0 (Transport: Arelion (IC-398708) [20Gbps]) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:40:55] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [15:43:03] PROBLEM - OSPF status on cr2-drmrs is CRITICAL: OSPFv2: 2/4 UP : OSPFv3: 2/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:43:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr2-codfw and cr2-drmrs (185.15.58.159) - group Confed_drmrs - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:43:43] RECOVERY - OSPF status on cr2-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:44:03] RECOVERY - OSPF status on cr2-drmrs is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:44:31] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:45:25] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr2-drmrs:et-0/0/0 (Transport: Arelion (IC-398708) [20Gbps]) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:48:39] RESOLVED: [3x] CoreBGPDown: Core BGP session down between cr2-codfw and cr2-drmrs (185.15.58.159) - group Confed_drmrs - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:49:10] RESOLVED: [2x] BFDdown: BFD session down between cr2-codfw and 185.15.58.159 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:53:13] (03PS1) 10FNegri: toolhub: bump container to 2026-11-12-120620-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369093 (https://phabricator.wikimedia.org/T444468) [16:00:55] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [16:04:29] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:10:39] FIRING: CirrusSearchNodeIndexingNotIncreasing: OpenSearch instance cirrussearch2081-production-search-codfw is not indexing - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&from=now-3d&to=now&viewPanel=57 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchNodeIndexingNotIncreasing [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:22:41] (03PS1) 10Dzahn: codesearch: drop the base_dir param and the "undefined" defaults [puppet] - 10https://gerrit.wikimedia.org/r/1369098 [16:23:03] (03CR) 10BPirkle: [C:03+1] "Looks good, had one nonblocking question." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368286 (https://phabricator.wikimedia.org/T444478) (owner: 10Gmodena) [16:23:59] (03CR) 10BPirkle: [C:03+1] rest-gateway: set wdqs v2 GA rate limits [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358194 (https://phabricator.wikimedia.org/T437892) (owner: 10Gmodena) [16:27:43] (03PS1) 10Dzahn: doc: drop the pre-bookworm PHP branch, fix the class name in admin [puppet] - 10https://gerrit.wikimedia.org/r/1369099 [16:29:03] (03PS1) 10FNegri: H:cloudweb: bump striker container version [puppet] - 10https://gerrit.wikimedia.org/r/1369100 (https://phabricator.wikimedia.org/T444468) [16:36:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [16:41:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [16:42:39] (03CR) 10Aleksandar Mastilovic: [C:03+1] "LGTM! Thank you." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368992 (https://phabricator.wikimedia.org/T444600) (owner: 10Brouberol) [16:46:01] (03CR) 10Raymond Ndibe: [C:03+1] toolhub: bump container to 2026-11-12-120620-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369093 (https://phabricator.wikimedia.org/T444468) (owner: 10FNegri) [16:55:47] (03CR) 10Raymond Ndibe: [C:03+1] H:cloudweb: bump striker container version [puppet] - 10https://gerrit.wikimedia.org/r/1369100 (https://phabricator.wikimedia.org/T444468) (owner: 10FNegri) [17:00:04] jhathaway and rzl: I, the Bot under the Fountain, call upon thee, The Deployer, to do Puppet request window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T1700). [17:00:04] No Gerrit patches in the queue for this window AFAICS. [17:03:15] !log bking@localhost unban cirrussearch2081 (hot shards) T444363 [17:03:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:03:19] T444363: Cirrussearch: Investigate CirrusSearchBackendMemoryIssue alerts in CODFW - https://phabricator.wikimedia.org/T444363 [17:10:39] RESOLVED: CirrusSearchNodeIndexingNotIncreasing: OpenSearch instance cirrussearch2081-production-search-codfw is not indexing - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&from=now-3d&to=now&viewPanel=57 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchNodeIndexingNotIncreasing [17:13:04] (03CR) 10FNegri: [C:03+2] toolhub: bump container to 2026-11-12-120620-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369093 (https://phabricator.wikimedia.org/T444468) (owner: 10FNegri) [17:13:12] (03CR) 10FNegri: [C:03+2] H:cloudweb: bump striker container version [puppet] - 10https://gerrit.wikimedia.org/r/1369100 (https://phabricator.wikimedia.org/T444468) (owner: 10FNegri) [17:14:50] (03Merged) 10jenkins-bot: toolhub: bump container to 2026-11-12-120620-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369093 (https://phabricator.wikimedia.org/T444468) (owner: 10FNegri) [17:16:23] !log fnegri@deploy2003 helmfile [staging] START helmfile.d/services/toolhub: apply [17:16:58] !log fnegri@deploy2003 helmfile [staging] DONE helmfile.d/services/toolhub: apply [17:22:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [17:24:04] (03PS1) 10Hasan Akgün (WMDE): hCaptcha: Enable hCaptcha for Wikibase edits on testwikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369108 (https://phabricator.wikimedia.org/T440845) [17:27:43] (03PS1) 10Ottomata: EventStreamConfig: Add html_edit_suggestions_counts_change stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1369109 (https://phabricator.wikimedia.org/T444212) [17:27:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [17:28:28] !log fnegri@deploy2003 helmfile [codfw] START helmfile.d/services/toolhub: apply [17:29:16] !log fnegri@deploy2003 helmfile [codfw] DONE helmfile.d/services/toolhub: apply [17:29:28] !log fnegri@deploy2003 helmfile [eqiad] START helmfile.d/services/toolhub: apply [17:30:20] !log fnegri@deploy2003 helmfile [eqiad] DONE helmfile.d/services/toolhub: apply [17:32:00] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [17:42:00] RESOLVED: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [17:49:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12550080 (10phaultfinder) [17:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.51% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [17:52:16] (03CR) 10Ahmon Dancy: [C:03+1] Add growthbook kubernetes service to spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1368697 (https://phabricator.wikimedia.org/T444502) (owner: 10Santiago Faci) [17:54:27] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for fab - https://phabricator.wikimedia.org/T442665#12550093 (10fkaelin) Sorry for the delay - I confirm that the yubikey works. [18:00:05] bd808: Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T1800). Please do the needful. [18:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T1800) [18:06:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [18:11:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [18:14:34] (03CR) 10Krinkle: varnish: Don't strip browser Cache-Control:maxage from action=raw (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) (owner: 10Krinkle) [18:26:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [18:36:16] (03CR) 10Krinkle: varnish: Don't strip browser Cache-Control:maxage from action=raw (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) (owner: 10Krinkle) [18:36:42] (03PS1) 10Krinkle: RawAction: Reduce action=raw maxage without CSS/JSON/JS ctype to 5s [core] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1369119 (https://phabricator.wikimedia.org/T279120) [18:41:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [18:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:43:06] (03CR) 10CI reject: [V:04-1] RawAction: Reduce action=raw maxage without CSS/JSON/JS ctype to 5s [core] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1369119 (https://phabricator.wikimedia.org/T279120) (owner: 10Krinkle) [18:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [18:49:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [18:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [19:00:05] andre and brennen: MediaWiki train - Utc-0+Utc-7 Version (secondary timeslot) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T1900). Please do the needful. [19:00:39] 10ops-eqiad, 06DC-Ops: Alert for device ps1-a4-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T444641 (10phaultfinder) 03NEW [19:04:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [19:06:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 29d 23h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [19:08:43] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369123 [19:10:11] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2191.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2291.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-work [19:10:11] odfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2301.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2062.codfw.wmnet, wikikube-worker2272.codfw.wmnet, wikikube-worker2276.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2369.codfw.wmnet, wikikube-worker2151.codfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2324.codfw.wmnet, wikikube-worker2087.codfw.wmnet, w [19:10:11] worker2013.codfw.wmnet, wikikube-worker2049.codfw.wmnet, wikikube-worker2308.codfw.wmnet, wikikube-worker2352.codfw.wmnet, wikikube-worker2317.codfw.wmnet, wikikube-worker2037.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [19:11:11] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:12:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:15:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:15:57] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369124 [19:16:08] (03PS3) 10Dillon: PersonalDashboard: Clean up config and set QA values for test and beta wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356990 (https://phabricator.wikimedia.org/T441729) [19:17:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:17:42] (03Abandoned) 10Dbrant: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369123 (owner: 10PipelineBot) [19:19:23] (03CR) 10Dillon: [C:03+1] "PS3 rebases onto master, which now turns on PersonalDashboard on every beta wiki for Special:Home (T444353) and on the Home pilot wikis (T" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356990 (https://phabricator.wikimedia.org/T441729) (owner: 10Dillon) [19:21:23] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2191.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-work [19:21:23] odfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2114.codfw.wmnet, wikikube-worker2268.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker2087.codfw.wmnet, wikikube-worker2370.codfw.wmnet, wikikube-worker2126.codfw.wmnet, wikikube-worker2308.codfw.wmnet, wikikube-worker2298.codfw.wmnet, wikikube-worker2352.codfw.wmnet, wikikube-worker2367.codfw.wmnet, wikikube-worker2137.codfw.wmnet, w [19:21:23] worker2316.codfw.wmnet, wikikube-worker2342.codfw.wmnet, wikikube-worker2341.codfw.wmnet, wikikube-worker2042.codfw.wmnet, wikikube-worker2186.codfw.wmnet, wikikube-worker2175.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [19:22:23] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:26:29] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 12 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356990 (https://phabricator.wikimedia.org/T441729) (owner: 10Dillon) [19:26:42] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 12 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356877 (https://phabricator.wikimedia.org/T417847) (owner: 10Samtar) [19:31:16] (03CR) 10Bking: [C:03+2] data_purge: Drop raw HDFS fsimage files older than 56 days [puppet] - 10https://gerrit.wikimedia.org/r/1360975 (https://phabricator.wikimedia.org/T438056) (owner: 10Xcollazo) [19:32:55] (03PS1) 10Ahmon Dancy: admin: Limit the scap and spiderpig accounts to release-engineering [puppet] - 10https://gerrit.wikimedia.org/r/1369127 [19:40:23] (03CR) 10Ahmon Dancy: "Tightening up SpiderPig security now that we're using it for more deployment types." [puppet] - 10https://gerrit.wikimedia.org/r/1369127 (owner: 10Ahmon Dancy) [19:41:07] (03PS1) 10Ahmon Dancy: admin: Add deployment-jenkins to the release-engineering description [puppet] - 10https://gerrit.wikimedia.org/r/1369128 [19:47:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:52:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:56:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [20:04:29] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:06:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [20:16:23] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2301.codfw.wmnet, wikikube-worker2062.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-work [20:16:23] odfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2018.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker2374.codfw.wmnet, wikikube-worker2035.codfw.wmnet, wikikube-worker2284.codfw.wmnet, wikikube-worker2354.codfw.wmnet, wikikube-worker2348.codfw.wmnet, wikikube-worker2115.codfw.wmnet, wikikube-worker2316.codfw.wmnet, wikikube-worker2074.codfw.wmnet, wikikube-worker2342.codfw.wmnet, wikikube-worker2341.codfw.wmnet, w [20:16:23] worker2130.codfw.wmnet, wikikube-worker2175.codfw.wmnet, wikikube-worker2006.codfw.wmnet, wikikube-worker2168.codfw.wmnet, wikikube-worker2359.codfw.wmnet, wikikube-worker2362.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [20:17:23] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:23:32] (03CR) 10Aghirelli: [C:03+1] REST: Correct test server hostname to test.wikipedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366554 (https://phabricator.wikimedia.org/T408816) (owner: 10KineticPelagic) [20:24:01] (03CR) 10BPirkle: [C:03+2] rest-gateway: set wdqs v2 GA rate limits [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358194 (https://phabricator.wikimedia.org/T437892) (owner: 10Gmodena) [20:25:53] (03Merged) 10jenkins-bot: rest-gateway: set wdqs v2 GA rate limits [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358194 (https://phabricator.wikimedia.org/T437892) (owner: 10Gmodena) [20:34:30] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 12 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366554 (https://phabricator.wikimedia.org/T408816) (owner: 10KineticPelagic) [20:45:01] !log gmodena@deploy2003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [20:45:15] !log gmodena@deploy2003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [20:46:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:47:16] (03CR) 10Brouberol: [C:03+2] blunderbuss: drop the obsolete gitlab-wikimedia external_services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368992 (https://phabricator.wikimedia.org/T444600) (owner: 10Brouberol) [20:51:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:51:08] (03PS1) 10Brouberol: blunderbuss: bump chart version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369136 (https://phabricator.wikimedia.org/T444600) [20:54:24] (03CR) 10Brouberol: [C:03+2] blunderbuss: bump chart version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369136 (https://phabricator.wikimedia.org/T444600) (owner: 10Brouberol) [20:56:33] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:57:59] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [20:59:08] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [21:00:02] !log gmodena@deploy2003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [21:00:05] RoanKattouw, urbanecm, TheresNoTime, and cjming: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for UTC late backport window . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T2100). [21:00:05] sbassett, stephanebisson, dhardy, and AGhirelli: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [21:00:19] !log gmodena@deploy2003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [21:00:39] o/ [21:00:52] jouncebot now [21:00:52] For the next 0 hour(s) and 59 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T2100) [21:01:00] hey everyone o/ [21:01:18] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:01:53] sbassett is first on the list, are you around? [21:02:30] !log gmodena@deploy2003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [21:02:40] I'll get started with my patch without wasting any time [21:02:40] yes, here [21:02:43] !log gmodena@deploy2003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [21:02:47] oh, go for it [21:02:48] mine is a config patch, could go out with others… [21:03:06] though last time some log breakage did occur and i had to roll back. but i don’t think that’ll happen this time. [21:03:29] You do it yourself or do you need someone to drive? [21:04:18] (03CR) 10SBassett: [C:03+1] Content Security Policy: add stricter report-only policy [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366869 (https://phabricator.wikimedia.org/T419612) (owner: 10SBassett) [21:04:43] i can deploy it [21:04:55] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbassett@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366869 (https://phabricator.wikimedia.org/T419612) (owner: 10SBassett) [21:05:45] I keep getting disconnected [21:06:28] (03Merged) 10jenkins-bot: Content Security Policy: add stricter report-only policy [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366869 (https://phabricator.wikimedia.org/T419612) (owner: 10SBassett) [21:06:44] !log sbassett@deploy2003 Started scap sync-world: Backport for [[gerrit:1366869|Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]] [21:06:51] T431144: CSP: Unlock loading data and block loading scripts for execution - https://phabricator.wikimedia.org/T431144 [21:06:51] T327588: Consider adding more CSP directives to MediaWIki core - https://phabricator.wikimedia.org/T327588 [21:06:51] T245424: Consider adding base-uri 'none' to mediawiki csp policy - https://phabricator.wikimedia.org/T245424 [21:08:52] !log sbassett@deploy2003 sbassett: Backport for [[gerrit:1366869|Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:09:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.542% from eqiad to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [21:09:31] !log sbassett@deploy2003 sbassett: Continuing with deployment [21:11:23] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-work [21:11:23] odfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2058.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2090.codfw.wmnet, wikikube-worker2062.codfw.wmnet, wikikube-worker2296.codfw.wmnet, wikikube-worker2369.codfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2133.codfw.wmnet, wikikube-worker2324.codfw.wmnet, wikikube-worker2075.codfw.wmnet, w [21:11:23] worker-exp2001.codfw.wmnet, wikikube-worker2056.codfw.wmnet, wikikube-worker2126.codfw.wmnet, wikikube-worker2374.codfw.wmnet, wikikube-worker2267.codfw.wmnet, wikikube-worker2138.codfw https://wikitech.wikimedia.org/wiki/PyBal [21:12:23] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:15:37] !log sbassett@deploy2003 Finished scap sync-world: Backport for [[gerrit:1366869|Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]] (duration: 08m 53s) [21:15:44] T431144: CSP: Unlock loading data and block loading scripts for execution - https://phabricator.wikimedia.org/T431144 [21:15:45] T327588: Consider adding more CSP directives to MediaWIki core - https://phabricator.wikimedia.org/T327588 [21:15:45] T245424: Consider adding base-uri 'none' to mediawiki csp policy - https://phabricator.wikimedia.org/T245424 [21:16:10] i’m done, logs look fine. [21:16:30] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbisson@deploy2003 using scap backport" [extensions/ContentTranslation] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1369052 (https://phabricator.wikimedia.org/T444489) (owner: 10Sbisson) [21:16:48] sorry, i just realized i hadn’t regained my irc nick.. [21:22:14] (03Merged) 10jenkins-bot: CX: Skip unregistered VE plugin modules [extensions/ContentTranslation] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1369052 (https://phabricator.wikimedia.org/T444489) (owner: 10Sbisson) [21:22:31] !log sbisson@deploy2003 Started scap sync-world: Backport for [[gerrit:1369052|CX: Skip unregistered VE plugin modules (T444489)]] (1.48.0-wmf.4) [21:22:34] T444489: Content translation tool not loading - https://phabricator.wikimedia.org/T444489 [21:24:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [21:24:42] !log sbisson@deploy2003 sbisson: Backport for [[gerrit:1369052|CX: Skip unregistered VE plugin modules (T444489)]] (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:27:12] !log sbisson@deploy2003 sbisson: Continuing with deployment [21:33:10] !log sbisson@deploy2003 Finished scap sync-world: Backport for [[gerrit:1369052|CX: Skip unregistered VE plugin modules (T444489)]] (1.48.0-wmf.4) (duration: 10m 39s) [21:33:14] T444489: Content translation tool not loading - https://phabricator.wikimedia.org/T444489 [21:33:37] I'm done. Over to you dhardy [21:34:01] thx! I have two config patches to deploy [21:34:26] AGhirelli: do you want your changes to ride along? [21:34:37] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369143 [21:35:28] (03CR) 10Cklimas: [C:03+2] wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1369143 (owner: 10PipelineBot) [21:36:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dhardy@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356990 (https://phabricator.wikimedia.org/T441729) (owner: 10Dillon) [21:36:15] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dhardy@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356877 (https://phabricator.wikimedia.org/T417847) (owner: 10Samtar) [21:37:50] (03Merged) 10jenkins-bot: PersonalDashboard: Clean up config and set QA values for test and beta wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356990 (https://phabricator.wikimedia.org/T441729) (owner: 10Dillon) [21:37:52] (03CR) 10CI reject: [V:04-1] IS: Enable watchstar popover everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356877 (https://phabricator.wikimedia.org/T417847) (owner: 10Samtar) [21:39:27] !log cklimas@deploy2003 helmfile [staging] START helmfile.d/services/wikifeeds: apply [21:39:55] !log cklimas@deploy2003 helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [21:40:28] (03PS3) 10Dillon: IS: Enable watchstar popover everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356877 (https://phabricator.wikimedia.org/T417847) (owner: 10Samtar) [21:40:38] !log cklimas@deploy2003 helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [21:41:39] !log cklimas@deploy2003 helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [21:41:56] !log cklimas@deploy2003 helmfile [codfw] START helmfile.d/services/wikifeeds: apply [21:42:21] !log cklimas@deploy2003 helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [21:42:52] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dhardy@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356877 (https://phabricator.wikimedia.org/T417847) (owner: 10Samtar) [21:44:29] (03Merged) 10jenkins-bot: IS: Enable watchstar popover everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356877 (https://phabricator.wikimedia.org/T417847) (owner: 10Samtar) [21:44:43] !log dhardy@deploy2003 Started scap sync-world: Backport for [[gerrit:1356990|PersonalDashboard: Clean up config and set QA values for test and beta wikis (T441729)]], [[gerrit:1356877|IS: Enable watchstar popover everywhere (T417847)]] [21:44:48] T441729: Clean up PersonalDashboard and ModeratorToolkit config - https://phabricator.wikimedia.org/T441729 [21:44:49] T417847: Add labels field to watchstar popup on all wikis - https://phabricator.wikimedia.org/T417847 [21:46:52] !log dhardy@deploy2003 samtar, dhardy: Backport for [[gerrit:1356990|PersonalDashboard: Clean up config and set QA values for test and beta wikis (T441729)]], [[gerrit:1356877|IS: Enable watchstar popover everywhere (T417847)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:49:36] !log dhardy@deploy2003 samtar, dhardy: Continuing with deployment [21:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.54% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [21:55:41] !log dhardy@deploy2003 Finished scap sync-world: Backport for [[gerrit:1356990|PersonalDashboard: Clean up config and set QA values for test and beta wikis (T441729)]], [[gerrit:1356877|IS: Enable watchstar popover everywhere (T417847)]] (duration: 10m 57s) [21:55:45] T441729: Clean up PersonalDashboard and ModeratorToolkit config - https://phabricator.wikimedia.org/T441729 [21:55:46] T417847: Add labels field to watchstar popup on all wikis - https://phabricator.wikimedia.org/T417847 [21:55:59] \o done! [22:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261112T2200) [22:01:12] Is anyone planning to use the Readers deployment window? If not, I would like to deploy a security patch [22:05:26] Alright, deploying [22:19:38] Done [22:30:35] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [22:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [22:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [22:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [22:51:30] Going to deploy AGhirelli's change now, he missed dhardy's ping by 10 minutes earlier [22:52:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366554 (https://phabricator.wikimedia.org/T408816) (owner: 10KineticPelagic) [22:53:01] Many thanks Roan! [22:53:21] AGhirelli: Do you have the WikimediaDebug browser extension installed? You'll need that to test your change later (in about 10 minutes or so) [22:53:31] Yup [22:53:45] Great! The bot will post here when it's time to test [22:54:00] Thanks! I'll keep the eyes on the chat [22:54:01] (03Merged) 10jenkins-bot: REST: Correct test server hostname to test.wikipedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1366554 (https://phabricator.wikimedia.org/T408816) (owner: 10KineticPelagic) [22:54:19] !log catrope@deploy2003 Started scap sync-world: Backport for [[gerrit:1366554|REST: Correct test server hostname to test.wikipedia.org (T408816)]] [22:54:22] T408816: Add 'test' as a server option for Wikipedia projects within REST Sandbox - https://phabricator.wikimedia.org/T408816 [22:56:26] !log catrope@deploy2003 kineticpelagic, catrope: Backport for [[gerrit:1366554|REST: Correct test server hostname to test.wikipedia.org (T408816)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:57:11] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2191.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-work [22:57:11] odfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2159.codfw.wmnet, wikikube-worker2276.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2285.codfw.wmnet, wikikube-worker2268.codfw.wmnet, wikikube-worker-exp2001.codfw.wmnet, wikikube-worker2013.codfw.wmnet, wikikube-worker2126.codfw.wmnet, wikikube-worker2374.codfw.wmnet, wikikube-worker2286.codfw.wmnet, wikikube-worker2275.codfw.wmne [22:57:11] ube-worker2034.codfw.wmnet, wikikube-worker2284.codfw.wmnet, wikikube-worker2271.codfw.wmnet, wikikube-worker2209.codfw.wmnet, wikikube-worker2046.codfw.wmnet, wikikube-worker2167.codfw https://wikitech.wikimedia.org/wiki/PyBal [22:58:11] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:59:01] AGhirelli: Please test [22:59:16] Yup, going to test [23:04:20] Ok, Roan it seems all right! :) [23:04:27] !log catrope@deploy2003 kineticpelagic, catrope: Continuing with deployment [23:06:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 29d 19h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [23:10:32] !log catrope@deploy2003 Finished scap sync-world: Backport for [[gerrit:1366554|REST: Correct test server hostname to test.wikipedia.org (T408816)]] (duration: 16m 14s) [23:10:35] T408816: Add 'test' as a server option for Wikipedia projects within REST Sandbox - https://phabricator.wikimedia.org/T408816 [23:10:54] AGhirelli: All done, should be live everywhere now. You should now see the effect of the patch with WMDebug turned off as well [23:11:21] Yup, confirmed! Many many thanks Roan!! [23:15:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:30:35] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state