[00:00:04] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [00:00:14] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [00:00:19] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [00:00:29] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [00:00:32] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [00:00:39] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [00:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:05:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker2015:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker2015 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [00:39:00] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1368420 [00:39:00] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1368420 (owner: 10TrainBranchBot) [00:45:20] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1368420 (owner: 10TrainBranchBot) [01:08:59] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1368424 [01:08:59] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1368424 (owner: 10TrainBranchBot) [01:15:16] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1368424 (owner: 10TrainBranchBot) [01:29:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12542863 (10phaultfinder) [01:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.37% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T0200) [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [02:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [02:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [03:20:23] FIRING: [11x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [03:37:23] FIRING: GnmiInterfaceCountersDrop: ... [03:37:23] cr1-esams is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr1-esams:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [04:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:24:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12543078 (10phaultfinder) [05:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.42% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [06:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [06:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [07:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T0700) [07:20:23] FIRING: [11x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:34:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12543238 (10phaultfinder) [07:35:07] (03CR) 10Marostegui: [C:03+2] team-data-persistence: Alert on high running thread [alerts] - 10https://gerrit.wikimedia.org/r/1366067 (https://phabricator.wikimedia.org/T442995) (owner: 10Marostegui) [07:37:23] FIRING: GnmiInterfaceCountersDrop: ... [07:37:23] cr1-esams is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr1-esams:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [07:44:22] (03PS1) 10Marostegui: team-data-persistence: alert connection exhaustion [alerts] - 10https://gerrit.wikimedia.org/r/1368597 (https://phabricator.wikimedia.org/T442995) [08:00:04] Amir1, urbanecm, awight, and chlod: UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T0800). Please do the needful. [08:00:04] WMDE-Fisch: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [08:00:23] o/ I'll self serve [08:01:52] gotcha, window's all yours :) [08:02:08] :-) [08:02:13] (03CR) 10TrainBranchBot: [C:03+2] "Approved by wmde-fisch@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368298 (https://phabricator.wikimedia.org/T443469) (owner: 10WMDE-Fisch) [08:03:15] (03Merged) 10jenkins-bot: Enable Citoid PDF fallback on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368298 (https://phabricator.wikimedia.org/T443469) (owner: 10WMDE-Fisch) [08:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:04:44] !log wmde-fisch@deploy2003 Started scap sync-world: Backport for [[gerrit:1368298|Enable Citoid PDF fallback on testwiki (T443469)]] [08:04:48] T443469: 1️⃣ Deploy PDF URL fallback to pilot wikis (S) - https://phabricator.wikimedia.org/T443469 [08:07:09] !log wmde-fisch@deploy2003 wmde-fisch: Backport for [[gerrit:1368298|Enable Citoid PDF fallback on testwiki (T443469)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:07:26] (03PS1) 10Jelto: admin: Add FIDO SSH key for cklimas [puppet] - 10https://gerrit.wikimedia.org/r/1368605 (https://phabricator.wikimedia.org/T442758) [08:09:38] !log wmde-fisch@deploy2003 wmde-fisch: Continuing with deployment [08:09:38] (03CR) 10Jelto: [C:03+1] "verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1368605 (https://phabricator.wikimedia.org/T442758) (owner: 10Jelto) [08:09:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12543308 (10phaultfinder) [08:10:34] (03CR) 10Jelto: [C:03+2] admin: Add FIDO SSH key for cklimas [puppet] - 10https://gerrit.wikimedia.org/r/1368605 (https://phabricator.wikimedia.org/T442758) (owner: 10Jelto) [08:15:43] !log wmde-fisch@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368298|Enable Citoid PDF fallback on testwiki (T443469)]] (duration: 10m 59s) [08:15:46] T443469: 1️⃣ Deploy PDF URL fallback to pilot wikis (S) - https://phabricator.wikimedia.org/T443469 [08:17:32] (03PS1) 10Jelto: aptrepo: update gitlab-ce package to 19.3 [puppet] - 10https://gerrit.wikimedia.org/r/1368608 (https://phabricator.wikimedia.org/T444442) [08:18:06] I'll push another config patch down that road, that I did not manage to add to the wikipage. [08:18:13] Its just cleanup though. [08:18:51] (03CR) 10TrainBranchBot: [C:03+2] "Approved by wmde-fisch@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1126134 (https://phabricator.wikimedia.org/T258917) (owner: 10Reedy) [08:19:40] (03Merged) 10jenkins-bot: Drop TemplateData EventStreams/EventLogging config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1126134 (https://phabricator.wikimedia.org/T258917) (owner: 10Reedy) [08:20:04] !log wmde-fisch@deploy2003 Started scap sync-world: Backport for [[gerrit:1126134|Drop TemplateData EventStreams/EventLogging config (T258917)]] [08:20:08] T258917: Record template use and dialog interaction metrics - https://phabricator.wikimedia.org/T258917 [08:24:02] (03CR) 10Jelto: [C:03+2] aptrepo: update gitlab-ce package to 19.3 [puppet] - 10https://gerrit.wikimedia.org/r/1368608 (https://phabricator.wikimedia.org/T444442) (owner: 10Jelto) [08:24:22] !log wmde-fisch@deploy2003 wmde-fisch, reedy: Backport for [[gerrit:1126134|Drop TemplateData EventStreams/EventLogging config (T258917)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:25:09] !log wmde-fisch@deploy2003 wmde-fisch, reedy: Continuing with deployment [08:31:13] !log wmde-fisch@deploy2003 Finished scap sync-world: Backport for [[gerrit:1126134|Drop TemplateData EventStreams/EventLogging config (T258917)]] (duration: 11m 08s) [08:31:16] T258917: Record template use and dialog interaction metrics - https://phabricator.wikimedia.org/T258917 [08:31:28] I'm done! [08:47:42] !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade [08:58:35] !log jelto@cumin1004 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade [08:59:15] !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade [09:00:05] andre and brennen: That opportune time for a MediaWiki train - Utc-0+Utc-7 Version deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T0900). [09:05:14] o/ [09:09:56] !log jelto@cumin1004 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade [09:10:28] (03PS1) 10TrainBranchBot: group1 to 1.48.0-wmf.4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368611 (https://phabricator.wikimedia.org/T442222) [09:10:32] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by aklapper@deploy2003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368611 (https://phabricator.wikimedia.org/T442222) (owner: 10TrainBranchBot) [09:10:57] !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade [09:11:16] (03Merged) 10jenkins-bot: group1 to 1.48.0-wmf.4 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368611 (https://phabricator.wikimedia.org/T442222) (owner: 10TrainBranchBot) [09:19:48] !log aklapper@deploy2003 rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.4 refs T442222 [09:19:51] T442222: 1.48.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T442222 [09:27:40] (03CR) 10Awight: "should be removed from -labs in the same patch, IMO" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368324 (https://phabricator.wikimedia.org/T438982) (owner: 10Seanleong-wmde) [09:30:05] jelto@cumin1004 jelto: The backup on gitlab1004 is complete, ready to proceed with upgrade. [09:36:53] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 3166 bytes in 0.031 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [09:37:53] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29853 bytes in 0.131 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [09:38:15] ^ that's me [09:40:44] (03PS6) 10Dpogorzelski: admin_ng: add KEDA, on ml-staging-codfw only [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364011 (https://phabricator.wikimedia.org/T436650) [09:40:47] !log jelto@cumin1004 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade [09:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.45% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [09:53:03] (03PS4) 10Hnowlan: alertmanager: remove the silences limited to the given hosts [software/spicerack] - 10https://gerrit.wikimedia.org/r/1355701 (https://phabricator.wikimedia.org/T395032) [09:53:03] (03PS1) 10Hnowlan: alertmanager: log dry-run silence deletions accurately [software/spicerack] - 10https://gerrit.wikimedia.org/r/1368623 (https://phabricator.wikimedia.org/T395032) [09:53:05] (03PS1) 10Hnowlan: alertmanager: add remove_downtimes() to remove silences by matchers [software/spicerack] - 10https://gerrit.wikimedia.org/r/1368624 (https://phabricator.wikimedia.org/T439099) [09:53:07] (03PS1) 10Hnowlan: alertmanager: warn about host silences that are not removed [software/spicerack] - 10https://gerrit.wikimedia.org/r/1368625 (https://phabricator.wikimedia.org/T395032) [09:56:03] (03CR) 10CI reject: [V:04-1] alertmanager: warn about host silences that are not removed [software/spicerack] - 10https://gerrit.wikimedia.org/r/1368625 (https://phabricator.wikimedia.org/T395032) (owner: 10Hnowlan) [09:56:24] (03CR) 10CI reject: [V:04-1] alertmanager: add remove_downtimes() to remove silences by matchers [software/spicerack] - 10https://gerrit.wikimedia.org/r/1368624 (https://phabricator.wikimedia.org/T439099) (owner: 10Hnowlan) [09:58:19] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [10:03:03] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [10:08:11] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - inference-staging_30443: Servers ml-staging2003.codfw.wmnet, ml-staging2002.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [10:09:11] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [10:19:49] (03CR) 10Bartosz Wójtowicz: [C:03+2] kserve: allow deploy users to delete Knative revisions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360421 (https://phabricator.wikimedia.org/T442872) (owner: 10Bartosz Wójtowicz) [10:21:18] (03PS1) 10Kevin Bazira: ml-services: deploy tts-section-generator server that uses trixie and ffmpeg 7.1.5 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368627 (https://phabricator.wikimedia.org/T444440) [10:21:46] (03CR) 10Matthias Mullie: "recheck" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368166 (owner: 10Matthias Mullie) [10:22:23] RESOLVED: GnmiInterfaceCountersDrop: ... [10:22:23] cr1-esams is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=cr1-esams:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [10:22:47] (03PS1) 10Majavah: dynamicproxy: Notify HAProxy on error page changes [puppet] - 10https://gerrit.wikimedia.org/r/1368628 [10:23:31] (03CR) 10Hnowlan: "recheck" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1368624 (https://phabricator.wikimedia.org/T439099) (owner: 10Hnowlan) [10:23:42] (03CR) 10CI reject: [V:04-1] dynamicproxy: Notify HAProxy on error page changes [puppet] - 10https://gerrit.wikimedia.org/r/1368628 (owner: 10Majavah) [10:25:42] (03PS2) 10Majavah: dynamicproxy: Notify HAProxy on error page changes [puppet] - 10https://gerrit.wikimedia.org/r/1368628 [10:25:52] (03Merged) 10jenkins-bot: kserve: allow deploy users to delete Knative revisions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360421 (https://phabricator.wikimedia.org/T442872) (owner: 10Bartosz Wójtowicz) [10:26:08] (03CR) 10Bartosz Wójtowicz: [C:03+1] ml-services: deploy tts-section-generator server that uses trixie and ffmpeg 7.1.5 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368627 (https://phabricator.wikimedia.org/T444440) (owner: 10Kevin Bazira) [10:26:35] (03CR) 10CI reject: [V:04-1] dynamicproxy: Notify HAProxy on error page changes [puppet] - 10https://gerrit.wikimedia.org/r/1368628 (owner: 10Majavah) [10:27:59] (03PS1) 10Matthias Mullie: Do not Implement UserThankedListener [extensions/WikimediaEvents] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368631 (https://phabricator.wikimedia.org/T441426) [10:28:52] (03CR) 10Matthias Mullie: [C:03+1] "Cherrypicking to wmf.4, as CI otherwise prevents backporting in the relevant extensions" [extensions/WikimediaEvents] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368631 (https://phabricator.wikimedia.org/T441426) (owner: 10Matthias Mullie) [10:29:08] (03PS2) 10Matthias Mullie: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368166 [10:29:11] (03CR) 10Hnowlan: "recheck" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1368625 (https://phabricator.wikimedia.org/T395032) (owner: 10Hnowlan) [10:29:32] !log dpogorzelski@deploy2003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [10:29:35] (03PS1) 10Gmodena: wdqs: route SERVICE traffic internally [puppet] - 10https://gerrit.wikimedia.org/r/1368632 (https://phabricator.wikimedia.org/T442059) [10:29:51] (03PS3) 10Majavah: dynamicproxy: Notify HAProxy on error page changes [puppet] - 10https://gerrit.wikimedia.org/r/1368628 [10:30:42] !log dpogorzelski@deploy2003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [10:33:34] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 11 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/WikimediaEvents] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368631 (https://phabricator.wikimedia.org/T441426) (owner: 10Matthias Mullie) [10:33:42] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 11 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368166 (owner: 10Matthias Mullie) [10:35:09] !log dpogorzelski@deploy2003 helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [10:36:16] !log dpogorzelski@deploy2003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [10:38:39] !log dpogorzelski@deploy2003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [10:40:42] !log dpogorzelski@deploy2003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [10:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:41:29] !log ammarpad@deploy2003 mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Language Converter' 'Language converter' Ammarpad --reason 'Requested at [[:phab:T444079|T444079]]' [10:41:32] T444079: Request to move translatable page: Language Converter - https://phabricator.wikimedia.org/T444079 [10:42:53] !log dpogorzelski@deploy2003 helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [10:43:59] !log dpogorzelski@deploy2003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [10:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [10:47:24] (03PS1) 10Muehlenhoff: Remove access for cdobbins [puppet] - 10https://gerrit.wikimedia.org/r/1368634 [10:47:53] (03CR) 10Ssingh: [C:03+1] Remove access for cdobbins [puppet] - 10https://gerrit.wikimedia.org/r/1368634 (owner: 10Muehlenhoff) [10:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [10:49:41] (03CR) 10Ladsgroup: [C:03+1] dynamicproxy: Notify HAProxy on error page changes [puppet] - 10https://gerrit.wikimedia.org/r/1368628 (owner: 10Majavah) [10:49:59] (03CR) 10Muehlenhoff: [C:03+2] Remove access for cdobbins [puppet] - 10https://gerrit.wikimedia.org/r/1368634 (owner: 10Muehlenhoff) [10:51:32] (03CR) 10Majavah: [C:03+2] dynamicproxy: Notify HAProxy on error page changes [puppet] - 10https://gerrit.wikimedia.org/r/1368628 (owner: 10Majavah) [10:52:36] (03CR) 10Kevin Bazira: [C:03+2] ml-services: deploy tts-section-generator server that uses trixie and ffmpeg 7.1.5 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368627 (https://phabricator.wikimedia.org/T444440) (owner: 10Kevin Bazira) [10:52:46] !log jmm@cumin2003 DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging CDobbins out of all services on: 2439 hosts [10:54:17] (03Merged) 10jenkins-bot: ml-services: deploy tts-section-generator server that uses trixie and ffmpeg 7.1.5 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368627 (https://phabricator.wikimedia.org/T444440) (owner: 10Kevin Bazira) [10:56:59] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [11:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1100) [11:00:46] !log kevinbazira@deploy2003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . [11:01:59] !log kevinbazira@deploy2003 helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [11:17:45] (03PS2) 10Jforrester: wikifunctions: Drop the evaluators' "request complete" log line [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363475 [11:17:45] (03PS1) 10Jforrester: wikifunctions: Upgrade evaluators from 2026-11-04-142036 to 2026-11-04-233007 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368638 [11:17:45] (03PS1) 10Jforrester: wikifunctions: Upgrade orchestrator from 2026-11-04-133739 to 2026-11-10-004058 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368639 (https://phabricator.wikimedia.org/T440244) [11:19:14] (03PS1) 10Zabe: Add script to remove own user page watches for users without edits [extensions/WikimediaMaintenance] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368640 (https://phabricator.wikimedia.org/T442779) [11:20:23] FIRING: [11x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:23:20] (03CR) 10Zabe: [C:03+2] Add script to remove own user page watches for users without edits [extensions/WikimediaMaintenance] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368640 (https://phabricator.wikimedia.org/T442779) (owner: 10Zabe) [11:24:49] (03Merged) 10jenkins-bot: Add script to remove own user page watches for users without edits [extensions/WikimediaMaintenance] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368640 (https://phabricator.wikimedia.org/T442779) (owner: 10Zabe) [11:25:19] !log zabe@deploy2003 Started scap sync-world: Backport for [[gerrit:1368640|Add script to remove own user page watches for users without edits (T442779)]] (1.48.0-wmf.4) [11:25:22] T442779: Stop users watching their own userpage on creation on loginwiki - https://phabricator.wikimedia.org/T442779 [11:29:40] !log zabe@deploy2003 zabe: Backport for [[gerrit:1368640|Add script to remove own user page watches for users without edits (T442779)]] (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:30:24] !log zabe@deploy2003 zabe: Continuing with deployment [11:36:24] !log zabe@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368640|Add script to remove own user page watches for users without edits (T442779)]] (1.48.0-wmf.4) (duration: 11m 05s) [11:36:27] T442779: Stop users watching their own userpage on creation on loginwiki - https://phabricator.wikimedia.org/T442779 [11:40:11] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2033.codfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2191.codfw.wmnet, wikikube-worker2356.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2176.codfw.wmnet, wikikube-work [11:40:11] odfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2355.codfw.wmnet, wikikube-worker2124.codfw.wmnet, wikikube-worker2090.codfw.wmnet, wikikube-worker2206.codfw.wmnet, wikikube-worker2251.codfw.wmnet, wikikube-worker2263.codfw.wmnet, wikikube-worker2062.codfw.wmnet, wikikube-worker2164.codfw.wmnet, wikikube-worker2134.codfw.wmnet, w [11:40:11] worker2272.codfw.wmnet, wikikube-worker2329.codfw.wmnet, wikikube-worker2339.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2014.codfw.wmnet, wikikube-worker2114.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [11:41:11] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [11:48:38] (03PS1) 10Gkyziridis: ml-services: place llm-qwen3-14b on 96G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368651 (https://phabricator.wikimedia.org/T444469) [11:58:39] (03PS1) 10Dreamy Jazz: hCaptcha: Require hCaptcha for all API edits on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368652 (https://phabricator.wikimedia.org/T444470) [11:58:42] jouncebot: nowandnext [11:58:42] For the next 0 hour(s) and 1 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1100) [11:58:42] In 0 hour(s) and 1 minute(s): Services – Citoid / Zotero (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1200) [11:58:42] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, 06tools-infrastructure-team: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12544243 (10fnegri) [11:59:23] Based on z abe just using scap, I'm going to assume scap is free and use it now [12:00:05] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:05] mvolz: Services – Citoid / Zotero (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1200). Please do the needful. [12:00:14] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:20] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:24] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368652 (https://phabricator.wikimedia.org/T444470) (owner: 10Dreamy Jazz) [12:00:30] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:00:33] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [12:00:40] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [12:00:56] (03Merged) 10jenkins-bot: hCaptcha: Require hCaptcha for all API edits on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368652 (https://phabricator.wikimedia.org/T444470) (owner: 10Dreamy Jazz) [12:01:22] (03CR) 10CWilliams: [C:03+2] Add lucian to the ops-limited group [puppet] - 10https://gerrit.wikimedia.org/r/1364304 (owner: 10Lucian) [12:01:26] !log dreamyjazz@deploy2003 Started scap sync-world: Backport for [[gerrit:1368652|hCaptcha: Require hCaptcha for all API edits on testwiki (T444470)]] [12:01:29] T444470: Require hCaptcha for all API edits on testwiki - https://phabricator.wikimedia.org/T444470 [12:02:30] (03CR) 10Lucian: "Done" [puppet] - 10https://gerrit.wikimedia.org/r/1364304 (owner: 10Lucian) [12:04:09] !log dreamyjazz@deploy2003 dreamyjazz: Backport for [[gerrit:1368652|hCaptcha: Require hCaptcha for all API edits on testwiki (T444470)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:05:20] 06SRE, 10SRE-swift-storage, 10Cloud-VPS, 10Observability-Metrics, 06tools-infrastructure-team: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#12544323 (10fnegri) [12:06:50] !log dreamyjazz@deploy2003 dreamyjazz: Continuing with deployment [12:11:55] (03PS1) 10Dreamy Jazz: hCaptcha: Drop CommunityRequests exemption [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368654 (https://phabricator.wikimedia.org/T426897) [12:12:56] !log dreamyjazz@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368652|hCaptcha: Require hCaptcha for all API edits on testwiki (T444470)]] (duration: 11m 30s) [12:12:59] T444470: Require hCaptcha for all API edits on testwiki - https://phabricator.wikimedia.org/T444470 [12:13:27] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368654 (https://phabricator.wikimedia.org/T426897) (owner: 10Dreamy Jazz) [12:14:43] (03Merged) 10jenkins-bot: hCaptcha: Drop CommunityRequests exemption [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368654 (https://phabricator.wikimedia.org/T426897) (owner: 10Dreamy Jazz) [12:15:07] !log dreamyjazz@deploy2003 Started scap sync-world: Backport for [[gerrit:1368654|hCaptcha: Drop CommunityRequests exemption (T426897)]] [12:15:10] T426897: hCaptcha: Add support in CommunityRequests extension - https://phabricator.wikimedia.org/T426897 [12:17:28] !log dreamyjazz@deploy2003 dreamyjazz: Backport for [[gerrit:1368654|hCaptcha: Drop CommunityRequests exemption (T426897)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:20:03] !log dreamyjazz@deploy2003 dreamyjazz: Continuing with deployment [12:21:12] (03CR) 10Kevin Bazira: [C:03+1] ml-services: place llm-qwen3-14b on 96G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368651 (https://phabricator.wikimedia.org/T444469) (owner: 10Gkyziridis) [12:24:27] (03CR) 10Bartosz Wójtowicz: [C:03+1] "LGTM, let's try it out!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368651 (https://phabricator.wikimedia.org/T444469) (owner: 10Gkyziridis) [12:24:34] (03CR) 10Urbanecm: [C:03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) (owner: 10Michael Große) [12:26:07] !log dreamyjazz@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368654|hCaptcha: Drop CommunityRequests exemption (T426897)]] (duration: 11m 00s) [12:26:10] T426897: hCaptcha: Add support in CommunityRequests extension - https://phabricator.wikimedia.org/T426897 [12:28:18] (03Restored) 10Mvolz: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364126 (owner: 10PipelineBot) [12:33:25] (03PS1) 10Bartosz Wójtowicz: ml-services: move llm-qwen3-14b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368662 (https://phabricator.wikimedia.org/T444464) [12:38:30] (03CR) 10Mvolz: [C:03+2] citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364126 (owner: 10PipelineBot) [12:38:32] (03PS2) 10Seanleong-wmde: Switch site lookup to sitematrix lookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368324 (https://phabricator.wikimedia.org/T438982) [12:39:18] (03CR) 10CI reject: [V:04-1] Switch site lookup to sitematrix lookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368324 (https://phabricator.wikimedia.org/T438982) (owner: 10Seanleong-wmde) [12:40:23] (03Merged) 10jenkins-bot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364126 (owner: 10PipelineBot) [12:42:48] !log mvolz@deploy2003 Started scap deploy-service citoid: PDF fallback [12:42:48] !log mvolz@deploy2003 scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid [12:42:52] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1355701 (https://phabricator.wikimedia.org/T395032) (owner: 10Hnowlan) [12:43:04] (03PS3) 10Seanleong-wmde: Switch site lookup to sitematrix lookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368324 (https://phabricator.wikimedia.org/T438982) [12:46:09] !log mvolz@deploy2003 scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid [12:47:57] !log mvolz@deploy2003 scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid [12:51:51] (03CR) 10Awight: [C:03+1] Switch site lookup to sitematrix lookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368324 (https://phabricator.wikimedia.org/T438982) (owner: 10Seanleong-wmde) [12:53:17] (03PS1) 10Dreamy Jazz: hCaptcha: Stop collecting risk scores for IP block notices [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368668 (https://phabricator.wikimedia.org/T444473) [12:53:49] jouncebot: nowandnext [12:53:49] For the next 0 hour(s) and 6 minute(s): Services – Citoid / Zotero (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1200) [12:53:49] In 1 hour(s) and 6 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1400) [12:57:58] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 12 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368324 (https://phabricator.wikimedia.org/T438982) (owner: 10Seanleong-wmde) [12:58:55] (03CR) 10Michael Große: [C:03+1] "I think this is ready to be deployed tomorrow, right? See the coordination with Experiment Platform here: https://wikimedia.slack.com/arch" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367855 (owner: 10Sergio Gimeno) [12:59:19] (03CR) 10Kevin Bazira: [C:03+1] ml-services: move llm-qwen3-14b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368662 (https://phabricator.wikimedia.org/T444464) (owner: 10Bartosz Wójtowicz) [13:02:57] (03PS1) 10Urbanecm: Hooks: Do not disable exempt beta features when auto-enroll is on [extensions/BetaFeatures] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368670 (https://phabricator.wikimedia.org/T444361) [13:03:13] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 11 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [extensions/BetaFeatures] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368670 (https://phabricator.wikimedia.org/T444361) (owner: 10Urbanecm) [13:03:24] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368668 (https://phabricator.wikimedia.org/T444473) (owner: 10Dreamy Jazz) [13:03:52] !log mvolz@deploy2003 Finished scap deploy-service citoid: PDF fallback (duration: 21m 04s) [13:04:10] (03Merged) 10jenkins-bot: hCaptcha: Stop collecting risk scores for IP block notices [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368668 (https://phabricator.wikimedia.org/T444473) (owner: 10Dreamy Jazz) [13:04:35] !log dreamyjazz@deploy2003 Started scap sync-world: Backport for [[gerrit:1368668|hCaptcha: Stop collecting risk scores for IP block notices (T444473)]] [13:04:38] T444473: hCaptcha: Disable collection of hCaptcha risk scores for users seeing IP block notices on WMF wikis - https://phabricator.wikimedia.org/T444473 [13:05:44] (03PS2) 10Gmodena: rest-gateway: add smoke test for the WDQSv2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368286 (https://phabricator.wikimedia.org/T444478) [13:06:55] !log dreamyjazz@deploy2003 dreamyjazz: Backport for [[gerrit:1368668|hCaptcha: Stop collecting risk scores for IP block notices (T444473)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:07:49] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 11 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) (owner: 10Michael Große) [13:07:55] !log dreamyjazz@deploy2003 dreamyjazz: Continuing with deployment [13:13:59] !log dreamyjazz@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368668|hCaptcha: Stop collecting risk scores for IP block notices (T444473)]] (duration: 09m 24s) [13:14:02] T444473: hCaptcha: Disable collection of hCaptcha risk scores for users seeing IP block notices on WMF wikis - https://phabricator.wikimedia.org/T444473 [13:18:15] (03PS1) 10STran: Handle multiple updates in case creation atomic section [extensions/CheckUser] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368674 (https://phabricator.wikimedia.org/T444117) [13:18:31] (03PS1) 10STran: Handle multiple updates in case creation atomic section [extensions/CheckUser] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368675 (https://phabricator.wikimedia.org/T444117) [13:24:56] (03CR) 10EarlyWarningBot: "[Failed command](https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83/130061/consoleFull): `composer run --timeout=0 phpuni" [extensions/CheckUser] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368675 (https://phabricator.wikimedia.org/T444117) (owner: 10STran) [13:25:24] (03CR) 10CI reject: [V:04-1] Handle multiple updates in case creation atomic section [extensions/CheckUser] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368675 (https://phabricator.wikimedia.org/T444117) (owner: 10STran) [13:36:46] (03PS1) 10Trueg: WDQSv2: Reduce number of CPU cores for index rebuilds on wdqs-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368679 (https://phabricator.wikimedia.org/T444487) [13:46:34] (03CR) 10Gmodena: [C:03+1] WDQSv2: Reduce number of CPU cores for index rebuilds on wdqs-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368679 (https://phabricator.wikimedia.org/T444487) (owner: 10Trueg) [13:48:39] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: move llm-qwen3-14b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368662 (https://phabricator.wikimedia.org/T444464) (owner: 10Bartosz Wójtowicz) [13:49:06] (03CR) 10Trueg: [C:03+2] WDQSv2: Reduce number of CPU cores for index rebuilds on wdqs-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368679 (https://phabricator.wikimedia.org/T444487) (owner: 10Trueg) [13:49:57] (03CR) 10STran: "recheck" [extensions/CheckUser] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368675 (https://phabricator.wikimedia.org/T444117) (owner: 10STran) [13:50:18] (03Merged) 10jenkins-bot: ml-services: move llm-qwen3-14b to vLLM 0.32 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368662 (https://phabricator.wikimedia.org/T444464) (owner: 10Bartosz Wójtowicz) [13:50:36] (03Merged) 10jenkins-bot: WDQSv2: Reduce number of CPU cores for index rebuilds on wdqs-next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368679 (https://phabricator.wikimedia.org/T444487) (owner: 10Trueg) [13:51:28] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [13:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.46% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [13:52:52] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [13:53:27] !log trueg@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [13:54:42] !log trueg@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [13:56:10] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 11 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [extensions/CheckUser] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368675 (https://phabricator.wikimedia.org/T444117) (owner: 10STran) [13:56:21] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 11 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [extensions/CheckUser] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368674 (https://phabricator.wikimedia.org/T444117) (owner: 10STran) [13:59:29] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [13:59:34] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . [13:59:34] jouncebot: nowandnext [13:59:34] No deployments scheduled for the next 0 hour(s) and 0 minute(s) [13:59:35] In 0 hour(s) and 0 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1400) [14:00:05] urbanecm and TheresNoTime: UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1400). Please do the needful. [14:00:05] Tran, urbanecm, and MichaelG_WMF: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [14:00:08] o/ [14:00:20] * MichaelG_WMF is here [14:00:34] \o (will wait to maybe make a config change after everyone else) [14:00:36] I have a private deploy ahead of my config change. I also have a change I'd like to backport separately but I can wait for that toward the end of the window or do it tomorrow [14:00:46] I'm going to get started on the private component of my deploy [14:01:39] I can deploy today [14:01:56] Tran: can you ping me once done, please? [14:01:59] ack [14:08:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [14:10:19] (03PS1) 10Dzahn: Revert "add amical.wm.o for Amical Wikimedia wiki" [dns] - 10https://gerrit.wikimedia.org/r/1368682 (https://phabricator.wikimedia.org/T444019) [14:11:41] (03PS1) 10Dzahn: Revert "Add DNS entries for initiativeswiki" [dns] - 10https://gerrit.wikimedia.org/r/1368683 (https://phabricator.wikimedia.org/T444019) [14:11:49] (03CR) 10CI reject: [V:04-1] Revert "Add DNS entries for initiativeswiki" [dns] - 10https://gerrit.wikimedia.org/r/1368683 (https://phabricator.wikimedia.org/T444019) (owner: 10Dzahn) [14:13:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [14:15:53] (03PS1) 10Dzahn: delete comcom.wikimedia.org, unused [dns] - 10https://gerrit.wikimedia.org/r/1368688 (https://phabricator.wikimedia.org/T444019) [14:18:32] private deploy done, moving on to config deploy [14:19:25] !log Deployed patch for T442582, T443646 [14:19:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:19:30] T442582: Enable Suggested Investigations on Portuguese Wikipedia (ptwiki) - https://phabricator.wikimedia.org/T442582 [14:19:31] T443646: Enable SI on nlwiki - https://phabricator.wikimedia.org/T443646 [14:20:13] (03CR) 10STran: Enable Special:SuggestedInvestigations on ptwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368331 (https://phabricator.wikimedia.org/T442582) (owner: 10STran) [14:20:25] (03CR) 10TrainBranchBot: [C:03+2] "Approved by stran@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368331 (https://phabricator.wikimedia.org/T442582) (owner: 10STran) [14:21:15] (03Merged) 10jenkins-bot: Enable Special:SuggestedInvestigations on ptwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368331 (https://phabricator.wikimedia.org/T442582) (owner: 10STran) [14:21:40] !log stran@deploy2003 Started scap sync-world: Backport for [[gerrit:1368331|Enable Special:SuggestedInvestigations on ptwiki (T442582)]] [14:24:05] !log stran@deploy2003 stran: Backport for [[gerrit:1368331|Enable Special:SuggestedInvestigations on ptwiki (T442582)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:25:15] !log stran@deploy2003 stran: Continuing with deployment [14:31:13] (03CR) 10Zabe: [C:03+1] Revert "add amical.wm.o for Amical Wikimedia wiki" [dns] - 10https://gerrit.wikimedia.org/r/1368682 (https://phabricator.wikimedia.org/T444019) (owner: 10Dzahn) [14:31:18] !log stran@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368331|Enable Special:SuggestedInvestigations on ptwiki (T442582)]] (duration: 09m 38s) [14:31:21] T442582: Enable Suggested Investigations on Portuguese Wikipedia (ptwiki) - https://phabricator.wikimedia.org/T442582 [14:31:30] urbanecm: I'm done [14:32:00] there were two backports too, those won't be done Tran ? [14:32:10] (03CR) 10Urbanecm: [C:03+2] Hooks: Do not disable exempt beta features when auto-enroll is on [extensions/BetaFeatures] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368670 (https://phabricator.wikimedia.org/T444361) (owner: 10Urbanecm) [14:32:17] It's not critical and I feel bad that I've taken up half the window already [14:32:25] But I can if no one minds [14:32:30] Tran: i can ship them with my backport if you can test them [14:32:32] that would be quicker [14:32:36] it would (hopefully) solve some scary looking prod errors [14:32:45] sure that would work too, thanks [14:32:49] (03CR) 10Urbanecm: [C:03+2] Handle multiple updates in case creation atomic section [extensions/CheckUser] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368674 (https://phabricator.wikimedia.org/T444117) (owner: 10STran) [14:32:51] (03CR) 10Urbanecm: [C:03+2] Handle multiple updates in case creation atomic section [extensions/CheckUser] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368675 (https://phabricator.wikimedia.org/T444117) (owner: 10STran) [14:33:08] (03Merged) 10jenkins-bot: Hooks: Do not disable exempt beta features when auto-enroll is on [extensions/BetaFeatures] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368670 (https://phabricator.wikimedia.org/T444361) (owner: 10Urbanecm) [14:34:34] (03CR) 10Urbanecm: [C:03+2] Growth: enable home beta on pilot and innovation wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) (owner: 10Michael Große) [14:36:06] (03Merged) 10jenkins-bot: Growth: enable home beta on pilot and innovation wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) (owner: 10Michael Große) [14:36:27] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for mazevedo - https://phabricator.wikimedia.org/T444497 (10Mazevedo) 03NEW [14:36:48] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for mazevedo - https://phabricator.wikimedia.org/T444497#12545162 (10Mazevedo) @Seddon need your approval! [14:37:01] hmm... [14:37:09] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for mazevedo - https://phabricator.wikimedia.org/T444497#12545165 (10Seddon) Approved [14:37:29] MichaelG_WMF: scap yells at me for trying to deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1361861 when the dependency is not going to be on all live branches [14:37:44] it is an _useful_ feature generally, but in this case it's working against us [14:38:12] and i don't want to backport https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1360890, because it has i18n changes [14:38:32] argh [14:38:41] ok, we can also do it tomorrow [14:38:41] MichaelG_WMF: unless you have any other thoughts, i'll momentarily revert the config patch to unblock the deployment, upload a new version w/o the depends-on and then deploy it after the backports [14:38:53] sounds also good [14:39:16] (03PS1) 10Urbanecm: Revert "Growth: enable home beta on pilot and innovation wikis" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368694 (https://phabricator.wikimedia.org/T443176) [14:39:19] (03CR) 10Gkyziridis: [C:03+2] ml-services: place llm-qwen3-14b on 96G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368651 (https://phabricator.wikimedia.org/T444469) (owner: 10Gkyziridis) [14:39:24] (03CR) 10Urbanecm: [C:03+2] Revert "Growth: enable home beta on pilot and innovation wikis" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368694 (https://phabricator.wikimedia.org/T443176) (owner: 10Urbanecm) [14:39:29] (03CR) 10CI reject: [V:04-1] ml-services: place llm-qwen3-14b on 96G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368651 (https://phabricator.wikimedia.org/T444469) (owner: 10Gkyziridis) [14:39:59] (03Merged) 10jenkins-bot: Handle multiple updates in case creation atomic section [extensions/CheckUser] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368674 (https://phabricator.wikimedia.org/T444117) (owner: 10STran) [14:40:00] (03Merged) 10jenkins-bot: Handle multiple updates in case creation atomic section [extensions/CheckUser] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1368675 (https://phabricator.wikimedia.org/T444117) (owner: 10STran) [14:40:59] (03Merged) 10jenkins-bot: Revert "Growth: enable home beta on pilot and innovation wikis" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368694 (https://phabricator.wikimedia.org/T443176) (owner: 10Urbanecm) [14:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:41:27] ok, starting a deployment for `1368670 1368675 1368674`, which is BetaFeatures patch and Tran's backports [14:41:52] scap now tells me about the merged patch and its revert, which is fine, and the diff should be empty. verifying. [14:42:18] it is indeed empty [14:42:19] proceeding [14:42:41] !log urbanecm@deploy2003 Started scap sync-world: Backport for [[gerrit:1368670|Hooks: Do not disable exempt beta features when auto-enroll is on (T444361)]] (1.48.0-wmf.4), Handle multiple updates in case creation atomic section (T444117) ([[gerrit:1368675|1.48.0-wmf.3]], [[gerrit:1368674|1.48.0-wmf.4]]) [14:42:45] preparing the revert of the revert [14:42:46] T444361: BetaFeatures: Features exempt from auto-enrollment cannot be toggled while "Automatically enable most beta features" is checked - https://phabricator.wikimedia.org/T444361 [14:42:46] T444117: Wikimedia\Rdbms\DBUnexpectedError: Cannot execute Wikimedia\Rdbms\Database::runOnTransactionIdleCallbacks critical section while session state is out of sync.A critical section from Wikimedia\Rdbms\Database::runOnTransactionI - https://phabricator.wikimedia.org/T444117 [14:43:09] (03PS2) 10Gkyziridis: ml-services: place llm-qwen3-14b on 96G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368651 (https://phabricator.wikimedia.org/T444469) [14:43:20] (03PS1) 10Urbanecm: Revert^2 "Growth: enable home beta on pilot and innovation wikis" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368696 (https://phabricator.wikimedia.org/T443176) [14:43:28] MichaelG_WMF: can i ask for a +1, please? [14:43:37] * MichaelG_WMF is looking [14:44:16] (03CR) 10Michael Große: [C:03+1] Revert^2 "Growth: enable home beta on pilot and innovation wikis" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368696 (https://phabricator.wikimedia.org/T443176) (owner: 10Urbanecm) [14:44:19] ty [14:44:45] (03CR) 10Gkyziridis: [C:03+2] ml-services: place llm-qwen3-14b on 96G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368651 (https://phabricator.wikimedia.org/T444469) (owner: 10Gkyziridis) [14:45:05] !log urbanecm@deploy2003 stran, urbanecm: Backport for [[gerrit:1368670|Hooks: Do not disable exempt beta features when auto-enroll is on (T444361)]] (1.48.0-wmf.4), Handle multiple updates in case creation atomic section (T444117) ([[gerrit:1368675|1.48.0-wmf.3]], [[gerrit:1368674|1.48.0-wmf.4]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:45:11] Tran: can you test? [14:45:19] 👀 [14:46:09] (03Merged) 10jenkins-bot: ml-services: place llm-qwen3-14b on 96G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368651 (https://phabricator.wikimedia.org/T444469) (owner: 10Gkyziridis) [14:46:18] my backport cannot be tested without the config, but the list of beta features is still there, and the //Automatically enable most beta features// checkbox is disabling all current checkboxes [14:46:34] cc MichaelG_WMF, not sure if you want to do any tests at this point (i assume not, but please confirm) [14:46:43] Well the page doesn't seem broken but my change needs logstash monitoring after the fact [14:46:51] good enough for me :) [14:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [14:47:48] MichaelG_WMF: waiting for your confirmation [14:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [14:51:17] !log urbanecm@deploy2003 stran, urbanecm: Continuing with deployment [14:51:22] proceeding [14:52:11] oh sorry, this notification got mixed up with something else [14:52:17] thank you for moving on! [14:52:51] (03CR) 10Urbanecm: [C:03+2] Revert^2 "Growth: enable home beta on pilot and innovation wikis" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368696 (https://phabricator.wikimedia.org/T443176) (owner: 10Urbanecm) [14:53:34] (03Merged) 10jenkins-bot: Revert^2 "Growth: enable home beta on pilot and innovation wikis" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368696 (https://phabricator.wikimedia.org/T443176) (owner: 10Urbanecm) [14:54:33] (03CR) 10Dzahn: [C:03+2] Revert "add amical.wm.o for Amical Wikimedia wiki" [dns] - 10https://gerrit.wikimedia.org/r/1368682 (https://phabricator.wikimedia.org/T444019) (owner: 10Dzahn) [14:55:12] !log dzahn@dns1004 START - running authdns-update [14:55:46] 06SRE, 10DNS, 06Traffic, 13Patch-For-Review: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019#12545248 (10Dzahn) [14:57:15] !log urbanecm@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368670|Hooks: Do not disable exempt beta features when auto-enroll is on (T444361)]] (1.48.0-wmf.4), Handle multiple updates in case creation atomic section (T444117) ([[gerrit:1368675|1.48.0-wmf.3]], [[gerrit:1368674|1.48.0-wmf.4]]) (duration: 14m 34s) [14:57:20] T444361: BetaFeatures: Features exempt from auto-enrollment cannot be toggled while "Automatically enable most beta features" is checked - https://phabricator.wikimedia.org/T444361 [14:57:20] T444117: Wikimedia\Rdbms\DBUnexpectedError: Cannot execute Wikimedia\Rdbms\Database::runOnTransactionIdleCallbacks critical section while session state is out of sync.A critical section from Wikimedia\Rdbms\Database::runOnTransactionI - https://phabricator.wikimedia.org/T444117 [14:57:31] !log dzahn@dns1004 END - running authdns-update [14:58:12] now scap doesn't complain [14:58:22] 🙌 [14:58:25] !log urbanecm@deploy2003 Started scap sync-world: Backport for [[gerrit:1368696|Revert^2 "Growth: enable home beta on pilot and innovation wikis" (T443176)]] [14:58:28] T443176: Beta Feature deployment of Home at pilot wikis - https://phabricator.wikimedia.org/T443176 [14:59:42] ty for backport 🙇 [15:00:05] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1500) [15:00:22] 06SRE, 06cloud-services-team, 06Infrastructure-Foundations, 10vm-requests: Site: 1 VM for codfw1dev bitu deployment - https://phabricator.wikimedia.org/T362128#12545285 (10Dzahn) Can we delete cloudtestidm.wikimedia.org from DNS ? --> T444019 [15:00:40] (03PS1) 10Santiago Faci: Add growthbook kubernetes service to spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1368697 (https://phabricator.wikimedia.org/T444502) [15:00:49] !log urbanecm@deploy2003 urbanecm: Backport for [[gerrit:1368696|Revert^2 "Growth: enable home beta on pilot and innovation wikis" (T443176)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:01:07] (03CR) 10Jforrester: [C:03+2] wikifunctions: Upgrade evaluators from 2026-11-04-142036 to 2026-11-04-233007 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368638 (owner: 10Jforrester) [15:01:50] James_F: note a scap is finishing, apologies for the overrun [15:01:59] !log gkyziridis@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [15:02:05] !log gkyziridis@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . [15:02:17] urbanecm: Don’t worry about it, we won’t clash I think. [15:02:31] (But I might be wrong! New tool makes for exciting times.) [15:02:31] i don't think so either, but just in case [15:02:41] MichaelG_WMF: can you test, please? [15:02:55] urbanecm: testing... [15:02:59] (03Merged) 10jenkins-bot: wikifunctions: Upgrade evaluators from 2026-11-04-142036 to 2026-11-04-233007 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368638 (owner: 10Jforrester) [15:04:32] !log jforrester@deploy2003 Started scap deploy-service wikifunctions: Weekly service deploy for evaluators (1368638) [15:04:32] !log jforrester@deploy2003 scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions [15:05:07] @urbanecm works as expected for me 👍 [15:05:12] perf [15:05:17] !log urbanecm@deploy2003 urbanecm: Continuing with deployment [15:05:21] proceeding [15:05:30] !log jforrester@deploy2003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions [15:05:40] urbanecm: For future knowledge, the deploy-service part of SpiderPig let me go ahead even with the MW deployment underway. Good news! [15:06:15] !log jforrester@deploy2003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions [15:07:37] !log jforrester@deploy2003 Finished scap deploy-service wikifunctions: Weekly service deploy for evaluators (1368638) (duration: 03m 05s) [15:07:43] (03CR) 10Jforrester: [C:03+2] wikifunctions: Upgrade orchestrator from 2026-11-04-133739 to 2026-11-10-004058 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368639 (https://phabricator.wikimedia.org/T440244) (owner: 10Jforrester) [15:10:04] (03Merged) 10jenkins-bot: wikifunctions: Upgrade orchestrator from 2026-11-04-133739 to 2026-11-10-004058 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368639 (https://phabricator.wikimedia.org/T440244) (owner: 10Jforrester) [15:11:14] !log jforrester@deploy2003 Started scap deploy-service wikifunctions: Weekly service deploy for orchestrator (1368639) [15:11:14] !log jforrester@deploy2003 scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions [15:11:16] !log urbanecm@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368696|Revert^2 "Growth: enable home beta on pilot and innovation wikis" (T443176)]] (duration: 12m 51s) [15:11:20] T443176: Beta Feature deployment of Home at pilot wikis - https://phabricator.wikimedia.org/T443176 [15:11:56] !log jforrester@deploy2003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions [15:12:07] MichaelG_WMF: and live [15:12:10] James_F: fully over to you [15:12:22] Thanks! [15:12:22] urbanecm: awesome, thanks! [15:12:37] (I’ll be done in ~5 mins if you want the window back for the i18n deploy?) [15:12:42] !log jforrester@deploy2003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions [15:12:51] James_F: nah, it'll rollout tomorrow, soon enough :) [15:13:01] Ack. [15:13:13] (Let’s not jinx the train. ;-)) [15:13:19] !log jforrester@deploy2003 Finished scap deploy-service wikifunctions: Weekly service deploy for orchestrator (1368639) (duration: 02m 05s) [15:13:44] (03PS3) 10Jforrester: wikifunctions: Drop the evaluators' "request complete" log line [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363475 [15:13:49] (03CR) 10Jforrester: [C:03+2] wikifunctions: Drop the evaluators' "request complete" log line [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363475 (owner: 10Jforrester) [15:13:51] jouncebot: nowandnext [15:13:51] For the next 0 hour(s) and 46 minute(s): Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1500) [15:13:51] In 0 hour(s) and 16 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1530) [15:15:23] FIRING: [11x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:15:29] (03Merged) 10jenkins-bot: wikifunctions: Drop the evaluators' "request complete" log line [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363475 (owner: 10Jforrester) [15:16:35] !log jforrester@deploy2003 Started scap deploy-service wikifunctions: Deploy evaluators config to silence logs from incoming calls (1363475) [15:16:35] !log jforrester@deploy2003 scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions [15:17:44] !log jforrester@deploy2003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions [15:18:20] !log jforrester@deploy2003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions [15:18:55] !log jforrester@deploy2003 Finished scap deploy-service wikifunctions: Deploy evaluators config to silence logs from incoming calls (1363475) (duration: 02m 20s) [15:19:03] All done at my end. [15:21:55] (03PS2) 10Urbanecm: cswiki: Remove celebration logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368266 (https://phabricator.wikimedia.org/T442387) [15:21:59] ok, then i'll ship ^^ [15:22:11] (03PS2) 10Urbanecm: GrowthExperiments: Remove the $wgGECampaignTopics setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367767 (https://phabricator.wikimedia.org/T444124) [15:22:40] (03CR) 10Urbanecm: [C:03+2] cswiki: Remove celebration logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368266 (https://phabricator.wikimedia.org/T442387) (owner: 10Urbanecm) [15:22:43] (03CR) 10Urbanecm: [C:03+2] GrowthExperiments: Remove the $wgGECampaignTopics setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367767 (https://phabricator.wikimedia.org/T444124) (owner: 10Urbanecm) [15:23:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368266 (https://phabricator.wikimedia.org/T442387) (owner: 10Urbanecm) [15:23:19] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367767 (https://phabricator.wikimedia.org/T444124) (owner: 10Urbanecm) [15:23:24] (03Merged) 10jenkins-bot: cswiki: Remove celebration logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368266 (https://phabricator.wikimedia.org/T442387) (owner: 10Urbanecm) [15:23:28] (03Merged) 10jenkins-bot: GrowthExperiments: Remove the $wgGECampaignTopics setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367767 (https://phabricator.wikimedia.org/T444124) (owner: 10Urbanecm) [15:23:52] !log urbanecm@deploy2003 Started scap sync-world: Backport for [[gerrit:1368266|cswiki: Remove celebration logo (T442387)]], [[gerrit:1367767|GrowthExperiments: Remove the $wgGECampaignTopics setting (T444124)]] [15:23:57] T442387: Add celebration logo for cswiki (600k articles) - https://phabricator.wikimedia.org/T442387 [15:23:57] T444124: GrowthExperiments: Remove the unused campaign topics feature - https://phabricator.wikimedia.org/T444124 [15:26:13] !log urbanecm@deploy2003 urbanecm: Backport for [[gerrit:1368266|cswiki: Remove celebration logo (T442387)]], [[gerrit:1367767|GrowthExperiments: Remove the $wgGECampaignTopics setting (T444124)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:26:30] (03PS1) 10Dreamy Jazz: hCaptcha: Enable on French Wikipedia for API editing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368699 (https://phabricator.wikimedia.org/T444509) [15:27:14] !log urbanecm@deploy2003 urbanecm: Continuing with deployment [15:30:04] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1500) [15:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1530) [15:32:31] jouncebot: nowandnext [15:32:31] For the next 0 hour(s) and 27 minute(s): Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1500) [15:32:32] For the next 0 hour(s) and 27 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1530) [15:32:32] In 2 hour(s) and 27 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1800) [15:32:52] I'd like to deploy after you urbanecm, if you could ping when done :D [15:33:17] !log urbanecm@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368266|cswiki: Remove celebration logo (T442387)]], [[gerrit:1367767|GrowthExperiments: Remove the $wgGECampaignTopics setting (T444124)]] (duration: 09m 26s) [15:33:22] T442387: Add celebration logo for cswiki (600k articles) - https://phabricator.wikimedia.org/T442387 [15:33:22] T444124: GrowthExperiments: Remove the unused campaign topics feature - https://phabricator.wikimedia.org/T444124 [15:33:47] Aha, it seems you are done? :D [15:38:28] Dreamy_Jazz: yep yep [15:38:30] go on :) [15:45:09] Turns out we are delaying that deploy to tommorrow, so won't deploy right now [15:45:13] Thanks for the ping tho [16:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:24:17] (03CR) 10Kosta Harlan: [C:03+1] hCaptcha: Enable on French Wikipedia for API editing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368699 (https://phabricator.wikimedia.org/T444509) (owner: 10Dreamy Jazz) [16:24:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 12 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1368699 (https://phabricator.wikimedia.org/T444509) (owner: 10Dreamy Jazz) [16:33:41] 06SRE, 10SRE-Access-Requests: Requesting access to production for FSesini-WMF - https://phabricator.wikimedia.org/T444520 (10FSesini-WMF) 03NEW [16:34:32] !log bking@deploy2003 `apt-get install python3-swiftclient` T434970 [16:34:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:34:36] T434970: RdfStreamingUpdaterSpaceUsageTooHigh - https://phabricator.wikimedia.org/T434970 [16:36:44] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 12 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367855 (owner: 10Sergio Gimeno) [16:37:25] (03CR) 10Sergio Gimeno: "Scheduled" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367855 (owner: 10Sergio Gimeno) [16:41:05] (03PS3) 10Sergio Gimeno: GrowthExperiments: Scale early onboarding on ar/en/es/frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367855 [16:42:17] (03PS1) 10Snwachukwu: Rewrite edit per editor v3 gateway requests to edit per editor v4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368714 [16:46:27] 06SRE, 10SRE-Access-Requests: Requesting access to production for FSesini-WMF - https://phabricator.wikimedia.org/T444520#12545869 (10FSesini-WMF) [17:00:47] (03PS1) 10Santiago Faci: Enable validation for deployment window on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368717 [17:01:16] (03PS2) 10Santiago Faci: Enable deployment windows validation on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368717 [17:04:51] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for FSesini-WMF - https://phabricator.wikimedia.org/T444520#12545944 (10Dzahn) [17:05:12] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for FSesini-WMF - https://phabricator.wikimedia.org/T444520#12545946 (10Dzahn) @thcipriani Hi, a deployment access request for you. [17:06:32] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for mazevedo - https://phabricator.wikimedia.org/T444497#12545948 (10Dzahn) @thcipriani Hi, another deployment access request for you. [17:08:46] (03PS1) 10Snwachukwu: Roll out edit v3-to-v4 per_editor gateway rewrite to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368720 (https://phabricator.wikimedia.org/T434136) [17:09:29] 06SRE, 10LDAP-Access-Requests: Grant Access to nda LDAP group for Monrac5 - https://phabricator.wikimedia.org/T443971#12545957 (10Dzahn) @KFrancis I can't see a user called Monrac5 on the NDA+MOU spreadsheet. Is an NDA on file? [17:10:09] 06SRE, 10LDAP-Access-Requests: Grant Access to nda LDAP group for Monrac5 - https://phabricator.wikimedia.org/T443971#12545962 (10Dzahn) I found the real name from T430304 [17:11:33] 06SRE, 10LDAP-Access-Requests: Grant Access to nda LDAP group for Monrac5 - https://phabricator.wikimedia.org/T443971#12545965 (10Dzahn) It says the NDA is for specific access for "any data from the WMDE LDAP Group/analytics-privatedata". Does this cover this request? [17:12:59] (03PS2) 10Snwachukwu: Roll out edit v3-to-v4 per_editor gateway rewrite to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368720 (https://phabricator.wikimedia.org/T434136) [17:16:24] 06SRE, 10LDAP-Access-Requests: Grant Access to nda LDAP group for Monrac5 - https://phabricator.wikimedia.org/T443971#12545973 (10Dzahn) Based on T430304#12072406 I added @Monrac5 to the WMF-NDA Phabricator group. [17:36:57] PROBLEM - Blazegraph Port for wdqs-blazegraph on wdqs2008 is CRITICAL: connect to address 127.0.0.1 and port 9999: Connection refused https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [17:37:57] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs2008 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [17:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.45% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [18:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1800) [18:00:44] 06SRE, 10Wikimedia-Mailing-lists: New WikiLatinos Mailing List - https://phabricator.wikimedia.org/T437742#12546083 (10Oscar) Okay, but that doesn’t leave us many options. As organizers, we can either go without a mailing list or run one that isn’t hosted here, since we’re stuck at the recognition step. That m... [18:10:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:15:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:21:09] (03CR) 10Bking: [C:03+2] Roll out edit v3-to-v4 per_editor gateway rewrite to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368720 (https://phabricator.wikimedia.org/T434136) (owner: 10Snwachukwu) [18:31:35] (03CR) 10Phuedx: [C:03+1] Enable deployment windows validation on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368717 (owner: 10Santiago Faci) [18:36:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:41:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [18:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [19:00:05] andre and brennen: Time to snap out of that daydream and deploy MediaWiki train - Utc-0+Utc-7 Version (secondary timeslot). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T1900). [19:00:15] jouncebot: still nah [19:07:19] (03PS1) 10Bking: deployment-server: Install Swift client libraries [puppet] - 10https://gerrit.wikimedia.org/r/1368736 (https://phabricator.wikimedia.org/T434970) [19:10:35] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [19:10:39] (03CR) 10Bking: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1368736 (https://phabricator.wikimedia.org/T434970) (owner: 10Bking) [19:12:57] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [19:13:12] 06SRE, 10LDAP-Access-Requests: Grant Access to nda LDAP group for Monrac5 - https://phabricator.wikimedia.org/T443971#12546206 (10Aklapper) 05Open→03Resolved a:03Dzahn [19:14:07] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [19:14:41] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [19:15:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:16:21] (03CR) 10Santiago Faci: [C:03+2] Enable deployment windows validation on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368717 (owner: 10Santiago Faci) [19:18:13] (03Merged) 10jenkins-bot: Enable deployment windows validation on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368717 (owner: 10Santiago Faci) [19:20:24] !log sfaci@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply [19:21:01] !log sfaci@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply [19:26:56] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [19:27:55] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [19:27:56] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [19:27:59] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [19:28:05] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [19:28:15] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [19:28:17] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [19:28:59] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [19:29:00] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [19:30:00] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [19:30:01] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply [19:30:42] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply [19:30:44] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply [19:31:46] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply [19:31:48] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply [19:32:54] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply [19:32:55] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [19:33:53] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [19:33:54] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply [19:35:07] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply [19:35:08] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [19:36:07] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [19:36:09] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [19:36:22] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [19:36:23] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [19:37:26] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [19:37:27] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply [19:38:29] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply [19:38:30] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply [19:39:54] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply [19:39:55] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply [19:40:05] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply [19:40:06] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply [19:41:24] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply [19:41:25] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply [19:42:19] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply [19:42:20] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply [19:42:21] 06SRE, 10LDAP-Access-Requests: Grant Access to nda LDAP group for Monrac5 - https://phabricator.wikimedia.org/T443971#12546298 (10Aklapper) 05Resolved→03Open a:05Dzahn→03None Oops, closed the wrong ticket, sorry [19:43:23] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply [19:43:24] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply [19:44:38] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply [19:44:39] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [19:44:46] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [19:44:47] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply [19:45:33] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply [19:45:34] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply [19:46:25] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply [19:46:26] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply [19:46:41] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply [19:47:18] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/echoserver: apply [19:47:28] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/echoserver: apply [20:04:28] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:10:39] PROBLEM - MariaDB Replica Lag: s1 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 631.40 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:18:26] 06SRE, 10LDAP-Access-Requests: Grant Access to nda LDAP group for Monrac5 - https://phabricator.wikimedia.org/T443971#12546429 (10Monrac5) thank you! [20:21:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [20:26:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [20:40:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [20:45:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [20:58:39] RECOVERY - MariaDB Replica Lag: s1 on db2250 is OK: OK slave_sql_lag Replication lag: 0.22 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:00:05] RoanKattouw, urbanecm, TheresNoTime, and cjming: Time to snap out of that daydream and deploy UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T2100). [21:00:05] matthiasmullie: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [21:00:07] o/ [21:02:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy2003 using scap backport" [extensions/WikimediaEvents] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368631 (https://phabricator.wikimedia.org/T441426) (owner: 10Matthias Mullie) [21:02:19] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy2003 using scap backport" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368166 (owner: 10Matthias Mullie) [21:05:39] (03Merged) 10jenkins-bot: Do not Implement UserThankedListener [extensions/WikimediaEvents] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368631 (https://phabricator.wikimedia.org/T441426) (owner: 10Matthias Mullie) [21:05:39] (03Merged) 10jenkins-bot: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.4) - 10https://gerrit.wikimedia.org/r/1368166 (owner: 10Matthias Mullie) [21:06:13] !log mlitn@deploy2003 Started scap sync-world: Backport for [[gerrit:1368631|Do not Implement UserThankedListener]] (1.48.0-wmf.4), [[gerrit:1368166|Squashed diff to master]] (1.48.0-wmf.4) [21:08:33] !log mlitn@deploy2003 mlitn: Backport for [[gerrit:1368631|Do not Implement UserThankedListener]] (1.48.0-wmf.4), [[gerrit:1368166|Squashed diff to master]] (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:09:19] !log mlitn@deploy2003 mlitn: Continuing with deployment [21:15:28] !log mlitn@deploy2003 Finished scap sync-world: Backport for [[gerrit:1368631|Do not Implement UserThankedListener]] (1.48.0-wmf.4), [[gerrit:1368166|Squashed diff to master]] (1.48.0-wmf.4) (duration: 09m 15s) [21:16:31] !log UTC late backport window done [21:16:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:33:31] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 610.47 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:38:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:43:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:52:06] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.45% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [22:00:04] Deploy window Wikifunctions Services UTC Late (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T2200) [22:00:39] PROBLEM - MariaDB Replica Lag: x4 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 617.46 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:36:31] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.28 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:41:08] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [22:47:07] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [22:49:10] FIRING: [2x] GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [22:50:39] RECOVERY - MariaDB Replica Lag: x4 on db2250 is OK: OK slave_sql_lag Replication lag: 0.46 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:00:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.958% from codfw to magru - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=magru - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [23:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261111T2300) [23:04:11] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2170.codfw.wmnet, wikikube-worker2262.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-work [23:04:11] odfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2343.codfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2071.codfw.wmnet, wikikube-worker2059.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2139.codfw.wmnet, w [23:04:11] worker2213.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2289.codfw.wmnet, wikikube-worker2314.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2125.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [23:05:11] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:11:31] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 658.26 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:15:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:20:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.333% from codfw to magru - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=magru - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [23:23:31] (03CR) 10BCornwall: "Please also note that we already have redirection implemented via Porkbun: See the intro at https://wikitech.wikimedia.org/wiki/Wikimedias" [puppet] - 10https://gerrit.wikimedia.org/r/1242499 (https://phabricator.wikimedia.org/T419887) (owner: 10Cwhite) [23:27:38] (03CR) 10BCornwall: "Oh my, I just noticed the age of this. Ahem. Thanks, disregard." [puppet] - 10https://gerrit.wikimedia.org/r/1242499 (https://phabricator.wikimedia.org/T419887) (owner: 10Cwhite) [23:41:37] !log brett@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp3074.esams.wmnet with reason: troubleshooting cmos failure [23:55:19] (03PS1) 10Ryan Kemper: airflow3: fix slow graph view for dumps DAGs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1368786 (https://phabricator.wikimedia.org/T443252) [23:58:31] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.44 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response