[01:09:49] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [01:10:11] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Enable L3 routing on cloudsw nodes - https://phabricator.wikimedia.org/T265288#12198776 (10Adminbdso) [01:10:42] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Enable L3 routing on cloudsw nodes - https://phabricator.wikimedia.org/T265288#12198777 (10Adminbdso) [01:12:25] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1323833 [01:12:25] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1323833 (owner: 10TrainBranchBot) [01:22:11] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Enable L3 routing on cloudsw nodes - https://phabricator.wikimedia.org/T265288#12198793 (10Novem_Linguae) [01:22:40] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1323833 (owner: 10TrainBranchBot) [01:27:11] FIRING: Temperature: Temp issue on wdqs1023:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=wdqs1023 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [01:32:11] RESOLVED: Temperature: Temp issue on wdqs1023:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=wdqs1023 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [01:32:54] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:37:54] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:45:13] FIRING: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [02:00:15] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:07:03] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 48s) [02:07:54] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:10:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [02:14:44] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:34:24] (03PS4) 10BryanDavis: [WIP] httpbb: Add test suite for testwiki [puppet] - 10https://gerrit.wikimedia.org/r/1323829 (https://phabricator.wikimedia.org/T428972) [03:34:58] FIRING: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [03:39:58] FIRING: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:09:44] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [06:10:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:14:44] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:34:44] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:37:54] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:49:01] 06SRE, 06Infrastructure-Foundations, 10WMDE Analytics, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12198944 (10AndrewTavis_WMDE) Thank you all so much! I've updated our docs for data access onboar... [07:00:05] Amir1, urbanecm, and awight: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for UTC morning backport window . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260810T0700). [07:00:05] No Gerrit patches in the queue for this window AFAICS. [07:35:27] <_joe_> !log restarting squid on urldownloader1006 [07:35:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:40:13] FIRING: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:57:12] !log hashar@deploy1003 Started deploy [integration/docroot@7772132]: update build dependencies [07:57:26] !log hashar@deploy1003 Finished deploy [integration/docroot@7772132]: update build dependencies (duration: 00m 13s) [08:17:20] (03CR) 10Klausman: [C:03+1] Update to 1.29.4 [debs/istio] - 10https://gerrit.wikimedia.org/r/1321949 (https://phabricator.wikimedia.org/T434198) (owner: 10JMeybohm) [08:39:44] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [09:09:44] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [09:17:54] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260810T1000) [10:10:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:12:26] (03PS1) 10WMDE-Fisch: Enable sub-references on more group2 wikis (batch3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323939 (https://phabricator.wikimedia.org/T430941) [10:13:56] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, August 10 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323939 (https://phabricator.wikimedia.org/T430941) (owner: 10WMDE-Fisch) [10:18:48] (03PS2) 10WMDE-Fisch: Enable sub-references on more group2 wikis (batch3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323939 (https://phabricator.wikimedia.org/T432731) [10:21:31] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1323943 [10:22:12] (03CR) 10Johannes Richter WMDE: [C:03+1] Enable sub-references on more group2 wikis (batch3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323939 (https://phabricator.wikimedia.org/T432731) (owner: 10WMDE-Fisch) [10:39:44] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:13:49] (03CR) 10Mareike Heuer: [C:03+1] Enable sub-references on more group2 wikis (batch3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323939 (https://phabricator.wikimedia.org/T432731) (owner: 10WMDE-Fisch) [11:37:54] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:39:44] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:40:13] FIRING: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:28:02] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1323955 (owner: 10L10n-bot) [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: OwO what's this, a deployment window?? UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260810T1300). nyaa~ [13:00:05] Superpes and WMDE-Fisch: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:25] \o [13:01:06] WMDE-Fisch: wanna start with yours? :) [13:01:19] Sure! [13:01:44] So I'm deploying that now. [13:01:48] urbanecm: [13:02:06] ack [13:02:27] (03CR) 10TrainBranchBot: [C:03+2] "Approved by wmde-fisch@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323939 (https://phabricator.wikimedia.org/T432731) (owner: 10WMDE-Fisch) [13:03:36] (03Merged) 10jenkins-bot: Enable sub-references on more group2 wikis (batch3) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1323939 (https://phabricator.wikimedia.org/T432731) (owner: 10WMDE-Fisch) [13:05:21] !log wmde-fisch@deploy1003 Started scap sync-world: Backport for [[gerrit:1323939|Enable sub-references on more group2 wikis (batch3) (T432731)]] [13:05:26] T432731: Global rollout - Sub-ref deployments to group 2 wikis (batch 3) - https://phabricator.wikimedia.org/T432731 [13:12:54] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [13:13:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1147:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1147 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [13:18:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1147:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1147 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [13:22:04] !log wmde-fisch@deploy1003 wmde-fisch: Backport for [[gerrit:1323939|Enable sub-references on more group2 wikis (batch3) (T432731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:22:09] T432731: Global rollout - Sub-ref deployments to group 2 wikis (batch 3) - https://phabricator.wikimedia.org/T432731 [13:22:55] Checking [13:25:37] !log wmde-fisch@deploy1003 wmde-fisch: Continuing with deployment [13:31:27] (03PS1) 10DDesouza: miscweb(research-landing-page): bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1323965 (https://phabricator.wikimedia.org/T219903) [13:35:49] (03CR) 10DDesouza: [C:03+2] miscweb(research-landing-page): bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1323965 (https://phabricator.wikimedia.org/T219903) (owner: 10DDesouza) [13:37:54] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:38:40] (03Merged) 10jenkins-bot: miscweb(research-landing-page): bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1323965 (https://phabricator.wikimedia.org/T219903) (owner: 10DDesouza) [13:38:42] !log wmde-fisch@deploy1003 Finished scap sync-world: Backport for [[gerrit:1323939|Enable sub-references on more group2 wikis (batch3) (T432731)]] (duration: 33m 21s) [13:38:47] T432731: Global rollout - Sub-ref deployments to group 2 wikis (batch 3) - https://phabricator.wikimedia.org/T432731 [13:39:26] urbanecm: I'm done [13:39:33] sounds great! [13:39:35] Superpes: are you around? [13:39:44] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:40:43] (03PS1) 10Urbanecm: updateIsActiveFlagForMentees: Commit the final partial batch [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323967 (https://phabricator.wikimedia.org/T432959) [13:40:52] (03CR) 10Urbanecm: [C:03+2] updateIsActiveFlagForMentees: Commit the final partial batch [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323967 (https://phabricator.wikimedia.org/T432959) (owner: 10Urbanecm) [13:41:17] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323967 (https://phabricator.wikimedia.org/T432959) (owner: 10Urbanecm) [13:44:50] !log dani@deploy1003 helmfile [staging] START helmfile.d/services/miscweb: apply [13:45:05] !log dani@deploy1003 helmfile [staging] DONE helmfile.d/services/miscweb: apply [13:45:07] !log dani@deploy1003 helmfile [eqiad] START helmfile.d/services/miscweb: apply [13:45:24] !log dani@deploy1003 helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [13:45:26] !log dani@deploy1003 helmfile [codfw] START helmfile.d/services/miscweb: apply [13:45:47] !log dani@deploy1003 helmfile [codfw] DONE helmfile.d/services/miscweb: apply [13:48:44] (03CR) 10CI reject: [V:04-1] updateIsActiveFlagForMentees: Commit the final partial batch [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323967 (https://phabricator.wikimedia.org/T432959) (owner: 10Urbanecm) [13:49:50] (03PS1) 10Urbanecm: chore: update mediawiki/mediawiki-codesniffer to fix CI [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323971 (https://phabricator.wikimedia.org/T434187) [13:50:01] (03PS2) 10Urbanecm: updateIsActiveFlagForMentees: Commit the final partial batch [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323967 (https://phabricator.wikimedia.org/T432959) [13:50:09] (03CR) 10Urbanecm: [C:03+2] chore: update mediawiki/mediawiki-codesniffer to fix CI [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323971 (https://phabricator.wikimedia.org/T434187) (owner: 10Urbanecm) [13:50:12] (03CR) 10Urbanecm: updateIsActiveFlagForMentees: Commit the final partial batch [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323967 (https://phabricator.wikimedia.org/T432959) (owner: 10Urbanecm) [13:50:15] (03CR) 10Urbanecm: [C:03+2] updateIsActiveFlagForMentees: Commit the final partial batch [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323967 (https://phabricator.wikimedia.org/T432959) (owner: 10Urbanecm) [13:50:34] (03CR) 10TrainBranchBot: "Approved by urbanecm@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323967 (https://phabricator.wikimedia.org/T432959) (owner: 10Urbanecm) [13:50:35] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323971 (https://phabricator.wikimedia.org/T434187) (owner: 10Urbanecm) [13:55:28] (03CR) 10CI reject: [V:04-1] chore: update mediawiki/mediawiki-codesniffer to fix CI [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323971 (https://phabricator.wikimedia.org/T434187) (owner: 10Urbanecm) [13:55:33] (03CR) 10CI reject: [V:04-1] updateIsActiveFlagForMentees: Commit the final partial batch [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323967 (https://phabricator.wikimedia.org/T432959) (owner: 10Urbanecm) [13:56:14] (03PS3) 10Urbanecm: updateIsActiveFlagForMentees: Commit the final partial batch [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323967 (https://phabricator.wikimedia.org/T432959) [13:56:32] (03CR) 10Urbanecm: [V:03+2] "doesn't pass only due to the codesniffer issues" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323967 (https://phabricator.wikimedia.org/T432959) (owner: 10Urbanecm) [13:56:58] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1323967|updateIsActiveFlagForMentees: Commit the final partial batch (T432959)]] [13:57:02] T432959: updateisactiveflagformentees failed with a transaction error - https://phabricator.wikimedia.org/T432959 [14:07:30] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1323967|updateIsActiveFlagForMentees: Commit the final partial batch (T432959)]] (duration: 10m 33s) [14:07:35] T432959: updateisactiveflagformentees failed with a transaction error - https://phabricator.wikimedia.org/T432959 [14:10:24] (03PS1) 10Krinkle: mmv.bootstrap: Fix getUrlParam to account for TIFF lossy/lossless param [extensions/MultimediaViewer] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323973 (https://phabricator.wikimedia.org/T434333) [14:10:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:11:11] urbanecm: done with deployment? [14:11:22] Krinkle: yep, go ahead [14:11:43] (03CR) 10CI reject: [V:04-1] mmv.bootstrap: Fix getUrlParam to account for TIFF lossy/lossless param [extensions/MultimediaViewer] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323973 (https://phabricator.wikimedia.org/T434333) (owner: 10Krinkle) [14:13:46] (03PS1) 10Krinkle: build: Bump mediawiki/mediawiki-codesniffer to v52.0.0 for security warning [core] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323974 (https://phabricator.wikimedia.org/T434187) [14:14:22] urbanecm: fyi ^ [14:14:28] that might be a more sensible thing than what i did [14:14:50] I suspect maybe both are needed, not sure [14:14:55] it seems yours failed [14:15:08] +maybe vendor...? [14:15:16] dev packages aren't in vendor [14:15:19] they're loaded on top of vendor [14:15:19] right [14:15:31] so the quibble job on an extension repo doesn't run phpcs [14:15:39] but it does run composer install in core to install phpunit etc [14:15:47] and so is affected by phpcs installation, even though it isn't used [14:15:52] so yeah, core backport is required for it to pass [14:16:02] I suspect the composer-test job though indeed requires your extension repo backport as well [14:16:19] I'm guessing you initially saw 2 jobs fail, not just quibble? [14:16:45] yeah, but both failed on the same issue [14:17:03] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MultimediaViewer/+/1323973 [14:17:08] quibble phpunit fails because of core composer [14:17:14] then phan fails because of extension composer [14:17:33] bah, that's gonna be a lot of repos [14:17:42] I guess we'll just hope there arent too many wmf.14 backports needed [14:17:47] and focus on wmf.15 [14:18:50] yep :-/ [14:28:52] (03CR) 10Krinkle: [C:03+2] build: Bump mediawiki/mediawiki-codesniffer to v52.0.0 for security warning [core] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323974 (https://phabricator.wikimedia.org/T434187) (owner: 10Krinkle) [14:29:11] (03PS3) 10Krinkle: build: Updating dependencies [extensions/MultimediaViewer] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323975 (https://phabricator.wikimedia.org/T434187) [14:29:31] (03PS2) 10Krinkle: mmv.bootstrap: Fix getUrlParam to account for TIFF lossy/lossless param [extensions/MultimediaViewer] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323973 (https://phabricator.wikimedia.org/T434333) [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260810T1430) [14:33:49] (03Merged) 10jenkins-bot: build: Bump mediawiki/mediawiki-codesniffer to v52.0.0 for security warning [core] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323974 (https://phabricator.wikimedia.org/T434187) (owner: 10Krinkle) [14:33:53] (03Merged) 10jenkins-bot: build: Updating dependencies [extensions/MultimediaViewer] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323975 (https://phabricator.wikimedia.org/T434187) (owner: 10Krinkle) [14:38:16] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323973 (https://phabricator.wikimedia.org/T434333) (owner: 10Krinkle) [14:41:08] (03Merged) 10jenkins-bot: mmv.bootstrap: Fix getUrlParam to account for TIFF lossy/lossless param [extensions/MultimediaViewer] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1323973 (https://phabricator.wikimedia.org/T434333) (owner: 10Krinkle) [14:43:28] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1323973|mmv.bootstrap: Fix getUrlParam to account for TIFF lossy/lossless param (T434333)]] [14:43:33] T434333: Single page tiff file reports urlparamnormal error when attempting to load in mediaviewer - https://phabricator.wikimedia.org/T434333 [14:45:14] !log krinkle@deploy1003 krinkle: Backport for [[gerrit:1323973|mmv.bootstrap: Fix getUrlParam to account for TIFF lossy/lossless param (T434333)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:50:49] !log krinkle@deploy1003 krinkle: Continuing with deployment [14:54:52] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1323973|mmv.bootstrap: Fix getUrlParam to account for TIFF lossy/lossless param (T434333)]] (duration: 11m 24s) [14:54:57] T434333: Single page tiff file reports urlparamnormal error when attempting to load in mediaviewer - https://phabricator.wikimedia.org/T434333 [15:30:05] jan_drewniak: #bothumor My software never has bugs. It just develops random features. Rise for Wikimedia Portals Update. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260810T1530). [15:39:44] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:40:13] FIRING: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:42:54] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:07:54] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:14:44] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:32:16] 06SRE, 10Pywikibot, 06Traffic, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#12199850 (10Ladsgroup) Maxlag is way too high: https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts?orgId=1&from=2026-08-10T05:20:04.599Z&to=2026-08-10T16:30:51.031... [17:00:04] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260810T1700) [17:00:05] ryankemper: How many deployers does it take to do Wikidata Query Service weekly deploy deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260810T1700). [17:14:49] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [18:10:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [18:54:08] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 100% [18:54:34] RECOVERY - Host titan1002 is UP: PING WARNING - Packet loss = 0%, RTA = 1528.23 ms [18:55:50] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 50%, RTA = 5501.02 ms [18:56:00] RECOVERY - Host titan1002 is UP: PING WARNING - Packet loss = 0%, RTA = 1512.58 ms [18:56:03] FIRING: ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:01:03] RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:40:13] FIRING: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: I, the Bot under the Fountain, call upon thee, The Deployer, to do UTC late backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260810T2000). [20:00:05] Superpes: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:17] Hi :) [20:04:59] Is there any deployer available? [20:17:54] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:00:04] alexsanford, Reedy, sbassett, Maryum, and manfredi: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Weekly Security deployment window . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260810T2100). [21:14:49] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [22:10:38] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [22:45:12] (03PS1) 10Effie Mouzeli: thumbor: prevent pod creation in specific rows [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324025 [22:45:26] (03PS2) 10Effie Mouzeli: thumbor: prevent pod creation in specific rows [deployment-charts] - 10https://gerrit.wikimedia.org/r/1324025 [22:53:06] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [23:30:05] (03Abandoned) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1323830 (owner: 10TrainBranchBot) [23:40:13] FIRING: [4x] CertAlmostExpired: Certificate for service gerrit-https:443 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:42:16] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1324027 [23:42:16] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1324027 (owner: 10TrainBranchBot) [23:52:33] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1324027 (owner: 10TrainBranchBot) [23:53:06] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state