[00:19:10] FIRING: [3x] SystemdUnitFailed: prometheus-wmf-elasticsearch-exporter-9600.service on cloudelastic1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:26:25] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:33:18] 10SRE-swift-storage, 06Data-Persistence, 10MediaViewer, 10Thumbor, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11848087 (10Aklapper) [00:48:14] 06SRE, 06Traffic, 07affects-Kiwix-and-openZIM, 10Maps (Kartographer): Multiple wikipedia wikis have broken maps URLs in the infobox - https://phabricator.wikimedia.org/T424046#11848107 (10ssingh) T423815 also reports the same issue and I can confirm the above error message (when you set the referrer): ` "... [00:48:25] 06SRE, 10Maps, 06Traffic, 07affects-Kiwix-and-openZIM: Multiple wikipedia wikis have broken maps URLs in the infobox - https://phabricator.wikimedia.org/T424046#11848111 (10ssingh) [00:54:54] jhancock@cumin2002 provision (PID 1808505) is awaiting input [00:55:51] !log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host phab2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [01:04:56] jhancock@cumin2002 provision (PID 1808505) is awaiting input [01:09:50] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1276094 [01:09:50] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1276094 (owner: 10TrainBranchBot) [01:20:07] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1276094 (owner: 10TrainBranchBot) [01:23:23] FIRING: [10x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:24:38] !log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host phab2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [01:37:47] !log jhancock@cumin2002 START - Cookbook sre.hosts.reimage for host phab2003.codfw.wmnet with OS trixie [01:38:00] 10ops-codfw, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Q3:rack/setup/install phab2003 - https://phabricator.wikimedia.org/T418899#11848217 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host phab2003.codfw.wmnet with OS trixie [01:51:30] !log jhancock@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on phab2003.codfw.wmnet with reason: host reimage [01:59:12] !log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab2003.codfw.wmnet with reason: host reimage [02:00:58] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:07:04] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 06s) [02:09:18] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:22] !log jhancock@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [02:17:02] !log jhancock@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [02:17:04] !log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host phab2003.codfw.wmnet with OS trixie [02:17:15] 10ops-codfw, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Q3:rack/setup/install phab2003 - https://phabricator.wikimedia.org/T418899#11848283 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host phab2003.codfw.wmnet with OS trixie completed:... [02:23:24] 10ops-codfw, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Q3:rack/setup/install phab2003 - https://phabricator.wikimedia.org/T418899#11848284 (10Jhancock.wm) 05Open→03Resolved [02:24:11] 10ops-codfw, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Q3:rack/setup/install phab2003 - https://phabricator.wikimedia.org/T418899#11848287 (10Jhancock.wm) @Dzahn finished up and ready to go [02:32:11] !log ladsgroup@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2161 (T410589)', diff saved to https://phabricator.wikimedia.org/P91306 and previous config saved to /var/cache/conftool/dbconfig/20260422-023211-ladsgroup.json [02:32:15] T410589: Optimize all core tables, late 2025 - https://phabricator.wikimedia.org/T410589 [02:34:18] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:42:19] !log ladsgroup@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P91307 and previous config saved to /var/cache/conftool/dbconfig/20260422-024219-ladsgroup.json [02:52:28] !log ladsgroup@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P91308 and previous config saved to /var/cache/conftool/dbconfig/20260422-025227-ladsgroup.json [03:02:36] !log ladsgroup@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2161 (T410589)', diff saved to https://phabricator.wikimedia.org/P91309 and previous config saved to /var/cache/conftool/dbconfig/20260422-030235-ladsgroup.json [03:02:40] T410589: Optimize all core tables, late 2025 - https://phabricator.wikimedia.org/T410589 [03:02:52] !log ladsgroup@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance [03:03:01] !log ladsgroup@cumin1003 dbctl commit (dc=all): 'Depooling db2163 (T410589)', diff saved to https://phabricator.wikimedia.org/P91310 and previous config saved to /var/cache/conftool/dbconfig/20260422-030300-ladsgroup.json [03:34:02] FIRING: HelmReleaseBadStatus: Helm release mw-script/nngkzgw8 on k8s@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s&var-namespace=mw-script - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [04:19:10] FIRING: [3x] SystemdUnitFailed: prometheus-wmf-elasticsearch-exporter-9600.service on cloudelastic1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:19:12] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [04:35:41] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [05:23:23] FIRING: [10x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [05:25:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:49:08] (03PS1) 10WAN233: change logo at zh-classical wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276281 [05:51:06] (03PS2) 10WAN233: change logo at zh-classical wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276281 [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260422T0600) [06:04:51] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave) {#11372}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [06:08:28] (03PS1) 10WAN233: Bug: T424128 Change-Id: Ie7d4580aaedae1fe30f3ad169305f50623bfc8fc [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276284 (https://phabricator.wikimedia.org/T424128) [06:14:13] (03PS2) 10WAN233: change logo at zh-classical wikipedia Bug: T424128 Change-Id: Ie7d4580aaedae1fe30f3ad169305f50623bfc8fc [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276284 (https://phabricator.wikimedia.org/T424128) [06:14:32] (03PS3) 10WAN233: change logo at zh-classical wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276284 (https://phabricator.wikimedia.org/T424128) [06:14:48] (03Abandoned) 10WAN233: change logo at zh-classical wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276281 (owner: 10WAN233) [06:24:51] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave) {#11372}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [06:35:21] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave) {#11372}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [07:00:05] Amir1, Urbanecm, and awight: How many deployers does it take to do UTC morning backport window deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260422T0700). [07:00:05] samwilson and Yahya: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:01:10] ok [07:01:20] I'm here. [07:02:08] hello! [07:04:52] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samwilson@deploy1003 using scap backport" [extensions/ProofreadPage] (wmf/1.46.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1275808 (https://phabricator.wikimedia.org/T423548) (owner: 10Samwilson) [07:13:54] (03Merged) 10jenkins-bot: Use canvas rather than webgl for OpenSeadragon [extensions/ProofreadPage] (wmf/1.46.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1275808 (https://phabricator.wikimedia.org/T423548) (owner: 10Samwilson) [07:14:35] !log samwilson@deploy1003 Started scap sync-world: Backport for [[gerrit:1275808|Use canvas rather than webgl for OpenSeadragon (T423548)]] [07:14:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr1-magru and Telxius (2001:1498:1:966:1::251) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [07:14:39] T423548: Page images disappearing on edit - https://phabricator.wikimedia.org/T423548 [07:16:17] !log samwilson@deploy1003 samwilson: Backport for [[gerrit:1275808|Use canvas rather than webgl for OpenSeadragon (T423548)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:17:15] !log samwilson@deploy1003 samwilson: Continuing with deployment [07:23:06] !log samwilson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1275808|Use canvas rather than webgl for OpenSeadragon (T423548)]] (duration: 08m 31s) [07:23:10] T423548: Page images disappearing on edit - https://phabricator.wikimedia.org/T423548 [07:27:05] (03CR) 10A smart kitten: "recheck [trigger CI]" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276284 (https://phabricator.wikimedia.org/T424128) (owner: 10WAN233) [07:27:56] (03CR) 10CI reject: [V:04-1] change logo at zh-classical wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276284 (https://phabricator.wikimedia.org/T424128) (owner: 10WAN233) [07:31:51] (03CR) 10A smart kitten: change logo at zh-classical wikipedia (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276284 (https://phabricator.wikimedia.org/T424128) (owner: 10WAN233) [07:34:02] FIRING: HelmReleaseBadStatus: Helm release mw-script/nngkzgw8 on k8s@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s&var-namespace=mw-script - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [08:15:43] (03CR) 10WAN233: change logo at zh-classical wikipedia (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276284 (https://phabricator.wikimedia.org/T424128) (owner: 10WAN233) [08:19:10] FIRING: [3x] SystemdUnitFailed: prometheus-wmf-elasticsearch-exporter-9600.service on cloudelastic1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:27:09] (03PS4) 10WAN233: change logo at zh-classical wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276284 (https://phabricator.wikimedia.org/T424128) [08:33:59] 06SRE, 10Maps, 06Traffic, 07affects-Kiwix-and-openZIM: Wikipedia wikis have broken maps URLs in infobox: "Bad GeoJSON - unknown \"type\" property \"ExternalData\"" - https://phabricator.wikimedia.org/T424046#11848525 (10Aklapper) [08:39:25] (03PS1) 10Daniel Kinzler: redioscope: add more histogram buckets [deployment-charts] - 10https://gerrit.wikimedia.org/r/1276363 (https://phabricator.wikimedia.org/T419796) [08:47:26] (03CR) 10Jcrespo: [C:03+2] dbbackups: Backup only regularly clusters 32 & 33, the read-write ones [puppet] - 10https://gerrit.wikimedia.org/r/1271730 (https://phabricator.wikimedia.org/T421729) (owner: 10Jcrespo) [08:49:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [08:59:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr1-magru and Telxius (2001:1498:1:966:1::251) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [09:09:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [09:09:15] (03CR) 10A smart kitten: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276284 (https://phabricator.wikimedia.org/T424128) (owner: 10WAN233) [09:09:40] 06SRE, 10Maps, 06Traffic, 07affects-Kiwix-and-openZIM: Wikipedia wikis have broken maps URLs in infobox: "Bad GeoJSON - unknown \"type\" property \"ExternalData\"" - https://phabricator.wikimedia.org/T424046#11848567 (10TheDJ) It's not parsoid specific, if I switch the renderer to legacy, the image is also... [09:23:24] FIRING: [10x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:25:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:38:43] (03PS2) 10Daniel Kinzler: rest gateway: refactor ratelimit integration test [deployment-charts] - 10https://gerrit.wikimedia.org/r/1266995 [09:38:59] (03PS5) 10Daniel Kinzler: rest gateway: rate limits for liftwing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1272765 (https://phabricator.wikimedia.org/T413448) [09:42:57] (03PS1) 10Daniel Kinzler: rest-gateway: adjust rate limits [deployment-charts] - 10https://gerrit.wikimedia.org/r/1276372 (https://phabricator.wikimedia.org/T417779) [09:50:21] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave) {#11372}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [09:50:36] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave) {#11372}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [09:51:21] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave) {#11372}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqord:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260422T1000) [10:05:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [10:10:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [10:27:45] !log jynus@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[2141,2250].codfw.wmnet with reason: clone [10:42:05] (03PS1) 10Jcrespo: mariadb: Set db2250 as a new codfw s1 backup source [puppet] - 10https://gerrit.wikimedia.org/r/1276382 (https://phabricator.wikimedia.org/T418979) [10:44:33] (03PS2) 10Jcrespo: mariadb: Set db2250 as a new codfw s1 backup source [puppet] - 10https://gerrit.wikimedia.org/r/1276382 (https://phabricator.wikimedia.org/T418979) [10:44:35] (03CR) 10Mvolz: [C:03+2] citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1271950 (owner: 10PipelineBot) [10:44:38] (03CR) 10Jcrespo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1276382 (https://phabricator.wikimedia.org/T418979) (owner: 10Jcrespo) [10:46:40] (03Merged) 10jenkins-bot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1271950 (owner: 10PipelineBot) [10:52:54] (03PS1) 10Mvolz: Update zotero [deployment-charts] - 10https://gerrit.wikimedia.org/r/1276384 [11:00:05] mvolz: #bothumor I � Unicode. All rise for Services – Citoid / Zotero deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260422T1100). [11:05:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [11:06:12] !log mvolz@deploy1003 helmfile [staging] START helmfile.d/services/citoid: apply [11:06:43] !log mvolz@deploy1003 helmfile [staging] DONE helmfile.d/services/citoid: apply [11:07:39] !log mvolz@deploy1003 helmfile [codfw] START helmfile.d/services/citoid: apply [11:08:13] !log mvolz@deploy1003 helmfile [codfw] DONE helmfile.d/services/citoid: apply [11:10:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [11:12:26] !log mvolz@deploy1003 helmfile [eqiad] START helmfile.d/services/citoid: apply [11:12:55] !log mvolz@deploy1003 helmfile [eqiad] DONE helmfile.d/services/citoid: apply [11:21:39] (03PS1) 10Mvolz: Revert "citoid: pipeline bot promote" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1276385 [11:21:49] (03CR) 10Mvolz: [C:03+2] Revert "citoid: pipeline bot promote" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1276385 (owner: 10Mvolz) [11:22:48] !log mvolz@deploy1003 helmfile [staging] START helmfile.d/services/citoid: apply [11:22:52] !log mvolz@deploy1003 helmfile [staging] DONE helmfile.d/services/citoid: apply [11:23:48] (03Merged) 10jenkins-bot: Revert "citoid: pipeline bot promote" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1276385 (owner: 10Mvolz) [11:24:43] !log mvolz@deploy1003 helmfile [staging] START helmfile.d/services/citoid: apply [11:25:03] !log mvolz@deploy1003 helmfile [staging] DONE helmfile.d/services/citoid: apply [11:25:30] !log mvolz@deploy1003 helmfile [eqiad] START helmfile.d/services/citoid: apply [11:25:57] !log mvolz@deploy1003 helmfile [eqiad] DONE helmfile.d/services/citoid: apply [11:26:15] !log mvolz@deploy1003 helmfile [codfw] START helmfile.d/services/citoid: apply [11:26:45] !log mvolz@deploy1003 helmfile [codfw] DONE helmfile.d/services/citoid: apply [11:31:16] (03CR) 10Mvolz: [C:03+2] Update zotero [deployment-charts] - 10https://gerrit.wikimedia.org/r/1276384 (owner: 10Mvolz) [11:33:33] (03Merged) 10jenkins-bot: Update zotero [deployment-charts] - 10https://gerrit.wikimedia.org/r/1276384 (owner: 10Mvolz) [11:34:02] FIRING: HelmReleaseBadStatus: Helm release mw-script/nngkzgw8 on k8s@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s&var-namespace=mw-script - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:36:01] !log mvolz@deploy1003 helmfile [staging] START helmfile.d/services/zotero: apply [11:36:26] !log mvolz@deploy1003 helmfile [staging] DONE helmfile.d/services/zotero: apply [11:40:59] !log mvolz@deploy1003 helmfile [codfw] START helmfile.d/services/zotero: apply [11:41:28] !log mvolz@deploy1003 helmfile [codfw] DONE helmfile.d/services/zotero: apply [11:58:50] !log mvolz@deploy1003 helmfile [eqiad] START helmfile.d/services/zotero: apply [11:59:19] !log mvolz@deploy1003 helmfile [eqiad] DONE helmfile.d/services/zotero: apply [12:06:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [12:11:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [12:19:10] FIRING: [3x] SystemdUnitFailed: prometheus-wmf-elasticsearch-exporter-9600.service on cloudelastic1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:00:05] Lucas_WMDE, Urbanecm, and TheresNoTime: UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260422T1300). Please do the needful. [13:00:05] No Gerrit patches in the queue for this window AFAICS. [13:00:10] o/ [13:00:21] nothing to deploy :) [13:00:41] happy earth day [13:05:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [13:05:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:10:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [13:23:24] FIRING: [10x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:48:31] (03PS1) 10Giuseppe Lavagetto: cache::haproxy: support wikilink style usernames in UAs [puppet] - 10https://gerrit.wikimedia.org/r/1276396 (https://phabricator.wikimedia.org/T423992) [14:00:04] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260422T1400) [14:17:18] (03PS1) 10Ottomata: EventStreamConfig - add rc0 streams for html and feature count change [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276397 (https://phabricator.wikimedia.org/T423920) [14:18:57] (03CR) 10CI reject: [V:04-1] EventStreamConfig - add rc0 streams for html and feature count change [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276397 (https://phabricator.wikimedia.org/T423920) (owner: 10Ottomata) [14:19:55] (03PS2) 10Ottomata: EventStreamConfig - add rc0 streams for html and feature count change [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276397 (https://phabricator.wikimedia.org/T423920) [14:21:00] (03CR) 10CI reject: [V:04-1] EventStreamConfig - add rc0 streams for html and feature count change [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276397 (https://phabricator.wikimedia.org/T423920) (owner: 10Ottomata) [14:25:06] (03CR) 10Daniel Kinzler: [C:03+1] "Looks good, but I have a bunch of nitpicks." [puppet] - 10https://gerrit.wikimedia.org/r/1276396 (https://phabricator.wikimedia.org/T423992) (owner: 10Giuseppe Lavagetto) [14:28:00] (03PS3) 10Ottomata: EventStreamConfig - add rc0 streams for html and feature count change [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276397 (https://phabricator.wikimedia.org/T423920) [14:28:02] PROBLEM - mysqld processes on db2141 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [14:28:04] PROBLEM - MariaDB Replica IO: s1 on db2141 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [14:28:04] PROBLEM - MariaDB Replica SQL: s1 on db2141 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [14:28:16] PROBLEM - MariaDB read only s1 on db2141 is CRITICAL: Could not connect to localhost:3311 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [14:30:04] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260422T1400) [14:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260422T1430) [14:35:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr1-drmrs and Hurricane Electric (185.1.47.2) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [14:37:29] !log ladsgroup@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2163 (T410589)', diff saved to https://phabricator.wikimedia.org/P91311 and previous config saved to /var/cache/conftool/dbconfig/20260422-143728-ladsgroup.json [14:37:33] T410589: Optimize all core tables, late 2025 - https://phabricator.wikimedia.org/T410589 [14:47:37] !log ladsgroup@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P91312 and previous config saved to /var/cache/conftool/dbconfig/20260422-144736-ladsgroup.json [14:54:08] (03PS1) 10Lucas Werkmeister (WMDE): wikidata-query-gui: Bump query-gui image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1276402 (https://phabricator.wikimedia.org/T414376) [14:54:18] anyone around who could review ^ for deployment? :) [14:54:29] (I can deploy myself, just need a +1) [14:57:45] !log ladsgroup@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P91313 and previous config saved to /var/cache/conftool/dbconfig/20260422-145744-ladsgroup.json [14:59:10] (03PS1) 10Giuseppe Lavagetto: cache_misc: apply traffic classification and rate-limting [puppet] - 10https://gerrit.wikimedia.org/r/1276403 [15:00:20] (03PS1) 10Daniel Kinzler: rest gateways: EXPERIMENT: set rate limit by referer [deployment-charts] - 10https://gerrit.wikimedia.org/r/1276404 [15:07:53] !log ladsgroup@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2163 (T410589)', diff saved to https://phabricator.wikimedia.org/P91314 and previous config saved to /var/cache/conftool/dbconfig/20260422-150752-ladsgroup.json [15:07:57] T410589: Optimize all core tables, late 2025 - https://phabricator.wikimedia.org/T410589 [15:08:09] !log ladsgroup@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance [15:08:17] !log ladsgroup@cumin1003 dbctl commit (dc=all): 'Depooling db2164 (T410589)', diff saved to https://phabricator.wikimedia.org/P91315 and previous config saved to /var/cache/conftool/dbconfig/20260422-150817-ladsgroup.json [15:25:01] 10SRE-swift-storage, 06Data-Persistence, 10MediaViewer, 10Thumbor, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11849180 (10Krinkle) >>! In T414805#11849172, @gerritbot wrote: > Change #1233759 **restored** by Krinkle: > %%%[mediawiki/extensions... [15:31:30] 06SRE: wiki.openstreetmap.org Commons thumbs rate limit allowance - https://phabricator.wikimedia.org/T423570#11849182 (10Firefishy) My private email might be a good route for private data, my email is grant AT osmfoundation DOT org. I am also happy to sign an NDA if required. [15:34:02] FIRING: HelmReleaseBadStatus: Helm release mw-script/nngkzgw8 on k8s@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s&var-namespace=mw-script - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [15:40:16] !log jynus@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[2141,2250].codfw.wmnet with reason: clone [15:47:04] RECOVERY - mysqld processes on db2141 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [15:47:18] RECOVERY - MariaDB read only s1 on db2141 is OK: Version 10.11.16-MariaDB-log, Uptime 29s, read_only: True, event_scheduler: True, 277.53 QPS, connection latency: 0.024177s, query latency: 0.000439s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [15:48:04] RECOVERY - MariaDB Replica SQL: s1 on db2141 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [15:48:04] RECOVERY - MariaDB Replica IO: s1 on db2141 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [15:55:43] (03PS1) 10Jcrespo: mariadb: Pool db2250 for backups instead of db2141 [puppet] - 10https://gerrit.wikimedia.org/r/1276406 (https://phabricator.wikimedia.org/T418979) [15:55:46] (03PS1) 10Jcrespo: mariadb: Set db2141 as a spare for decommissioning [puppet] - 10https://gerrit.wikimedia.org/r/1276407 (https://phabricator.wikimedia.org/T418979) [15:56:44] (03PS2) 10Jcrespo: mariadb: Pool db2250 for backups instead of db2141 [puppet] - 10https://gerrit.wikimedia.org/r/1276406 (https://phabricator.wikimedia.org/T418979) [15:57:42] (03PS3) 10Jcrespo: mariadb: Pool db2250 for backups instead of db2141 [puppet] - 10https://gerrit.wikimedia.org/r/1276406 (https://phabricator.wikimedia.org/T418979) [15:58:23] (03PS2) 10Jcrespo: mariadb: Set db2141 as a spare for decommissioning [puppet] - 10https://gerrit.wikimedia.org/r/1276407 (https://phabricator.wikimedia.org/T418979) [15:59:15] (03CR) 10Jcrespo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1276406 (https://phabricator.wikimedia.org/T418979) (owner: 10Jcrespo) [15:59:20] (03CR) 10Jcrespo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1276407 (https://phabricator.wikimedia.org/T418979) (owner: 10Jcrespo) [16:09:18] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:10:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr1-drmrs and Hurricane Electric (185.1.47.2) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [16:13:55] FIRING: [3x] SystemdUnitFailed: prometheus-wmf-elasticsearch-exporter-9600.service on cloudelastic1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:15:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr1-drmrs and Hurricane Electric (185.1.47.2) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [16:34:18] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:00:04] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260422T1700) [17:17:53] (03CR) 10Marostegui: [C:03+1] mariadb: Set db2250 as a new codfw s1 backup source [puppet] - 10https://gerrit.wikimedia.org/r/1276382 (https://phabricator.wikimedia.org/T418979) (owner: 10Jcrespo) [17:23:24] FIRING: [10x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:34:02] FIRING: HelmReleaseBadStatus: Helm release mw-script/nngkzgw8 on k8s@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s&var-namespace=mw-script - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [20:00:04] RoanKattouw, Urbanecm, TheresNoTime, kindrobot, and cjming: Time to snap out of that daydream and deploy UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260422T2000). [20:00:04] No Gerrit patches in the queue for this window AFAICS. [20:14:10] FIRING: [2x] SystemdUnitFailed: prometheus-wmf-elasticsearch-exporter-9600.service on cloudelastic1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:00:04] Deploy window Wikifunctions Services UTC Late (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260422T2100) [21:23:24] FIRING: [10x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:33:40] PROBLEM - mailman list info on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [21:38:30] RECOVERY - mailman list info on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 9234 bytes in 0.175 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [22:00:04] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260422T2200) [22:38:09] FIRING: [12x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:46:14] (03PS1) 10Neriah: Enable Hebrew keyboard DWIM for namespace resolution on hewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276432 (https://phabricator.wikimedia.org/T412468) [22:54:10] (03PS2) 10Neriah: Enable Hebrew keyboard DWIM for namespace resolution on hewikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276432 (https://phabricator.wikimedia.org/T412468) [23:23:18] (03CR) 10Cwhite: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1275799 (https://phabricator.wikimedia.org/T423986) (owner: 10Tiziano Fogli) [23:30:05] (03Abandoned) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1276081 (owner: 10TrainBranchBot) [23:34:02] FIRING: HelmReleaseBadStatus: Helm release mw-script/nngkzgw8 on k8s@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s&var-namespace=mw-script - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [23:40:03] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1276433 [23:40:03] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1276433 (owner: 10TrainBranchBot) [23:49:22] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1276433 (owner: 10TrainBranchBot) [23:50:02] (03CR) 10Cwhite: [C:04-1] "A good start!" [puppet] - 10https://gerrit.wikimedia.org/r/1275800 (https://phabricator.wikimedia.org/T423986) (owner: 10Tiziano Fogli)