[00:22:18] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.19 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:38:58] FIRING: KafkaMirrorMakerConsumerMaxLag: Kafka MirrorMaker main-eqiad-to-main-codfw max lag in last 10 minutes - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-mirror_name=main-eqiad-to-main-codfw - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerConsumerMaxLag [00:47:18] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 647.08 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [01:08:56] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1350932 [01:08:56] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1350932 (owner: 10TrainBranchBot) [01:15:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:16:41] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1350932 (owner: 10TrainBranchBot) [01:18:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:30:50] RECOVERY - MariaDB Replica Lag: s4 on db1265 is OK: OK slave_sql_lag Replication lag: 0.14 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [01:46:50] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [02:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261004T0700) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T0200) [02:00:19] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:08:05] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 46s) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:11:50] PROBLEM - MariaDB Replica Lag: s3 on db1265 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 616.28 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:27:18] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.12 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [03:08:58] RESOLVED: KafkaMirrorMakerConsumerMaxLag: Kafka MirrorMaker main-eqiad-to-main-codfw max lag in last 10 minutes - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-mirror_name=main-eqiad-to-main-codfw - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerConsumerMaxLag [03:34:02] FIRING: NetworkDeviceAlarmActive: Alarm active on ssw1-f1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-f1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [03:54:50] RECOVERY - MariaDB Replica Lag: s3 on db1265 is OK: OK slave_sql_lag Replication lag: 0.27 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [05:15:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:18:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:19:14] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351005 [05:46:50] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [05:59:19] (03PS1) 10Marostegui: db1270: Remove note [puppet] - 10https://gerrit.wikimedia.org/r/1351016 [06:01:16] (03CR) 10Marostegui: [C:03+2] db1270: Remove note [puppet] - 10https://gerrit.wikimedia.org/r/1351016 (owner: 10Marostegui) [06:07:14] (03CR) 10Arnaudb: [C:03+2] "I'll run this on gerrit2003 (current replica) first with puppet disabled on primary:" [puppet] - 10https://gerrit.wikimedia.org/r/1345040 (https://phabricator.wikimedia.org/T439209) (owner: 10Arnaudb) [06:10:48] (03CR) 10Arnaudb: [C:03+2] "`" [puppet] - 10https://gerrit.wikimedia.org/r/1345040 (https://phabricator.wikimedia.org/T439209) (owner: 10Arnaudb) [06:31:01] (03PS2) 10Muehlenhoff: liberica: Run spec tests on Bookworm/Trixie [puppet] - 10https://gerrit.wikimedia.org/r/1240728 [06:59:31] (03CR) 10Slyngshede: [C:03+2] P:IDP Tomcat 11 for CAS 8.X [puppet] - 10https://gerrit.wikimedia.org/r/1345921 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [07:00:05] Amir1, urbanecm, and awight: UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T0700). Please do the needful. [07:00:05] awight, Volker_E, and Ameisenigel: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:15:57] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [07:16:30] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [07:17:35] !log brouberol@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [07:18:31] !log brouberol@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [07:32:22] jouncebot: nowandnext [07:32:23] For the next 0 hour(s) and 27 minute(s): UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T0700) [07:32:23] In 2 hour(s) and 27 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1000) [07:34:02] FIRING: NetworkDeviceAlarmActive: Alarm active on ssw1-f1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-f1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [07:35:33] (03PS1) 10Hashar: Merge tag 'stable-3.10-integration-issues-538883789-539032891-540877705-541287630-545820925' into wmf/stable-3.10 [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1351052 [07:36:55] (03PS6) 10Arnaudb: gerrit: add failover, DNS emergency and demotion cookbooks [cookbooks] - 10https://gerrit.wikimedia.org/r/1344697 (https://phabricator.wikimedia.org/T439025) [07:36:55] (03CR) 10Arnaudb: "Following up on @rcoccioli@wikimedia.org feedback, I updated the emergency DNS cookbook to fully automate the process in: https://wikitech" [cookbooks] - 10https://gerrit.wikimedia.org/r/1344697 (https://phabricator.wikimedia.org/T439025) (owner: 10Arnaudb) [07:37:11] (03CR) 10Dpogorzelski: [C:03+2] wikimedia.org: rename liftwing-studio to hummingbird [dns] - 10https://gerrit.wikimedia.org/r/1348507 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [07:38:04] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: serve on hummingbird.wikimedia.org [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348509 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [07:39:36] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [07:40:46] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [07:41:01] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [07:41:48] (03PS3) 10Hashar: Merge tag 'stable-3.10-integration-issues-538883789-539032891-540877705-541287630-545820925' into wmf/stable-3.10 [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 [07:42:16] (03PS1) 10Tiziano Fogli: kafka-logging: remove kafka-logging200[1-3] [puppet] - 10https://gerrit.wikimedia.org/r/1351141 (https://phabricator.wikimedia.org/T432444) [07:42:19] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [07:43:16] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: serve on hummingbird.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1348508 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [07:43:16] (03PS2) 10Tiziano Fogli: kafka-logging: remove kafka-logging200[1-3] [puppet] - 10https://gerrit.wikimedia.org/r/1351141 (https://phabricator.wikimedia.org/T432444) [07:44:41] (03PS4) 10Hashar: Merge tag 'stable-3.10-integration-issues-538883789-539032891-540877705-541287630-545820925' into wmf/stable-3.10 [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 [07:46:15] Amir1 hi! [07:46:22] late start here [07:46:53] Amir1 would you be able to help with https://gerrit.wikimedia.org/r/c/1350233/? [07:46:58] !log dpogorzelski@dns1004 START - running authdns-update [07:49:31] !log dpogorzelski@dns1004 END - running authdns-update [07:50:11] (03PS3) 10CWilliams: mediabackups: add mediabackup_worker metrics [puppet] - 10https://gerrit.wikimedia.org/r/1346031 (https://phabricator.wikimedia.org/T439526) [07:50:25] (03PS4) 10CWilliams: mediabackups: add mediabackup_worker metrics [puppet] - 10https://gerrit.wikimedia.org/r/1346031 (https://phabricator.wikimedia.org/T439526) [07:52:33] (03PS3) 10CWilliams: versitygw: require mount points for objectstorage [puppet] - 10https://gerrit.wikimedia.org/r/1345578 (https://phabricator.wikimedia.org/T439424) [07:54:01] (03CR) 10CI reject: [V:04-1] Merge tag 'stable-3.10-integration-issues-538883789-539032891-540877705-541287630-545820925' into wmf/stable-3.10 [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 (owner: 10Hashar) [07:54:56] RECOVERY - Backup freshness on backup1014 is OK: Fresh: 140 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [07:56:51] (03CR) 10Elukey: [C:03+1] Remove profile::base::cuminunpriv from installservers [puppet] - 10https://gerrit.wikimedia.org/r/1348724 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [08:06:03] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, October 05 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347674 (https://phabricator.wikimedia.org/T409145) (owner: 10Ameisenigel) [08:07:03] (03CR) 10Ozge: [C:03+1] liftwing-studio: add Qwen3.8 to list of models [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348836 (owner: 10Ilias Sarantopoulos) [08:18:31] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343917 (https://phabricator.wikimedia.org/T437577) (owner: 10Seanleong-wmde) [08:20:34] !log jmm@cumin2003 START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS bookworm [08:22:28] 06SRE: Bing Webmaster Tools access request for Ash Han - https://phabricator.wikimedia.org/T439646#12394453 (10LSobanski) 05Open→03Declined @AHan-WMF looking at https://office.wikimedia.org/wiki/Google_Search_Console, this should be an ITS request instead. I'll close this task, please cc me in the ITS re... [08:22:39] !log upgrading cloudcumin2001 in-place to trixie T436667 [08:22:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:22:42] T436667: Upgrade cloudcumin hosts to Trixie - https://phabricator.wikimedia.org/T436667 [08:27:57] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for mw-web* and mw-api* in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1347640 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [08:30:13] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw [08:30:33] (03CR) 10Elukey: [C:04-1] "Analyzed it with Claude and there are some follow ups, will patch this and get back to you :)" [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 (owner: 10Elukey) [08:33:12] (03PS3) 10Ayounsi: netbox: add an option to connect to netbox-next [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) [08:34:02] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [08:35:00] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [08:35:01] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw [08:37:52] !log jmm@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage [08:39:18] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [08:40:41] (03PS1) 10Federico Ceratto: team-data-persistence: Alarm on stale data in Zarcillo [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) [08:40:52] (03PS2) 10Jelto: service::catalog: Set ipip_encapsulation for mw-web* and mw-api* in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1347641 (https://phabricator.wikimedia.org/T420436) [08:42:37] (03CR) 10CI reject: [V:04-1] team-data-persistence: Alarm on stale data in Zarcillo [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) (owner: 10Federico Ceratto) [08:42:39] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage [08:45:04] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for mw-web* and mw-api* in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1347641 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [08:46:49] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad [08:47:44] (03CR) 10Jgiannelos: [C:03+1] mobile-apps: add mathjax csp rules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347615 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [08:49:35] (03CR) 10Jgiannelos: [C:04-1] "needs version bump" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347615 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [08:52:43] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [08:53:53] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [08:53:53] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad [08:58:52] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS bookworm [09:04:15] (03PS2) 10Federico Ceratto: team-data-persistence: Alarm on stale data in Zarcillo [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) [09:04:42] (03CR) 10Fabfur: [C:03+1] profile::cache::base: adjust the range of ephemeral ports [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [09:06:11] (03CR) 10Hashar: "Needs a backport for `ListProjectsImpl`: https://gerrit-review.googlesource.com/c/plugins/healthcheck/+/637649" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 (owner: 10Hashar) [09:06:39] (03PS1) 10Btullis: install_server: Install grub on all disks in druid reuse recipes [puppet] - 10https://gerrit.wikimedia.org/r/1351177 (https://phabricator.wikimedia.org/T440033) [09:06:50] (03CR) 10Vgutierrez: [C:03+1] "Done" [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [09:13:03] (03CR) 10Daniel Kertesz: [C:03+2] profile::cache::base: adjust the range of ephemeral ports [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [09:15:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:17:48] !log dkertesz@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [09:17:51] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [09:18:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [09:20:34] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet [09:23:08] 10ops-eqiad, 06SRE, 06DC-Ops: Enter Equinix's Rack Location in Netbox's "Facility ID" Field - https://phabricator.wikimedia.org/T440070#12394603 (10ayounsi) Good idea, before we start entering data we need to add validators to prevent entry mistakes (regex for the format, maybe coherence in the numbering) an... [09:23:12] 10ops-codfw, 06SRE, 06DC-Ops: Enter CyrusOne's Rack Location in Netbox's "Facility ID" Field - https://phabricator.wikimedia.org/T440071#12394605 (10ayounsi) Good idea, before we start entering data we need to add validators to prevent entry mistakes (regex for the format, maybe coherence in the numbering) a... [09:24:25] (03CR) 10Brouberol: [C:03+1] install_server: Install grub on all disks in druid reuse recipes [puppet] - 10https://gerrit.wikimedia.org/r/1351177 (https://phabricator.wikimedia.org/T440033) (owner: 10Btullis) [09:24:28] (03CR) 10Klausman: [C:03+1] install_server: Install grub on all disks in druid reuse recipes [puppet] - 10https://gerrit.wikimedia.org/r/1351177 (https://phabricator.wikimedia.org/T440033) (owner: 10Btullis) [09:24:32] (03PS5) 10Hashar: Merge tag 'stable-3.10-integration-issues-538883789-539032891-540877705-541287630-545820925' into wmf/stable-3.10 [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 [09:24:48] (03CR) 10Btullis: [C:03+2] install_server: Install grub on all disks in druid reuse recipes [puppet] - 10https://gerrit.wikimedia.org/r/1351177 (https://phabricator.wikimedia.org/T440033) (owner: 10Btullis) [09:26:31] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet [09:28:51] (03CR) 10Volans: ferm: Absent the NRPE check when migrating from ferm to nftables (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1283620 (owner: 10Muehlenhoff) [09:30:37] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet [09:33:22] PROBLEM - Ensure trafficserver_exporter is running for instance backend on cp4047 is CRITICAL: PROCS CRITICAL: 2 processes with args /usr/bin/python3 /usr/bin/prometheus-trafficserver-exporter --no-procstats --no-ssl-verification --endpoint http://127.0.0.1:3128/_stats --port 9122 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [09:34:11] (03PS2) 10Amire80: Rename project namespace on Madurese Wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1350687 (https://phabricator.wikimedia.org/T440137) [09:34:22] RECOVERY - Ensure trafficserver_exporter is running for instance backend on cp4047 is OK: PROCS OK: 1 process with args /usr/bin/python3 /usr/bin/prometheus-trafficserver-exporter --no-procstats --no-ssl-verification --endpoint http://127.0.0.1:3128/_stats --port 9122 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [09:36:11] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1004.eqiad.wmnet [09:36:13] (03CR) 10Marostegui: [C:03+2] orchestrator: allow healthchecking over HTTP [puppet] - 10https://gerrit.wikimedia.org/r/1343590 (https://phabricator.wikimedia.org/T407329) (owner: 10Hnowlan) [09:37:41] 06SRE, 10SRE-Access-Requests: Requesting access to ops for lizf - https://phabricator.wikimedia.org/T439933#12394666 (10Aklapper) @Lizimedia: Welcome! Please also [link your LDAP account to your Phabricator account](https://phabricator.wikimedia.org/settings/panel/external/), so your 'LDAP User' account is sho... [09:38:49] (03CR) 10Marostegui: [C:03+1] versitygw: require mount points for objectstorage [puppet] - 10https://gerrit.wikimedia.org/r/1345578 (https://phabricator.wikimedia.org/T439424) (owner: 10CWilliams) [09:39:18] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [09:42:06] (03CR) 10Volans: "As requested on IRC here are my thoughts on this approach." [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [09:46:50] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:49:02] (03PS2) 10Ayounsi: spicerack config: make netbox URLs their own dict [puppet] - 10https://gerrit.wikimedia.org/r/1348660 (https://phabricator.wikimedia.org/T439408) [09:50:00] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm [09:50:26] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm [09:51:02] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): an-druid1003 Bootlooping: able to reseat/power drain? - https://phabricator.wikimedia.org/T440033#12394748 (10BTullis) a:05Jclark-ctr→03BTullis [09:52:45] (03CR) 10Giuseppe Lavagetto: [C:04-1] "This actually doesn't work as intended with Dell servers, which get rebooted anyways. Need a bit more research." [cookbooks] - 10https://gerrit.wikimedia.org/r/1346499 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [09:53:24] (03PS9) 10Hnowlan: sre.dns.admin: add option to depool DNS hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 (https://phabricator.wikimedia.org/T440168) (owner: 10Ssingh) [09:53:56] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): an-druid1003 Bootlooping: able to reseat/power drain? - https://phabricator.wikimedia.org/T440033#12394787 (10BTullis) I think that this is a preseeding issue, so something that we should be able to sort out. My reckoning at the m... [09:59:10] (03CR) 10Blake: [C:03+2] sidecars: remove pre-stop sleep, stop envoy last [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346502 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1000) [10:00:53] (03PS3) 10Giuseppe Lavagetto: sre.hosts.provision: add --no-reboot option to skip host reboot [cookbooks] - 10https://gerrit.wikimedia.org/r/1346499 (https://phabricator.wikimedia.org/T438287) [10:00:53] (03PS2) 10Giuseppe Lavagetto: sre.hosts.provision: enable virtualization by default for wikikube-worker [cookbooks] - 10https://gerrit.wikimedia.org/r/1347480 (https://phabricator.wikimedia.org/T438287) [10:01:41] (03PS4) 10Ayounsi: netbox: add an option to connect to netbox-next [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) [10:01:57] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) [10:03:02] (03PS9) 10Elukey: sre.hosts.provision: move DHCP configs to the DHCP cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 [10:03:02] (03PS4) 10Elukey: sre.hosts.reimage: refactor DHCP configs as cookbook calls [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 [10:03:40] (03Merged) 10jenkins-bot: sidecars: remove pre-stop sleep, stop envoy last [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346502 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:03:47] (03CR) 10CI reject: [V:04-1] netbox: add an option to connect to netbox-next [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [10:03:47] (03PS5) 10Ayounsi: netbox: add an option to connect to netbox-next [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) [10:04:07] (03CR) 10Ayounsi: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1348660 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [10:04:19] (03CR) 10Ayounsi: "Thanks!" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [10:04:38] (03CR) 10Ayounsi: netbox: add an option to connect to netbox-next (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [10:05:41] !log elukey@cumin1004 START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [10:05:45] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) [10:05:49] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) [10:05:51] !log blake@deploy1003 Started scap sync-world: helm-only deploy for T417800 [10:05:54] T417800: Implement proper sidecar container support in mediawiki pods - https://phabricator.wikimedia.org/T417800 [10:06:09] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage [10:06:40] (03CR) 10CI reject: [V:04-1] netbox: add an option to connect to netbox-next [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [10:07:57] PROBLEM - Host sretest2001 is DOWN: PING CRITICAL - Packet loss = 100% [10:08:52] (03PS15) 10Cathal Mooney: LVS IPIP encapsulation: support backend servers using nftables [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) [10:08:57] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage [10:09:26] RECOVERY - Host sretest2001 is UP: PING OK - Packet loss = 0%, RTA = 34.13 ms [10:09:33] (03CR) 10Cathal Mooney: LVS IPIP encapsulation: support backend servers using nftables (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) (owner: 10Cathal Mooney) [10:10:32] PROBLEM - Host wikikube-worker1157 is DOWN: PING CRITICAL - Packet loss = 60%, RTA = 3832.92 ms [10:11:10] RECOVERY - Host wikikube-worker1157 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [10:11:58] (03PS6) 10Ayounsi: netbox: add an option to connect to netbox-next [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) [10:12:56] PROBLEM - Host sretest2001 is DOWN: PING CRITICAL - Packet loss = 100% [10:14:01] (03PS6) 10Hashar: Merge tag 'stable-3.10-integration-issues-538883789-539032891-540877705-541287630-545820925' into wmf/stable-3.10 [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 (https://phabricator.wikimedia.org/T437620) [10:14:26] RECOVERY - Host sretest2001 is UP: PING OK - Packet loss = 0%, RTA = 30.32 ms [10:14:31] (03CR) 10Hashar: "Both got quickly merged upstream by Paladox! 🎉" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 (https://phabricator.wikimedia.org/T437620) (owner: 10Hashar) [10:14:58] !log blake@deploy1003 Finished scap sync-world: helm-only deploy for T417800 (duration: 09m 40s) [10:15:01] T417800: Implement proper sidecar container support in mediawiki pods - https://phabricator.wikimedia.org/T417800 [10:15:59] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) [10:16:04] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) [10:16:15] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [10:19:42] !log elukey@cumin1004 START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie [10:20:37] !log elukey@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS trixie [10:21:31] (03PS3) 10Blake: mw-experimental: enable native_sidecars on mw-experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333740 [10:24:42] (03CR) 10Btullis: [C:03+2] admin_ng: move dse-k8s-codfw to ceph-csi-rbd v3.14.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343624 (https://phabricator.wikimedia.org/T407166) (owner: 10Btullis) [10:24:46] (03CR) 10Marostegui: team-data-persistence: Alarm on stale data in Zarcillo (032 comments) [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) (owner: 10Federico Ceratto) [10:25:06] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1007.eqiad.wmnet [10:25:42] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1007.eqiad.wmnet [10:26:11] (03CR) 10Volans: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1348660 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [10:27:14] !log marostegui@cumin1004 START - Cookbook sre.mysql.depool depool db2207: stress tests [10:27:56] PROBLEM - Kafka broker TLS certificate validity on kafka-logging2003 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/Kafka/Administration%23Renew_TLS_certificate [10:27:56] PROBLEM - Kafka broker TLS certificate validity on kafka-logging2001 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/Kafka/Administration%23Renew_TLS_certificate [10:27:56] PROBLEM - Kafka broker TLS certificate validity on kafka-logging2002 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/Kafka/Administration%23Renew_TLS_certificate [10:27:56] PROBLEM - Kafka Broker Server on kafka-logging2003 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [10:27:56] PROBLEM - Kafka Broker Server on kafka-logging2001 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [10:27:57] PROBLEM - Kafka Broker Server on kafka-logging2002 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [10:28:09] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2207: stress tests [10:28:17] (03PS3) 10Ayounsi: spicerack config: make netbox URLs their own dict [puppet] - 10https://gerrit.wikimedia.org/r/1348660 (https://phabricator.wikimedia.org/T439408) [10:28:25] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1003.eqiad.wmnet with OS bookworm [10:28:26] Hi there, I'm going to merge and then deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/1351141. It's to remove kafka-logging200[1-3] from the codfw cluster. Any objections? [10:28:27] (03CR) 10Ayounsi: spicerack config: make netbox URLs their own dict (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1348660 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [10:28:34] PROBLEM - Host wikikube-worker1007 is DOWN: PING CRITICAL - Packet loss = 100% [10:28:51] FIRING: [4x] ProbeDown: Service kafka-logging2002:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging2002_ip4) - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:29:04] (03CR) 10Volans: [C:03+1] "LGTM from my side, check if that's ok for Luca too ;)" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [10:29:05] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2207.codfw.wmnet with reason: stress testing [10:29:06] (03PS5) 10Elukey: sre.hosts.reimage: refactor DHCP configs as cookbook calls [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 [10:29:20] !log elukey@cumin1004 START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie [10:29:28] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) [10:29:32] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) [10:30:08] (03PS1) 10Zabe: Move reads/writes of wbc_entity_usage to x1 for frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351194 (https://phabricator.wikimedia.org/T439753) [10:30:08] (03CR) 10Tiziano Fogli: [C:03+2] kafka-logging: remove kafka-logging200[1-3] [puppet] - 10https://gerrit.wikimedia.org/r/1351141 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [10:30:18] (03CR) 10Ayounsi: [C:03+2] spicerack config: make netbox URLs their own dict [puppet] - 10https://gerrit.wikimedia.org/r/1348660 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [10:30:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) [10:31:02] RECOVERY - Host wikikube-worker1007 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms [10:31:23] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1007.eqiad.wmnet [10:31:25] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1007.eqiad.wmnet [10:31:53] !log Stop MariaDB on db2207 and db2209 for stress testing [10:31:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:32:12] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2209.codfw.wmnet with reason: stress testing [10:32:24] !log marostegui@cumin1004 START - Cookbook sre.mysql.depool depool db2209: stress tests [10:32:38] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2209: stress tests [10:32:48] (03CR) 10Muehlenhoff: [C:03+1] "Looks good!" [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) (owner: 10Cathal Mooney) [10:33:35] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host an-druid1004.eqiad.wmnet with OS bookworm [10:33:41] !log marostegui@cumin1004 START - Cookbook sre.mysql.depool depool db1218: stress tests [10:33:49] !log Stop MariaDB on db1218, db2207 and db2209 for stress testing [10:33:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:33:51] FIRING: [6x] ProbeDown: Service kafka-logging2001:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging2001_ip4) - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:33:53] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: stress tests [10:34:17] (03CR) 10Cathal Mooney: [C:03+2] LVS IPIP encapsulation: support backend servers using nftables [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) (owner: 10Cathal Mooney) [10:34:21] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1218.eqiad.wmnet with reason: stress testing [10:35:26] (03CR) 10Elukey: "LGTM! I am very ignorant about nftables so I cannot really comment on the logic, but the python script and the puppet code look sane :)" [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) (owner: 10Cathal Mooney) [10:35:53] (03Merged) 10jenkins-bot: admin_ng: move dse-k8s-codfw to ceph-csi-rbd v3.14.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343624 (https://phabricator.wikimedia.org/T407166) (owner: 10Btullis) [10:36:07] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 07Sustainability (Incident Followup): Setup url-downloader-next.w.o to simplify tests - https://phabricator.wikimedia.org/T430166#12394996 (10LSobanski) [10:36:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:37:49] !log dkertesz@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [10:37:52] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [10:38:04] (03CR) 10Elukey: "After a chat with Claude I fixed some corner cases, and re-provisioned sretest2001. All good, but I also need to provision a brand new hos" [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 (owner: 10Elukey) [10:38:13] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) [10:38:19] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) [10:39:56] !log tappof@deploy1003 helmfile [eqiad] START helmfile.d/admin 'sync'. [10:40:09] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet [10:40:12] !log tappof@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'sync'. [10:40:43] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet [10:40:44] !log elukey@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage [10:40:49] !log tappof@deploy1003 helmfile [codfw] START helmfile.d/admin 'sync'. [10:40:53] (03CR) 10Cathal Mooney: [C:03+2] LVS IPIP encapsulation: support backend servers using nftables (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) (owner: 10Cathal Mooney) [10:41:25] !log tappof@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'sync'. [10:41:56] (03PS4) 10CWilliams: versitygw: require mount points for objectstorage [puppet] - 10https://gerrit.wikimedia.org/r/1345578 (https://phabricator.wikimedia.org/T439424) [10:42:44] jouncebot: nowandnext [10:42:44] For the next 0 hour(s) and 17 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1000) [10:42:45] In 2 hour(s) and 17 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1300) [10:42:50] (03CR) 10Zabe: [C:03+2] Move reads/writes of wbc_entity_usage to x1 for frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351194 (https://phabricator.wikimedia.org/T439753) (owner: 10Zabe) [10:42:57] !log tappof@deploy1003 Started scap sync-world: T432444 - Remove kafka-logging200[1-3] [10:43:00] T432444: Provision kafka-logging100[6-8] and kafka-logging200[6-8] - https://phabricator.wikimedia.org/T432444 [10:43:12] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet [10:43:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet [10:43:16] (03CR) 10CWilliams: [C:03+2] versitygw: require mount points for objectstorage [puppet] - 10https://gerrit.wikimedia.org/r/1345578 (https://phabricator.wikimedia.org/T439424) (owner: 10CWilliams) [10:43:26] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:43:27] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage [10:46:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:46:15] PROBLEM - Host wikikube-worker2001 is DOWN: PING CRITICAL - Packet loss = 100% [10:46:23] !log tappof@deploy1003 Finished scap sync-world: T432444 - Remove kafka-logging200[1-3] (duration: 04m 44s) [10:47:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:50:04] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage [10:50:53] RECOVERY - Host wikikube-worker2001 is UP: PING OK - Packet loss = 0%, RTA = 31.76 ms [10:51:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:51:58] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2001.codfw.wmnet [10:52:01] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2001.codfw.wmnet [10:52:09] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2002.codfw.wmnet [10:52:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:52:44] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2002.codfw.wmnet [10:52:54] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage [10:55:27] !log btullis@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [10:55:51] oblivian@cumin1004 provision (PID 2895962) is awaiting input [10:57:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:57:37] (03PS1) 10Tiziano Fogli: eventgate-logging-external: replace kafka-logging200[1-3] with 200[6-8] [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351204 (https://phabricator.wikimedia.org/T432444) [10:57:38] !log btullis@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [10:58:02] !log elukey@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" [11:00:58] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:01:05] elukey@cumin1004 reimage (PID 2889394) is awaiting input [11:03:00] (03PS1) 10DCausse: search: test impact of the MLR window size [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351207 (https://phabricator.wikimedia.org/T438984) [11:03:02] (03PS1) 10DCausse: Revert "search: test impact of the MLR window size" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351208 (https://phabricator.wikimedia.org/T438984) [11:03:04] (03PS1) 10DCausse: search: test impact of the function score window size [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351209 (https://phabricator.wikimedia.org/T438984) [11:03:06] (03PS1) 10DCausse: Revert "search: test impact of the function score window size" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351210 (https://phabricator.wikimedia.org/T438984) [11:03:08] (03PS1) 10DCausse: search: test impact of cross project searches [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351211 (https://phabricator.wikimedia.org/T438984) [11:03:10] (03PS1) 10DCausse: Revert "search: test impact of cross project searches" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351212 (https://phabricator.wikimedia.org/T438984) [11:03:12] (03PS1) 10DCausse: search: measure impact of enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351213 (https://phabricator.wikimedia.org/T438984) [11:03:15] (03PS1) 10DCausse: Revert "search: measure impact of enwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351214 (https://phabricator.wikimedia.org/T438984) [11:03:17] (03PS1) 10DCausse: search: test perf of moving completion out of eqiad [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351215 (https://phabricator.wikimedia.org/T438984) [11:03:20] (03PS1) 10DCausse: Revert "search: test perf of moving completion out of eqiad" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351216 (https://phabricator.wikimedia.org/T438984) [11:04:12] PROBLEM - Host wikikube-worker2002 is DOWN: PING CRITICAL - Packet loss = 100% [11:04:23] (03CR) 10CI reject: [V:04-1] search: test impact of cross project searches [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351211 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [11:04:23] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1348986 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [11:08:02] (03PS2) 10DCausse: search: test impact of cross project searches [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351211 (https://phabricator.wikimedia.org/T438984) [11:08:02] (03PS2) 10DCausse: Revert "search: test impact of cross project searches" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351212 (https://phabricator.wikimedia.org/T438984) [11:08:03] (03PS2) 10DCausse: search: measure impact of enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351213 (https://phabricator.wikimedia.org/T438984) [11:08:03] (03PS2) 10DCausse: Revert "search: measure impact of enwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351214 (https://phabricator.wikimedia.org/T438984) [11:08:04] (03PS2) 10DCausse: search: test perf of moving completion out of eqiad [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351215 (https://phabricator.wikimedia.org/T438984) [11:08:05] (03PS2) 10DCausse: Revert "search: test perf of moving completion out of eqiad" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351216 (https://phabricator.wikimedia.org/T438984) [11:08:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [11:08:17] (03PS1) 10JavierMonton: stream: reconcile + HTML enrichment streams [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351220 (https://phabricator.wikimedia.org/T415264) [11:08:22] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:08:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet [11:08:46] RECOVERY - Host wikikube-worker2002 is UP: PING OK - Packet loss = 0%, RTA = 31.78 ms [11:09:02] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2001.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:09:20] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2002.codfw.wmnet [11:10:02] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet [11:10:05] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2002.codfw.wmnet [11:10:33] (03PS1) 10Btullis: ATS: Configure datatsets.wikimedia.org for normal caching [puppet] - 10https://gerrit.wikimedia.org/r/1351221 (https://phabricator.wikimedia.org/T434046) [11:11:20] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1351221 (https://phabricator.wikimedia.org/T434046) (owner: 10Btullis) [11:11:59] (03PS2) 10DCausse: search: test impact of the function score window size [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351209 (https://phabricator.wikimedia.org/T438984) [11:11:59] (03PS2) 10DCausse: Revert "search: test impact of the function score window size" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351210 (https://phabricator.wikimedia.org/T438984) [11:11:59] (03PS3) 10DCausse: search: test impact of cross project searches [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351211 (https://phabricator.wikimedia.org/T438984) [11:12:00] (03PS3) 10DCausse: Revert "search: test impact of cross project searches" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351212 (https://phabricator.wikimedia.org/T438984) [11:12:01] (03PS3) 10DCausse: search: measure impact of enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351213 (https://phabricator.wikimedia.org/T438984) [11:12:02] (03PS3) 10DCausse: Revert "search: measure impact of enwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351214 (https://phabricator.wikimedia.org/T438984) [11:12:05] (sorry for the noise..) [11:12:06] (03PS3) 10DCausse: search: test perf of moving completion out of eqiad [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351215 (https://phabricator.wikimedia.org/T438984) [11:12:10] (03PS3) 10DCausse: Revert "search: test perf of moving completion out of eqiad" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351216 (https://phabricator.wikimedia.org/T438984) [11:12:14] (03PS1) 10DCausse: search: test impact of MLR [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351222 (https://phabricator.wikimedia.org/T438984) [11:12:18] (03PS1) 10DCausse: Revert "search: test impact of MLR" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351223 (https://phabricator.wikimedia.org/T438984) [11:13:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [11:13:42] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2005.codfw.wmnet [11:14:02] RESOLVED: KubernetesCalicoDown: wikikube-worker2002.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2002.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:14:16] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2005.codfw.wmnet [11:14:27] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:14:56] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:15:10] (03CR) 10CI reject: [V:04-1] search: test impact of MLR [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351222 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [11:17:38] (03CR) 10Ilias Sarantopoulos: [C:03+1] ATS: Configure datatsets.wikimedia.org for normal caching [puppet] - 10https://gerrit.wikimedia.org/r/1351221 (https://phabricator.wikimedia.org/T434046) (owner: 10Btullis) [11:18:36] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:19:24] (03PS7) 10Cathal Mooney: Interface validator: check Nokia breakout ports have parent [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) [11:19:26] (03CR) 10Muehlenhoff: [C:03+2] Add conftool-data for ldap-ro-test [puppet] - 10https://gerrit.wikimedia.org/r/1347474 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [11:21:16] PROBLEM - Host wikikube-worker2005 is DOWN: PING CRITICAL - Packet loss = 100% [11:22:46] (03PS3) 10Isabelle Hurbain-Palatin: mobile-apps: add mathjax csp rules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347615 (https://phabricator.wikimedia.org/T431693) [11:23:12] (03CR) 10Jgiannelos: [C:03+1] mobile-apps: add mathjax csp rules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347615 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [11:23:42] (03CR) 10Isabelle Hurbain-Palatin: "done" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347615 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [11:23:47] (03PS2) 10DCausse: search: test impact of MLR [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351222 (https://phabricator.wikimedia.org/T438984) [11:23:47] (03PS2) 10DCausse: Revert "search: test impact of MLR" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351223 (https://phabricator.wikimedia.org/T438984) [11:23:47] (03PS3) 10DCausse: search: test impact of the function score window size [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351209 (https://phabricator.wikimedia.org/T438984) [11:23:48] (03PS3) 10DCausse: Revert "search: test impact of the function score window size" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351210 (https://phabricator.wikimedia.org/T438984) [11:23:49] (03PS4) 10DCausse: search: test impact of cross project searches [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351211 (https://phabricator.wikimedia.org/T438984) [11:23:50] (03PS4) 10DCausse: Revert "search: test impact of cross project searches" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351212 (https://phabricator.wikimedia.org/T438984) [11:23:54] (03PS4) 10DCausse: search: measure impact of enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351213 (https://phabricator.wikimedia.org/T438984) [11:23:58] (03PS4) 10DCausse: Revert "search: measure impact of enwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351214 (https://phabricator.wikimedia.org/T438984) [11:24:02] (03PS4) 10DCausse: search: test perf of moving completion out of eqiad [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351215 (https://phabricator.wikimedia.org/T438984) [11:24:06] (03PS4) 10DCausse: Revert "search: test perf of moving completion out of eqiad" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351216 (https://phabricator.wikimedia.org/T438984) [11:25:01] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:25:07] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1004.eqiad.wmnet with OS bookworm [11:25:22] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2005.codfw.wmnet [11:25:41] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2005.codfw.wmnet [11:25:46] RECOVERY - Host wikikube-worker2005 is UP: PING OK - Packet loss = 0%, RTA = 30.46 ms [11:26:45] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2006.codfw.wmnet [11:27:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2006.codfw.wmnet [11:27:30] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): an-druid1003 Bootlooping: able to reseat/power drain? - https://phabricator.wikimedia.org/T440033#12395117 (10BTullis) 05Open→03Resolved Yes, this is fixed now. `an-druid1003` is reimaged and back in the cluster. I'll reim... [11:27:33] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:28:47] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [11:29:29] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, October 05 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1350233 (https://phabricator.wikimedia.org/T439887) (owner: 10VolkerE) [11:29:32] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [11:30:12] PROBLEM - Host wikikube-worker2006 is DOWN: PING CRITICAL - Packet loss = 100% [11:31:23] btullis@cumin1004 reimage (PID 2903088) is awaiting input [11:31:39] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:31:44] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2006.codfw.wmnet [11:34:02] FIRING: NetworkDeviceAlarmActive: Alarm active on ssw1-f1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-f1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [11:34:03] FIRING: KubernetesCalicoDown: wikikube-worker2006.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2006.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:34:32] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2006.codfw.wmnet [11:34:40] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet [11:34:46] RECOVERY - Host wikikube-worker2006 is UP: PING OK - Packet loss = 0%, RTA = 31.76 ms [11:35:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet [11:35:45] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:38:14] PROBLEM - Host wikikube-worker2011 is DOWN: PING CRITICAL - Packet loss = 100% [11:39:03] RESOLVED: KubernetesCalicoDown: wikikube-worker2006.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2006.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:40:20] jouncebot: nowandnext [11:40:20] No deployments scheduled for the next 1 hour(s) and 19 minute(s) [11:40:20] In 1 hour(s) and 19 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1300) [11:40:34] I’ll deploy an urgent backport for Volker_E2 [11:40:46] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1350233 (https://phabricator.wikimedia.org/T439887) (owner: 10VolkerE) [11:41:19] !log dkertesz@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [11:41:22] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [11:41:46] RECOVERY - Host wikikube-worker2011 is UP: PING OK - Packet loss = 0%, RTA = 30.52 ms [11:42:03] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:42:07] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet [11:42:13] (03Merged) 10jenkins-bot: instrument(AccountSetup): expose motivation data for control [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1350233 (https://phabricator.wikimedia.org/T439887) (owner: 10VolkerE) [11:42:58] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1350233|instrument(AccountSetup): expose motivation data for control (T439887)]] [11:43:01] T439887: Recommended Reading exp: Align Reader pageview metric for control - https://phabricator.wikimedia.org/T439887 [11:43:02] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2011.codfw.wmnet [11:43:36] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet [11:43:51] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet [11:44:00] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2012.codfw.wmnet [11:44:10] <_joe_> jouncebot: now [11:44:10] No deployments scheduled for the next 1 hour(s) and 15 minute(s) [11:44:26] PROBLEM - Confd vcl based reload on cp6012 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [11:44:26] PROBLEM - Confd vcl based reload on cp6016 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [11:44:34] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2012.codfw.wmnet [11:44:49] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host an-druid1005.eqiad.wmnet with OS bookworm [11:45:18] !log btullis@cumin1004 START - Cookbook sre.hosts.move-vlan for host an-druid1005 [11:45:25] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [11:47:26] RECOVERY - Confd vcl based reload on cp6016 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [11:47:39] (03PS1) 10Btullis: dse-k8s-codfw: Bring 3 new workers into service [puppet] - 10https://gerrit.wikimedia.org/r/1351249 (https://phabricator.wikimedia.org/T430654) [11:47:41] oblivian@cumin1004 provision (PID 2906308) is awaiting input [11:48:05] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:49:05] hm, feels like build-and-push-container-images is unusually slow for a backport without i18n changes [11:49:26] PROBLEM - Confd vcl based reload on cp6014 is CRITICAL: reload-vcl failed to run since 0h, 1 minutes. https://wikitech.wikimedia.org/wiki/Varnish [11:50:56] btullis@cumin1004 reimage (PID 2903088) is awaiting input [11:51:18] PROBLEM - Host wikikube-worker2012 is DOWN: PING CRITICAL - Packet loss = 100% [11:52:26] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" [11:52:35] _joe_: re your jouncebot: now – are you doing anything that would interact with scap? or unrelated stuff? [11:52:56] !log jmm@puppetserver1001 conftool action : set/weight=10; selector: name=ldap-replica2008.wikimedia.org [11:52:59] <_joe_> Lucas_WMDE: I am doing stuff that might interact with scap [11:53:10] !log jmm@puppetserver1001 conftool action : set/weight=10; selector: name=ldap-replica2007.wikimedia.org [11:53:12] <_joe_> but I can pause between them [11:53:12] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" [11:53:13] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:53:13] !log btullis@cumin1004 START - Cookbook sre.dns.wipe-cache an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [11:53:16] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [11:53:16] !log btullis@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host an-druid1005 [11:53:18] !log jmm@puppetserver1001 conftool action : set/weight=10; selector: name=ldap-replica1005.wikimedia.org [11:53:20] hm [11:53:23] !log jmm@puppetserver1001 conftool action : set/weight=10; selector: name=ldap-replica1006.wikimedia.org [11:53:24] I’m currently deploying https://spiderpig.wikimedia.org/jobs/2923 [11:53:36] currently waiting for the image build to finish (it’s at the “pushing” phase) [11:53:42] !log jmm@puppetserver1001 conftool action : set/pooled=yes; selector: name=ldap-replica2008.wikimedia.org [11:54:08] <_joe_> yeah dw you shouldn't be impacted [11:54:13] ok thanks [11:54:14] RECOVERY - Host wikikube-worker2012 is UP: PING OK - Packet loss = 0%, RTA = 31.67 ms [11:54:18] guess I’ll just keep waiting for the image build then ^^ [11:54:24] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:54:26] RECOVERY - Confd vcl based reload on cp6014 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [11:54:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet [11:54:32] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet [11:54:40] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2013.codfw.wmnet [11:54:53] <_joe_> Lucas_WMDE: ping me when you're done [11:54:56] (“it’s one JS file, Michael. how long could it take to sync? ten minutes?”) [11:54:58] _joe_: ack [11:55:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2013.codfw.wmnet [11:55:21] !log btullis@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-druid1005 [11:55:22] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-druid1005 [11:56:32] (03CR) 10Btullis: [C:03+2] ATS: Configure datatsets.wikimedia.org for normal caching [puppet] - 10https://gerrit.wikimedia.org/r/1351221 (https://phabricator.wikimedia.org/T434046) (owner: 10Btullis) [11:58:01] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [11:58:23] oblivian@cumin1004 provision (PID 2907128) is awaiting input [11:59:38] PROBLEM - Druid historical on an-druid1007 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [12:00:15] !log jmm@puppetserver1001 conftool action : set/pooled=yes; selector: name=ldap-replica2007.wikimedia.org [12:00:16] (03PS3) 10Scott French: hieradata: Tweak supervision messages on shellbox deploy-service config [puppet] - 10https://gerrit.wikimedia.org/r/1347761 (https://phabricator.wikimedia.org/T412941) [12:00:26] !log jmm@puppetserver1001 conftool action : set/pooled=yes; selector: name=ldap-replica1006.wikimedia.org [12:00:41] !log jmm@puppetserver1001 conftool action : set/pooled=yes; selector: name=ldap-replica1005.wikimedia.org [12:00:42] (03CR) 10Scott French: hieradata: Tweak supervision messages on shellbox deploy-service config (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1347761 (https://phabricator.wikimedia.org/T412941) (owner: 10Scott French) [12:00:44] something weird is going on with this image, sync-testservers-k8s is also slow… [12:00:52] am I doing the first deployment of the week? ^^ [12:01:10] huh, last one in spiderpig is on thursday. okay then [12:01:19] !log lucaswerkmeister-wmde@deploy1003 volker-e, lucaswerkmeister-wmde: Backport for [[gerrit:1350233|instrument(AccountSetup): expose motivation data for control (T439887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:01:23] T439887: Recommended Reading exp: Align Reader pageview metric for control - https://phabricator.wikimedia.org/T439887 [12:01:23] (03CR) 10Scott French: [C:03+2] hieradata: Tweak supervision messages on shellbox deploy-service config [puppet] - 10https://gerrit.wikimedia.org/r/1347761 (https://phabricator.wikimedia.org/T412941) (owner: 10Scott French) [12:01:38] RECOVERY - Druid historical on an-druid1007 is OK: PROCS OK: 1 process with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [12:01:45] FIRING: WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [12:01:46] now where did Volker_E2 go [12:06:02] Volker_E2: please test on mwdebug :) [12:06:15] (03CR) 10Federico Ceratto: "We could not find a quick solution so we discussed closing this change and leaving the mariadb configuration as it is." [puppet] - 10https://gerrit.wikimedia.org/r/1341257 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [12:06:19] (03CR) 10VolkerE: [C:03+1] instrument(AccountSetup): expose motivation data for control [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1350233 (https://phabricator.wikimedia.org/T439887) (owner: 10VolkerE) [12:06:19] (03CR) 10Jelto: [C:03+1] "looks reasonable and should be a noop for all other clusters (except dse-k8s-codfw with the next patch in the chain)." [puppet] - 10https://gerrit.wikimedia.org/r/1347693 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [12:06:40] (03Abandoned) 10Federico Ceratto: core-mysql.my.cnf.erb: add role:: in role name syntax [puppet] - 10https://gerrit.wikimedia.org/r/1341257 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [12:08:57] Lucas_WMDE on it [12:09:02] ok, thanks [12:10:00] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker2009 has a BGP session which is not in the 'established' state. [12:12:30] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage [12:12:48] (03CR) 10Jelto: [V:03+1] "PCC SUCCESS (CORE_DIFF 2 NOOP 4): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1347694 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [12:15:00] FIRING: [4x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker2009 has a BGP session which is not in the 'established' state. [12:15:52] (03CR) 10Jelto: [V:03+1 C:03+1] "technically this looks good and is a noop for other clusters." [puppet] - 10https://gerrit.wikimedia.org/r/1347694 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [12:16:42] !log installing mesa security updates [12:16:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:17:26] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage [12:19:30] Volker_E2: still testing? [12:19:58] (03CR) 10Federico Ceratto: team-data-persistence: Alarm on stale data in Zarcillo (032 comments) [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) (owner: 10Federico Ceratto) [12:20:00] FIRING: [6x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker2009 has a BGP session which is not in the 'established' state. [12:20:19] (03CR) 10Brouberol: [C:03+1] dse-k8s-codfw: Bring 3 new workers into service [puppet] - 10https://gerrit.wikimedia.org/r/1351249 (https://phabricator.wikimedia.org/T430654) (owner: 10Btullis) [12:23:13] (03CR) 10Aqu: [C:03+1] stream: reconcile + HTML enrichment streams [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351220 (https://phabricator.wikimedia.org/T415264) (owner: 10JavierMonton) [12:24:19] (03CR) 10Marostegui: "Adding @tfogli@wikimedia.org for some more input." [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) (owner: 10Federico Ceratto) [12:25:49] Lucas_WMDE it works [12:26:17] thanks! [12:26:18] (03PS1) 10Filippo Giunchedi: install_server: remove unused recipes [puppet] - 10https://gerrit.wikimedia.org/r/1351274 (https://phabricator.wikimedia.org/T439831) [12:26:20] !log lucaswerkmeister-wmde@deploy1003 volker-e, lucaswerkmeister-wmde: Continuing with deployment [12:26:20] (03PS1) 10Filippo Giunchedi: install_server: create efi partitions in the lvm case [puppet] - 10https://gerrit.wikimedia.org/r/1351275 (https://phabricator.wikimedia.org/T439831) [12:26:22] (03PS1) 10Filippo Giunchedi: install_server: new script detect-os-device.sh [puppet] - 10https://gerrit.wikimedia.org/r/1351276 (https://phabricator.wikimedia.org/T439831) [12:26:25] (03PS1) 10Filippo Giunchedi: install_server: use raid1-jbod-efi for cloudcephosd >= 1053 [puppet] - 10https://gerrit.wikimedia.org/r/1351277 (https://phabricator.wikimedia.org/T439831) [12:27:08] (03CR) 10Cathal Mooney: Interface validator: check Nokia breakout ports have parent (031 comment) [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [12:28:24] (03CR) 10Klausman: [C:03+1] dse-k8s-codfw: Bring 3 new workers into service [puppet] - 10https://gerrit.wikimedia.org/r/1351249 (https://phabricator.wikimedia.org/T430654) (owner: 10Btullis) [12:31:44] !log installing sqlite3 security updates [12:31:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:32:28] RECOVERY - Confd vcl based reload on cp6012 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [12:34:28] PROBLEM - Confd vcl based reload on cp6011 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [12:35:44] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [12:36:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [12:38:24] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1350233|instrument(AccountSetup): expose motivation data for control (T439887)]] (duration: 55m 26s) [12:38:28] T439887: Recommended Reading exp: Align Reader pageview metric for control - https://phabricator.wikimedia.org/T439887 [12:38:28] _joe_: I’m done [12:38:34] !log elukey@cumin1004 END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" [12:38:34] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS trixie [12:38:40] that took much longer than expected, sorry :/ [12:38:41] <_joe_> Lucas_WMDE: thanks [12:39:00] <_joe_> np, but in general I'd like us to stick to the schedule if possible :) [12:39:16] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:39:30] <_joe_> jouncebot: next [12:39:30] In 0 hour(s) and 20 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1300) [12:39:48] fair, I was told it was an emergency [12:40:28] PROBLEM - Confd vcl based reload on cp6012 is CRITICAL: reload-vcl failed to run since 0h, 1 minutes. https://wikitech.wikimedia.org/wiki/Varnish [12:41:37] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181 (10tappof) 03NEW [12:41:56] PROBLEM - Host wikikube-worker2013 is DOWN: PING CRITICAL - Packet loss = 100% [12:42:08] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181#12395344 (10tappof) [12:44:20] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181#12395351 (10tappof) [12:44:28] RECOVERY - Confd vcl based reload on cp6012 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [12:45:04] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:45:10] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet [12:45:22] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [12:45:28] RECOVERY - Confd vcl based reload on cp6011 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [12:45:33] (03CR) 10Btullis: [C:03+2] kubernetes: Add an option to select the service account key type [puppet] - 10https://gerrit.wikimedia.org/r/1347693 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [12:45:46] RECOVERY - Host wikikube-worker2013 is UP: PING OK - Packet loss = 0%, RTA = 30.45 ms [12:46:02] !log tappof@cumin1004 START - Cookbook sre.hosts.decommission for hosts kafka-logging2001.codfw.wmnet [12:46:04] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet [12:46:37] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [12:46:39] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet [12:47:16] (03PS1) 10Brouberol: Add fixtures for the codfw cephosd admin [labs/private] - 10https://gerrit.wikimedia.org/r/1351299 [12:47:28] PROBLEM - Confd vcl based reload on cp6012 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [12:47:29] (03CR) 10Brouberol: [C:03+2] Add fixtures for the codfw cephosd admin [labs/private] - 10https://gerrit.wikimedia.org/r/1351299 (owner: 10Brouberol) [12:47:33] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet [12:47:33] (03CR) 10Brouberol: [V:03+2 C:03+2] Add fixtures for the codfw cephosd admin [labs/private] - 10https://gerrit.wikimedia.org/r/1351299 (owner: 10Brouberol) [12:47:39] (03CR) 10Ayounsi: [C:03+1] Interface validator: check Nokia breakout ports have parent [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [12:48:15] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [12:48:19] (03PS1) 10Gkyziridis: ml-services: deploy latest reference-quality image in revision-models [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351300 (https://phabricator.wikimedia.org/T435491) [12:48:21] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, October 05 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1349752 (https://phabricator.wikimedia.org/T439991) (owner: 10Tryvix1509) [12:48:40] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet [12:48:43] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2013.codfw.wmnet [12:49:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2014.codfw.wmnet [12:49:36] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host an-druid1005.eqiad.wmnet with OS bookworm [12:49:59] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2014.codfw.wmnet [12:50:30] (03PS3) 10Clément Goubert: admin_ng: Add lac-lambdas namespace to wikikube [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347653 (https://phabricator.wikimedia.org/T439879) [12:50:30] (03PS6) 10Clément Goubert: article-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) [12:50:52] !log tappof@cumin1004 START - Cookbook sre.dns.netbox [12:51:28] RECOVERY - Confd vcl based reload on cp6012 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [12:51:37] jouncebot: nowandnext [12:51:37] No deployments scheduled for the next 0 hour(s) and 8 minute(s) [12:51:37] In 0 hour(s) and 8 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1300) [12:51:42] (03CR) 10Btullis: [C:03+2] dse-k8s-codfw: Bring 3 new workers into service [puppet] - 10https://gerrit.wikimedia.org/r/1351249 (https://phabricator.wikimedia.org/T430654) (owner: 10Btullis) [12:51:50] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [12:51:55] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:52:13] (03PS3) 10Elukey: profile::pki::multirooca: simplify vhost config [puppet] - 10https://gerrit.wikimedia.org/r/1346615 (https://phabricator.wikimedia.org/T436809) [12:52:13] (03PS3) 10Elukey: Disable pki vhost on port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346616 (https://phabricator.wikimedia.org/T436809) [12:52:14] (03PS5) 10Elukey: profile::pki::multirootca: remove support for port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346617 (https://phabricator.wikimedia.org/T436809) [12:52:14] (03PS3) 10Elukey: Remove mTLS support for pki clients [puppet] - 10https://gerrit.wikimedia.org/r/1346630 (https://phabricator.wikimedia.org/T436809) [12:52:16] (03PS1) 10Elukey: role::pki::multiroot: add profile::lvs::realserver::ipip [puppet] - 10https://gerrit.wikimedia.org/r/1351301 (https://phabricator.wikimedia.org/T436809) [12:52:22] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, October 05 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351194 (https://phabricator.wikimedia.org/T439753) (owner: 10Zabe) [12:53:07] (03PS2) 10Elukey: role::pki::multiroot: add profile::lvs::realserver::ipip [puppet] - 10https://gerrit.wikimedia.org/r/1351301 (https://phabricator.wikimedia.org/T436809) [12:53:07] (03PS4) 10Elukey: profile::pki::multirooca: simplify vhost config [puppet] - 10https://gerrit.wikimedia.org/r/1346615 (https://phabricator.wikimedia.org/T436809) [12:53:07] (03PS4) 10Elukey: Disable pki vhost on port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346616 (https://phabricator.wikimedia.org/T436809) [12:53:08] (03PS6) 10Elukey: profile::pki::multirootca: remove support for port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346617 (https://phabricator.wikimedia.org/T436809) [12:53:09] (03PS4) 10Elukey: Remove mTLS support for pki clients [puppet] - 10https://gerrit.wikimedia.org/r/1346630 (https://phabricator.wikimedia.org/T436809) [12:53:34] (03CR) 10Cathal Mooney: [C:03+1] role::pki::multiroot: add profile::lvs::realserver::ipip [puppet] - 10https://gerrit.wikimedia.org/r/1351301 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [12:53:38] (03CR) 10Clément Goubert: article-feature-counts: add deployment chart files (034 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [12:53:48] (03CR) 10Muehlenhoff: [C:03+1] "Looks good!" [puppet] - 10https://gerrit.wikimedia.org/r/1351274 (https://phabricator.wikimedia.org/T439831) (owner: 10Filippo Giunchedi) [12:54:06] PROBLEM - Host wikikube-worker2014 is DOWN: PING CRITICAL - Packet loss = 100% [12:54:19] (03PS1) 10Filippo Giunchedi: team-wmcs: add backup alerts [alerts] - 10https://gerrit.wikimedia.org/r/1344316 (https://phabricator.wikimedia.org/T428893) [12:54:20] (03PS1) 10Gkyziridis: ml-services: deploy latest liftwing-openapi-server image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351302 (https://phabricator.wikimedia.org/T435491) [12:54:26] (03CR) 10Filippo Giunchedi: [C:03+2] team-wmcs: add backup alerts [alerts] - 10https://gerrit.wikimedia.org/r/1344316 (https://phabricator.wikimedia.org/T428893) (owner: 10Filippo Giunchedi) [12:54:30] (03PS7) 10Clément Goubert: article-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) [12:54:32] (03CR) 10Clément Goubert: article-feature-counts: add deployment chart files (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [12:55:09] !log tappof@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" [12:55:18] (03CR) 10Elukey: [C:03+2] role::pki::multiroot: add profile::lvs::realserver::ipip [puppet] - 10https://gerrit.wikimedia.org/r/1351301 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [12:55:23] !log tappof@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" [12:55:23] !log tappof@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:55:24] !log tappof@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2001.codfw.wmnet [12:55:34] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181#12395428 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by tappof@cumin1004 for hosts: `kafka-logging2001.codfw.wmne... [12:56:00] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1351275 (https://phabricator.wikimedia.org/T439831) (owner: 10Filippo Giunchedi) [12:56:18] !log tappof@cumin1004 START - Cookbook sre.hosts.decommission for hosts kafka-logging2002.codfw.wmnet [12:56:26] (03CR) 10Andrew Bogott: [C:03+1] "Thank you for spending all this time with partman :/" [puppet] - 10https://gerrit.wikimedia.org/r/1351277 (https://phabricator.wikimedia.org/T439831) (owner: 10Filippo Giunchedi) [12:56:27] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [12:57:46] RECOVERY - Host wikikube-worker2014 is UP: PING OK - Packet loss = 0%, RTA = 31.75 ms [12:57:59] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm [12:58:02] (03CR) 10Gkyziridis: [C:03+1] liftwing-studio: add Qwen3.8 to list of models [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348836 (owner: 10Ilias Sarantopoulos) [12:58:40] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [12:59:03] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2013.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:59:50] !log btullis@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [13:00:00] FIRING: [6x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker2009 has a BGP session which is not in the 'established' state. [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: Your horoscope predicts another UTC afternoon backport window deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1300). [13:00:05] Volker_E, annet, Hide_on_rosie, and zabe: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:29] o/ [13:01:01] o/ [13:01:08] o/ [13:01:16] !log btullis@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [13:01:18] annet: Hide_on_rosie: do you need a deployer? [13:01:25] zabe: yes please [13:01:27] yes [13:01:32] Ok, I can do it [13:01:35] (03CR) 10Zabe: [C:03+2] Enable ReadingList CTA on Minerva for all reading list wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346053 (https://phabricator.wikimedia.org/T438779) (owner: 10Jdlrobson) [13:01:37] (03CR) 10Zabe: [C:03+2] Update pulsating blue dot behavior for anonymous bookmark [extensions/ReadingLists] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346793 (https://phabricator.wikimedia.org/T438779) (owner: 10Anne Tomasevich) [13:01:50] !log tappof@cumin1004 START - Cookbook sre.dns.netbox [13:02:57] (03CR) 10Filippo Giunchedi: [C:03+2] install_server: remove unused recipes [puppet] - 10https://gerrit.wikimedia.org/r/1351274 (https://phabricator.wikimedia.org/T439831) (owner: 10Filippo Giunchedi) [13:03:06] (03CR) 10Filippo Giunchedi: [C:03+2] install_server: create efi partitions in the lvm case [puppet] - 10https://gerrit.wikimedia.org/r/1351275 (https://phabricator.wikimedia.org/T439831) (owner: 10Filippo Giunchedi) [13:04:03] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2013.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:04:25] (03CR) 10Zabe: [C:03+2] throttle.php: Lift IP cap for edit-a-thon [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1349752 (https://phabricator.wikimedia.org/T439991) (owner: 10Tryvix1509) [13:04:50] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2009.codfw.wmnet [13:05:00] RESOLVED: [6x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker2009 has a BGP session which is not in the 'established' state. [13:05:20] (03CR) 10Xcollazo: [C:03+1] stream: reconcile + HTML enrichment streams [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351220 (https://phabricator.wikimedia.org/T415264) (owner: 10JavierMonton) [13:05:21] (03PS3) 10Btullis: admin_ng: move dse-k8s-codfw to ceph-csi-cephfs v3.14.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343625 (https://phabricator.wikimedia.org/T407166) [13:05:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:05:29] (03Merged) 10jenkins-bot: throttle.php: Lift IP cap for edit-a-thon [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1349752 (https://phabricator.wikimedia.org/T439991) (owner: 10Tryvix1509) [13:05:38] !log tappof@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" [13:05:50] !log tappof@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" [13:05:50] !log tappof@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:05:51] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1349752|throttle.php: Lift IP cap for edit-a-thon (T439991)]] [13:05:51] !log tappof@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2002.codfw.wmnet [13:05:55] T439991: Lift IP cap on these dates 2026-10-15 and 2026-10-29 for edit-a-thon for eswiki, commons and wikidata - https://phabricator.wikimedia.org/T439991 [13:05:56] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181#12395467 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by tappof@cumin1004 for hosts: `kafka-logging2002.codfw.wmne... [13:06:32] PROBLEM - Confd vcl based reload on cp6013 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [13:06:36] (03PS1) 10Brouberol: Add fixtures for the codfw cephosd admin [labs/private] - 10https://gerrit.wikimedia.org/r/1351305 [13:06:53] (03CR) 10Brouberol: [C:03+2] Add fixtures for the codfw cephosd admin [labs/private] - 10https://gerrit.wikimedia.org/r/1351305 (owner: 10Brouberol) [13:06:54] (03CR) 10Brouberol: [V:03+2 C:03+2] Add fixtures for the codfw cephosd admin [labs/private] - 10https://gerrit.wikimedia.org/r/1351305 (owner: 10Brouberol) [13:08:48] (03PS3) 10Brouberol: ceph::admin::cephosd: install and configure opentofu [puppet] - 10https://gerrit.wikimedia.org/r/1351303 (https://phabricator.wikimedia.org/T435595) [13:09:22] (03PS4) 10Brouberol: ceph::admin::cephosd: install and configure opentofu [puppet] - 10https://gerrit.wikimedia.org/r/1351303 (https://phabricator.wikimedia.org/T435595) [13:09:30] (03Merged) 10jenkins-bot: Update pulsating blue dot behavior for anonymous bookmark [extensions/ReadingLists] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346793 (https://phabricator.wikimedia.org/T438779) (owner: 10Anne Tomasevich) [13:09:49] (03CR) 10Zabe: [C:03+2] "retry" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346053 (https://phabricator.wikimedia.org/T438779) (owner: 10Jdlrobson) [13:10:13] (03CR) 10Muehlenhoff: [C:03+1] "Looks great!" [puppet] - 10https://gerrit.wikimedia.org/r/1351276 (https://phabricator.wikimedia.org/T439831) (owner: 10Filippo Giunchedi) [13:10:20] !log tappof@cumin1004 START - Cookbook sre.hosts.decommission for hosts kafka-logging2003.codfw.wmnet [13:10:34] RECOVERY - Confd vcl based reload on cp6013 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [13:10:56] (03Merged) 10jenkins-bot: Enable ReadingList CTA on Minerva for all reading list wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346053 (https://phabricator.wikimedia.org/T438779) (owner: 10Jdlrobson) [13:11:44] (03CR) 10Muehlenhoff: install_server: use raid1-jbod-efi for cloudcephosd >= 1053 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1351277 (https://phabricator.wikimedia.org/T439831) (owner: 10Filippo Giunchedi) [13:11:50] (03PS1) 10Elukey: Revert^2 "Move pki.discovery.wmnet to lvs_setup state" [puppet] - 10https://gerrit.wikimedia.org/r/1351310 [13:11:54] !log zabe@deploy1003 zabe, tryvix1509: Backport for [[gerrit:1349752|throttle.php: Lift IP cap for edit-a-thon (T439991)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:11:56] T439991: Lift IP cap on these dates 2026-10-15 and 2026-10-29 for edit-a-thon for eswiki, commons and wikidata - https://phabricator.wikimedia.org/T439991 [13:12:03] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet [13:12:06] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2010.codfw.wmnet [13:12:47] (03PS5) 10Brouberol: ceph::admin::cephosd: install and configure opentofu [puppet] - 10https://gerrit.wikimedia.org/r/1351303 (https://phabricator.wikimedia.org/T435595) [13:12:53] !log zabe@deploy1003 zabe, tryvix1509: Continuing with deployment [13:13:27] tappof@cumin1004 decommission (PID 2919494) is awaiting input [13:13:32] PROBLEM - Confd vcl based reload on cp6010 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [13:13:34] PROBLEM - Confd vcl based reload on cp6014 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [13:14:08] (03PS6) 10Brouberol: ceph::admin::cephosd: install and configure opentofu [puppet] - 10https://gerrit.wikimedia.org/r/1351303 (https://phabricator.wikimedia.org/T435595) [13:14:50] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:15:03] (03CR) 10Brouberol: [V:03+1] "PCC SUCCESS (CORE_DIFF 1 DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1351303 (https://phabricator.wikimedia.org/T435595) (owner: 10Brouberol) [13:15:36] (03CR) 10Btullis: ceph::admin::cephosd: install and configure opentofu (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1351303 (https://phabricator.wikimedia.org/T435595) (owner: 10Brouberol) [13:15:58] (03PS2) 10Zabe: Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351194 (https://phabricator.wikimedia.org/T439753) [13:16:27] (03CR) 10JavierMonton: [C:03+2] stream: reconcile + HTML enrichment streams [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351220 (https://phabricator.wikimedia.org/T415264) (owner: 10JavierMonton) [13:16:34] RECOVERY - Confd vcl based reload on cp6014 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [13:16:34] RECOVERY - Confd vcl based reload on cp6010 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [13:18:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:18:44] (03CR) 10Brouberol: [V:03+1] ceph::admin::cephosd: install and configure opentofu (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1351303 (https://phabricator.wikimedia.org/T435595) (owner: 10Brouberol) [13:18:48] (03Merged) 10jenkins-bot: stream: reconcile + HTML enrichment streams [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351220 (https://phabricator.wikimedia.org/T415264) (owner: 10JavierMonton) [13:19:00] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:19:12] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet [13:19:15] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2011.codfw.wmnet [13:19:25] (03CR) 10Btullis: [C:03+1] ceph::admin::cephosd: install and configure opentofu [puppet] - 10https://gerrit.wikimedia.org/r/1351303 (https://phabricator.wikimedia.org/T435595) (owner: 10Brouberol) [13:19:32] PROBLEM - Confd vcl based reload on cp6015 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [13:19:53] (03CR) 10Btullis: [C:03+1] ceph::admin::cephosd: install and configure opentofu (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1351303 (https://phabricator.wikimedia.org/T435595) (owner: 10Brouberol) [13:20:09] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [13:20:13] !log oblivian@cumin1004 END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:20:20] (03CR) 10Brouberol: [V:03+1 C:03+2] ceph::admin::cephosd: install and configure opentofu [puppet] - 10https://gerrit.wikimedia.org/r/1351303 (https://phabricator.wikimedia.org/T435595) (owner: 10Brouberol) [13:20:30] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet [13:20:32] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet [13:20:39] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1349752|throttle.php: Lift IP cap for edit-a-thon (T439991)]] (duration: 14m 48s) [13:20:42] T439991: Lift IP cap on these dates 2026-10-15 and 2026-10-29 for edit-a-thon for eswiki, commons and wikidata - https://phabricator.wikimedia.org/T439991 [13:21:13] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1346053|Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793|Update pulsating blue dot behavior for anonymous bookmark (T438779)]] [13:21:16] T438779: Mobile CTA: Turn on the anonymous mobile bookmark experience for all Wikipedia users - https://phabricator.wikimedia.org/T438779 [13:21:17] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [13:21:34] PROBLEM - Confd vcl based reload on cp6009 is CRITICAL: reload-vcl failed to run since 0h, 4 minutes. https://wikitech.wikimedia.org/wiki/Varnish [13:22:07] !log dkertesz@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [13:22:10] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [13:23:34] RECOVERY - Confd vcl based reload on cp6009 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [13:23:34] RECOVERY - Confd vcl based reload on cp6015 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [13:23:55] Hide_on_rosie: your patch should be live :) [13:24:10] Thanks! [13:24:37] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [13:24:53] !log dkertesz@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [13:25:27] !log zabe@deploy1003 jdlrobson, zabe, annet: Backport for [[gerrit:1346053|Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793|Update pulsating blue dot behavior for anonymous bookmark (T438779)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:25:34] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [13:25:49] annet: could you test the patch? [13:26:17] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet [13:26:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2015.codfw.wmnet [13:27:07] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2015.codfw.wmnet [13:27:14] zabe: LGTM! [13:27:26] nice! [13:27:27] !log zabe@deploy1003 jdlrobson, zabe, annet: Continuing with deployment [13:28:43] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:29:04] jouncebot: nowandnext [13:29:04] For the next 0 hour(s) and 30 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1300) [13:29:04] In 0 hour(s) and 30 minute(s): Distributed search queries experiment (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1400) [13:29:05] (03CR) 10Zabe: [C:03+2] Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351194 (https://phabricator.wikimedia.org/T439753) (owner: 10Zabe) [13:30:37] (03PS3) 10Federico Ceratto: team-data-persistence: Alarm on stale data in Zarcillo [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) [13:30:42] (03Merged) 10jenkins-bot: Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351194 (https://phabricator.wikimedia.org/T439753) (owner: 10Zabe) [13:31:10] (03CR) 10Fabfur: [C:03+1] Revert^2 "Move pki.discovery.wmnet to lvs_setup state" [puppet] - 10https://gerrit.wikimedia.org/r/1351310 (owner: 10Elukey) [13:31:40] (03CR) 10Dpogorzelski: [C:04-2] liftwing-studio: add Qwen3.8 to list of models [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348836 (owner: 10Ilias Sarantopoulos) [13:32:16] PROBLEM - Host wikikube-worker2015 is DOWN: PING CRITICAL - Packet loss = 100% [13:32:32] (03CR) 10Elukey: [C:03+2] Revert^2 "Move pki.discovery.wmnet to lvs_setup state" [puppet] - 10https://gerrit.wikimedia.org/r/1351310 (owner: 10Elukey) [13:32:36] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346053|Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793|Update pulsating blue dot behavior for anonymous bookmark (T438779)]] (duration: 11m 23s) [13:32:39] T438779: Mobile CTA: Turn on the anonymous mobile bookmark experience for all Wikipedia users - https://phabricator.wikimedia.org/T438779 [13:32:51] annet: done :) [13:32:57] (03PS1) 10DCausse: Find canonical pages on special page completion search [core] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351316 (https://phabricator.wikimedia.org/T440151) [13:32:57] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1351194|Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki (T439753)]] [13:33:00] T439753: Switch reads/writes of wbc_entity_usage from local cluster to x1 - https://phabricator.wikimedia.org/T439753 [13:33:13] zabe: awesome, thank you! [13:33:23] yw! [13:33:36] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [core] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351316 (https://phabricator.wikimedia.org/T440151) (owner: 10DCausse) [13:34:00] (03PS2) 10Filippo Giunchedi: install_server: new script detect-os-device.sh [puppet] - 10https://gerrit.wikimedia.org/r/1351276 (https://phabricator.wikimedia.org/T439831) [13:34:00] (03PS2) 10Filippo Giunchedi: install_server: use raid1-jbod-efi for new cloudcephosd hosts [puppet] - 10https://gerrit.wikimedia.org/r/1351277 (https://phabricator.wikimedia.org/T439831) [13:34:11] (03CR) 10Filippo Giunchedi: install_server: use raid1-jbod-efi for new cloudcephosd hosts (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1351277 (https://phabricator.wikimedia.org/T439831) (owner: 10Filippo Giunchedi) [13:34:26] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:34:31] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet [13:35:16] (03CR) 10Btullis: [C:03+2] dse-k8s-codfw: Sign service account tokens with an RSA key [puppet] - 10https://gerrit.wikimedia.org/r/1347694 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [13:35:29] !log tappof@cumin1004 START - Cookbook sre.dns.netbox [13:35:53] RECOVERY - Host wikikube-worker2015 is UP: PING OK - Packet loss = 0%, RTA = 30.73 ms [13:36:33] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet [13:36:56] !log zabe@deploy1003 zabe: Backport for [[gerrit:1351194|Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:37:08] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet [13:37:15] !log restart pybal on lvs201[34] to pick up a new LVS service [13:37:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:37:37] !log zabe@deploy1003 zabe: Continuing with deployment [13:38:02] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet [13:38:15] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet [13:38:24] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2015.codfw.wmnet [13:39:03] FIRING: KubernetesCalicoDown: wikikube-worker2015.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2015.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:39:22] !log tappof@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" [13:39:23] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2016.codfw.wmnet [13:39:36] !log tappof@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" [13:39:36] !log tappof@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:39:37] !log tappof@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2003.codfw.wmnet [13:39:43] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181#12395669 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by tappof@cumin1004 for hosts: `kafka-logging2003.codfw.wmne... [13:39:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2016.codfw.wmnet [13:40:05] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181#12395671 (10tappof) [13:40:28] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:41:46] (03PS1) 10Slyngshede: data.yaml offboarding of cmadeo [puppet] - 10https://gerrit.wikimedia.org/r/1351320 [13:41:59] (03CR) 10Marostegui: team-data-persistence: Alarm on stale data in Zarcillo (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) (owner: 10Federico Ceratto) [13:42:08] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1351277 (https://phabricator.wikimedia.org/T439831) (owner: 10Filippo Giunchedi) [13:42:10] !log btullis@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [13:42:40] !log restart pybal on lvs10[20,19] to pick up a new LVS service [13:42:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:42:46] (03CR) 10Btullis: [C:03+2] admin_ng: move dse-k8s-codfw to ceph-csi-cephfs v3.14.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343625 (https://phabricator.wikimedia.org/T407166) (owner: 10Btullis) [13:42:56] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1351194|Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki (T439753)]] (duration: 09m 59s) [13:42:59] T439753: Switch reads/writes of wbc_entity_usage from local cluster to x1 - https://phabricator.wikimedia.org/T439753 [13:43:15] (03PS1) 10Btullis: install_server: Rebuild an-druid100[67] RAID10 across all 8 disks [puppet] - 10https://gerrit.wikimedia.org/r/1351321 (https://phabricator.wikimedia.org/T440033) [13:43:18] (03PS1) 10Btullis: install_server: Reuse the 8 disk recipe for an-druid100[67] [puppet] - 10https://gerrit.wikimedia.org/r/1351322 (https://phabricator.wikimedia.org/T440033) [13:43:24] Dreamy_Jazz: I am done, in case you wanted to deploy something :) [13:43:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:43:53] (03CR) 10JHathaway: [C:03+2] stdlib: replace deprecated function names [puppet] - 10https://gerrit.wikimedia.org/r/1348986 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [13:44:03] RESOLVED: KubernetesCalicoDown: wikikube-worker2015.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2015.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:44:12] Thanks, yeah [13:44:17] PROBLEM - Host wikikube-worker2016 is DOWN: PING CRITICAL - Packet loss = 100% [13:44:22] jouncebot: nowandnext [13:44:22] For the next 0 hour(s) and 15 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1300) [13:44:22] In 0 hour(s) and 15 minute(s): Distributed search queries experiment (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1400) [13:45:25] (03CR) 10Bking: [C:03+1] install_server: Rebuild an-druid100[67] RAID10 across all 8 disks [puppet] - 10https://gerrit.wikimedia.org/r/1351321 (https://phabricator.wikimedia.org/T440033) (owner: 10Btullis) [13:45:56] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:46:31] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet [13:46:38] Using scap but logs won't appear for a moment [13:46:38] btullis@cumin1004 reimage (PID 2917878) is awaiting input [13:46:42] (03PS1) 10Elukey: Move the pki lvs service to state production [puppet] - 10https://gerrit.wikimedia.org/r/1351325 (https://phabricator.wikimedia.org/T436809) [13:47:59] (03PS2) 10Ozge: liftwing-studio: add Qwen3.8 to list of models [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348836 (owner: 10Ilias Sarantopoulos) [13:48:24] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:48:30] (03CR) 10Btullis: [C:03+2] install_server: Rebuild an-druid100[67] RAID10 across all 8 disks [puppet] - 10https://gerrit.wikimedia.org/r/1351321 (https://phabricator.wikimedia.org/T440033) (owner: 10Btullis) [13:48:45] RECOVERY - Host wikikube-worker2016 is UP: PING OK - Packet loss = 0%, RTA = 30.33 ms [13:48:51] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet [13:48:53] (03CR) 10Muehlenhoff: [C:03+1] "Looks good!" [puppet] - 10https://gerrit.wikimedia.org/r/1351320 (owner: 10Slyngshede) [13:49:03] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2015.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:49:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet [13:49:34] (03CR) 10Gkyziridis: [C:03+1] "LGTM!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348836 (owner: 10Ilias Sarantopoulos) [13:49:48] (03PS1) 10Tiziano Fogli: alertamanager: add task receiver for mediawiki-engineering [puppet] - 10https://gerrit.wikimedia.org/r/1351327 (https://phabricator.wikimedia.org/T435644) [13:49:48] (03PS3) 10Ilias Sarantopoulos: liftwing-studio: add Qwen3.8 to list of models [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348836 [13:49:49] (03CR) 10Muehlenhoff: [C:03+2] Remove profile::base::cuminunpriv from installservers [puppet] - 10https://gerrit.wikimedia.org/r/1348724 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [13:50:12] (03PS1) 10Daniel Kertesz: varnish: redact logs for 5XX response + Set-Cookie [puppet] - 10https://gerrit.wikimedia.org/r/1351328 (https://phabricator.wikimedia.org/T440187) [13:50:20] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet [13:50:28] (03PS4) 10Ozge: liftwing-studio: add Qwen3.8 to list of models [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348836 (owner: 10Ilias Sarantopoulos) [13:50:34] (03CR) 10Dpogorzelski: [C:03+1] "Copied votes on follow-up patch sets have been updated:" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348836 (owner: 10Ilias Sarantopoulos) [13:51:12] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet [13:51:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2016.codfw.wmnet [13:51:42] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181#12395735 (10tappof) [13:51:55] (03CR) 10Ozge: [C:03+2] liftwing-studio: add Qwen3.8 to list of models [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348836 (owner: 10Ilias Sarantopoulos) [13:51:58] (03CR) 10Slyngshede: [C:03+2] data.yaml offboarding of cmadeo [puppet] - 10https://gerrit.wikimedia.org/r/1351320 (owner: 10Slyngshede) [13:52:05] (03PS2) 10Btullis: install_server: Reuse the 8 disk recipe for an-druid100[67] [puppet] - 10https://gerrit.wikimedia.org/r/1351322 (https://phabricator.wikimedia.org/T440033) [13:52:10] (03CR) 10Bking: [C:03+1] install_server: Reuse the 8 disk recipe for an-druid100[67] [puppet] - 10https://gerrit.wikimedia.org/r/1351322 (https://phabricator.wikimedia.org/T440033) (owner: 10Btullis) [13:52:12] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181#12395738 (10tappof) a:03tappof [13:52:18] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181#12395739 (10tappof) a:05tappof→03None [13:53:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2017.codfw.wmnet [13:53:49] (03CR) 10Klausman: [C:03+1] install_server: Reuse the 8 disk recipe for an-druid100[67] [puppet] - 10https://gerrit.wikimedia.org/r/1351322 (https://phabricator.wikimedia.org/T440033) (owner: 10Btullis) [13:53:56] (03Merged) 10jenkins-bot: admin_ng: move dse-k8s-codfw to ceph-csi-cephfs v3.14.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343625 (https://phabricator.wikimedia.org/T407166) (owner: 10Btullis) [13:54:00] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2017.codfw.wmnet [13:54:03] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2015.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:54:45] (03CR) 10Btullis: dumps: web: Add a brownout for v1 meta-current and meta-history (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1346007 (https://phabricator.wikimedia.org/T437461) (owner: 10Btullis) [13:54:53] !log btullis@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [13:56:21] !log btullis@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [13:56:52] (03PS1) 10SBassett: Bump security landing page image version to 2026-10-05-135051 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351331 [13:57:07] oblivian@cumin1004 provision (PID 2927889) is awaiting input [13:57:16] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:57:16] !log btullis@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [13:58:52] !log dreamyjazz Deployed security patch for T439968 [13:59:28] (03PS1) 10Filippo Giunchedi: team-wmcs: deploy backup alerts to 'ops' Prometheus [alerts] - 10https://gerrit.wikimedia.org/r/1351332 (https://phabricator.wikimedia.org/T428893) [14:00:04] (03PS1) 10JavierMonton: stream: reconcile + enrichment streams [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351333 (https://phabricator.wikimedia.org/T420353) [14:00:04] Deploy window Distributed search queries experiment (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1400) [14:00:15] PROBLEM - Host wikikube-worker2017 is DOWN: PING CRITICAL - Packet loss = 100% [14:01:59] o/ [14:02:53] (03Merged) 10jenkins-bot: liftwing-studio: add Qwen3.8 to list of models [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348836 (owner: 10Ilias Sarantopoulos) [14:03:04] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:03:19] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [14:03:26] (03CR) 10Filippo Giunchedi: [C:03+2] team-wmcs: deploy backup alerts to 'ops' Prometheus [alerts] - 10https://gerrit.wikimedia.org/r/1351332 (https://phabricator.wikimedia.org/T428893) (owner: 10Filippo Giunchedi) [14:03:27] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12395783 (10MoritzMuehlenhoff) [14:03:39] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2017.codfw.wmnet [14:03:42] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2017.codfw.wmnet [14:03:46] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [14:03:47] RECOVERY - Host wikikube-worker2017 is UP: PING OK - Packet loss = 0%, RTA = 31.66 ms [14:03:51] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2018.codfw.wmnet [14:04:27] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2018.codfw.wmnet [14:04:58] (03PS1) 10Slyngshede: switchdc: Skip set-readonly in live-test [cookbooks] - 10https://gerrit.wikimedia.org/r/1351336 (https://phabricator.wikimedia.org/T436846) [14:05:29] for the next 3hours ("Distributed search queries experiment" window) we'll be deploying a set of mw-config patches to test search perfs [14:07:22] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1006.eqiad.wmnet with OS bookworm [14:07:35] oblivian@cumin1004 provision (PID 2928412) is awaiting input [14:07:43] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm [14:08:08] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351207 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [14:09:05] (03Merged) 10jenkins-bot: search: test impact of the MLR window size [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351207 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [14:09:18] !log dcausse@deploy1003 Started scap sync-world: Backport for [[gerrit:1351207|search: test impact of the MLR window size (T438984)]] [14:09:22] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [14:09:39] (03PS1) 10Gmodena: wdqs: proxy: version bump. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351340 (https://phabricator.wikimedia.org/T438488) [14:10:00] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:10:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:10:23] (03CR) 10AKhatun: [C:03+1] "lgtm!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351333 (https://phabricator.wikimedia.org/T420353) (owner: 10JavierMonton) [14:10:51] (03CR) 10JavierMonton: [C:03+2] stream: reconcile + enrichment streams [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351333 (https://phabricator.wikimedia.org/T420353) (owner: 10JavierMonton) [14:12:10] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [14:12:26] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [14:12:33] PROBLEM - Host wikikube-worker2018 is DOWN: PING CRITICAL - Packet loss = 100% [14:13:21] !log dcausse@deploy1003 dcausse: Backport for [[gerrit:1351207|search: test impact of the MLR window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:13:26] (03Merged) 10jenkins-bot: stream: reconcile + enrichment streams [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351333 (https://phabricator.wikimedia.org/T420353) (owner: 10JavierMonton) [14:15:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:15:14] !log dcausse@deploy1003 dcausse: Continuing with deployment [14:15:45] RECOVERY - Host wikikube-worker2018 is UP: PING OK - Packet loss = 0%, RTA = 30.34 ms [14:16:19] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:16:54] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2018.codfw.wmnet [14:16:56] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2018.codfw.wmnet [14:17:05] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2033.codfw.wmnet [14:18:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2033.codfw.wmnet [14:18:33] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:20:06] !log btullis@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-druid1006.eqiad.wmnet with OS bookworm [14:20:22] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm [14:20:30] !log dcausse@deploy1003 Finished scap sync-world: Backport for [[gerrit:1351207|search: test impact of the MLR window size (T438984)]] (duration: 11m 12s) [14:20:33] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [14:21:17] PROBLEM - Host wikikube-worker2033 is DOWN: PING CRITICAL - Packet loss = 100% [14:21:49] (03PS2) 10Hnowlan: mediawiki-cache-warmup: log to file [puppet] - 10https://gerrit.wikimedia.org/r/1345119 (https://phabricator.wikimedia.org/T440193) [14:22:01] (03PS2) 10Hnowlan: mediawiki-cache-warmup: use ThreadPoolExecutor for workers [puppet] - 10https://gerrit.wikimedia.org/r/1345079 (https://phabricator.wikimedia.org/T440193) [14:22:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:22:17] (03CR) 10Elukey: [C:03+2] Move the pki lvs service to state production [puppet] - 10https://gerrit.wikimedia.org/r/1351325 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [14:22:17] (03CR) 10Clément Goubert: [C:03+1] rest-gateway: Prepare the cluster specifier plugin for XWD routing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335285 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [14:24:15] RECOVERY - Host wikikube-worker2033 is UP: PING OK - Packet loss = 0%, RTA = 30.38 ms [14:24:19] (03CR) 10Jelto: [C:03+1] "looks mostly good" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341857 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [14:24:40] 10ops-eqiad, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T440195 (10phaultfinder) 03NEW [14:24:51] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:25:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2033.codfw.wmnet [14:25:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2033.codfw.wmnet [14:25:38] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2034.codfw.wmnet [14:26:14] (03PS1) 10Elukey: Revert "Remove the pki discovery settings" [dns] - 10https://gerrit.wikimedia.org/r/1351349 [14:26:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2034.codfw.wmnet [14:27:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:27:10] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351208 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [14:27:10] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351222 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [14:27:35] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:28:21] (03Merged) 10jenkins-bot: Revert "search: test impact of the MLR window size" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351208 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [14:28:24] (03Merged) 10jenkins-bot: search: test impact of MLR [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351222 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [14:28:37] !log dcausse@deploy1003 Started scap sync-world: Backport for [[gerrit:1351208|Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222|search: test impact of MLR (T438984)]] [14:28:40] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [14:29:15] (03PS4) 10Blake: mw-experimental: enable native_sidecars on mw-experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333740 (https://phabricator.wikimedia.org/T417800) [14:30:05] Deploy window Distributed search queries experiment (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1400) [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1430) [14:30:07] PROBLEM - Host wikikube-worker2034 is DOWN: PING CRITICAL - Packet loss = 100% [14:30:17] !log btullis@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [14:30:29] (03CR) 10Rsilvola: [C:03+2] Bump security landing page image version to 2026-10-05-135051 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351331 (owner: 10SBassett) [14:30:34] !log btullis@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [14:32:27] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage [14:32:42] !log dcausse@deploy1003 dcausse: Backport for [[gerrit:1351208|Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222|search: test impact of MLR (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:33:12] (03Merged) 10jenkins-bot: Bump security landing page image version to 2026-10-05-135051 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351331 (owner: 10SBassett) [14:33:23] !log dcausse@deploy1003 dcausse: Continuing with deployment [14:33:24] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:33:45] RECOVERY - Host wikikube-worker2034 is UP: PING OK - Packet loss = 0%, RTA = 31.69 ms [14:33:59] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet [14:34:03] FIRING: KubernetesCalicoDown: wikikube-worker2034.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2034.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:34:53] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2034.codfw.wmnet [14:35:23] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: eqiad row A&B host migration details request for Infrastructure Foundations - https://phabricator.wikimedia.org/T432646#12396020 (10cmooney) p:05Triage→03Medium a:03cmooney [14:35:28] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet [14:35:30] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2034.codfw.wmnet [14:35:40] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2035.codfw.wmnet [14:36:05] !log cdobbins@cumin1004 START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox [14:36:20] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2035.codfw.wmnet [14:36:38] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:36:41] 06SRE, 06Infrastructure-Foundations: Unresolved failures from url-downloader - https://phabricator.wikimedia.org/T439560#12396024 (10cmooney) 05Open→03Resolved a:03cmooney I'm going to close this one for now as we know the cause. [14:37:05] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12396029 (10MoritzMuehlenhoff) p:05Triage→03Medium a:03MoritzMuehlenhoff [14:37:16] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage [14:37:41] !log sbassett@deploy1003 helmfile [codfw] START helmfile.d/services/miscweb: apply [14:37:56] 06SRE, 06Infrastructure-Foundations, 06Traffic, 07LDAP, 13Patch-For-Review: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730#12396033 (10MoritzMuehlenhoff) p:05Triage→03Medium a:03MoritzMuehlenhoff [14:38:03] !log sbassett@deploy1003 helmfile [codfw] DONE helmfile.d/services/miscweb: apply [14:38:18] !log sbassett@deploy1003 helmfile [eqiad] START helmfile.d/services/miscweb: apply [14:38:41] !log dcausse@deploy1003 Finished scap sync-world: Backport for [[gerrit:1351208|Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222|search: test impact of MLR (T438984)]] (duration: 10m 04s) [14:38:41] 10ops-codfw, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181#12396038 (10Jclark-ctr) [14:38:44] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [14:38:46] !log sbassett@deploy1003 helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [14:39:08] RESOLVED: KubernetesCalicoDown: wikikube-worker2034.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2034.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:41:17] PROBLEM - Host wikikube-worker2035 is DOWN: PING CRITICAL - Packet loss = 100% [14:42:47] !log dkertesz@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [14:42:47] (03CR) 10Ssingh: "This is ready to be merged with the only caveat that once we merge it, we should do a test run and depool a site and DNS hosts. An actual " [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 (https://phabricator.wikimedia.org/T440168) (owner: 10Ssingh) [14:42:51] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [14:43:02] !log osleger@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [14:43:23] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:43:25] (03CR) 10Kevin Bazira: [C:03+1] ml-services: deploy latest reference-quality image in revision-models [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351300 (https://phabricator.wikimedia.org/T435491) (owner: 10Gkyziridis) [14:43:40] !log osleger@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [14:43:41] !log osleger@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [14:44:22] (03CR) 10Kevin Bazira: [C:03+1] ml-services: deploy latest liftwing-openapi-server image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351302 (https://phabricator.wikimedia.org/T435491) (owner: 10Gkyziridis) [14:44:23] !log osleger@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [14:44:31] !log dancy@deploy1003 Installing scap version "4.294.0" for 3 host(s) [14:44:42] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351223 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [14:44:42] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351209 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [14:44:53] RECOVERY - Host wikikube-worker2035 is UP: PING OK - Packet loss = 0%, RTA = 30.42 ms [14:45:00] 06SRE, 06Infrastructure-Foundations, 10observability, 10Puppet-Infrastructure: Add monitoring for the puppet-netbox repository - https://phabricator.wikimedia.org/T310639#12396073 (10LSobanski) 05Open→03Resolved a:03LSobanski The hook to call sre.puppet.netbox-sync has since been added to the sre... [14:45:25] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:45:39] !log osleger@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [14:45:41] !log osleger@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [14:45:43] !log osleger@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [14:45:46] !log osleger@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [14:45:48] (03Merged) 10jenkins-bot: Revert "search: test impact of MLR" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351223 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [14:45:51] (03Merged) 10jenkins-bot: search: test impact of the function score window size [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351209 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [14:46:00] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet [14:46:15] !log btullis@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [14:46:28] !log dancy@deploy1003 Installation of scap version "4.294.0" completed for 3 hosts [14:46:53] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [14:46:54] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2035.codfw.wmnet [14:46:58] !log btullis@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [14:47:08] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [14:47:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet [14:47:31] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2035.codfw.wmnet [14:47:40] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2036.codfw.wmnet [14:47:54] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [14:48:05] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [14:48:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2036.codfw.wmnet [14:48:43] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply [14:48:44] !log dcausse@deploy1003 Started scap sync-world: Backport for [[gerrit:1351223|Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209|search: test impact of the function score window size (T438984)]] [14:48:47] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [14:48:52] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply [14:49:06] 06SRE, 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE: Reduce Kerberos logs produced by Presto - https://phabricator.wikimedia.org/T353802#12396090 (10LSobanski) [14:49:22] !log osleger@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [14:49:25] !log osleger@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [14:49:26] !log osleger@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [14:49:30] !log osleger@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [14:50:53] 06SRE, 06Fundraising-Backlog, 06Infrastructure-Foundations, 10Mail: Add granularity limiter (g=) to wikimedia.org DKIM record(s) - https://phabricator.wikimedia.org/T140316#12396097 (10LSobanski) @greg could you take a look and see if this is still a valid request? [14:51:22] oblivian@cumin1004 provision (PID 2938134) is awaiting input [14:51:44] (03PS1) 10Elukey: profile::k8s::deployment_server::global_config: add PKI LVS IPs [puppet] - 10https://gerrit.wikimedia.org/r/1351364 (https://phabricator.wikimedia.org/T436809) [14:52:37] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:52:54] !log dcausse@deploy1003 dcausse: Backport for [[gerrit:1351223|Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209|search: test impact of the function score window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:53:08] 06SRE, 06Infrastructure-Foundations: Adapt firewall logging for nftables - https://phabricator.wikimedia.org/T348736#12396103 (10LSobanski) Let's review if the correct scope here is to add ulogd for nftables or remove it for ferm. [14:53:17] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [14:53:21] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [14:53:27] !log dcausse@deploy1003 dcausse: Continuing with deployment [14:53:38] 06SRE, 06Infrastructure-Foundations: Adapt firewall logging for nftables - https://phabricator.wikimedia.org/T348736#12396104 (10LSobanski) p:05Medium→03Low [14:53:41] 10ops-codfw, 10ops-eqiad, 06SRE, 06DC-Ops: Audit Serial Console Connections (2026) - https://phabricator.wikimedia.org/T438876#12396105 (10Jhancock.wm) **scs-a1-codfw: ** ps1-b7-codfw on port 15 was being used to console to a test server. i moved it back for now. if they yell, i'll run a new connection ins... [14:56:17] PROBLEM - Host wikikube-worker2036 is DOWN: PING CRITICAL - Packet loss = 100% [14:56:26] 10ops-codfw, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181#12396112 (10Jhancock.wm) a:03Jhancock.wm [14:57:57] (03PS1) 10Ozge: inference-services: increasing max replica of llm-qwen36-27b to 3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351366 [14:58:56] 10ops-codfw, 10ops-eqiad, 06SRE, 06DC-Ops: Audit Serial Console Connections (2026) - https://phabricator.wikimedia.org/T438876#12396159 (10cmooney) >>! In T438876#12396105, @Jhancock.wm wrote: > @cmooney the console port on the atlas on port 47, there is no cable connected to it and it doesn't seem to have... [14:58:56] !log dcausse@deploy1003 Finished scap sync-world: Backport for [[gerrit:1351223|Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209|search: test impact of the function score window size (T438984)]] (duration: 10m 12s) [14:58:57] (03PS1) 10Elukey: profile::netops: move pki1003 references to pki.discovery [puppet] - 10https://gerrit.wikimedia.org/r/1351367 (https://phabricator.wikimedia.org/T436809) [14:59:00] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [14:59:22] (03CR) 10Ozge: "Increased the maxReplica to 3. min is still 1." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351366 (owner: 10Ozge) [14:59:37] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T440195#12396167 (10phaultfinder) [15:00:20] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1006.eqiad.wmnet with OS bookworm [15:00:40] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:00:43] (03PS1) 10Elukey: admin_ng: move cfssl-issuer's config to pki.discovery in staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351368 (https://phabricator.wikimedia.org/T436809) [15:00:47] RECOVERY - Host wikikube-worker2036 is UP: PING OK - Packet loss = 0%, RTA = 31.71 ms [15:01:14] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2036.codfw.wmnet [15:01:18] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2036.codfw.wmnet [15:01:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2037.codfw.wmnet [15:02:05] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2037.codfw.wmnet [15:02:16] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [15:02:23] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:02:24] (03CR) 10Elukey: [C:03+1] alertamanager: add task receiver for mediawiki-engineering [puppet] - 10https://gerrit.wikimedia.org/r/1351327 (https://phabricator.wikimedia.org/T435644) (owner: 10Tiziano Fogli) [15:02:27] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [15:03:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351210 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [15:03:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351211 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [15:04:22] 06SRE, 06Infrastructure-Foundations, 10Packaging: Create an upload queue for reprepro - https://phabricator.wikimedia.org/T115349#12396193 (10MoritzMuehlenhoff) 05Open→03Declined We're not planning to work on this anymore, the current setup for manual builds and Gitlab runners works fine. [15:04:46] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [dns] - 10https://gerrit.wikimedia.org/r/1351349 (owner: 10Elukey) [15:04:54] (03Merged) 10jenkins-bot: Revert "search: test impact of the function score window size" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351210 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [15:04:57] (03Merged) 10jenkins-bot: search: test impact of cross project searches [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351211 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [15:05:10] !log dcausse@deploy1003 Started scap sync-world: Backport for [[gerrit:1351210|Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211|search: test impact of cross project searches (T438984)]] [15:05:13] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [15:05:19] PROBLEM - Host wikikube-worker2037 is DOWN: PING CRITICAL - Packet loss = 100% [15:05:47] (03CR) 10Ilias Sarantopoulos: [C:03+1] inference-services: increasing max replica of llm-qwen36-27b to 3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351366 (owner: 10Ozge) [15:08:36] (03CR) 10Ozge: [C:03+2] inference-services: increasing max replica of llm-qwen36-27b to 3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351366 (owner: 10Ozge) [15:09:03] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2036.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:09:07] !log dcausse@deploy1003 dcausse: Backport for [[gerrit:1351210|Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211|search: test impact of cross project searches (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:09:35] (03CR) 10Urbanecm: "This appears to cause T440201." [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) (owner: 10Cathal Mooney) [15:09:47] RECOVERY - Host wikikube-worker2037 is UP: PING OK - Packet loss = 0%, RTA = 30.43 ms [15:09:49] (03PS1) 10Muehlenhoff: Remove obsolete stub keytabs for apt/install [labs/private] - 10https://gerrit.wikimedia.org/r/1351370 (https://phabricator.wikimedia.org/T439522) [15:09:57] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1351364 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [15:10:00] (03CR) 10Umherirrender: scap: Remove unused php7_admin_port option (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1219164 (https://phabricator.wikimedia.org/T224491) (owner: 10Krinkle) [15:10:06] !log dcausse@deploy1003 dcausse: Continuing with deployment [15:10:09] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:10:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:10:35] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete stub keytabs for apt/install [labs/private] - 10https://gerrit.wikimedia.org/r/1351370 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [15:10:44] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2037.codfw.wmnet [15:10:58] (03Merged) 10jenkins-bot: inference-services: increasing max replica of llm-qwen36-27b to 3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351366 (owner: 10Ozge) [15:11:07] 06SRE, 06Infrastructure-Foundations, 06Traffic: Support LVS backend servers using nftables - https://phabricator.wikimedia.org/T433601#12396223 (10Urbanecm_WMF) This appears to cause {T440201} [15:11:31] (03CR) 10Tiziano Fogli: [C:03+2] alertamanager: add task receiver for mediawiki-engineering [puppet] - 10https://gerrit.wikimedia.org/r/1351327 (https://phabricator.wikimedia.org/T435644) (owner: 10Tiziano Fogli) [15:11:39] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2037.codfw.wmnet [15:11:48] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2038.codfw.wmnet [15:12:06] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [15:12:07] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [15:12:13] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [15:12:22] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2038.codfw.wmnet [15:12:23] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [15:12:27] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [15:12:36] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:13:05] (03CR) 10Elukey: [C:03+2] profile::k8s::deployment_server::global_config: add PKI LVS IPs [puppet] - 10https://gerrit.wikimedia.org/r/1351364 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [15:13:44] (03CR) 10Elukey: [C:03+2] Revert "Remove the pki discovery settings" [dns] - 10https://gerrit.wikimedia.org/r/1351349 (owner: 10Elukey) [15:14:03] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2036.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:14:25] !log elukey@dns1004 START - running authdns-update [15:14:51] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:14:51] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:14:58] 10ops-eqiad, 06DC-Ops: Power Supply - Status - issue on wikikube-worker1080:9290 - https://phabricator.wikimedia.org/T440202 (10phaultfinder) 03NEW [15:15:19] PROBLEM - Host wikikube-worker2038 is DOWN: PING CRITICAL - Packet loss = 100% [15:15:21] !log dcausse@deploy1003 Finished scap sync-world: Backport for [[gerrit:1351210|Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211|search: test impact of cross project searches (T438984)]] (duration: 10m 11s) [15:15:24] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [15:16:42] !log elukey@dns1004 END - running authdns-update [15:16:45] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host an-druid1007.eqiad.wmnet with OS bookworm [15:17:26] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [15:17:50] ACKNOWLEDGEMENT - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1200 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T440203 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [15:17:56] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T440203 (10ops-monitoring-bot) 03NEW [15:18:03] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [15:18:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:19:27] FIRING: [3x] KubernetesCalicoDown: wikikube-worker2036.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:19:31] (03CR) 10Elukey: [C:03+2] admin_ng: move cfssl-issuer's config to pki.discovery in staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351368 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [15:19:41] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T440195#12396268 (10phaultfinder) [15:19:46] RECOVERY - Host wikikube-worker2038 is UP: PING OK - Packet loss = 0%, RTA = 31.70 ms [15:19:52] 10ops-eqiad, 06DC-Ops: Power Supply - PS Redundancy - issue on wikikube-worker1080:9290 - https://phabricator.wikimedia.org/T440204 (10phaultfinder) 03NEW [15:19:55] FIRING: [5x] KubernetesCalicoDown: wikikube-worker2034.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:20:00] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:20:06] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:20:41] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet [15:20:53] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351212 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [15:20:53] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351213 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [15:21:00] (03CR) 10Muehlenhoff: [C:03+2] "The keytabs and the compat symlink were cleaned up from the install* hosts" [puppet] - 10https://gerrit.wikimedia.org/r/1348724 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [15:21:35] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2038.codfw.wmnet [15:22:10] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet [15:22:11] (03Merged) 10jenkins-bot: Revert "search: test impact of cross project searches" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351212 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [15:22:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2038.codfw.wmnet [15:22:14] (03Merged) 10jenkins-bot: search: measure impact of enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351213 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [15:22:21] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2039.codfw.wmnet [15:22:25] !log dcausse@deploy1003 Started scap sync-world: Backport for [[gerrit:1351212|Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213|search: measure impact of enwiki (T438984)]] [15:22:28] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [15:22:56] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2039.codfw.wmnet [15:23:05] (03PS1) 10Muehlenhoff: Remove test_krb role [puppet] - 10https://gerrit.wikimedia.org/r/1351374 (https://phabricator.wikimedia.org/T439522) [15:23:32] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:24:24] (03PS1) 10Muehlenhoff: Remove profile::base::cuminunpriv from apt servers [puppet] - 10https://gerrit.wikimedia.org/r/1351375 (https://phabricator.wikimedia.org/T439522) [15:24:48] RESOLVED: [4x] KubernetesCalicoDown: wikikube-worker2035.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:26:33] !log dcausse@deploy1003 dcausse: Backport for [[gerrit:1351212|Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213|search: measure impact of enwiki (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:27:10] PROBLEM - Host wikikube-worker2039 is DOWN: PING CRITICAL - Packet loss = 100% [15:27:44] !log dcausse@deploy1003 dcausse: Continuing with deployment [15:28:44] (03PS1) 10Matthias Mullie: Properly render carousel preview dialog sizing in Webkit [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351378 (https://phabricator.wikimedia.org/T439007) [15:28:54] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:28:55] (03PS1) 10Matthias Mullie: Change lead section infobox exclusion [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351379 (https://phabricator.wikimedia.org/T439943) [15:29:10] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage [15:29:13] !log elukey@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'sync'. [15:29:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet [15:29:43] !log elukey@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. [15:30:04] Deploy window Distributed search queries experiment (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1400) [15:30:05] jan_drewniak: May I have your attention please! Wikimedia Portals Update. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1530) [15:30:08] !log elukey@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'sync'. [15:30:32] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351378 (https://phabricator.wikimedia.org/T439007) (owner: 10Matthias Mullie) [15:30:44] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351379 (https://phabricator.wikimedia.org/T439943) (owner: 10Matthias Mullie) [15:30:50] !log elukey@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [15:31:50] RECOVERY - Host wikikube-worker2039 is UP: PING OK - Packet loss = 0%, RTA = 30.48 ms [15:32:04] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage [15:32:09] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet [15:32:21] !log elukey@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'sync'. [15:32:24] !log elukey@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. [15:32:44] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet [15:32:54] !log elukey@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'sync'. [15:32:58] !log elukey@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [15:33:10] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12396365 (10fgiunchedi) 05Resolved→03Open reopening, we'll be rebalancing hosts slated for decom too 1040,1041,1042,1043,1044,1045,1046 af... [15:33:37] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet [15:34:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:34:48] FIRING: NetworkDeviceAlarmActive: Alarm active on ssw1-f1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-f1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [15:34:48] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2038.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:35:05] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox [15:36:03] 06SRE, 06Collaboration-Services, 06Traffic: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12396389 (10LSobanski) 05Open→03Resolved a:03LSobanski I believe we can consider this one resolved, with {T440068} being the follow up task. [15:37:18] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [15:39:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:39:48] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2038.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:47:13] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet [15:47:16] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2039.codfw.wmnet [15:47:21] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1007.eqiad.wmnet with OS bookworm [15:47:37] !log dcausse@deploy1003 Finished scap sync-world: Backport for [[gerrit:1351212|Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213|search: measure impact of enwiki (T438984)]] (duration: 25m 12s) [15:47:40] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [15:49:16] PROBLEM - Druid broker on an-druid1007 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.druid.cli.Main server broker https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [15:49:16] PROBLEM - Druid historical on an-druid1007 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [15:51:17] !log cdobbins@cumin1004 START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqiad [15:53:03] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351214 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [15:53:03] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351215 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [15:54:17] (03Merged) 10jenkins-bot: Revert "search: measure impact of enwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351214 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [15:54:20] (03Merged) 10jenkins-bot: search: test perf of moving completion out of eqiad [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351215 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [15:54:36] !log dcausse@deploy1003 Started scap sync-world: Backport for [[gerrit:1351214|Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215|search: test perf of moving completion out of eqiad (T438984)]] [15:54:39] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [15:55:12] (03CR) 10Subramanya Sastry: alertamanager: add task receiver for mediawiki-engineering (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1351327 (https://phabricator.wikimedia.org/T435644) (owner: 10Tiziano Fogli) [15:56:54] (03PS1) 10Hnowlan: orchestrator: remove absented check [puppet] - 10https://gerrit.wikimedia.org/r/1351385 (https://phabricator.wikimedia.org/T407329) [15:56:54] (03CR) 10Clément Goubert: [C:03+1] mw-experimental: enable native_sidecars on mw-experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333740 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [15:58:12] (03PS3) 10Krinkle: scap: Remove unused php7_admin_port option [puppet] - 10https://gerrit.wikimedia.org/r/1219164 (https://phabricator.wikimedia.org/T224491) [15:58:20] (03CR) 10Krinkle: scap: Remove unused php7_admin_port option (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1219164 (https://phabricator.wikimedia.org/T224491) (owner: 10Krinkle) [15:58:36] !log dcausse@deploy1003 dcausse: Backport for [[gerrit:1351214|Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215|search: test perf of moving completion out of eqiad (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:59:55] !log dcausse@deploy1003 dcausse: Continuing with deployment [16:02:16] RECOVERY - Druid historical on an-druid1007 is OK: PROCS OK: 1 process with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [16:02:16] RECOVERY - Druid broker on an-druid1007 is OK: PROCS OK: 1 process with command name java, args org.apache.druid.cli.Main server broker https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [16:05:20] !log dcausse@deploy1003 Finished scap sync-world: Backport for [[gerrit:1351214|Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215|search: test perf of moving completion out of eqiad (T438984)]] (duration: 10m 44s) [16:05:24] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [16:07:28] 10ops-eqiad, 06SRE, 06DC-Ops: Power Supply - PS Redundancy - issue on wikikube-worker1080:9290 - https://phabricator.wikimedia.org/T440204#12396540 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr [16:11:09] (03PS1) 10Federico Ceratto: mysql: Fix fleetwide RO/RW setting [cookbooks] - 10https://gerrit.wikimedia.org/r/1351376 (https://phabricator.wikimedia.org/T438831) [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:12:46] (03CR) 10Federico Ceratto: "The change in the test output show we are now committing on both DCs, the question is how to test this for real." [cookbooks] - 10https://gerrit.wikimedia.org/r/1351376 (https://phabricator.wikimedia.org/T438831) (owner: 10Federico Ceratto) [16:16:29] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqiad [16:16:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:18:05] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351216 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [16:19:19] (03Merged) 10jenkins-bot: Revert "search: test perf of moving completion out of eqiad" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351216 (https://phabricator.wikimedia.org/T438984) (owner: 10DCausse) [16:19:31] !log dcausse@deploy1003 Started scap sync-world: Backport for [[gerrit:1351216|Revert "search: test perf of moving completion out of eqiad" (T438984)]] [16:19:34] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [16:19:39] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T440195#12396609 (10phaultfinder) [16:19:46] !log cdobbins@cumin1004 START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox [16:19:48] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [16:21:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [16:23:39] !log dcausse@deploy1003 dcausse: Backport for [[gerrit:1351216|Revert "search: test perf of moving completion out of eqiad" (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:24:49] (03PS1) 10Tiziano Fogli: alertamanager: fix typo in task receiver for mediawiki-engineering [puppet] - 10https://gerrit.wikimedia.org/r/1351391 (https://phabricator.wikimedia.org/T435644) [16:25:01] !log dcausse@deploy1003 dcausse: Continuing with deployment [16:25:44] (03CR) 10Tiziano Fogli: [C:03+2] alertamanager: add task receiver for mediawiki-engineering (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1351327 (https://phabricator.wikimedia.org/T435644) (owner: 10Tiziano Fogli) [16:25:52] (03CR) 10Tiziano Fogli: [C:03+2] alertamanager: add task receiver for mediawiki-engineering (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1351327 (https://phabricator.wikimedia.org/T435644) (owner: 10Tiziano Fogli) [16:26:00] (03CR) 10Tiziano Fogli: [C:03+2] alertamanager: fix typo in task receiver for mediawiki-engineering [puppet] - 10https://gerrit.wikimedia.org/r/1351391 (https://phabricator.wikimedia.org/T435644) (owner: 10Tiziano Fogli) [16:29:13] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:30:19] !log dcausse@deploy1003 Finished scap sync-world: Backport for [[gerrit:1351216|Revert "search: test perf of moving completion out of eqiad" (T438984)]] (duration: 10m 48s) [16:30:22] T438984: Exploration: Route queries based on query type and/or target wiki and/or query cost - https://phabricator.wikimedia.org/T438984 [16:32:58] !log closing the "Distributed search queries experiment" window [16:32:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:37:18] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:39:39] (03CR) 10Marostegui: [C:03+1] "This is easy to test: sudo cookbook sre.mysql.depool es1035 -r "testing"" [cookbooks] - 10https://gerrit.wikimedia.org/r/1351376 (https://phabricator.wikimedia.org/T438831) (owner: 10Federico Ceratto) [16:44:40] (03PS1) 10Vgutierrez: liberica: Sync with liberica 0.25 idletcpconnectioncheck config [puppet] - 10https://gerrit.wikimedia.org/r/1351400 (https://phabricator.wikimedia.org/T439840) [16:46:45] (03CR) 10Vgutierrez: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1351400 (https://phabricator.wikimedia.org/T439840) (owner: 10Vgutierrez) [16:50:39] !log cdobbins@cumin1004 START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-magru [16:56:35] 06SRE: Bing Webmaster Tools access request for Ash Han - https://phabricator.wikimedia.org/T439646#12396776 (10AHan-WMF) @LSobanski ITS had actually directed me to SRE, saying that SRE will handle this request: https://wikimediainternal.zendesk.com/hc/en-us/requests/130810 [17:00:05] swfrench-wmf: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for MediaWiki infrastructure (UTC late) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1700). [17:00:05] ryankemper: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Wikidata Query Service weekly deploy . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T1700). [17:00:15] o/ [17:02:25] (03CR) 10Scott French: [C:03+2] mw-(jobrunner|pretrain): Set route-level stream idle time on jobrunners [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347604 (https://phabricator.wikimedia.org/T439577) (owner: 10Scott French) [17:03:28] (03PS1) 10Giuseppe Lavagetto: Use the provision cookbook to just apply BIOS changes quickly [cookbooks] - 10https://gerrit.wikimedia.org/r/1351408 [17:04:11] (03Merged) 10jenkins-bot: mw-(jobrunner|pretrain): Set route-level stream idle time on jobrunners [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347604 (https://phabricator.wikimedia.org/T439577) (owner: 10Scott French) [17:06:21] (03CR) 10CI reject: [V:04-1] Use the provision cookbook to just apply BIOS changes quickly [cookbooks] - 10https://gerrit.wikimedia.org/r/1351408 (owner: 10Giuseppe Lavagetto) [17:06:38] I'll be deploying some changes to mw-jobrunner shortly, followed by some additional testing of scap deploy-service, then *maybe* (time permitting) some changes to REST Gateway [17:06:39] (03CR) 10Clément Goubert: [C:03+1] eventschemas: sparse-checkout the mediawiki repo to its mediawiki/ dir [puppet] - 10https://gerrit.wikimedia.org/r/1343109 (owner: 10Aghirelli) [17:06:46] (03CR) 10Clément Goubert: [C:03+1] eventschemas: add api-platform-schemas repository [puppet] - 10https://gerrit.wikimedia.org/r/1343092 (https://phabricator.wikimedia.org/T438208) (owner: 10Aghirelli) [17:06:54] (03CR) 10Clément Goubert: [C:03+1] git::clone: add optional sparse-checkout support [puppet] - 10https://gerrit.wikimedia.org/r/1343101 (owner: 10Aghirelli) [17:11:35] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [17:12:02] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [17:12:25] (03PS3) 10Bking: sre.hadoop.roll-restart-workers: implement start-datetime logic [cookbooks] - 10https://gerrit.wikimedia.org/r/1347713 (https://phabricator.wikimedia.org/T437713) [17:12:57] (03CR) 10Hnowlan: [C:03+2] orchestrator: remove absented check [puppet] - 10https://gerrit.wikimedia.org/r/1351385 (https://phabricator.wikimedia.org/T407329) (owner: 10Hnowlan) [17:13:21] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply [17:14:15] (03CR) 10Bking: sre.hadoop.roll-restart-workers: implement start-datetime logic (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1347713 (https://phabricator.wikimedia.org/T437713) (owner: 10Bking) [17:14:26] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply [17:15:08] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply [17:15:26] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply [17:15:38] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply [17:15:55] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-magru [17:16:08] !log cdobbins@cumin1004 START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo [17:16:46] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply [17:27:54] !log swfrench@deploy1003 Started scap deploy-service shellbox: Test deployment after logging and configuration changes [17:27:54] !log swfrench@deploy1003 scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video [17:28:17] !log swfrench@deploy1003 scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video [17:29:05] !log swfrench@deploy1003 scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video [17:29:25] !log swfrench@deploy1003 Finished scap deploy-service shellbox: Test deployment after logging and configuration changes (duration: 01m 31s) [17:29:34] very nuce [17:29:41] s/nuce/nice/ :) [17:33:44] !log urbanecm@deploy1003 mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Maliner --performer=Maliner --as-job # T439577 [17:33:47] T439577: changeprop timeouts are not configured consistently - https://phabricator.wikimedia.org/T439577 [17:34:43] 🎉 [17:35:55] (03CR) 10Scott French: "Thanks for the review! I'll follow up with you on the remaining patches in the series about testing / pacing of the change." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335285 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [17:35:57] (03CR) 10Scott French: [C:03+2] rest-gateway: Prepare the cluster specifier plugin for XWD routing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335285 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [17:37:26] (03Merged) 10jenkins-bot: rest-gateway: Prepare the cluster specifier plugin for XWD routing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335285 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [17:39:16] alright, last round of infra-window changes coming shortly [17:39:52] !log swfrench@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:40:08] !log swfrench@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:41:07] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo [17:42:43] !log deployed csv file to security.wikimedia.org (miscweb) [17:42:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:43:02] !log cdobbins@cumin1004 START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo [17:46:00] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [17:46:42] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [17:46:56] 10ops-codfw, 06SRE, 06DC-Ops: Enter CyrusOne's Rack Location in Netbox's "Facility ID" Field - https://phabricator.wikimedia.org/T440071#12397024 (10Jhancock.wm) i don't mind filling these in once we have a format. @RobH do you know C1's labeling layout? I honestly don't recall seeing it. if you have exampl... [17:49:13] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:52:58] !log swfrench@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:53:15] !log swfrench@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:57:42] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [17:58:40] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [17:58:46] !log swfrench@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:58:58] !log swfrench@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:59:45] (03CR) 10Dzahn: [C:03+2] zuul: export the project encryption keys daily for backup [puppet] - 10https://gerrit.wikimedia.org/r/1349165 (https://phabricator.wikimedia.org/T436131) (owner: 10Dzahn) [18:01:48] (03PS1) 10Hnowlan: team-sre: add exporter-based host_down check [alerts] - 10https://gerrit.wikimedia.org/r/1351431 (https://phabricator.wikimedia.org/T395448) [18:03:18] 14SRE-Sprint-Week-Sustainability-March2023, 06Release-Engineering-Team, 10Scap, 13Patch-For-Review, 07Sustainability (Incident Followup): Add failure rate triggered rollback to scap - https://phabricator.wikimedia.org/T317405#12397128 (10dancy) 05Open→03Resolved a:03dancy [18:04:33] 14SRE-Sprint-Week-Sustainability-March2023, 06Release-Engineering-Team, 10Scap, 13Patch-For-Review, 07Sustainability (Incident Followup): Add failure rate triggered rollback to scap - https://phabricator.wikimedia.org/T317405#12397133 (10dancy) 05Resolved→03Open Re-opening after re-reading the descri... [18:04:42] 14SRE-Sprint-Week-Sustainability-March2023, 06Release-Engineering-Team, 10Scap, 13Patch-For-Review, 07Sustainability (Incident Followup): Add failure rate triggered rollback to scap - https://phabricator.wikimedia.org/T317405#12397136 (10dancy) a:05dancy→03None [18:05:15] 14SRE-Sprint-Week-Sustainability-March2023, 06Release-Engineering-Team, 10Scap, 13Patch-For-Review, 07Sustainability (Incident Followup): Add failure rate triggered rollback to scap - https://phabricator.wikimedia.org/T317405#12397141 (10dancy) @Clement_Goubert Any current feelings on this topic? [18:08:20] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo [18:14:50] !log cdobbins@cumin1004 START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-drmrs [18:15:35] 10ops-codfw, 06SRE, 06DC-Ops: Enter CyrusOne's Rack Location in Netbox's "Facility ID" Field - https://phabricator.wikimedia.org/T440071#12397197 (10RobH) The most recent request to access our cage from CyrusOne listed our 'door' as Cage Door 7006.066. IIRC our racks are then labeled based from that. imo... [18:15:37] (03CR) 10Eric Gardner: "I can +2 this once the -2 is lifted, assuming we're ready now." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347724 (https://phabricator.wikimedia.org/T439914) (owner: 10DErenrich) [18:22:02] 06SRE, 10SRE-Access-Requests: Requesting access to Superset for DLin - https://phabricator.wikimedia.org/T440228 (10DLin-WMF) 03NEW [18:22:48] 06SRE, 10SRE-Access-Requests: Requesting access to Superset for DLin - https://phabricator.wikimedia.org/T440228#12397256 (10Maryana) I'm Damian's manager and I approve this request! [18:23:09] !log isaranto@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [18:24:12] 10ops-codfw, 06SRE, 06DC-Ops: Enter CyrusOne's Rack Location in Netbox's "Facility ID" Field - https://phabricator.wikimedia.org/T440071#12397261 (10RobH) 3621872 opened, listed myself and Jenn on notify list. > Support, > > We recently had to open a case and refer to a rack in our datahall 7, but our ra... [18:26:52] PROBLEM - Host wikikube-worker1156 is DOWN: PING CRITICAL - Packet loss = 77%, RTA = 8510.90 ms [18:27:00] RECOVERY - Host wikikube-worker1156 is UP: PING OK - Packet loss = 0%, RTA = 0.71 ms [18:34:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:37:38] 06SRE, 06Fundraising-Backlog, 06Infrastructure-Foundations, 10Mail: Add granularity limiter (g=) to wikimedia.org DKIM record(s) - https://phabricator.wikimedia.org/T140316#12397309 (10greg) 05Open→03Declined >>! In T140316#12396096, @LSobanski wrote: > @greg could you take a look and see if this i... [18:39:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:39:51] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-drmrs [18:44:21] !log cdobbins@cumin1004 START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-esams [18:45:39] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [18:46:26] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [18:54:59] (03PS1) 10Btullis: apereo_cas: Add a grant_types parameter for OIDC services [puppet] - 10https://gerrit.wikimedia.org/r/1351438 (https://phabricator.wikimedia.org/T435602) [18:55:05] 06SRE, 10DNS, 06Fundraising-Backlog, 10fundraising-tech-ops, and 4 others: Investigate having combowiki on donate.wikipedia.org - https://phabricator.wikimedia.org/T433108#12397418 (10Jgreen) The load balancers and servers have been configured to accept the new domain, ready to go once the new cert and DNS... [18:55:50] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1351438 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [18:58:05] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [18:58:14] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [19:04:13] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:04:13] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:04:24] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:09:24] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-esams [19:10:08] (03PS1) 10DLynch: services_proxy: Enable the linked-artifacts listener for MediaWiki [puppet] - 10https://gerrit.wikimedia.org/r/1351442 (https://phabricator.wikimedia.org/T432733) [19:10:12] (03PS1) 10Dzahn: zuul: add hosts to Bacula, for backups of private keys [puppet] - 10https://gerrit.wikimedia.org/r/1351443 (https://phabricator.wikimedia.org/T436131) [19:11:05] (03PS1) 10DLynch: VisualEditor: Precompute edit suggestion counts on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351444 (https://phabricator.wikimedia.org/T432733) [19:11:24] (03PS2) 10Dzahn: zuul: add hosts to Bacula, for backups of private keys [puppet] - 10https://gerrit.wikimedia.org/r/1351443 (https://phabricator.wikimedia.org/T436131) [19:11:25] (03CR) 10CI reject: [V:04-1] services_proxy: Enable the linked-artifacts listener for MediaWiki [puppet] - 10https://gerrit.wikimedia.org/r/1351442 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [19:12:18] (03CR) 10DLynch: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/1351442 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [19:15:02] (03PS2) 10DLynch: services_proxy: Enable the linked-artifacts listener for MediaWiki [puppet] - 10https://gerrit.wikimedia.org/r/1351442 (https://phabricator.wikimedia.org/T432733) [19:15:41] (03CR) 10Dzahn: [C:03+2] zuul: add hosts to Bacula, for backups of private keys [puppet] - 10https://gerrit.wikimedia.org/r/1351443 (https://phabricator.wikimedia.org/T436131) (owner: 10Dzahn) [19:22:37] (03PS2) 10Scott French: rest-gateway: Adopt cluster specifier and XWD routing in staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346837 (https://phabricator.wikimedia.org/T433752) [19:22:37] (03PS2) 10Scott French: rest-gateway: Adopt cluster specifier and XWD routing in production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346838 (https://phabricator.wikimedia.org/T433752) [19:32:03] (03CR) 10Scott French: "Thanks for the review on Id63c6c0468! Following up on the discussion in [0], let me know if you'd like me to go about this in a different " [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346837 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [19:33:33] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [19:33:47] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [19:33:49] 06SRE, 10DNS, 06Fundraising-Backlog, 10fundraising-tech-ops, and 4 others: Investigate having combowiki on donate.wikipedia.org - https://phabricator.wikimedia.org/T433108#12397620 (10XenoRyet) 05Open→03Resolved [19:34:49] (03PS1) 10Dzahn: site: make zuul1006 and zuul1007 trusted build nodes [puppet] - 10https://gerrit.wikimedia.org/r/1351451 (https://phabricator.wikimedia.org/T427353) [19:35:10] FIRING: BFDdown: BFD session down between cr1-magru and fe80::6687:8807:11f2:7018 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [19:36:30] (03CR) 10Dzahn: [C:03+2] site: make zuul1006 and zuul1007 trusted build nodes [puppet] - 10https://gerrit.wikimedia.org/r/1351451 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [19:39:48] jouncebot: nowandnext [19:39:48] No deployments scheduled for the next 0 hour(s) and 20 minute(s) [19:39:48] In 0 hour(s) and 20 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T2000) [19:40:10] RESOLVED: BFDdown: BFD session down between cr1-magru and fe80::6687:8807:11f2:7018 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [19:51:56] !log dreamyjazz Deployed security patch for T439932 [19:55:32] (03PS1) 10Dzahn: zuul: only export the encryption keys where the scheduler runs [puppet] - 10https://gerrit.wikimedia.org/r/1351462 (https://phabricator.wikimedia.org/T436131) [19:56:50] (03CR) 10Dzahn: [C:03+2] zuul: only export the encryption keys where the scheduler runs [puppet] - 10https://gerrit.wikimedia.org/r/1351462 (https://phabricator.wikimedia.org/T436131) (owner: 10Dzahn) [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: OwO what's this, a deployment window?? UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T2000). nyaa~ [20:00:05] Ameisenigel: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:01:10] Ameisenigel: unfortunately, this kind of patch is not suitable for a backport window, without some prep, it will fully render the wiki unusable for absolutely everyone. i can _try_ to help with this later, but not in this window :) [20:01:12] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [20:01:27] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [20:03:36] urbanecm OK, thanks [20:09:40] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox [20:15:22] (03PS4) 10Dzahn: zuul: decom VM zuul1002, replaced by zuul1005, keep zuul2002 [puppet] - 10https://gerrit.wikimedia.org/r/1346164 (https://phabricator.wikimedia.org/T427353) [20:15:30] (03PS5) 10Dzahn: zuul: decom VM zuul1002, replaced by zuul1005, keep zuul2002 [puppet] - 10https://gerrit.wikimedia.org/r/1346164 (https://phabricator.wikimedia.org/T427353) [20:24:51] (03PS2) 10Xcollazo: dumps: Keep File Exports on the distribution hosts, not in HDFS [puppet] - 10https://gerrit.wikimedia.org/r/1346752 (https://phabricator.wikimedia.org/T437463) [20:26:30] (03CR) 10Xcollazo: "You ok if I try to slim this down on a follow up patch just doing that?" [puppet] - 10https://gerrit.wikimedia.org/r/1346752 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [20:28:58] (03PS1) 10Gmodena: wdqs: enforce quarkus rest client connection pool [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351482 (https://phabricator.wikimedia.org/T438488) [20:33:43] (03PS2) 10Gmodena: wdqs: enforce quarkus rest client connection pool [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351482 (https://phabricator.wikimedia.org/T438488) [20:36:19] jouncebot: nowandnext [20:36:19] For the next 0 hour(s) and 23 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T2000) [20:36:19] In 0 hour(s) and 23 minute(s): Weekly Security deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T2100) [20:36:52] (03PS3) 10Gmodena: wdqs: enforce quarkus rest client connection pool [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351482 (https://phabricator.wikimedia.org/T438488) [20:39:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:39:54] (03PS1) 10Dzahn: zuul: decom VM zuul1003, replaced by zuul1006/zuul1007, keep zuul2003 [puppet] - 10https://gerrit.wikimedia.org/r/1351484 (https://phabricator.wikimedia.org/T427353) [20:44:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:45:14] 06SRE, 10DNS, 06Fundraising-Backlog, 10fundraising-tech-ops, and 4 others: Investigate having combowiki on donate.wikipedia.org - https://phabricator.wikimedia.org/T433108#12397979 (10ssingh) Please let us know when you want us to retire donate.wikimedia.org. There are a few Puppet-related cleanups we... [20:45:42] !log dreamyjazz Deployed security patch for T439974 [20:50:36] !log bking@cumin2003 START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [20:50:53] (03PS1) 10Gmodena: wdqs: set request timeout to 60s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351488 (https://phabricator.wikimedia.org/T439942) [20:58:42] !log dreamyjazz Deployed security patch for T439719 [21:00:05] alexsanford, Reedy, sbassett, Maryum, and manfredi: #bothumor I � Unicode. All rise for Weekly Security deployment window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T2100). [21:01:45] (03CR) 10DErenrich: Enable the discord previews from WikimediaCustomizations on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347724 (https://phabricator.wikimedia.org/T439914) (owner: 10DErenrich) [21:02:20] (03CR) 10Eric Gardner: [C:03+1] "+1 I mean, since this is config" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347724 (https://phabricator.wikimedia.org/T439914) (owner: 10DErenrich) [21:02:50] !log bking@cumin2003 END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [21:05:08] !log bking@cumin2003 START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [21:10:47] !log bking@cumin2003 END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [21:21:09] (03PS4) 10Bking: sre.hadoop.roll-restart-workers: implement start-datetime logic [cookbooks] - 10https://gerrit.wikimedia.org/r/1347713 (https://phabricator.wikimedia.org/T437713) [21:21:59] i'm going to merge a config change during the reader window in ~2h [21:33:03] (03CR) 10Dzahn: [C:03+2] zuul: decom VM zuul1002, replaced by zuul1005, keep zuul2002 [puppet] - 10https://gerrit.wikimedia.org/r/1346164 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [21:33:52] !log dzahn@cumin1004 START - Cookbook sre.hosts.decommission for hosts zuul1002.eqiad.wmnet [21:34:12] !log jasmine@cumin2003 START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet [21:35:55] !log jasmine@cumin2003 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet [21:38:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-jobrunner - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [21:38:50] !log dzahn@cumin1004 START - Cookbook sre.dns.netbox [21:39:25] !log jasmine@cumin2003 START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet [21:39:58] !log jasmine@cumin2003 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet [21:43:15] RESOLVED: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-jobrunner - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [21:45:17] !log dzahn@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" [21:45:34] !log dzahn@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" [21:45:34] !log dzahn@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [21:45:36] !log dzahn@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1002.eqiad.wmnet [21:45:50] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12398194 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1004 for hosts: `zuul1002.eqiad.wmnet` - zuul1002.eq... [21:48:33] !log dzahn@cumin1004 START - Cookbook sre.hosts.decommission for hosts zuul1003.eqiad.wmnet [21:49:24] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:53:24] !log dzahn@cumin1004 START - Cookbook sre.dns.netbox [21:58:56] dzahn@cumin1004 decommission (PID 2991423) is awaiting input [22:00:17] !log dzahn@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" [22:00:49] !log dzahn@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" [22:00:49] !log dzahn@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [22:00:50] !log dzahn@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1003.eqiad.wmnet [22:01:10] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12398211 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1004 for hosts: `zuul1003.eqiad.wmnet` - zuul1003.eq... [22:01:39] (03CR) 10Dzahn: [C:03+2] zuul: decom VM zuul1003, replaced by zuul1006/zuul1007, keep zuul2003 [puppet] - 10https://gerrit.wikimedia.org/r/1351484 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [22:01:45] (03PS2) 10Dzahn: zuul: decom VM zuul1003, replaced by zuul1006/zuul1007, keep zuul2003 [puppet] - 10https://gerrit.wikimedia.org/r/1351484 (https://phabricator.wikimedia.org/T427353) [22:02:40] (03CR) 10Dzahn: [C:03+2] zuul: decom VM zuul1003, replaced by zuul1006/zuul1007, keep zuul2003 [puppet] - 10https://gerrit.wikimedia.org/r/1351484 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [22:10:06] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12398221 (10Dzahn) 05Open→03Resolved All four former ganeti102[3456] hosts are reimaged on trixie and in service under their new... [22:10:58] (03PS1) 10Kimberly Sarabia: [Image Browsing] Prevent layout shift when loading carousel experiment [extensions/ReaderExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351518 (https://phabricator.wikimedia.org/T439948) [22:11:51] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/ReaderExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351518 (https://phabricator.wikimedia.org/T439948) (owner: 10Kimberly Sarabia) [22:22:34] (03PS4) 10Scott French: mediawiki: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338000 (https://phabricator.wikimedia.org/T427666) [22:22:36] (03PS3) 10Scott French: Update mesh.{configuration,networkpolicy} and dependents in all charts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338030 (https://phabricator.wikimedia.org/T427666) [22:22:36] (03PS1) 10Scott French: changeprop: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351523 (https://phabricator.wikimedia.org/T427666) [22:25:24] (03PS1) 10Jasmine: wmnet: remove testreduce CNAME record [dns] - 10https://gerrit.wikimedia.org/r/1351524 (https://phabricator.wikimedia.org/T436917) [22:35:03] (03CR) 10Dzahn: "This is still in CDN config. I would get that removed before DNS." [dns] - 10https://gerrit.wikimedia.org/r/1351524 (https://phabricator.wikimedia.org/T436917) (owner: 10Jasmine) [22:37:18] (03CR) 10Dzahn: "maybe the ticket is a bit too narrow in focus just looking at a host name, but then it seems the only host that is left for this" [dns] - 10https://gerrit.wikimedia.org/r/1351524 (https://phabricator.wikimedia.org/T436917) (owner: 10Jasmine) [22:41:36] (03PS2) 10Jforrester: Factor out Abstract Wikipedia client mode into a dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346643 [22:41:36] (03PS2) 10Jforrester: Abstractwiki: Drop temporary memcached/mainstash config, no longer needed [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346644 [22:41:36] (03PS1) 10Jforrester: Wikifunctions: Switch the editor from Ace to CodeMirror [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351531 (https://phabricator.wikimedia.org/T400015) [22:44:23] (03PS1) 10Dzahn: gerrit: block Google fonts on the Documentation pages with CSP [puppet] - 10https://gerrit.wikimedia.org/r/1351534 (https://phabricator.wikimedia.org/T393763) [22:47:35] (03CR) 10Dzahn: "given that it is inside a tag that only affects Documentation I think the risk isn't very high and we can just do this. waiting" [puppet] - 10https://gerrit.wikimedia.org/r/1351534 (https://phabricator.wikimedia.org/T393763) (owner: 10Dzahn) [22:58:24] (03PS2) 10Scott French: changeprop: Support enabling the mesh as a sidecar [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351528 (https://phabricator.wikimedia.org/T434925) [23:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261005T2300) [23:04:07] (03CR) 10Scott French: [C:04-1] Update mesh.{configuration,networkpolicy} and dependents in all charts (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338030 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [23:05:08] ok i'm about to merge a config change [23:10:53] (03CR) 10TrainBranchBot: [C:03+2] "Approved by derenrich@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347724 (https://phabricator.wikimedia.org/T439914) (owner: 10DErenrich) [23:12:08] (03Merged) 10jenkins-bot: Enable the discord previews from WikimediaCustomizations on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347724 (https://phabricator.wikimedia.org/T439914) (owner: 10DErenrich) [23:12:20] !log derenrich@deploy1003 Started scap sync-world: Backport for [[gerrit:1347724|Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] [23:12:24] T439914: Enable custom discord previews on enwiki - https://phabricator.wikimedia.org/T439914 [23:13:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [23:16:16] !log derenrich@deploy1003 derenrich: Backport for [[gerrit:1347724|Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [23:18:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [23:18:58] !log derenrich@deploy1003 derenrich: Continuing with deployment [23:24:19] !log derenrich@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347724|Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] (duration: 11m 59s) [23:24:23] T439914: Enable custom discord previews on enwiki - https://phabricator.wikimedia.org/T439914 [23:25:30] ok i'm done [23:38:57] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1351549 [23:38:57] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1351549 (owner: 10TrainBranchBot) [23:46:45] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1351549 (owner: 10TrainBranchBot)