[00:01:38] (03PS2) 10Scott French: api-gateway: Support PHP_ENGINE routing in the mw-api plugin [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335270 (https://phabricator.wikimedia.org/T433752) [00:01:38] (03CR) 10Scott French: "This is the last of the traffic-diversion features that we know will be needed in the near term: PHP_ENGINE diversion to support the 8.5 m" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335270 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [00:09:02] FIRING: SystemdUnitFailed: prometheus_arelion_exporter.service on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:41:11] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 1284.18 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:52:07] (03PS4) 10Scott French: rest-gateway: Prepare the cluster specifier plugin for XWD routing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335285 (https://phabricator.wikimedia.org/T433752) [00:52:07] (03PS1) 10Scott French: rest-gateway: Adopt cluster specifier and XWD routing in staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346837 (https://phabricator.wikimedia.org/T433752) [00:52:11] (03PS1) 10Scott French: rest-gateway: Adopt cluster specifier and XWD routing in production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346838 (https://phabricator.wikimedia.org/T433752) [01:03:20] (03CR) 10Scott French: "Thanks again for iterating on this with me! (and apologies for the delayed reply on my part)" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335285 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [01:09:01] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1346853 [01:09:01] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1346853 (owner: 10TrainBranchBot) [01:13:43] (03CR) 10CI reject: [V:04-1] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1346853 (owner: 10TrainBranchBot) [01:16:47] RECOVERY - MariaDB Replica Lag: s4 on db1265 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [01:24:42] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:29:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:52:24] (03CR) 10Fcvgov: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/827566 (owner: 10BBlack) [01:52:39] (03CR) 10Fcvgov: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/827566 (owner: 10BBlack) [01:52:41] (03CR) 10Fcvgov: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/827566 (owner: 10BBlack) [02:00:04] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T0200) [02:05:11] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:05:47] PROBLEM - MariaDB Replica Lag: s3 on db1265 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 617.96 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:07:56] (03CR) 10Ryan Kemper: [C:03+2] kafka-configurator: install on brokers [puppet] - 10https://gerrit.wikimedia.org/r/1346361 (https://phabricator.wikimedia.org/T435282) (owner: 10Ryan Kemper) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [03:00:17] 06SRE, 06serviceops-deprecated, 07Wikimedia-Performance-recommendation: Evaluate using igbinary for MW php-apcu at WMF - https://phabricator.wikimedia.org/T225074#12383372 (10MGoncalves-WMF) @Krinkle Yes, the comparison was run with the latest MediaWiki-Docker PHP-FPM image, for previous and actual runs: -... [03:02:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-magru (195.200.68.139) - group Confed_magru - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_magru&var-bgp_neighbor=cr1-magru - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [03:03:00] FIRING: PingLossPercent: Blackbox probe packet loss 4.583% from eqiad to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [03:04:10] FIRING: BFDdown: BFD session down between cr1-magru and fe80::6687:8807:11f2:7018 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [03:07:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-magru (195.200.68.139) - group Confed_magru - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [03:09:10] RESOLVED: BFDdown: BFD session down between cr1-magru and fe80::6687:8807:11f2:7018 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [03:13:00] FIRING: [2x] PingLossPercent: Blackbox probe packet loss 5% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [03:18:00] FIRING: [2x] PingLossPercent: Blackbox probe packet loss 3.333% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [03:20:29] PROBLEM - OSPF status on cr1-codfw is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [03:21:10] FIRING: BFDdown: BFD session down between cr1-magru and 195.200.68.138 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [03:21:39] FIRING: CoreBGPDown: Core BGP session down between cr1-magru and cr1-codfw (195.200.68.138) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_codfw&var-bgp_neighbor=cr1-codfw - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [03:22:29] RECOVERY - OSPF status on cr1-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [03:22:54] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-magru (195.200.68.139) - group Confed_magru - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [03:23:00] RESOLVED: [2x] PingLossPercent: Blackbox probe packet loss 3.333% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [03:26:10] FIRING: [3x] BFDdown: BFD session down between cr1-codfw and fe80::b6f9:5d07:1130:c93f - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [03:26:29] PROBLEM - OSPF status on cr1-codfw is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [03:26:39] RESOLVED: CoreBGPDown: Core BGP session down between cr1-magru and cr1-codfw (195.200.68.138) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_codfw&var-bgp_neighbor=cr1-codfw - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [03:27:29] RECOVERY - OSPF status on cr1-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [03:30:29] PROBLEM - OSPF status on cr1-codfw is CRITICAL: OSPFv2: 8/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [03:30:29] PROBLEM - OSPF status on cr1-magru is CRITICAL: OSPFv2: 5/5 UP : OSPFv3: 4/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [03:31:10] RESOLVED: [3x] BFDdown: BFD session down between cr1-codfw and fe80::b6f9:5d07:1130:c93f - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [03:31:29] RECOVERY - OSPF status on cr1-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [03:31:29] RECOVERY - OSPF status on cr1-magru is OK: OSPFv2: 5/5 UP : OSPFv3: 5/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [03:34:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:54:51] (03CR) 10Ladsgroup: "Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1346769 (owner: 10Majavah) [03:54:55] (03PS2) 10Majavah: Remove meet accountmanager classes [puppet] - 10https://gerrit.wikimedia.org/r/1346769 [03:54:58] (03CR) 10Ladsgroup: [C:03+2] Remove meet accountmanager classes [puppet] - 10https://gerrit.wikimedia.org/r/1346769 (owner: 10Majavah) [03:54:59] (03CR) 10Ladsgroup: [V:03+2 C:03+2] Remove meet accountmanager classes [puppet] - 10https://gerrit.wikimedia.org/r/1346769 (owner: 10Majavah) [03:55:47] RECOVERY - MariaDB Replica Lag: s3 on db1265 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [04:01:29] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12383404 (10Eric) >>! In T439644#12381049, @eniroomand wrote: > @ayounsi I confirm that I have read th... [04:09:02] FIRING: SystemdUnitFailed: prometheus_arelion_exporter.service on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:20:10] FIRING: BFDdown: BFD session down between cr1-codfw and fe80::b6f9:5d07:1130:c93f - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [04:25:10] RESOLVED: BFDdown: BFD session down between cr1-codfw and fe80::b6f9:5d07:1130:c93f - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [04:44:10] FIRING: BFDdown: BFD session down between cr1-magru and fe80::6687:8807:11f2:7018 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [04:49:10] RESOLVED: BFDdown: BFD session down between cr1-magru and fe80::6687:8807:11f2:7018 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [05:09:10] FIRING: BFDdown: BFD session down between cr1-magru and fe80::6687:8807:11f2:7018 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [05:14:10] RESOLVED: BFDdown: BFD session down between cr1-magru and fe80::6687:8807:11f2:7018 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [05:19:31] (03CR) 10Gmodena: [C:03+1] Annotate wdqs-proxy pods so metrics are scraped. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346743 (https://phabricator.wikimedia.org/T439449) (owner: 10Lerickson) [06:00:04] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T0600) [06:00:04] marostegui, cezmunsta, and federico3: #bothumor I � Unicode. All rise for Primary database switchover deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T0600). [06:07:49] (03PS1) 10Kosta Harlan: AbuseReview: Send the abuse review tag in interaction events [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347210 (https://phabricator.wikimedia.org/T439238) [06:08:06] (03PS1) 10Kosta Harlan: AbuseReview: Instrument when content policy match adds tags [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347211 (https://phabricator.wikimedia.org/T439242) [06:08:49] If the infra window isn't being used, I'd like to do some wmf.22 backports [06:15:30] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347210 (https://phabricator.wikimedia.org/T439238) (owner: 10Kosta Harlan) [06:15:30] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347211 (https://phabricator.wikimedia.org/T439242) (owner: 10Kosta Harlan) [06:16:34] (03Merged) 10jenkins-bot: AbuseReview: Send the abuse review tag in interaction events [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347210 (https://phabricator.wikimedia.org/T439238) (owner: 10Kosta Harlan) [06:17:30] (03Merged) 10jenkins-bot: AbuseReview: Instrument when content policy match adds tags [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347211 (https://phabricator.wikimedia.org/T439242) (owner: 10Kosta Harlan) [06:18:37] !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1347210|AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211|AbuseReview: Instrument when content policy match adds tags (T439242)]] [06:18:42] T439238: Include tag in metadata for ReviewVerdictHandler instrumentation - https://phabricator.wikimedia.org/T439238 [06:18:43] T439242: Create an event when a tagged tracked in AbuseReview is added - https://phabricator.wikimedia.org/T439242 [06:20:03] (03CR) 10Tiziano Fogli: [C:03+2] kafka-logging: update brokers list [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346631 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [06:22:31] (03Merged) 10jenkins-bot: kafka-logging: update brokers list [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346631 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [06:22:53] !log kharlan@deploy1003 kharlan: Backport for [[gerrit:1347210|AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211|AbuseReview: Instrument when content policy match adds tags (T439242)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [06:23:50] !log kharlan@deploy1003 kharlan: Continuing with deployment [06:24:05] (03PS1) 10Kosta Harlan: Add extra UI elements to Special:AbuseReview following user feedback [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347223 (https://phabricator.wikimedia.org/T439694) [06:29:10] !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347210|AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211|AbuseReview: Instrument when content policy match adds tags (T439242)]] (duration: 10m 33s) [06:29:14] T439238: Include tag in metadata for ReviewVerdictHandler instrumentation - https://phabricator.wikimedia.org/T439238 [06:29:14] T439242: Create an event when a tagged tracked in AbuseReview is added - https://phabricator.wikimedia.org/T439242 [06:30:47] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T439559#12383559 (10tappof) [06:31:31] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347223 (https://phabricator.wikimedia.org/T439694) (owner: 10Kosta Harlan) [06:32:48] (03Merged) 10jenkins-bot: Add extra UI elements to Special:AbuseReview following user feedback [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347223 (https://phabricator.wikimedia.org/T439694) (owner: 10Kosta Harlan) [06:33:11] !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1347223|Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] [06:33:14] T439694: Add some extra UI elements to Special:AbuseReview - https://phabricator.wikimedia.org/T439694 [06:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [06:35:16] !log tappof@cumin1004 START - Cookbook sre.hosts.decommission for hosts kafka-logging1001.eqiad.wmnet [06:41:37] !log tappof@cumin1004 START - Cookbook sre.dns.netbox [06:45:57] !log tappof@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" [06:46:12] !log tappof@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" [06:46:12] !log tappof@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [06:46:13] !log tappof@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1001.eqiad.wmnet [06:46:22] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T439559#12383620 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by tappof@cumin1004 for hosts: `kafka-logging1001.eqiad.wmne... [06:46:47] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 01 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346542 (https://phabricator.wikimedia.org/T401808) (owner: 10Kosta Harlan) [06:47:12] !log tappof@cumin1004 START - Cookbook sre.hosts.decommission for hosts kafka-logging1002.eqiad.wmnet [06:53:13] !log kharlan@deploy1003 kharlan: Backport for [[gerrit:1347223|Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [06:53:16] T439694: Add some extra UI elements to Special:AbuseReview - https://phabricator.wikimedia.org/T439694 [06:53:23] !log tappof@cumin1004 START - Cookbook sre.dns.netbox [06:54:30] !log kharlan@deploy1003 kharlan: Continuing with deployment [06:55:36] (03CR) 10Trueg: [C:03+1] "nice" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346743 (https://phabricator.wikimedia.org/T439449) (owner: 10Lerickson) [06:57:14] !log tappof@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" [06:57:26] !log tappof@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" [06:57:26] !log tappof@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [06:57:27] !log tappof@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1002.eqiad.wmnet [06:57:35] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T439559#12383631 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by tappof@cumin1004 for hosts: `kafka-logging1002.eqiad.wmne... [07:00:05] Amir1, urbanecm, and awight: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for UTC morning backport window . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T0700). [07:00:05] kostajh: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:22] syncing out a wmf.22 patch at the moment [07:01:56] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T439559#12383648 (10tappof) [07:02:21] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T439559#12383651 (10tappof) a:05tappof→03None [07:04:29] (03PS1) 10Slyngshede: data.yaml offboarding of andreawest [puppet] - 10https://gerrit.wikimedia.org/r/1347268 [07:05:25] (03CR) 10Muehlenhoff: [C:03+2] thumbor: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346363 (owner: 10Muehlenhoff) [07:05:43] (03CR) 10CI reject: [V:04-1] data.yaml offboarding of andreawest [puppet] - 10https://gerrit.wikimedia.org/r/1347268 (owner: 10Slyngshede) [07:06:29] !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347223|Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] (duration: 33m 18s) [07:06:33] T439694: Add some extra UI elements to Special:AbuseReview - https://phabricator.wikimedia.org/T439694 [07:07:05] (03PS2) 10Slyngshede: data.yaml offboarding of andreawest [puppet] - 10https://gerrit.wikimedia.org/r/1347268 [07:07:27] !log jmm@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [07:08:08] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346542 (https://phabricator.wikimedia.org/T401808) (owner: 10Kosta Harlan) [07:08:08] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346541 (https://phabricator.wikimedia.org/T401808) (owner: 10Kosta Harlan) [07:09:03] (03Merged) 10jenkins-bot: CheckUser: Enable ClientHintsInEditRequest on group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346541 (https://phabricator.wikimedia.org/T401808) (owner: 10Kosta Harlan) [07:09:07] (03Merged) 10jenkins-bot: CheckUser: Enable ClientHintsInEditRequest on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346542 (https://phabricator.wikimedia.org/T401808) (owner: 10Kosta Harlan) [07:09:32] !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1346542|CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541|CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] [07:09:35] T401808: Allow Client Hints data to be used to prevent the save of edits - https://phabricator.wikimedia.org/T401808 [07:11:56] !log jmm@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [07:11:59] (03CR) 10Giuseppe Lavagetto: [C:03+2] buildx: New driver for building images via Buildx/BuildKit [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) (owner: 10Dduvall) [07:15:37] !log kharlan@deploy1003 kharlan: Backport for [[gerrit:1346542|CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541|CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:15:40] T401808: Allow Client Hints data to be used to prevent the save of edits - https://phabricator.wikimedia.org/T401808 [07:16:30] !log kharlan@deploy1003 kharlan: Continuing with deployment [07:16:42] (03Merged) 10jenkins-bot: buildx: New driver for building images via Buildx/BuildKit [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) (owner: 10Dduvall) [07:22:06] (03PS1) 10Slyngshede: data.yaml: offboarding jan-e [puppet] - 10https://gerrit.wikimedia.org/r/1347283 [07:23:50] !log jmm@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [07:24:09] !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346542|CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541|CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] (duration: 14m 37s) [07:24:12] T401808: Allow Client Hints data to be used to prevent the save of edits - https://phabricator.wikimedia.org/T401808 [07:26:26] (03CR) 10Brouberol: [C:03+2] sqoop_mediawiki: Add weekly change_tag sqoop [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [07:26:34] (03CR) 10Brouberol: [C:03+2] sqoop_mediawiki: Add weekly full snapshot of the actor table [puppet] - 10https://gerrit.wikimedia.org/r/1343954 (https://phabricator.wikimedia.org/T437961) (owner: 10A-pizzata) [07:28:49] !log jmm@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [07:29:05] !log jmm@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [07:30:47] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for mw-api-int mw-jobrunner in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1347293 (https://phabricator.wikimedia.org/T420436) [07:30:50] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for mw-api-int mw-jobrunner in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1347294 (https://phabricator.wikimedia.org/T420436) [07:31:28] !log jmm@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [07:34:09] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1347268 (owner: 10Slyngshede) [07:34:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:39:08] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1347283 (owner: 10Slyngshede) [07:39:33] (03PS4) 10Vgutierrez: purged: use the new host prefixes [puppet] - 10https://gerrit.wikimedia.org/r/1346014 [07:41:48] (03CR) 10Ayounsi: [C:03+2] Add lwatson to deployment and analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1346660 (https://phabricator.wikimedia.org/T438926) (owner: 10Ayounsi) [07:45:57] (03CR) 10Slyngshede: [C:03+2] data.yaml offboarding of andreawest [puppet] - 10https://gerrit.wikimedia.org/r/1347268 (owner: 10Slyngshede) [07:46:42] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1346616 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [07:46:54] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users and deployment for lwatson - https://phabricator.wikimedia.org/T438926#12383709 (10ayounsi) 05Open→03Resolved a:05thcipriani→03None Give it ~30min for the change to propagate and give it a try. Plea... [07:47:14] (03CR) 10Ayounsi: [C:03+2] add tsev to deployment [puppet] - 10https://gerrit.wikimedia.org/r/1346661 (https://phabricator.wikimedia.org/T439138) (owner: 10Ayounsi) [07:47:38] (03CR) 10Slyngshede: [C:03+2] data.yaml: offboarding jan-e [puppet] - 10https://gerrit.wikimedia.org/r/1347283 (owner: 10Slyngshede) [07:48:22] hello team, not sure if the right place but could anyone approve this one: https://gerrit.wikimedia.org/r/c/operations/dns/+/1346553 ? [07:48:48] (03PS2) 10Ayounsi: add tsev to deployment [puppet] - 10https://gerrit.wikimedia.org/r/1346661 (https://phabricator.wikimedia.org/T439138) [07:49:14] dpogorzelski: usually best to ask in #wikimedia-traffic for DNS related questions [07:49:42] -operations is quite noisy so question can be missed [07:50:57] kk :) [07:52:09] (03CR) 10Vgutierrez: [C:03+2] purged: use the new host prefixes [puppet] - 10https://gerrit.wikimedia.org/r/1346014 (owner: 10Vgutierrez) [07:52:20] (03CR) 10Ayounsi: [C:03+2] add tsev to deployment [puppet] - 10https://gerrit.wikimedia.org/r/1346661 (https://phabricator.wikimedia.org/T439138) (owner: 10Ayounsi) [07:52:56] (03PS1) 10Sadiya.mohammed13: Enable mobile editing for all Wikidata editors [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347373 (https://phabricator.wikimedia.org/T437054) [07:54:35] !log upload purged 0.25 to trixie-wikimedia (apt.wm.o) [07:54:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:57:35] (03CR) 10Muehlenhoff: [C:03+1] "Looka good" [puppet] - 10https://gerrit.wikimedia.org/r/1346617 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [07:59:28] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Tsevener - https://phabricator.wikimedia.org/T439138#12383751 (10ayounsi) 05Open→03Resolved Give it 30min for the change to propagate. Please re-open the task if you have any issue. [08:01:33] (03PS12) 10Cathal Mooney: LVS IPIP encapsulation: support backend servers using nftables [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) [08:02:39] (03CR) 10CI reject: [V:04-1] LVS IPIP encapsulation: support backend servers using nftables [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) (owner: 10Cathal Mooney) [08:04:19] (03PS13) 10Cathal Mooney: LVS IPIP encapsulation: support backend servers using nftables [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) [08:05:12] !log installing pcre2/openssl security updates [08:05:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:07:30] 06SRE-OnFire, 06MediaWiki-Engineering, 06ServiceOps, 06SRE Observability, 07Sustainability (Incident Followup): Reduce the amount of messages sent through channel:Memcached during failures - https://phabricator.wikimedia.org/T390529#12383770 (10MLechvien-WMF) This incident is from March last year so it w... [08:09:02] FIRING: SystemdUnitFailed: prometheus_arelion_exporter.service on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:09:03] (03CR) 10Cathal Mooney: LVS IPIP encapsulation: support backend servers using nftables (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) (owner: 10Cathal Mooney) [08:10:20] (03CR) 10Cathal Mooney: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) (owner: 10Cathal Mooney) [08:15:12] (03PS2) 10Giuseppe Lavagetto: sre.hosts.provision: add --no-reboot option to skip host reboot [cookbooks] - 10https://gerrit.wikimedia.org/r/1346499 (https://phabricator.wikimedia.org/T438287) [08:16:22] (03CR) 10Giuseppe Lavagetto: sre.hosts.provision: add --no-reboot option to skip host reboot (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1346499 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [08:19:47] (03PS14) 10Cathal Mooney: LVS IPIP encapsulation: support backend servers using nftables [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) [08:21:41] (03PS2) 10Giuseppe Lavagetto: shellbox: enable polling under gVisor for TLS certificate changes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345556 (https://phabricator.wikimedia.org/T439409) [08:21:42] (03PS1) 10Giuseppe Lavagetto: shellbox-timeline: enable gVisor in codfw as well [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347398 (https://phabricator.wikimedia.org/T438287) [08:22:07] (03PS2) 10Giuseppe Lavagetto: shellbox-timeline: enable gVisor in codfw as well [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347398 (https://phabricator.wikimedia.org/T438287) [08:22:15] !log restart FPM on phab1005 to pick up openssl/PCRE security updates [08:22:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:25:40] (03CR) 10Cathal Mooney: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) (owner: 10Cathal Mooney) [08:26:42] (03PS5) 10Daniel Kertesz: profile::cache::base: adjust the range of ephemeral ports [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) [08:28:03] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Eqiad: Replacement top-of-rack switch for rack C1 - https://phabricator.wikimedia.org/T403031#12383802 (10cmooney) [08:28:28] (03CR) 10Daniel Kertesz: "right, so I changed the approach and instead I bumped to lower bound to 10k like Brett mentioned in his comment above" [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [08:29:14] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12383807 (10Tchanders) Ah, that should be @EMill-WMF [08:30:08] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1346630 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [08:32:40] (03PS1) 10Ayounsi: Prometheus Arelion exporter: handle new cycle [puppet] - 10https://gerrit.wikimedia.org/r/1347412 (https://phabricator.wikimedia.org/T311005) [08:33:46] (03CR) 10Ayounsi: "Tested manually, works as expected." [puppet] - 10https://gerrit.wikimedia.org/r/1347412 (https://phabricator.wikimedia.org/T311005) (owner: 10Ayounsi) [08:34:02] RESOLVED: SystemdUnitFailed: prometheus_arelion_exporter.service on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:34:02] (03Abandoned) 10Giuseppe Lavagetto: shellbox: allow inotify probes to be set for tls certs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345535 (https://phabricator.wikimedia.org/T439409) (owner: 10Giuseppe Lavagetto) [08:34:59] !log dkertesz@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [08:35:03] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [08:35:17] (03Abandoned) 10Giuseppe Lavagetto: shellbox: enable polling under gVisor for TLS certificate changes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345556 (https://phabricator.wikimedia.org/T439409) (owner: 10Giuseppe Lavagetto) [08:36:06] (03CR) 10Vgutierrez: [C:04-1] "app servers are missing liftwing-studio.wm.o in their TLS material:" [puppet] - 10https://gerrit.wikimedia.org/r/1346557 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [08:38:58] !log hashar@deploy1003 Started deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 [08:39:02] FIRING: SystemdUnitFailed: prometheus_arelion_exporter.service on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:39:08] !log hashar@deploy1003 Finished deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 (duration: 00m 10s) [08:46:18] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 761059352 and 28 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [08:47:18] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3761008 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [08:50:22] (03CR) 10Ilias Sarantopoulos: [C:03+1] ml-services: move llm-qwen36-27b to ml-serve1013/1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346621 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [08:51:24] (03PS1) 10Jelto: miscweb wmf-navigator: update urldownloader URL to LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347421 (https://phabricator.wikimedia.org/T439713) [08:51:36] (03PS1) 10Michael Große: instrument(AccountSetup): don't double count finish/skip clicks [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347422 [08:51:52] (03PS1) 10Michael Große: stats(AccountSetup): Record onboarding ready time in Prometheus [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347423 (https://phabricator.wikimedia.org/T432810) [08:52:07] (03PS1) 10Michael Große: instrument(AccountSetup): Send exposure from server, send ready event [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347424 (https://phabricator.wikimedia.org/T432164) [08:52:41] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 01 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347422 (owner: 10Michael Große) [08:52:47] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 01 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347423 (https://phabricator.wikimedia.org/T432810) (owner: 10Michael Große) [08:52:53] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 01 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347424 (https://phabricator.wikimedia.org/T432164) (owner: 10Michael Große) [08:54:30] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347421 (https://phabricator.wikimedia.org/T439713) (owner: 10Jelto) [09:06:38] 06SRE: Bing Webmaster Tools access request for Ash Han - https://phabricator.wikimedia.org/T439646#12384015 (10ayounsi) a:03LSobanski @LSobanski is that something you can help with ? [09:11:31] (03PS1) 10Ayounsi: WIP: Add cookbook to sync BGP sessions with Netbox [cookbooks] - 10https://gerrit.wikimedia.org/r/1347441 (https://phabricator.wikimedia.org/T439408) [09:12:25] (03PS2) 10Dpogorzelski: liftwing-studio: serve on liftwing-studio.wikimedia.org [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346563 (https://phabricator.wikimedia.org/T439416) [09:13:30] (03PS1) 10Elukey: sre.hosts.provision: re-add a 10s sleep before checking ipmi [cookbooks] - 10https://gerrit.wikimedia.org/r/1347442 (https://phabricator.wikimedia.org/T426180) [09:13:55] (03CR) 10CI reject: [V:04-1] WIP: Add cookbook to sync BGP sessions with Netbox [cookbooks] - 10https://gerrit.wikimedia.org/r/1347441 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [09:18:39] (03CR) 10Cathal Mooney: [C:03+1] "lgtm!" [puppet] - 10https://gerrit.wikimedia.org/r/1347412 (https://phabricator.wikimedia.org/T311005) (owner: 10Ayounsi) [09:23:34] (03CR) 10Cathal Mooney: [C:03+1] "I'm not 100% what these are used for but looks good in terms of the change, same result." [puppet] - 10https://gerrit.wikimedia.org/r/1346800 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [09:24:27] (03PS1) 10Ayounsi: netbox: add an option to connect to netbox-next [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) [09:24:46] (03CR) 10Cathal Mooney: [C:03+1] "Thanks for taking care of this Ben much obliged :) I will merge and push out to the network devices." [homer/public] - 10https://gerrit.wikimedia.org/r/1346596 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [09:24:53] (03PS1) 10Ayounsi: Spicerack config: add netbox-next url [puppet] - 10https://gerrit.wikimedia.org/r/1347451 (https://phabricator.wikimedia.org/T439408) [09:24:58] (03CR) 10CI reject: [V:04-1] netbox: add an option to connect to netbox-next [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [09:25:06] (03CR) 10Cathal Mooney: [C:03+2] dse-k8s: remove the old POD IPv4 block 10.67.24.0/21 from prefix lists [homer/public] - 10https://gerrit.wikimedia.org/r/1346596 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [09:26:34] (03Merged) 10jenkins-bot: dse-k8s: remove the old POD IPv4 block 10.67.24.0/21 from prefix lists [homer/public] - 10https://gerrit.wikimedia.org/r/1346596 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [09:26:36] (03PS2) 10Ayounsi: WIP: Add cookbook to sync BGP sessions with Netbox [cookbooks] - 10https://gerrit.wikimedia.org/r/1347441 (https://phabricator.wikimedia.org/T439408) [09:27:30] (03CR) 10Jelto: [C:03+2] miscweb wmf-navigator: update urldownloader URL to LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347421 (https://phabricator.wikimedia.org/T439713) (owner: 10Jelto) [09:27:34] (03PS2) 10Ayounsi: Spicerack config: add netbox-next url [puppet] - 10https://gerrit.wikimedia.org/r/1347451 (https://phabricator.wikimedia.org/T439408) [09:27:40] (03CR) 10CI reject: [V:04-1] WIP: Add cookbook to sync BGP sessions with Netbox [cookbooks] - 10https://gerrit.wikimedia.org/r/1347441 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [09:27:43] (03CR) 10Ayounsi: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347451 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [09:30:05] (03CR) 10Arnaudb: [C:03+1] "looks good to me" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347421 (https://phabricator.wikimedia.org/T439713) (owner: 10Jelto) [09:30:26] (03Merged) 10jenkins-bot: miscweb wmf-navigator: update urldownloader URL to LVS endpoint [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347421 (https://phabricator.wikimedia.org/T439713) (owner: 10Jelto) [09:34:24] (03PS3) 10Ayounsi: WIP: Add cookbook to sync BGP sessions with Netbox [cookbooks] - 10https://gerrit.wikimedia.org/r/1347441 (https://phabricator.wikimedia.org/T439408) [09:35:23] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply [09:35:38] (03CR) 10Vgutierrez: liftwing-studio: serve on liftwing-studio.wikimedia.org (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346563 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [09:35:41] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply [09:41:14] (03CR) 10Kosta Harlan: "I've verified that both of my Yubikeys work" [puppet] - 10https://gerrit.wikimedia.org/r/1346424 (https://phabricator.wikimedia.org/T439665) (owner: 10Kosta Harlan) [09:41:24] (03PS4) 10Kosta Harlan: admin: Remove non-FIDO SSH key for kharlan [puppet] - 10https://gerrit.wikimedia.org/r/1346424 (https://phabricator.wikimedia.org/T439665) [09:42:59] (03CR) 10Cathal Mooney: [C:03+1] "lgtm" [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [09:45:28] (03CR) 10Gkyziridis: [C:03+2] ml-services: move llm-qwen36-27b to ml-serve1013/1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346621 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [09:45:37] (03CR) 10Ayounsi: "recheck" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [09:47:04] !log jelto@deploy1003 helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply [09:47:50] !log jelto@deploy1003 helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply [09:48:00] (03Merged) 10jenkins-bot: ml-services: move llm-qwen36-27b to ml-serve1013/1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346621 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [09:48:03] (03CR) 10Cathal Mooney: "Lgtm based on initial scan. I was hesitating about using the stateful session status (i.e. "show bgp neighbor") rather than the configura" [cookbooks] - 10https://gerrit.wikimedia.org/r/1347441 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [09:55:02] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06tools-platform-team, 06Traffic: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12384184 (10Jelto) The legacy CNAME `url-downloader` URL was removed from wmf-navigator with the patch above. [09:58:20] (03PS1) 10Brouberol: dse-k8s-eqiad/mediawiki-dumps-legacy: increase max pod cpu limits [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347463 [09:59:36] !log dkertesz@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [09:59:39] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1000) [10:00:28] (03CR) 10Kosta Harlan: [C:03+1] admin: Remove non-FIDO SSH key for kharlan [puppet] - 10https://gerrit.wikimedia.org/r/1346424 (https://phabricator.wikimedia.org/T439665) (owner: 10Kosta Harlan) [10:08:33] (03PS2) 10Tiziano Fogli: kafka-logging: add kafka-logging200[6-8] to codfwcluster [puppet] - 10https://gerrit.wikimedia.org/r/1347454 (https://phabricator.wikimedia.org/T432444) [10:08:43] (03PS1) 10Tiziano Fogli: kafka-logging: temporarily disable notifications for kafka-logging100[7-8] [puppet] - 10https://gerrit.wikimedia.org/r/1347456 (https://phabricator.wikimedia.org/T432444) [10:10:48] (03PS1) 10Elukey: WIP: sre.hosts.provision: move DHCP configs to the DHCP cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 [10:12:29] (03PS1) 10Brouberol: airlfow-dumps: inscrease the number of tasks selected by the scheduler in a single pass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347468 [10:12:49] (03CR) 10Jelto: [C:03+1] "lgtm, this is just a copy of 1.16.0" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338959 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [10:13:45] (03CR) 10Btullis: [C:03+1] airlfow-dumps: inscrease the number of tasks selected by the scheduler in a single pass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347468 (owner: 10Brouberol) [10:17:43] 06SRE, 06Infrastructure-Foundations, 06Traffic, 07LDAP: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730#12384261 (10MoritzMuehlenhoff) [10:17:59] (03PS2) 10Brouberol: airlfow-dumps: inscrease the number of tasks selected by the scheduler in a single pass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347468 [10:18:21] (03PS3) 10Brouberol: airlfow-dumps: increase the number of tasks selected by the scheduler in a single pass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347468 [10:18:52] (03CR) 10Btullis: [C:03+1] airlfow-dumps: increase the number of tasks selected by the scheduler in a single pass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347468 (owner: 10Brouberol) [10:20:18] (03PS1) 10Muehlenhoff: Add conftool-data for ldap-ro-test [puppet] - 10https://gerrit.wikimedia.org/r/1347474 (https://phabricator.wikimedia.org/T437730) [10:21:49] (03CR) 10Brouberol: [C:03+2] airlfow-dumps: increase the number of tasks selected by the scheduler in a single pass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347468 (owner: 10Brouberol) [10:22:12] (03CR) 10Elukey: [C:03+1] Spicerack config: add netbox-next url [puppet] - 10https://gerrit.wikimedia.org/r/1347451 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [10:22:32] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [10:22:54] (03CR) 10Elukey: [C:03+1] Prometheus Arelion exporter: handle new cycle [puppet] - 10https://gerrit.wikimedia.org/r/1347412 (https://phabricator.wikimedia.org/T311005) (owner: 10Ayounsi) [10:22:54] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [10:23:01] (03PS3) 10Giuseppe Lavagetto: shellbox-timeline: enable gVisor in codfw as well [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347398 (https://phabricator.wikimedia.org/T438287) [10:23:50] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:24:48] !log ammarpad@deploy1003 mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=metawiki Jakehekd Ivyeclipse # T439805 [10:24:51] T439805: Unblock stuck global rename of Ivyeclipse - https://phabricator.wikimedia.org/T439805 [10:27:30] (03CR) 10Giuseppe Lavagetto: [C:03+2] shellbox-timeline: enable gVisor in codfw as well [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347398 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [10:28:16] (03PS1) 10Giuseppe Lavagetto: sre.hosts.provision: enable virtualization by default for wikikube-worker [cookbooks] - 10https://gerrit.wikimedia.org/r/1347480 (https://phabricator.wikimedia.org/T438287) [10:29:50] (03Merged) 10jenkins-bot: shellbox-timeline: enable gVisor in codfw as well [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347398 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [10:30:23] (03PS1) 10Svantje Lilienthal: Revert "Use explicit numbers in reference lists and Parsoid rendering" [extensions/Cite] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347483 (https://phabricator.wikimedia.org/T439790) [10:30:27] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [10:30:39] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST events) on k8s-dse@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s-dse&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [10:30:59] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [10:33:10] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 01 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/Cite] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347483 (https://phabricator.wikimedia.org/T439790) (owner: 10Svantje Lilienthal) [10:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [10:38:19] (03CR) 10TrainBranchBot: [C:03+2] "Approved by javiermonton@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346516 (https://phabricator.wikimedia.org/T437875) (owner: 10JavierMonton) [10:39:36] (03Merged) 10jenkins-bot: stream: mediawiki.job.pageviewTrendingRelativeUpdate [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346516 (https://phabricator.wikimedia.org/T437875) (owner: 10JavierMonton) [10:40:00] !log javiermonton@deploy1003 Started scap sync-world: Backport for [[gerrit:1346516|stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] [10:40:04] T437875: New Schema for Mediawiki output job - https://phabricator.wikimedia.org/T437875 [10:42:37] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12384367 (10MoritzMuehlenhoff) [10:44:10] !log javiermonton@deploy1003 javiermonton: Backport for [[gerrit:1346516|stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [10:45:34] !log javiermonton@deploy1003 javiermonton: Continuing with deployment [10:50:51] !log javiermonton@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346516|stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] (duration: 10m 51s) [10:50:55] T437875: New Schema for Mediawiki output job - https://phabricator.wikimedia.org/T437875 [10:53:14] (03CR) 10Slyngshede: [C:03+1] "I think this is a good solution for now. The concern is that it could be a little confusing that a sign out from IDM also signs you out fr" [software/bitu] - 10https://gerrit.wikimedia.org/r/1341185 (https://phabricator.wikimedia.org/T392350) (owner: 10Arendpieter) [10:53:57] (03CR) 10Slyngshede: [C:03+1] "We'll just get a second +1 and I can merge it and plan a new release." [software/bitu] - 10https://gerrit.wikimedia.org/r/1341185 (https://phabricator.wikimedia.org/T392350) (owner: 10Arendpieter) [10:57:40] !log jmm@cumin2003 START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw [11:00:14] (03CR) 10Jelto: [C:03+1] "looks good" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338751 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [11:05:39] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST events) on k8s-dse@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s-dse&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [11:05:48] (03CR) 10Arnaudb: "thanks for the review! no problem for me, if you want to test it with etherpad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338751 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [11:06:11] !log jmm@cumin2003 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw [11:08:41] (03PS3) 10Dpogorzelski: liftwing-studio: serve on liftwing-studio.wikimedia.org [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346563 (https://phabricator.wikimedia.org/T439416) [11:09:01] (03CR) 10Dpogorzelski: liftwing-studio: serve on liftwing-studio.wikimedia.org (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346563 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [11:09:07] (03PS1) 10Gkyziridis: ml-services: move llm-qwen38-27b to ml-serve1013/1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347502 (https://phabricator.wikimedia.org/T439555) [11:12:10] (03CR) 10Hnowlan: "One nit but lgtm after that!" [puppet] - 10https://gerrit.wikimedia.org/r/1347454 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [11:12:42] (03CR) 10Hnowlan: kafka-logging: temporarily disable notifications for kafka-logging100[7-8] (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1347456 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [11:13:23] (03PS3) 10Tiziano Fogli: kafka-logging: add kafka-logging200[6-8] to codfwcluster [puppet] - 10https://gerrit.wikimedia.org/r/1347454 (https://phabricator.wikimedia.org/T432444) [11:13:48] jouncebot: nowandnext [11:13:48] No deployments scheduled for the next 0 hour(s) and 46 minute(s) [11:13:48] In 0 hour(s) and 46 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1200) [11:13:56] (03CR) 10JMeybohm: [C:03+1] bird: add a new prometheus-exporter for bird. [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1343943 (https://phabricator.wikimedia.org/T423851) (owner: 10Blake) [11:14:04] (03CR) 10Hnowlan: [C:03+1] kafka-logging: add kafka-logging200[6-8] to codfwcluster [puppet] - 10https://gerrit.wikimedia.org/r/1347454 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [11:14:13] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for mw-api-int mw-jobrunner in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1347293 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [11:14:24] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for mw-api-int mw-jobrunner in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1347294 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [11:14:42] (03CR) 10Blake: bird: add a new prometheus-exporter for bird. (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1343943 (https://phabricator.wikimedia.org/T423851) (owner: 10Blake) [11:14:48] (03PS6) 10Blake: bird: add a new prometheus-exporter for bird. [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1343943 (https://phabricator.wikimedia.org/T423851) [11:14:55] (03CR) 10Tiziano Fogli: kafka-logging: add kafka-logging200[6-8] to codfwcluster (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1347454 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [11:15:32] (03CR) 10Vgutierrez: [C:03+1] liftwing-studio: serve on liftwing-studio.wikimedia.org [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346563 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [11:15:58] (03PS2) 10Tiziano Fogli: kafka-logging: temporarily disable notifications for kafka-logging200[6-8] [puppet] - 10https://gerrit.wikimedia.org/r/1347456 (https://phabricator.wikimedia.org/T432444) [11:15:58] (03PS4) 10Tiziano Fogli: kafka-logging: add kafka-logging200[6-8] to codfw cluster [puppet] - 10https://gerrit.wikimedia.org/r/1347454 (https://phabricator.wikimedia.org/T432444) [11:16:23] (03CR) 10Ilias Sarantopoulos: [C:03+1] "all good!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347502 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [11:16:26] (03CR) 10JMeybohm: [C:03+1] sre.hosts.provision: add --no-reboot option to skip host reboot (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1346499 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [11:17:03] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345600 (https://phabricator.wikimedia.org/T439409) (owner: 10Giuseppe Lavagetto) [11:19:28] (03CR) 10Blake: [V:03+2 C:03+2] bird: add a new prometheus-exporter for bird. [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1343943 (https://phabricator.wikimedia.org/T423851) (owner: 10Blake) [11:19:39] (03PS1) 10JavierMonton: stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347509 (https://phabricator.wikimedia.org/T437875) [11:20:11] (03PS1) 10Majavah: P:openstack: Remove pointless inline_template() usage [puppet] - 10https://gerrit.wikimedia.org/r/1347510 [11:20:11] (03PS1) 10Majavah: cloudnfs: Replace use of inline_template [puppet] - 10https://gerrit.wikimedia.org/r/1347511 [11:21:17] (03CR) 10Muehlenhoff: [C:03+1] "Looks good, thanks! This will be helpful for those users who mostly only interact with IDM in terms of SSO." [software/bitu] - 10https://gerrit.wikimedia.org/r/1341185 (https://phabricator.wikimedia.org/T392350) (owner: 10Arendpieter) [11:21:39] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST events) on k8s-dse@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s-dse&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [11:21:58] (03CR) 10Slyngshede: [C:03+2] Logout: End the OpenID Connect session at the provider [software/bitu] - 10https://gerrit.wikimedia.org/r/1341185 (https://phabricator.wikimedia.org/T392350) (owner: 10Arendpieter) [11:24:07] (03Merged) 10jenkins-bot: Logout: End the OpenID Connect session at the provider [software/bitu] - 10https://gerrit.wikimedia.org/r/1341185 (https://phabricator.wikimedia.org/T392350) (owner: 10Arendpieter) [11:24:17] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: serve on liftwing-studio.wikimedia.org [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346563 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [11:24:34] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] liftwing-studio: serve on liftwing-studio.wikimedia.org [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346563 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [11:26:58] (03CR) 10Gkyziridis: [C:03+2] ml-services: move llm-qwen38-27b to ml-serve1013/1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347502 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [11:28:04] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [11:29:17] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [11:29:21] (03Merged) 10jenkins-bot: ml-services: move llm-qwen38-27b to ml-serve1013/1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347502 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [11:30:21] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [11:30:30] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [11:30:52] (03Abandoned) 10Dpogorzelski: dse-k8s: add liftwing-studio.wikimedia.org to the namespace certificate [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346562 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [11:31:39] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST events) on k8s-dse@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s-dse&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [11:31:47] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [11:32:22] !log jmm@cumin2003 START - Cookbook sre.dns.netbox [11:34:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:35:15] (03PS3) 10Giuseppe Lavagetto: kubernetes::deployment_server: setup daily timer to reload certificates [puppet] - 10https://gerrit.wikimedia.org/r/1345600 (https://phabricator.wikimedia.org/T439409) [11:36:43] !log jmm@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" [11:39:47] jmm@cumin2003 netbox (PID 2257043) is awaiting input [11:39:55] (03PS1) 10Jforrester: RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users [extensions/WikiLambda] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347530 (https://phabricator.wikimedia.org/T437533) [11:40:02] (03CR) 10Dpogorzelski: "changes are now applied on admin_ng/service level" [puppet] - 10https://gerrit.wikimedia.org/r/1346557 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [11:40:13] (03CR) 10Giuseppe Lavagetto: [V:03+1] "PCC SUCCESS (CORE_DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9492/co" [puppet] - 10https://gerrit.wikimedia.org/r/1345600 (https://phabricator.wikimedia.org/T439409) (owner: 10Giuseppe Lavagetto) [11:40:18] (03CR) 10Vgutierrez: [C:03+1] "looking good now:" [puppet] - 10https://gerrit.wikimedia.org/r/1346557 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [11:40:36] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jforrester@deploy1003 using scap backport" [extensions/WikiLambda] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347530 (https://phabricator.wikimedia.org/T437533) (owner: 10Jforrester) [11:43:02] (03Merged) 10jenkins-bot: RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users [extensions/WikiLambda] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347530 (https://phabricator.wikimedia.org/T437533) (owner: 10Jforrester) [11:43:13] (03CR) 10Hnowlan: kafka-logging: add kafka-logging200[6-8] to codfw cluster [puppet] - 10https://gerrit.wikimedia.org/r/1347454 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [11:43:24] (03CR) 10Hnowlan: [C:03+1] kafka-logging: temporarily disable notifications for kafka-logging200[6-8] [puppet] - 10https://gerrit.wikimedia.org/r/1347456 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [11:43:28] !log jforrester@deploy1003 Started scap sync-world: Backport for [[gerrit:1347530|RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] [11:43:31] T437533: Arrange for "Freshen Result" requests to be rate limited - https://phabricator.wikimedia.org/T437533 [11:45:03] (03CR) 10Ozge: [C:03+1] ml-services: move llm-qwen38-27b to ml-serve1013/1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347502 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [11:45:44] (03CR) 10A-pizzata: [C:03+1] stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347509 (https://phabricator.wikimedia.org/T437875) (owner: 10JavierMonton) [11:47:32] (03PS1) 10Gkyziridis: ml-services: move cope-b-a4b to ml-serve1013/1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347534 (https://phabricator.wikimedia.org/T439555) [11:47:38] !log jforrester@deploy1003 jforrester: Backport for [[gerrit:1347530|RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:49:37] (03PS2) 10Gkyziridis: ml-services: move cope-b-a4b to ml-serve1013/1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347534 (https://phabricator.wikimedia.org/T439555) [11:50:12] !log jmm@cumin2003 START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad [11:50:27] !log jforrester@deploy1003 jforrester: Continuing with deployment [11:51:16] 06SRE, 06Infrastructure-Foundations, 06Traffic, 07LDAP: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730#12384563 (10MoritzMuehlenhoff) Initial steps: 1. Patch for conftool-data is at https://gerrit.wikimedia.org/r/1347474 2. No discovery record is used/needed 3. VIPs were cr... [11:55:55] !log jforrester@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347530|RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] (duration: 12m 27s) [11:55:59] T437533: Arrange for "Freshen Result" requests to be rate limited - https://phabricator.wikimedia.org/T437533 [11:58:26] !log jmm@cumin2003 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad [12:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1200) [12:00:35] 10SRE-SLO, 03Abstract Wikipedia Fix-It tasks, 06Abstract Wikipedia team (27Q2 (Oct–Dec)), 07Essential-Work: Sloth 30-day SLI over-counts errors for our low-traffic SLO, wikilambda-parsoid-combined - https://phabricator.wikimedia.org/T439571#12384651 (10Jdforrester-WMF) [12:02:06] (03CR) 10Ayounsi: [C:03+2] admin: Remove non-FIDO SSH key for kharlan [puppet] - 10https://gerrit.wikimedia.org/r/1346424 (https://phabricator.wikimedia.org/T439665) (owner: 10Kosta Harlan) [12:02:18] (03CR) 10Brouberol: [C:03+2] dse-k8s-eqiad/mediawiki-dumps-legacy: increase max pod cpu limits [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347463 (owner: 10Brouberol) [12:02:56] 06SRE, 06Infrastructure-Foundations, 10netops: Missing series for BGP session_state from eqiad CRs since upgrade to 23.4R2-S8.7 - https://phabricator.wikimedia.org/T435909#12384669 (10cmooney) [12:03:11] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for kharlan - https://phabricator.wikimedia.org/T439665#12384672 (10ayounsi) 05Open→03Resolved a:03ayounsi All done, give it 30min for the removal to propagate. Please reopen if any issue. [12:03:37] (03CR) 10Ayounsi: [C:03+2] Prometheus Arelion exporter: handle new cycle [puppet] - 10https://gerrit.wikimedia.org/r/1347412 (https://phabricator.wikimedia.org/T311005) (owner: 10Ayounsi) [12:05:44] !log dkertesz@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [12:05:47] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [12:05:52] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [12:06:40] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [12:06:55] (03CR) 10Ayounsi: [C:03+1] sre.hosts.provision: re-add a 10s sleep before checking ipmi (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1347442 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [12:07:59] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for kharlan - https://phabricator.wikimedia.org/T439665#12384679 (10kostajh) Thanks! [12:08:32] (03CR) 10Ayounsi: [C:03+2] Spicerack config: add netbox-next url [puppet] - 10https://gerrit.wikimedia.org/r/1347451 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [12:09:02] RESOLVED: SystemdUnitFailed: prometheus_arelion_exporter.service on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:10:04] jouncebot: nowandnext [12:10:04] For the next 0 hour(s) and 49 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1200) [12:10:04] In 0 hour(s) and 49 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1300) [12:14:49] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply [12:15:06] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply [12:15:30] (03CR) 10JMeybohm: [C:04-1] kubernetes::deployment_server: setup daily timer to reload certificates (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345600 (https://phabricator.wikimedia.org/T439409) (owner: 10Giuseppe Lavagetto) [12:19:21] (03CR) 10Ozge: [C:03+1] ml-services: move cope-b-a4b to ml-serve1013/1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347534 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [12:19:42] !log a-pizzata@deploy1003 Started deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] [12:20:29] !log a-pizzata@deploy1003 Finished deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] (duration: 00m 46s) [12:21:16] !log a-pizzata@deploy1003 Started deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] [12:22:34] !log installing glibc security updates [12:22:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:23:36] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [12:24:16] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [12:25:16] (03PS4) 10Giuseppe Lavagetto: kubernetes::deployment_server: setup daily timer to reload certificates [puppet] - 10https://gerrit.wikimedia.org/r/1345600 (https://phabricator.wikimedia.org/T439409) [12:25:31] (03CR) 10Giuseppe Lavagetto: kubernetes::deployment_server: setup daily timer to reload certificates (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345600 (https://phabricator.wikimedia.org/T439409) (owner: 10Giuseppe Lavagetto) [12:25:43] !log a-pizzata@deploy1003 Finished deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] (duration: 04m 27s) [12:27:09] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T439299#12384743 (10Jclark-ctr) [12:27:10] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T439570#12384745 (10Jclark-ctr) →14Duplicate dup:03T439299 [12:27:28] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply [12:27:54] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply [12:28:23] (03CR) 10Gkyziridis: [C:03+2] ml-services: move cope-b-a4b to ml-serve1013/1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347534 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [12:28:46] (03Abandoned) 10Thiemo Kreuz (WMDE): Helmfile for continuous deployment [deployment-charts] - 10https://gerrit.wikimedia.org/r/634354 (https://phabricator.wikimedia.org/T214158) (owner: 10Jeena Huneidi) [12:29:33] (03Abandoned) 10Thiemo Kreuz (WMDE): Add mounts property to scaffold values [deployment-charts] - 10https://gerrit.wikimedia.org/r/570137 (owner: 10Jeena Huneidi) [12:30:53] (03Merged) 10jenkins-bot: ml-services: move cope-b-a4b to ml-serve1013/1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347534 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [12:32:13] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [12:33:28] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [12:34:19] !log installing libtasn1-6 security updates [12:34:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:34:28] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [12:37:36] (03CR) 10Lerickson: [C:03+2] Annotate wdqs-proxy pods so metrics are scraped. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346743 (https://phabricator.wikimedia.org/T439449) (owner: 10Lerickson) [12:37:41] (03CR) 10Arnaudb: "following https://w.wiki/W3GU and adding @jmeybohm@wikimedia.org to reviewers 😊" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341855 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [12:37:45] (03PS2) 10Atsuko: airflow3: smtp_default connection [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346713 (https://phabricator.wikimedia.org/T433383) [12:38:11] jouncebot: nowandnext [12:38:11] For the next 0 hour(s) and 21 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1200) [12:38:11] In 0 hour(s) and 21 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1300) [12:39:02] (03Merged) 10jenkins-bot: Annotate wdqs-proxy pods so metrics are scraped. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346743 (https://phabricator.wikimedia.org/T439449) (owner: 10Lerickson) [12:39:19] (03CR) 10Atsuko: airflow3: smtp_default connection (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346713 (https://phabricator.wikimedia.org/T433383) (owner: 10Atsuko) [12:40:22] !log ihurbain@deploy1003 helmfile [eqiad] START helmfile.d/services/mobileapps: apply [12:41:37] (03PS1) 10Btullis: dse-k8s-eqiad: Reassign dse-k8s-worker100[1-4] to the right storageclass [puppet] - 10https://gerrit.wikimedia.org/r/1347580 (https://phabricator.wikimedia.org/T430654) [12:42:11] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347580 (https://phabricator.wikimedia.org/T430654) (owner: 10Btullis) [12:44:09] !log ihurbain@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [12:44:26] !log ihurbain@deploy1003 helmfile [codfw] START helmfile.d/services/mobileapps: apply [12:44:32] (03PS1) 10Filippo Giunchedi: installserver: use cephosd for cloudcephosd1* [puppet] - 10https://gerrit.wikimedia.org/r/1347585 (https://phabricator.wikimedia.org/T419892) [12:45:26] !log ihurbain@deploy1003 helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [12:45:56] (03CR) 10Klausman: [C:03+1] cache: expose liftwing-studio at liftwing-studio.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1346557 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [12:47:02] (03CR) 10Btullis: [C:03+1] "Looks good to me." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346713 (https://phabricator.wikimedia.org/T433383) (owner: 10Atsuko) [12:47:13] Hi! I have a private code change to deploy. Would anyone mind if I do it now? [12:47:57] (03CR) 10Klausman: [C:03+1] dse-k8s-eqiad: Reassign dse-k8s-worker100[1-4] to the right storageclass [puppet] - 10https://gerrit.wikimedia.org/r/1347580 (https://phabricator.wikimedia.org/T430654) (owner: 10Btullis) [12:47:59] (03CR) 10Filippo Giunchedi: [C:03+2] "Hosts not yet in service, self-merging" [puppet] - 10https://gerrit.wikimedia.org/r/1347585 (https://phabricator.wikimedia.org/T419892) (owner: 10Filippo Giunchedi) [12:50:05] Deploying [12:50:25] (03PS1) 10Cathal Mooney: Interface validator: allow non-virtual child int on SR-Linux breakout [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) [12:51:11] (03CR) 10Atsuko: [C:03+2] airflow3: smtp_default connection [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346713 (https://phabricator.wikimedia.org/T433383) (owner: 10Atsuko) [12:52:01] (03PS2) 10Cathal Mooney: Interface validator: allow non-virtual child int on SR-Linux breakout [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) [12:53:18] (03CR) 10JavierMonton: [C:03+2] stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347509 (https://phabricator.wikimedia.org/T437875) (owner: 10JavierMonton) [12:53:25] (03Merged) 10jenkins-bot: airflow3: smtp_default connection [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346713 (https://phabricator.wikimedia.org/T433383) (owner: 10Atsuko) [12:53:40] !log mszwarc@deploy1003 mszwarc: Sync PrivateSettings.php synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:53:41] (03CR) 10Arnaudb: "@hashar@free.fr is there a reason for this not to be merged as a "double safety" in addition to the rules that are already at the edge? ha" [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) (owner: 10Arnaudb) [12:54:24] !log mszwarc@deploy1003 mszwarc: Continuing with deployment [12:54:37] !log filippo@cumin1004 START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm [12:55:34] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12384852 (10EMill-WMF) Yes, I approve Eva for access here. And sorry @Eric! I think the fact that we... [12:55:54] (03Merged) 10jenkins-bot: stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347509 (https://phabricator.wikimedia.org/T437875) (owner: 10JavierMonton) [12:57:09] (03CR) 10CI reject: [V:04-1] Interface validator: allow non-virtual child int on SR-Linux breakout [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [12:58:06] (03PS3) 10Cathal Mooney: Interface validator: allow non-virtual child int on SR-Linux breakout [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) [12:59:32] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [12:59:43] !log mszwarc@deploy1003 Synchronized private/PrivateSettings.php: Sync PrivateSettings.php (duration: 10m 02s) [12:59:55] Finished deploying [12:59:58] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: gettimeofday() says it's time for UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1300) [13:00:05] MichaelG_WMF and awight: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:01:17] (03PS2) 10Ayounsi: Add eniroomand to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1346659 (https://phabricator.wikimedia.org/T439644) [13:02:25] (03CR) 10CI reject: [V:04-1] Add eniroomand to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1346659 (https://phabricator.wikimedia.org/T439644) (owner: 10Ayounsi) [13:02:28] * MichaelG_WMF is here [13:03:46] Do you need a deployer? [13:03:52] (03PS1) 10Majavah: O:wmcs::waf: Add empty role [puppet] - 10https://gerrit.wikimedia.org/r/1347595 (https://phabricator.wikimedia.org/T439418) [13:04:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:05:03] (03CR) 10Awight: [C:03+1] Revert "Use explicit numbers in reference lists and Parsoid rendering" [extensions/Cite] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347483 (https://phabricator.wikimedia.org/T439790) (owner: 10Svantje Lilienthal) [13:05:45] !log jmm@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" [13:05:46] !log jmm@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:06:38] (03CR) 10JMeybohm: [C:03+1] kubernetes::deployment_server: setup daily timer to reload certificates [puppet] - 10https://gerrit.wikimedia.org/r/1345600 (https://phabricator.wikimedia.org/T439409) (owner: 10Giuseppe Lavagetto) [13:06:42] !log oblivian@deploy1003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply [13:07:03] !log oblivian@deploy1003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply [13:07:31] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [13:08:04] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [13:08:06] !log jmm@cumin2003 START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe [13:08:29] MichaelG_WMF: I showed up late so maybe I missed it: are you self-deploying, or can I help? [13:08:31] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12384952 (10Eric) >>! In T439644#12384852, @EMill-WMF wrote: > And sorry @Eric! I think the fact that... [13:08:53] awight: I cannot deploy, it would be great if you could help! [13:08:56] (03PS1) 10Majavah: Add dummy hiddenparma password [labs/private] - 10https://gerrit.wikimedia.org/r/1347598 (https://phabricator.wikimedia.org/T439418) [13:09:00] sure! [13:09:41] (03CR) 10TrainBranchBot: [C:03+2] "Approved by awight@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347422 (owner: 10Michael Große) [13:09:41] (03CR) 10TrainBranchBot: [C:03+2] "Approved by awight@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347423 (https://phabricator.wikimedia.org/T432810) (owner: 10Michael Große) [13:09:42] (03CR) 10TrainBranchBot: [C:03+2] "Approved by awight@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347424 (https://phabricator.wikimedia.org/T432164) (owner: 10Michael Große) [13:09:43] Msz2001: sorry, I did not see your message. I was in a different tab and am usually listening to pings [13:09:55] np [13:10:42] (03CR) 10Giuseppe Lavagetto: [C:03+2] kubernetes::deployment_server: setup daily timer to reload certificates [puppet] - 10https://gerrit.wikimedia.org/r/1345600 (https://phabricator.wikimedia.org/T439409) (owner: 10Giuseppe Lavagetto) [13:11:20] (03CR) 10Awight: [C:03+1] instrument(AccountSetup): don't double count finish/skip clicks [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347422 (owner: 10Michael Große) [13:12:01] (03CR) 10Awight: [C:03+1] stats(AccountSetup): Record onboarding ready time in Prometheus [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347423 (https://phabricator.wikimedia.org/T432810) (owner: 10Michael Große) [13:12:18] !log jmm@cumin2003 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe [13:13:25] (03CR) 10Dpogorzelski: [C:03+2] cache: expose liftwing-studio at liftwing-studio.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1346557 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [13:14:00] (03CR) 10Awight: [C:03+1] instrument(AccountSetup): Send exposure from server, send ready event [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347424 (https://phabricator.wikimedia.org/T432164) (owner: 10Michael Große) [13:15:28] (03CR) 10Anne Tomasevich: [C:03+1] Enable ReadingList CTA on Minerva for all reading list wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346053 (https://phabricator.wikimedia.org/T438779) (owner: 10Jdlrobson) [13:15:56] (03CR) 10Brouberol: [C:03+1] "PCC diff looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1347580 (https://phabricator.wikimedia.org/T430654) (owner: 10Btullis) [13:16:54] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:16:57] FIRING: [3x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:16:59] (03CR) 10Dpogorzelski: [C:03+2] idp: switch liftwing-studio to liftwing-studio.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1346556 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [13:17:53] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:18:04] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:18:24] (03Merged) 10jenkins-bot: instrument(AccountSetup): don't double count finish/skip clicks [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347422 (owner: 10Michael Große) [13:18:26] (03Merged) 10jenkins-bot: stats(AccountSetup): Record onboarding ready time in Prometheus [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347423 (https://phabricator.wikimedia.org/T432810) (owner: 10Michael Große) [13:18:27] (03Merged) 10jenkins-bot: instrument(AccountSetup): Send exposure from server, send ready event [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347424 (https://phabricator.wikimedia.org/T432164) (owner: 10Michael Große) [13:18:55] !log awight@deploy1003 Started scap sync-world: Backport for [[gerrit:1347422|instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423|stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424|instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] [13:18:57] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply [13:19:00] T432810: Account setup: Aggregate stats — Grafana (DE1.3.1) - https://phabricator.wikimedia.org/T432810 [13:19:00] T432164: Instrumentation: DE1.3.1 Early onboarding experiment - https://phabricator.wikimedia.org/T432164 [13:19:24] PROBLEM - Druid historical on an-druid1006 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [13:19:24] PROBLEM - Druid historical on an-druid1007 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [13:19:42] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply [13:20:56] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [13:21:29] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [13:21:40] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [13:21:40] (03PS2) 10Majavah: Add dummy hiddenparma database password [labs/private] - 10https://gerrit.wikimedia.org/r/1347598 (https://phabricator.wikimedia.org/T439418) [13:21:57] RESOLVED: [3x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:22:18] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [13:22:35] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply [13:23:07] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply [13:23:12] !log awight@deploy1003 awight, migr: Backport for [[gerrit:1347422|instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423|stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424|instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be ve [13:23:12] rified there. [13:23:35] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:23:40] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:24:15] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply [13:24:19] MichaelG_WMF: let me know whether it tests okay [13:24:30] awight: testing... [13:24:42] !log oblivian@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: sync [13:24:45] !log oblivian@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [13:24:47] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply [13:24:49] !log oblivian@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [13:24:54] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply [13:25:00] !log oblivian@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [13:25:32] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply [13:25:42] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:25:46] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:25:53] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply [13:26:03] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:26:08] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:26:27] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply [13:26:59] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12385020 (10MoritzMuehlenhoff) [13:27:07] (03PS1) 10Scott French: mw-(jobrunner|pretrain): Set route-level stream idle time on jobrunners [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347604 (https://phabricator.wikimedia.org/T439577) [13:28:32] (03PS1) 10VolkerE: fix(ReadingRecommendations): Update button style to primary [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347606 (https://phabricator.wikimedia.org/T436332) [13:29:22] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [13:29:23] awight: do I have to do something special? [13:29:34] MichaelG_WMF: to test, or to let me know? hehe [13:29:35] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [13:29:37] because I'm not seeing the expected changes with mwdebug [13:29:41] ooh! [13:29:55] I wonder if it's a ResourceLoader caching thing? [13:29:56] or did the test-servers change? [13:30:08] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply [13:30:10] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [13:30:12] Last I tested (yesterday) it was k8s-mwdebug [13:30:15] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [13:30:22] I tried that as well yesterday [13:30:25] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [13:30:31] it worked yesterday is what I mean [13:30:34] but not today? [13:30:40] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 01 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347606 (https://phabricator.wikimedia.org/T436332) (owner: 10VolkerE) [13:30:42] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply [13:30:43] though there is k8s-mwdebug-next [13:30:47] let me try that [13:30:50] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply [13:30:51] O_O [13:30:55] I doubt it [13:31:19] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply [13:31:20] > Finished sync-testservers-k8s [13:31:25] RECOVERY - Druid historical on an-druid1007 is OK: PROCS OK: 1 process with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [13:31:29] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply [13:31:44] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply [13:32:00] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply [13:32:02] awight: ugh I'm an idiot [13:32:17] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply [13:32:21] * MichaelG_WMF goes to testwiki, which actually has -wmf.22 [13:32:22] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply [13:32:24] MichaelG_WMF: jfyi https://en.wikipedia.org/w/extensions/GrowthExperiments/modules/ext.growthExperiments.AccountSetup/init.js [13:32:30] haha kk I'm glad there was an answer [13:32:53] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [13:32:59] do you need to backport to wmf.22 as well? [13:33:00] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply [13:33:15] (because wmf.21 is going away today, if the train is successful) [13:33:26] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply [13:33:29] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply [13:34:00] awight: now it worked as expected [13:34:17] !log filippo@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS bookworm [13:34:19] kk! I will finish deploying then. Let me know if you want to do wmf.22 too [13:34:36] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [13:34:38] awight: this backport was to wmf.22 and on testwiki it works :). It will land on enwiki later today [13:35:02] but that was the reason that my testing on enwiki failed 🫠 [13:36:29] !log awight@deploy1003 awight, migr: Continuing with deployment [13:37:33] okay that's right. I'm now surprised that I could see the new file on enwiki. Nothing to see here... [13:38:25] RECOVERY - Druid historical on an-druid1006 is OK: PROCS OK: 1 process with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [13:40:32] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [13:40:47] (03PS2) 10Muehlenhoff: Add conftool-data for ldap-ro-test [puppet] - 10https://gerrit.wikimedia.org/r/1347474 (https://phabricator.wikimedia.org/T437730) [13:41:24] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [13:41:48] !log awight@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347422|instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423|stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424|instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] (duration: 22m 53s) [13:41:52] T432810: Account setup: Aggregate stats — Grafana (DE1.3.1) - https://phabricator.wikimedia.org/T432810 [13:41:52] T432164: Instrumentation: DE1.3.1 Early onboarding experiment - https://phabricator.wikimedia.org/T432164 [13:42:59] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [13:43:21] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [13:44:24] lilients: okay here we go! [13:45:37] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [13:45:56] (03CR) 10TrainBranchBot: [C:03+2] "Approved by awight@deploy1003 using scap backport" [extensions/Cite] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347483 (https://phabricator.wikimedia.org/T439790) (owner: 10Svantje Lilienthal) [13:45:56] (03CR) 10Btullis: [C:03+2] dse-k8s-eqiad: Reassign dse-k8s-worker100[1-4] to the right storageclass [puppet] - 10https://gerrit.wikimedia.org/r/1347580 (https://phabricator.wikimedia.org/T430654) (owner: 10Btullis) [13:45:57] I am here [13:45:59] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [13:46:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [13:47:08] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [13:47:41] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [13:47:43] (03PS2) 10Ayounsi: netbox: add an option to connect to netbox-next [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) [13:48:10] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [13:48:36] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [13:49:57] (03PS1) 10Isabelle Hurbain-Palatin: mobile-apps: add mathjax csp rules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347615 [13:50:26] 10SRE-swift-storage, 10Cloud-VPS (Debian Bullseye Deprecation): Migrate swift away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435783#12385202 (10komla) Will you be able to meet a deadline of October 31st? [13:51:37] (03PS3) 10Ayounsi: Add eniroomand to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1346659 (https://phabricator.wikimedia.org/T439644) [13:51:44] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12385216 (10fgiunchedi) Ok `cephosd.cfg` is not going to work, and that's not surprising since I used the wrong recipe as these systems are efi, not bios,... [13:52:56] 10SRE-SLO, 06SRE Observability, 10Test Kitchen: Support SLOs based on Test Kitchen instruments - https://phabricator.wikimedia.org/T438274#12385224 (10JVanderhoop-WMF) Hello @RLazarus! Yes, this has long been in our minds (as @Ottomata 's links suggest). Do you have a list of potential client-side performa... [13:52:57] (03Merged) 10jenkins-bot: Revert "Use explicit numbers in reference lists and Parsoid rendering" [extensions/Cite] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347483 (https://phabricator.wikimedia.org/T439790) (owner: 10Svantje Lilienthal) [13:53:02] 10ops-eqiad, 06SRE, 06DC-Ops, 10ServiceOps-Upgrades-Hardware, 06ServiceOps (Next quarter): Q1:rack/setup/install conf101[0-2] - https://phabricator.wikimedia.org/T435426#12385226 (10Scott_French) a:05Scott_French→03None Just realized I was still assigned to this. Un-assigning per T435426#12234869, si... [13:53:19] !log awight@deploy1003 Started scap sync-world: Backport for [[gerrit:1347483|Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] [13:53:22] T439790: References' footnote numbers no longer display on en.Wiktionary.org - https://phabricator.wikimedia.org/T439790 [13:53:37] (03CR) 10Jgiannelos: [C:03+1] mobile-apps: add mathjax csp rules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347615 (owner: 10Isabelle Hurbain-Palatin) [13:53:52] (03CR) 10Ayounsi: [C:03+2] Add eniroomand to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1346659 (https://phabricator.wikimedia.org/T439644) (owner: 10Ayounsi) [13:54:01] (03CR) 10Jgiannelos: mobile-apps: add mathjax csp rules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347615 (owner: 10Isabelle Hurbain-Palatin) [13:55:31] (03PS2) 10Isabelle Hurbain-Palatin: mobile-apps: add mathjax csp rules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347615 (https://phabricator.wikimedia.org/T431693) [13:55:53] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12385235 (10ayounsi) 05Open→03Resolved Give it 30min for the change to propagate and you shoul... [13:57:38] !log awight@deploy1003 lilients, awight: Backport for [[gerrit:1347483|Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:57:46] (03PS2) 10Krinkle: scap: Remove unused php7_admin_port option [puppet] - 10https://gerrit.wikimedia.org/r/1219164 (https://phabricator.wikimedia.org/T224491) [13:58:30] !log awight@deploy1003 lilients, awight: Continuing with deployment [14:01:47] !log dkertesz@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [14:01:51] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [14:02:08] (03CR) 10Ayounsi: "Thanks!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [14:02:23] (03PS7) 10Ayounsi: network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) [14:02:57] (03CR) 10JHathaway: [C:03+2] pybal: replace legacy fact [puppet] - 10https://gerrit.wikimedia.org/r/1346800 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [14:03:45] !log awight@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347483|Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] (duration: 10m 26s) [14:03:49] T439790: References' footnote numbers no longer display on en.Wiktionary.org - https://phabricator.wikimedia.org/T439790 [14:04:06] lol 'replace legacy fact' it's like i've stepped into the ministry of truth [14:04:10] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12385299 (10aputhin) [14:04:26] !log EU afternoon backport complete [14:04:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:04:47] (03PS1) 10Slyngshede: Signout: Handle custom OIDC backend for signout [software/bitu] - 10https://gerrit.wikimedia.org/r/1347622 [14:05:02] ejegg: All truths must be compatible with pybal3 [14:05:44] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for mw-api-int mw-jobrunner in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1347293 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [14:06:26] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw [14:07:36] (03PS1) 10Muehlenhoff: Add service catalogue entry for ldap-ro-test [puppet] - 10https://gerrit.wikimedia.org/r/1347624 (https://phabricator.wikimedia.org/T439730) [14:08:27] (03CR) 10Ayounsi: [C:03+2] network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [14:09:41] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [14:10:46] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [14:10:46] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw [14:10:56] (03PS2) 10Jelto: service::catalog: Set ipip_encapsulation for mw-api-int mw-jobrunner in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1347294 (https://phabricator.wikimedia.org/T420436) [14:11:50] (03Merged) 10jenkins-bot: network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [14:14:16] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for mw-api-int mw-jobrunner in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1347294 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [14:14:40] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad [14:17:39] (03PS1) 10Muehlenhoff: Configure both LDAP replicas to be manageable by ldap-ro-test [puppet] - 10https://gerrit.wikimedia.org/r/1347630 (https://phabricator.wikimedia.org/T439730) [14:18:23] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [14:18:51] (03CR) 10Ayounsi: [C:03+1] "overall lgtm, 2 nits" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [14:19:33] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [14:19:33] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad [14:26:30] (03CR) 10Muehlenhoff: [C:03+1] "Looks good!" [software/bitu] - 10https://gerrit.wikimedia.org/r/1347622 (owner: 10Slyngshede) [14:26:47] 06SRE, 06Infrastructure-Foundations, 06Traffic, 07LDAP, 13Patch-For-Review: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730#12385512 (10MoritzMuehlenhoff) And: 4. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347624 to add it to the service catalogue 5. https://ger... [14:26:54] (03PS1) 10Urbanecm: NotificationController: Use lazyPush() for the delete job [extensions/Echo] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347638 (https://phabricator.wikimedia.org/T439626) [14:27:08] (03PS1) 10Urbanecm: NotificationController: Use lazyPush() for the delete job [extensions/Echo] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1347639 (https://phabricator.wikimedia.org/T439626) [14:27:16] jouncebot: nowandnext [14:27:16] No deployments scheduled for the next 0 hour(s) and 2 minute(s) [14:27:16] In 0 hour(s) and 2 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1430) [14:27:31] (03CR) 10Urbanecm: [C:03+2] NotificationController: Use lazyPush() for the delete job [extensions/Echo] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347638 (https://phabricator.wikimedia.org/T439626) (owner: 10Urbanecm) [14:27:33] (03CR) 10Urbanecm: [C:03+2] NotificationController: Use lazyPush() for the delete job [extensions/Echo] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1347639 (https://phabricator.wikimedia.org/T439626) (owner: 10Urbanecm) [14:29:12] !log dpogorzelski@dns1004 START - running authdns-update [14:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1430) [14:31:36] !log dpogorzelski@dns1004 END - running authdns-update [14:32:16] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for mw-web* and mw-api* in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1347640 (https://phabricator.wikimedia.org/T420436) [14:32:18] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for mw-web* and mw-api* in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1347641 (https://phabricator.wikimedia.org/T420436) [14:32:47] !log installing e2fsprogs bugfix update from trixie point release [14:32:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [14:34:15] (03CR) 10LorenMora: [C:03+1] fix(ReadingRecommendations): Update button style to primary [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347606 (https://phabricator.wikimedia.org/T436332) (owner: 10VolkerE) [14:34:31] (03Merged) 10jenkins-bot: NotificationController: Use lazyPush() for the delete job [extensions/Echo] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347638 (https://phabricator.wikimedia.org/T439626) (owner: 10Urbanecm) [14:36:54] (03Merged) 10jenkins-bot: NotificationController: Use lazyPush() for the delete job [extensions/Echo] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1347639 (https://phabricator.wikimedia.org/T439626) (owner: 10Urbanecm) [14:37:06] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy1003 using scap backport" [extensions/Echo] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1347639 (https://phabricator.wikimedia.org/T439626) (owner: 10Urbanecm) [14:37:24] (03PS4) 10Cathal Mooney: Interface validator: check Nokia breakout ports have parent [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) [14:37:29] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1347638|NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639|NotificationController: Use lazyPush() for the delete job (T439626)]] [14:37:32] T439626: Echo: NotificationController::enqueueDeleteJob should use lazyPush rather than push - https://phabricator.wikimedia.org/T439626 [14:38:10] (03PS1) 10JHathaway: puppetdb: rename spec and format [puppet] - 10https://gerrit.wikimedia.org/r/1347642 [14:38:40] (03PS2) 10JHathaway: puppetdb: rename spec and format [puppet] - 10https://gerrit.wikimedia.org/r/1347642 (https://phabricator.wikimedia.org/T372666) [14:41:06] (03CR) 10JHathaway: [C:03+2] puppetdb: rename spec and format [puppet] - 10https://gerrit.wikimedia.org/r/1347642 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [14:41:15] (03CR) 10Cathal Mooney: Interface validator: check Nokia breakout ports have parent (032 comments) [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [14:41:45] !log urbanecm@deploy1003 urbanecm: Backport for [[gerrit:1347638|NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639|NotificationController: Use lazyPush() for the delete job (T439626)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:42:17] !log installing zsh bugfix update from trixie point release [14:42:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:42:20] !log urbanecm@deploy1003 urbanecm: Continuing with deployment [14:44:12] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12385651 (10MoritzMuehlenhoff) [14:44:46] (03CR) 10Ayounsi: [C:03+1] Interface validator: check Nokia breakout ports have parent [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [14:47:34] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347638|NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639|NotificationController: Use lazyPush() for the delete job (T439626)]] (duration: 10m 05s) [14:47:38] T439626: Echo: NotificationController::enqueueDeleteJob should use lazyPush rather than push - https://phabricator.wikimedia.org/T439626 [14:47:42] (03CR) 10Ahmon Dancy: "Still LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1219164 (https://phabricator.wikimedia.org/T224491) (owner: 10Krinkle) [14:48:28] !log installing busybox bugfix updates from trixie point release [14:48:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:48:33] (03CR) 10Clément Goubert: [C:03+1] mw-(jobrunner|pretrain): Set route-level stream idle time on jobrunners [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347604 (https://phabricator.wikimedia.org/T439577) (owner: 10Scott French) [14:52:57] (03CR) 10Jdlrobson: [C:03+1] Enable ReadingList CTA on Minerva for all reading list wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346053 (https://phabricator.wikimedia.org/T438779) (owner: 10Jdlrobson) [14:53:45] FIRING: WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:55:22] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12385715 (10MoritzMuehlenhoff) [14:56:28] !log installing initramfs-tools bugfix updates from trixie point release [14:56:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:00:29] (03PS1) 10Clément Goubert: wikikube: Add lac-lambdas namespace [puppet] - 10https://gerrit.wikimedia.org/r/1347649 (https://phabricator.wikimedia.org/T439879) [15:00:59] (03PS1) 10Zabe: Add missing alias for filerevision [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347650 [15:01:11] (03PS1) 10Zabe: Add missing alias for filerevision [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1347652 [15:01:30] (03PS2) 10Clément Goubert: wikikube: Add lac-lambdas namespace [puppet] - 10https://gerrit.wikimedia.org/r/1347649 (https://phabricator.wikimedia.org/T439879) [15:02:36] jouncebot: nowandnext [15:02:36] No deployments scheduled for the next 0 hour(s) and 57 minute(s) [15:02:36] In 0 hour(s) and 57 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1600) [15:03:13] (03PS1) 10Clément Goubert: admin_ng: Add lac-lambdas namespace to wikikube [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347653 (https://phabricator.wikimedia.org/T439879) [15:03:58] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.5 point update - https://phabricator.wikimedia.org/T427072#12385776 (10MoritzMuehlenhoff) [15:07:02] (03CR) 10Zabe: [C:03+2] Add missing alias for filerevision [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1347652 (owner: 10Zabe) [15:07:03] (03CR) 10Zabe: [C:03+2] Add missing alias for filerevision [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347650 (owner: 10Zabe) [15:07:36] (03PS3) 10Muehlenhoff: Remove Python 2 exception from Hadoop nodes [puppet] - 10https://gerrit.wikimedia.org/r/1338126 [15:08:08] !log cdobbins@cumin1004 conftool action : set/pooled=no; selector: name=tcp-proxy.* [15:08:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:08:56] !log cdobbins@cumin1004 conftool action : set/pooled=yes; selector: name=tcp-proxy.* [15:09:51] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1347652|Add missing alias for filerevision]], [[gerrit:1347650|Add missing alias for filerevision]] [15:11:24] !log installing postgresql security updates [15:11:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:12:58] (03CR) 10Muehlenhoff: [C:03+2] Remove Python 2 exception from Hadoop nodes [puppet] - 10https://gerrit.wikimedia.org/r/1338126 (owner: 10Muehlenhoff) [15:13:31] (03CR) 10Muehlenhoff: "check" [alerts] - 10https://gerrit.wikimedia.org/r/1345966 (https://phabricator.wikimedia.org/T439440) (owner: 10Muehlenhoff) [15:14:02] !log zabe@deploy1003 zabe: Backport for [[gerrit:1347652|Add missing alias for filerevision]], [[gerrit:1347650|Add missing alias for filerevision]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:14:47] !log zabe@deploy1003 zabe: Continuing with deployment [15:20:04] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347652|Add missing alias for filerevision]], [[gerrit:1347650|Add missing alias for filerevision]] (duration: 10m 12s) [15:20:29] 10SRE-SLO, 03Abstract Wikipedia Fix-It tasks, 06Abstract Wikipedia team (27Q2 (Oct–Dec)), 07Essential-Work: Sloth 30-day SLI over-counts errors for our low-traffic SLO, wikilambda-parsoid-combined - https://phabricator.wikimedia.org/T439571#12385818 (10Jdforrester-WMF) SLOs are just moving to https://grafa... [15:24:05] (03CR) 10Vgutierrez: [C:03+1] wikimedia.org: add liftwing-studio [dns] - 10https://gerrit.wikimedia.org/r/1346553 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [15:24:48] (03CR) 10Dpogorzelski: [C:03+2] wikimedia.org: add liftwing-studio [dns] - 10https://gerrit.wikimedia.org/r/1346553 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [15:25:09] !log dpogorzelski@dns1004 START - running authdns-update [15:25:25] (03PS1) 10Giuseppe Lavagetto: kubernetes::deployment_server: setup daily timer to reload certificates [puppet] - 10https://gerrit.wikimedia.org/r/1347668 (https://phabricator.wikimedia.org/T439409) [15:25:27] (03PS1) 10Giuseppe Lavagetto: kubernetes::deployment_server: check and reload jaeger for cert changes [puppet] - 10https://gerrit.wikimedia.org/r/1347669 (https://phabricator.wikimedia.org/T439246) [15:25:47] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439786#12385857 (10Jhancock.wm) a:03Jhancock.wm last alert @ Oct 1, 2026, 9:08:24 AM. attempted to balance. check back later today. [15:26:20] !log cdobbins@cumin1004 START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy [15:29:22] !log dpogorzelski@dns1004 END - running authdns-update [15:32:10] !log installing socat security updates [15:32:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:32:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:34:02] FIRING: NetworkDeviceAlarmActive: Alarm active on ssw1-f1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-f1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [15:35:10] (03CR) 10Giuseppe Lavagetto: [C:03+2] kubernetes::deployment_server: check and reload jaeger for cert changes [puppet] - 10https://gerrit.wikimedia.org/r/1347669 (https://phabricator.wikimedia.org/T439246) (owner: 10Giuseppe Lavagetto) [15:35:43] (03CR) 10Giuseppe Lavagetto: [C:03+2] kubernetes::deployment_server: setup daily timer to reload certificates [puppet] - 10https://gerrit.wikimedia.org/r/1347668 (https://phabricator.wikimedia.org/T439409) (owner: 10Giuseppe Lavagetto) [15:35:53] (03CR) 10Giuseppe Lavagetto: kubernetes::deployment_server: check and reload jaeger for cert changes [puppet] - 10https://gerrit.wikimedia.org/r/1347669 (https://phabricator.wikimedia.org/T439246) (owner: 10Giuseppe Lavagetto) [15:37:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:38:56] (03PS1) 10Jdlrobson: EditWatchlist: use same view on mobile and desktop [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347658 (https://phabricator.wikimedia.org/T363032) [15:39:12] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy [15:39:58] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () [15:40:00] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () [15:40:26] (03CR) 10Elukey: sre.hosts.provision: re-add a 10s sleep before checking ipmi (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1347442 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [15:40:46] (03PS1) 10Zabe: Add dblist to control wbc_entity_usage migration progress [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347671 (https://phabricator.wikimedia.org/T439753) [15:42:07] (03CR) 10CI reject: [V:04-1] Add dblist to control wbc_entity_usage migration progress [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347671 (https://phabricator.wikimedia.org/T439753) (owner: 10Zabe) [15:43:52] (03PS2) 10Zabe: Add dblist to control wbc_entity_usage migration progress [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347671 (https://phabricator.wikimedia.org/T439753) [15:44:17] (03PS24) 10Bking: WIP: Add Matomo frontend helm chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) [15:44:45] (03CR) 10Bking: "The chart path has been fixed." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [15:45:19] (03PS1) 10Zabe: Move reads/writes of wbc_entity_usage to x1 for s5 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347672 (https://phabricator.wikimedia.org/T439753) [15:49:37] (03Abandoned) 10Zabe: Add dblist to control wbc_entity_usage migration progress [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347671 (https://phabricator.wikimedia.org/T439753) (owner: 10Zabe) [15:50:36] (03PS25) 10Bking: Add Matomo frontend helm chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) [15:51:51] (03PS2) 10Zabe: Move reads/writes of wbc_entity_usage to x1 for s5 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347672 (https://phabricator.wikimedia.org/T439753) [15:52:18] (03PS1) 10Ameisenigel: Make azwikimedia a SUL wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347674 (https://phabricator.wikimedia.org/T409145) [15:52:27] (03PS3) 10Zabe: Move reads/writes of wbc_entity_usage to x1 for s5 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347672 (https://phabricator.wikimedia.org/T439753) [15:55:42] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:00:04] jhathaway and rzl: gettimeofday() says it's time for Puppet request window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1600) [16:00:04] No Gerrit patches in the queue for this window AFAICS. [16:00:32] (03CR) 10Btullis: "How are you ensuring that requests to the tracker submissions endpoint (matomo.php I believe) do not requre the use of the CAS SSO authent" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [16:05:42] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:06:43] (03PS1) 10Kamila Součková: php8.3: Rebuild to pick up 8.3.35-1+wmf12u2 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1347678 (https://phabricator.wikimedia.org/T439100) [16:12:09] !log cdobbins@cumin1004 START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum [16:12:22] (03CR) 10Scott French: [C:03+1] php8.3: Rebuild to pick up 8.3.35-1+wmf12u2 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1347678 (https://phabricator.wikimedia.org/T439100) (owner: 10Kamila Součková) [16:14:19] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 608789360 and 40 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [16:16:12] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:19] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2641616 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [16:20:34] (03CR) 10Kamila Součková: [V:03+1] "verified locally, see on task" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1347678 (https://phabricator.wikimedia.org/T439100) (owner: 10Kamila Součková) [16:20:58] (03CR) 10Kamila Součková: [V:03+1 C:03+2] php8.3: Rebuild to pick up 8.3.35-1+wmf12u2 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1347678 (https://phabricator.wikimedia.org/T439100) (owner: 10Kamila Součková) [16:21:02] (03CR) 10Kamila Součková: [V:03+2 C:03+2] php8.3: Rebuild to pick up 8.3.35-1+wmf12u2 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1347678 (https://phabricator.wikimedia.org/T439100) (owner: 10Kamila Součková) [16:21:12] RESOLVED: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:23:35] (03PS5) 10Cathal Mooney: Interface validator: check Nokia breakout ports have parent [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) [16:25:08] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum [16:25:57] (03CR) 10RLazarus: [C:03+1] wikikube: Add lac-lambdas namespace [puppet] - 10https://gerrit.wikimedia.org/r/1347649 (https://phabricator.wikimedia.org/T439879) (owner: 10Clément Goubert) [16:26:02] (03CR) 10RLazarus: [C:03+1] admin_ng: Add lac-lambdas namespace to wikikube [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347653 (https://phabricator.wikimedia.org/T439879) (owner: 10Clément Goubert) [16:26:31] (03PS1) 10Matthias Mullie: Lose DB call for page prop [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347686 (https://phabricator.wikimedia.org/T439182) [16:26:38] (03PS2) 10Clément Goubert: admin_ng: Add lac-lambdas namespace to wikikube [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347653 (https://phabricator.wikimedia.org/T439879) [16:26:38] (03PS4) 10Clément Goubert: article-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) [16:27:12] (03PS1) 10Matthias Mullie: Extract images in parser post-processing [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347687 (https://phabricator.wikimedia.org/T439182) [16:27:56] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 01 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347686 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [16:28:04] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 01 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347687 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [16:29:36] (03PS5) 10Clément Goubert: article-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) [16:30:12] (03PS1) 10JHathaway: puppetdb::query_facts: switch to inventory endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1347689 (https://phabricator.wikimedia.org/T372666) [16:31:15] jouncebot: nowandnext [16:31:15] For the next 0 hour(s) and 28 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1600) [16:31:15] In 0 hour(s) and 28 minute(s): Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1700) [16:31:15] In 0 hour(s) and 28 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1700) [16:31:30] (03CR) 10CI reject: [V:04-1] puppetdb::query_facts: switch to inventory endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1347689 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [16:31:57] Raine: nothing in flight, all yours [16:32:24] (03PS2) 10JHathaway: puppetdb::query_facts: switch to inventory endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1347689 (https://phabricator.wikimedia.org/T372666) [16:36:27] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347689 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [16:37:34] (03PS1) 10Srapoj: Fix UTF-8 corruption in newline sanitization [extensions/timeline] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347692 (https://phabricator.wikimedia.org/T431960) [16:38:14] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 01 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/timeline] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347692 (https://phabricator.wikimedia.org/T431960) (owner: 10Srapoj) [16:40:07] (03PS1) 10Btullis: kubernetes: Add an option to select the service account key type [puppet] - 10https://gerrit.wikimedia.org/r/1347693 (https://phabricator.wikimedia.org/T439865) [16:40:10] (03PS1) 10Btullis: dse-k8s-codfw: Sign service account tokens with an RSA key [puppet] - 10https://gerrit.wikimedia.org/r/1347694 (https://phabricator.wikimedia.org/T439865) [16:42:30] (03PS1) 10Clément Goubert: article-feature-counts: service catalog, listener [puppet] - 10https://gerrit.wikimedia.org/r/1347695 (https://phabricator.wikimedia.org/T439879) [16:42:57] (03PS2) 10Clément Goubert: article-feature-counts: service catalog, listener [puppet] - 10https://gerrit.wikimedia.org/r/1347695 (https://phabricator.wikimedia.org/T437000) [16:43:22] 10SRE-Access-Requests, 06Infrastructure-Foundations, 10LDAP-Access-Requests, 13Patch-For-Review: Superset data access request for abibendall - https://phabricator.wikimedia.org/T430938#12386391 (10herron) 05Stalled→03Invalid Temporarily closing this as invalid (incomplete) due to inactivity, please... [16:43:49] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347693 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [16:45:19] (03PS1) 10Clément Goubert: Add article-features-counts ingress CNAME [dns] - 10https://gerrit.wikimedia.org/r/1347696 (https://phabricator.wikimedia.org/T437000) [16:47:35] (03PS2) 10Clément Goubert: Add article-feature-counts ingress CNAME [dns] - 10https://gerrit.wikimedia.org/r/1347696 (https://phabricator.wikimedia.org/T437000) [16:48:44] (03CR) 10Bking: "Sorry, I should have made it clear that I'm not enabling CAS in this patch (see L228 in `charts/matomo/values.yaml` . I was planning on ad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [16:54:29] jouncebot: nowandnext [16:54:29] For the next 0 hour(s) and 5 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1600) [16:54:30] In 0 hour(s) and 5 minute(s): Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1700) [16:54:30] In 0 hour(s) and 5 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1700) [16:55:34] (03CR) 10Tiziano Fogli: kafka-logging: temporarily disable notifications for kafka-logging200[6-8] (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1347456 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [16:55:37] (03CR) 10Tiziano Fogli: [C:03+2] kafka-logging: temporarily disable notifications for kafka-logging200[6-8] [puppet] - 10https://gerrit.wikimedia.org/r/1347456 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [16:56:23] 06SRE, 10SRE-Access-Requests: Requesting access to Deployment for GreyOlson/GOlson-WMF - https://phabricator.wikimedia.org/T439143#12386472 (10herron) Hi @thcipriani @dancy, could you please review/approve the groupadd to deployment? Thanks! [16:57:00] (03CR) 10Xcollazo: "This one could use careful review as I am in no way good at puppet stuff." [puppet] - 10https://gerrit.wikimedia.org/r/1346764 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [16:57:20] (03CR) 10Xcollazo: "This one could also use careful review as I am in no way good at puppet stuff." [puppet] - 10https://gerrit.wikimedia.org/r/1346765 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [16:57:32] 06SRE, 10LDAP-Access-Requests: Grant Access to <500> for  - https://phabricator.wikimedia.org/T437488#12386493 (10herron) 05Stalled→03Invalid Temporarily closing this as invalid (incomplete) due to inactivity, please re-open with the requested details when ready to resume. Thanks! [16:58:11] 06SRE, 10LDAP-Access-Requests: Grant Access to wmf, for WMF staff/contractors nda group for gsduser - https://phabricator.wikimedia.org/T435852#12386498 (10herron) 05Stalled→03Invalid Temporarily closing this as invalid (incomplete) due to inactivity, please re-open with the requested details when read... [16:59:35] Hi there, I'm going to merge (and deploy) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347454 to add the new hosts to the kafka-logging cluster in codfw. [17:00:03] FIRING: ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:00:05] bd808: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1700). [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1700) [17:00:25] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for mw-web* and mw-api* in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1347640 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [17:00:40] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for mw-web* and mw-api* in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1347641 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [17:00:47] (03CR) 10Herron: [C:03+1] kafka-logging: add kafka-logging200[6-8] to codfw cluster [puppet] - 10https://gerrit.wikimedia.org/r/1347454 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [17:00:57] (03CR) 10Tiziano Fogli: [C:03+2] kafka-logging: add kafka-logging200[6-8] to codfw cluster [puppet] - 10https://gerrit.wikimedia.org/r/1347454 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [17:01:11] (03CR) 10Herron: [C:03+1] alert: add a dummy Klaxon managers incident URL [labs/private] - 10https://gerrit.wikimedia.org/r/1345218 (owner: 10Andrea Denisse) [17:01:14] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [17:01:18] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [17:01:56] rzl: thanks! [17:02:16] Raine: was true then, not sure if it is now, check with tappof :) [17:02:23] oh, meep [17:02:44] tappof: ok to start a MW deploy? I want to rebuild images [17:03:32] Raine: I just need to run puppet-agent on deploy1003, and then you can start. Ok? [17:03:39] ok, thanks [17:04:14] (03PS2) 10Btullis: kubernetes: Add an option to select the service account key type [puppet] - 10https://gerrit.wikimedia.org/r/1347693 (https://phabricator.wikimedia.org/T439865) [17:04:14] (03PS2) 10Btullis: dse-k8s-codfw: Sign service account tokens with an RSA key [puppet] - 10https://gerrit.wikimedia.org/r/1347694 (https://phabricator.wikimedia.org/T439865) [17:04:26] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347693 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [17:04:29] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347694 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [17:04:51] tappof: ping me when done, please :-) [17:05:03] RESOLVED: ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:05:17] Raine: You'll see some changes in the diff because changing the hosts in the Kafka logging cluster affects many deployments. I'll let you know once puppet has finished, thanks again. [17:07:17] ack [17:09:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [17:12:00] !log tappof@deploy1003 helmfile [eqiad] START helmfile.d/admin 'sync'. [17:12:15] !log tappof@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'sync'. [17:12:59] !log tappof@deploy1003 helmfile [codfw] START helmfile.d/admin 'sync'. [17:13:35] !log tappof@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'sync'. [17:13:40] Raine: You're free to go ahead [17:14:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [17:15:29] (03PS1) 10Xcollazo: alertmanager: send message to #data-engineering-emergency when alert is resolved [puppet] - 10https://gerrit.wikimedia.org/r/1347702 (https://phabricator.wikimedia.org/T432312) [17:16:16] !log kamila@deploy1003 Started scap sync-world: rebuild for T439100 [17:16:54] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:17:23] (03CR) 10Aleksandar Mastilovic: [C:03+1] alertmanager: send message to #data-engineering-emergency when alert is resolved [puppet] - 10https://gerrit.wikimedia.org/r/1347702 (https://phabricator.wikimedia.org/T432312) (owner: 10Xcollazo) [17:20:01] (03CR) 10Xcollazo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347702 (https://phabricator.wikimedia.org/T432312) (owner: 10Xcollazo) [17:20:20] (03PS1) 10Ssingh: admin: add backup Yubikey for sukhe [puppet] - 10https://gerrit.wikimedia.org/r/1347704 [17:20:59] (03PS2) 10Ssingh: admin: add backup Yubikey for sukhe [puppet] - 10https://gerrit.wikimedia.org/r/1347704 [17:24:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [17:29:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [17:33:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [17:37:37] (03PS1) 10Zabe: Add missing select of fr_timestamp [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1347710 [17:37:47] (03PS1) 10Zabe: Add missing select of fr_timestamp [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347711 [17:38:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [17:40:14] (03CR) 10RLazarus: [C:03+1] "Some suggestions, happy to take another look if you like but feel free to merge without waiting for another roundtrip." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [17:40:58] (03CR) 10RLazarus: [C:03+1] admin_ng: Add lac-lambdas namespace to wikikube [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347653 (https://phabricator.wikimedia.org/T439879) (owner: 10Clément Goubert) [17:42:10] (03CR) 10RLazarus: [C:03+1] article-feature-counts: service catalog, listener [puppet] - 10https://gerrit.wikimedia.org/r/1347695 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [17:42:20] (03CR) 10RLazarus: [C:03+1] Add article-feature-counts ingress CNAME [dns] - 10https://gerrit.wikimedia.org/r/1347696 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [17:44:52] (03PS1) 10VolkerE: instrument(WelcomeSurvey): align motivation event with AccountSetup [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347712 (https://phabricator.wikimedia.org/T439887) [17:45:29] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 01 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347712 (https://phabricator.wikimedia.org/T439887) (owner: 10VolkerE) [17:46:19] (03CR) 10Kamila Součková: [C:03+1] "LGTM based on the linked changes" [puppet] - 10https://gerrit.wikimedia.org/r/1219164 (https://phabricator.wikimedia.org/T224491) (owner: 10Krinkle) [17:46:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:49:05] !log kamila@deploy1003 Finished scap sync-world: rebuild for T439100 (duration: 33m 25s) [17:49:36] Raine: any concerns if I sneak a noop deploy to shellbox* services into the remainder of the window? [17:49:55] (literal noop, in that there are no diffs; this is just to test an orchestration layer bolted on top) [17:50:19] (03CR) 10Kamila Součková: "@jmeybohm@wikimedia.org the previous rsyslog image change was yours, do you remember what the reasoning was for going to both experimental" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346712 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [17:50:29] swfrench-wmf: go ahead [17:50:37] and yes please :D [17:50:42] Raine: awesome, thank you :) [17:50:56] (you know, we should talk about shellbox someday) [17:51:26] (("we need to talk about shellbox" sounds like a version of "I'm breaking up with you", just more threatening)) [17:51:47] haha [17:52:03] !log swfrench@deploy1003 Started scap deploy-service shellbox: Test deploy-service configuration for Shellbox [17:52:10] !log spiderpig@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply [17:52:10] !log spiderpig@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: apply [17:52:10] !log spiderpig@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: apply [17:52:11] !log spiderpig@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-media: apply [17:52:11] !log spiderpig@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-constraints: apply [17:52:11] !log spiderpig@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-video: apply [17:52:13] !log spiderpig@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: apply [17:52:13] !log spiderpig@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-media: apply [17:52:13] !log spiderpig@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [17:52:13] !log spiderpig@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply [17:52:13] !log spiderpig@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-video: apply [17:52:14] !log spiderpig@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply [17:53:14] !log spiderpig@deploy1003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply [17:53:14] !log spiderpig@deploy1003 helmfile [codfw] START helmfile.d/services/shellbox-media: apply [17:53:14] !log spiderpig@deploy1003 helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply [17:53:14] !log spiderpig@deploy1003 helmfile [codfw] START helmfile.d/services/shellbox-video: apply [17:53:14] !log spiderpig@deploy1003 helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply [17:53:14] w00t w00t [17:53:15] !log spiderpig@deploy1003 helmfile [codfw] START helmfile.d/services/shellbox: apply [17:53:17] !log spiderpig@deploy1003 helmfile [codfw] DONE helmfile.d/services/shellbox: apply [17:53:17] !log spiderpig@deploy1003 helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply [17:53:17] !log spiderpig@deploy1003 helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [17:53:17] !log spiderpig@deploy1003 helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply [17:53:17] !log spiderpig@deploy1003 helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply [17:53:18] !log spiderpig@deploy1003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply [17:53:29] * Raine didn't realise scap deploy-service was already that alive [17:53:34] :D [17:54:28] Oooh. [17:54:48] !log spiderpig@deploy1003 helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply [17:54:48] !log spiderpig@deploy1003 helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply [17:54:48] !log spiderpig@deploy1003 helmfile [eqiad] START helmfile.d/services/shellbox-media: apply [17:54:48] !log spiderpig@deploy1003 helmfile [eqiad] START helmfile.d/services/shellbox: apply [17:54:48] !log spiderpig@deploy1003 helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply [17:54:49] !log spiderpig@deploy1003 helmfile [eqiad] START helmfile.d/services/shellbox-video: apply [17:54:50] !log spiderpig@deploy1003 helmfile [eqiad] DONE helmfile.d/services/shellbox: apply [17:54:50] !log spiderpig@deploy1003 helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply [17:54:51] !log spiderpig@deploy1003 helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply [17:54:51] !log spiderpig@deploy1003 helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply [17:54:51] !log spiderpig@deploy1003 helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [17:54:52] !log spiderpig@deploy1003 helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply [17:55:32] !log swfrench@deploy1003 Finished scap deploy-service shellbox: Test deploy-service configuration for Shellbox (duration: 03m 28s) [17:56:00] a great many thanks to dancy, this is indeed a thing now :) [17:57:27] oh wow that's a lot more simultaneous than `charlie -s shellbox*` [17:57:35] (03CR) 10Xcollazo: "Not sure why puppet is unhappy. I does seem like the right `Hosts:` tag?" [puppet] - 10https://gerrit.wikimedia.org/r/1347702 (https://phabricator.wikimedia.org/T432312) (owner: 10Xcollazo) [17:57:45] also presumably involves a lot less "reviewing the same diff twelve to eighteen times" [17:58:57] (03PS1) 10Bking: sre.hadoop.roll-restart-workers: implement start-datetime logic [cookbooks] - 10https://gerrit.wikimedia.org/r/1347713 (https://phabricator.wikimedia.org/T437713) [17:59:09] rzl: indeed, it's a lot more MediaWiki deployment shaped, in that you can separate releases into a (fixed) set of deployment stages, but within stage, the updates are generally concurrent. and yes, similar to what we have for MediaWiki, diff review is front-loaded [17:59:23] (with the caveat that it may drift if the repo changes) [17:59:59] nod, cool [18:00:05] brennen and jeena: Your horoscope predicts another MediaWiki train - Utc-7 Version deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1800). [18:00:28] I had "named service sets" somewhere in the back of my mind as a future charlie feature, but it sounds like that's obsoleted, that's great :) [18:00:38] o/ [18:00:47] holding train for a bit while doing some log triage [18:00:58] swfrench-wmf: yay! [18:01:04] (if anybody knows what gives with T439902, i welcome input.) [18:01:04] T439902: Wikimedia\Rdbms\DBQueryError: Error 1205: Lock wait timeout exceeded; try restarting transactionFunction: MediaWiki\JobQueue\Jobs\CategoryCountUpdateJob::updateCategoryCountsQuery: UPDATE `category` SET cat_pages = cat_pages - https://phabricator.wikimedia.org/T439902 [18:01:46] (03CR) 10Dzahn: [V:03+1 C:03+2] "dzahn@wikistats-bookworm:~$ /usr/bin/php /usr/lib/wikistats/update.php wk" [puppet] - 10https://gerrit.wikimedia.org/r/1346817 (owner: 10Dzahn) [18:03:05] (03CR) 10Dzahn: [V:03+1 C:03+2] "dzahn@wikistats-bookworm:~$ /usr/bin/php /usr/lib/wikistats/update.php lx" [puppet] - 10https://gerrit.wikimedia.org/r/1346814 (https://phabricator.wikimedia.org/T262148) (owner: 10Dzahn) [18:03:34] (03PS3) 10Dzahn: wikistats: absent the update timers for gentoo and wikkii [puppet] - 10https://gerrit.wikimedia.org/r/1346817 [18:05:42] (03CR) 10Dzahn: [C:03+2] wikistats: absent the update timers for gentoo and wikkii [puppet] - 10https://gerrit.wikimedia.org/r/1346817 (owner: 10Dzahn) [18:06:06] (03PS2) 10Dzahn: wikistats: remove absented gentoo and wikkii jobs and tables [puppet] - 10https://gerrit.wikimedia.org/r/1346819 [18:10:42] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:12:19] !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () [18:16:25] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [18:16:29] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [18:20:42] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:25:15] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12386911 (10eniroomand) @ayounsi It works, I can now see the data that was previously hidden. A follow-up question (let... [18:25:19] !log 1.47.0-wmf.22 train status (T438218): logs kind of a mess, but no current blockers, rolling to all wikis [18:25:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:25:22] T438218: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218 [18:25:36] (03PS1) 10TrainBranchBot: group2 to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347715 (https://phabricator.wikimedia.org/T438218) [18:25:39] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by brennen@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347715 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [18:27:08] (03Merged) 10jenkins-bot: group2 to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347715 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [18:30:42] (03CR) 10Muehlenhoff: [C:03+1] "Looks good!" [puppet] - 10https://gerrit.wikimedia.org/r/1347689 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [18:31:48] (03PS1) 10Eric Gardner: Preferred Sources: Move all eligibility checks to the client [extensions/ReaderExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347718 (https://phabricator.wikimedia.org/T436692) [18:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [18:36:23] !log disabling puppet fleet wide to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347689 [18:36:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:39:17] (03PS1) 10DErenrich: Enable the discord previews from WikimediaCustomizations on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347724 (https://phabricator.wikimedia.org/T439914) [18:40:07] (03CR) 10DErenrich: [C:04-2] "waiting on code merge before enabling" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347724 (https://phabricator.wikimedia.org/T439914) (owner: 10DErenrich) [18:40:51] !log brennen@deploy1003 rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.22 refs T438218 [18:40:55] T438218: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218 [18:42:00] !log brett@cumin1004 END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () [18:47:02] (03CR) 10JHathaway: [C:03+2] puppetdb::query_facts: switch to inventory endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1347689 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [18:51:12] (03PS1) 10Kimberly Sarabia: Make carousel open MMV directly [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347729 (https://phabricator.wikimedia.org/T439611) [18:55:38] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 01 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347729 (https://phabricator.wikimedia.org/T439611) (owner: 10Kimberly Sarabia) [18:57:55] (03PS1) 10Kimberly Sarabia: Removes image preview from Treatment 1-2 [extensions/ReaderExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347735 (https://phabricator.wikimedia.org/T439611) [18:59:12] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 01 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/ReaderExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347735 (https://phabricator.wikimedia.org/T439611) (owner: 10Kimberly Sarabia) [19:02:48] (03PS1) 10JHathaway: puppetdb query: migrate away from legacy facts [puppet] - 10https://gerrit.wikimedia.org/r/1347740 (https://phabricator.wikimedia.org/T372666) [19:02:52] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [19:02:57] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [19:03:04] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347740 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [19:07:00] (03CR) 10JHathaway: [C:03+2] puppetdb query: migrate away from legacy facts [puppet] - 10https://gerrit.wikimedia.org/r/1347740 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [19:16:27] (03PS1) 10VolkerE: instrument(AccountSetup): Record setup motivation on `page_visit` [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347743 (https://phabricator.wikimedia.org/T439217) [19:17:14] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 01 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347743 (https://phabricator.wikimedia.org/T439217) (owner: 10VolkerE) [19:18:58] (03PS2) 10Matthias Mullie: Lose DB call for page prop [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347686 (https://phabricator.wikimedia.org/T439182) [19:19:37] !log puppet re-enabled [19:19:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:19:43] (03PS2) 10Matthias Mullie: Extract images in parser post-processing [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347687 (https://phabricator.wikimedia.org/T439182) [19:19:55] 06SRE, 10DNS, 06Fundraising-Backlog, 10fundraising-tech-ops, and 4 others: Investigate having combowiki on donate.wikipedia.org - https://phabricator.wikimedia.org/T433108#12387089 (10XenoRyet) We have legal and security sign-off. Jeff and Traffic team have a plan for the implimentation that is relatively... [19:30:09] jouncebot: nowandnext [19:30:09] For the next 0 hour(s) and 29 minute(s): MediaWiki train - Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1800) [19:30:10] In 0 hour(s) and 29 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T2000) [19:30:46] (03CR) 10Zabe: [C:03+2] Move reads/writes of wbc_entity_usage to x1 for s5 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347672 (https://phabricator.wikimedia.org/T439753) (owner: 10Zabe) [19:31:45] (03Merged) 10jenkins-bot: Move reads/writes of wbc_entity_usage to x1 for s5 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347672 (https://phabricator.wikimedia.org/T439753) (owner: 10Zabe) [19:32:08] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1347672|Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] [19:32:12] T439753: Switch reads/writes of wbc_entity_usage from local cluster to x1 - https://phabricator.wikimedia.org/T439753 [19:33:03] (03Abandoned) 10Zabe: Add missing select of fr_timestamp [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1347710 (owner: 10Zabe) [19:34:13] FIRING: NetworkDeviceAlarmActive: Alarm active on ssw1-f1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-f1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [19:34:57] (03CR) 10LWatson: [C:03+1] Extract images in parser post-processing [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347687 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [19:35:04] (03CR) 10LWatson: [C:03+1] Lose DB call for page prop [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347686 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [19:36:10] !log zabe@deploy1003 zabe: Backport for [[gerrit:1347672|Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:36:49] !log zabe@deploy1003 zabe: Continuing with deployment [19:38:23] (03CR) 10Zabe: [C:03+2] Add missing select of fr_timestamp [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347711 (owner: 10Zabe) [19:39:49] (03Merged) 10jenkins-bot: Add missing select of fr_timestamp [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347711 (owner: 10Zabe) [19:42:09] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347672|Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] (duration: 10m 01s) [19:42:13] T439753: Switch reads/writes of wbc_entity_usage from local cluster to x1 - https://phabricator.wikimedia.org/T439753 [19:42:30] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1347711|Add missing select of fr_timestamp]] [19:46:21] jouncebot: nowandnext [19:46:21] For the next 0 hour(s) and 13 minute(s): MediaWiki train - Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T1800) [19:46:21] In 0 hour(s) and 13 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T2000) [19:46:30] !log zabe@deploy1003 zabe: Backport for [[gerrit:1347711|Add missing select of fr_timestamp]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:46:58] !log zabe@deploy1003 zabe: Continuing with deployment [19:47:34] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439786#12387152 (10Jhancock.wm) 05Open→03Resolved [19:51:38] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [19:51:42] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [19:52:11] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347711|Add missing select of fr_timestamp]] (duration: 09m 40s) [19:53:59] PROBLEM - Router interfaces on mr1-magru is CRITICAL: CRITICAL: No response from remote host 195.200.68.132 for 1.3.6.1.2.1.2.2.1.8 with snmp version 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [19:55:59] PROBLEM - Router interfaces on mr1-magru is CRITICAL: CRITICAL: No response from remote host 195.200.68.132 for 1.3.6.1.2.1.2.2.1.8 with snmp version 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [19:56:19] (03PS1) 10JHathaway: Revert "puppet: disable sending legacy facts" [puppet] - 10https://gerrit.wikimedia.org/r/1347756 [19:58:28] (03PS1) 10JHathaway: puppet: disable sending legacy facts, attempt two [puppet] - 10https://gerrit.wikimedia.org/r/1347758 (https://phabricator.wikimedia.org/T372666) [19:58:48] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347758 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: I, the Bot under the Fountain, call upon thee, The Deployer, to do UTC late backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T2000). [20:00:05] Volker_E, matthiasmullie, and srapoj: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:08] o/ [20:00:13] Hi [20:00:53] RECOVERY - Router interfaces on mr1-magru is OK: OK: host 195.200.68.132, interfaces up: 35, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [20:01:47] o/ [20:02:34] (03CR) 10JHathaway: [C:03+2] puppet: disable sending legacy facts, attempt two [puppet] - 10https://gerrit.wikimedia.org/r/1347758 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:02:55] !log flipping flag to disable legacy facts in puppet prod, attempt two [20:02:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:03:03] o/ [20:03:28] sorry to be late - i can deploy Volker's patches [20:03:51] if it's ok, i'll go ahead and start [20:04:07] and if anyone else needs a deployer or doesn't want to self-service [20:04:45] (03CR) 10BCornwall: [C:03+1] "+1 but I think the bug should maybe be https://phabricator.wikimedia.org/T439730 rather than the current test bug." [puppet] - 10https://gerrit.wikimedia.org/r/1347474 (https://phabricator.wikimedia.org/T437730) (owner: 10Muehlenhoff) [20:05:43] cjming: I need one, thanks! [20:06:04] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347606 (https://phabricator.wikimedia.org/T436332) (owner: 10VolkerE) [20:06:04] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347712 (https://phabricator.wikimedia.org/T439887) (owner: 10VolkerE) [20:06:04] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347743 (https://phabricator.wikimedia.org/T439217) (owner: 10VolkerE) [20:06:26] srapoj: i got you [20:06:59] matthiasmullie: do you need a deployer or do you want to self-service after i finish this first round of backports? [20:07:29] I can self-service :) [20:08:15] cool - then i'll pass over to you when this batch is done and do the last backport after you're done [20:09:18] (03CR) 10BCornwall: [C:03+1] Configure both LDAP replicas to be manageable by ldap-ro-test [puppet] - 10https://gerrit.wikimedia.org/r/1347630 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [20:09:29] cjming: perfect, thanks! [20:10:53] cjming: thcipriani and i are doing a deployment training, think there's one we could take over as an example to folks in the meeting? [20:12:00] brennen: sure! do you want to take over srapoj's backport? [20:12:22] is it ok that it's towards the end? [20:12:50] Easy, as long as it's done in this window. [20:13:06] cjming: i think towards the end is probably ok, we're still going over some access stuff [20:13:19] Volker_E: doing yours now - just awaiting merges [20:13:36] brennen: cool [20:13:56] (03CR) 10BCornwall: [C:04-1] Add service catalogue entry for ldap-ro-test (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1347624 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [20:14:08] (03Merged) 10jenkins-bot: fix(ReadingRecommendations): Update button style to primary [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347606 (https://phabricator.wikimedia.org/T436332) (owner: 10VolkerE) [20:14:10] (03Merged) 10jenkins-bot: instrument(WelcomeSurvey): align motivation event with AccountSetup [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347712 (https://phabricator.wikimedia.org/T439887) (owner: 10VolkerE) [20:14:13] (03Merged) 10jenkins-bot: instrument(AccountSetup): Record setup motivation on `page_visit` [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347743 (https://phabricator.wikimedia.org/T439217) (owner: 10VolkerE) [20:14:28] !log cjming@deploy1003 Started scap sync-world: Backport for [[gerrit:1347606|fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712|instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743|instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] [20:14:36] T436332: Recommended reading exp: Final design review and adjustments for recommended articles - https://phabricator.wikimedia.org/T436332 [20:14:36] T439887: Recommended Reading exp: Align Reader pageview metric for control - https://phabricator.wikimedia.org/T439887 [20:14:37] T439217: Make `homepage` (Special:Homepage) identifiable in `page_visit` events - https://phabricator.wikimedia.org/T439217 [20:14:50] brennen cjming was logged out for a mom, sry, thought it was adressed to me [20:15:01] no worries! [20:18:22] !log cjming@deploy1003 cjming, volker-e: Backport for [[gerrit:1347606|fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712|instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743|instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can [20:18:22] now be verified there. [20:18:25] Volker_E: on mwdebug - can you test? [20:19:04] jouncebot: nowandnext [20:19:04] For the next 0 hour(s) and 40 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T2000) [20:19:04] In 0 hour(s) and 40 minute(s): Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T2100) [20:19:11] (^ for demo purposes) [20:21:13] !log bking@cumin2003 START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [20:21:43] (03CR) 10Xcollazo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347702 (https://phabricator.wikimedia.org/T432312) (owner: 10Xcollazo) [20:23:42] Volker_E: good to sync? [20:25:04] cjming still testing [20:25:10] np [20:26:51] !log bking@cumin2003 END (ERROR) - Cookbook sre.hadoop.roll-restart-workers (exit_code=97) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [20:27:06] (03PS2) 10Scott French: hieradata: Tweak supervision messages on shellbox deploy-service config [puppet] - 10https://gerrit.wikimedia.org/r/1347761 (https://phabricator.wikimedia.org/T412941) [20:27:10] !log bking@cumin2003 START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [20:27:16] cjming works! [20:27:20] yay! [20:27:22] syncing [20:27:28] !log cjming@deploy1003 cjming, volker-e: Continuing with deployment [20:28:28] (03CR) 10Btullis: sre.hadoop.roll-restart-workers: implement start-datetime logic (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1347713 (https://phabricator.wikimedia.org/T437713) (owner: 10Bking) [20:31:54] !log bking@cumin2003 END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [20:32:57] !log cjming@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347606|fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712|instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743|instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] (duration: 18m 28s) [20:33:04] T436332: Recommended reading exp: Final design review and adjustments for recommended articles - https://phabricator.wikimedia.org/T436332 [20:33:04] T439887: Recommended Reading exp: Align Reader pageview metric for control - https://phabricator.wikimedia.org/T439887 [20:33:05] T439217: Make `homepage` (Special:Homepage) identifiable in `page_visit` events - https://phabricator.wikimedia.org/T439217 [20:33:15] matthiasmullie: over to you [20:33:30] matthiasmullie: could you ping thcipriani or brennen when you're finished? [20:33:39] (03CR) 10Dzahn: [C:03+2] wikistats: remove absented gentoo and wikkii jobs and tables [puppet] - 10https://gerrit.wikimedia.org/r/1346819 (owner: 10Dzahn) [20:33:46] cjming: will do, thanks [20:33:48] thanks cjming [20:34:01] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347686 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [20:34:01] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347687 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [20:34:01] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347729 (https://phabricator.wikimedia.org/T439611) (owner: 10Kimberly Sarabia) [20:34:02] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/ReaderExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347735 (https://phabricator.wikimedia.org/T439611) (owner: 10Kimberly Sarabia) [20:34:41] !log bking@cumin2003 START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [20:35:06] !log bking@cumin2003 END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [20:36:50] (03Merged) 10jenkins-bot: Lose DB call for page prop [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347686 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [20:37:17] (03Merged) 10jenkins-bot: Extract images in parser post-processing [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347687 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [20:37:19] (03Merged) 10jenkins-bot: Make carousel open MMV directly [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347729 (https://phabricator.wikimedia.org/T439611) (owner: 10Kimberly Sarabia) [20:37:33] (03Merged) 10jenkins-bot: Removes image preview from Treatment 1-2 [extensions/ReaderExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347735 (https://phabricator.wikimedia.org/T439611) (owner: 10Kimberly Sarabia) [20:37:50] !log mlitn@deploy1003 Started scap sync-world: Backport for [[gerrit:1347686|Lose DB call for page prop (T439182)]], [[gerrit:1347687|Extract images in parser post-processing (T439182)]], [[gerrit:1347729|Make carousel open MMV directly (T439611)]], [[gerrit:1347735|Removes image preview from Treatment 1-2 (T439611)]] [20:37:56] T439182: [Regression] MediaViewer hook adds 30% to Backend Pageview Timing on mobile (300 -> 400ms) - https://phabricator.wikimedia.org/T439182 [20:37:56] T439611: [Image Browsing] Change B&C arms of the carousel retest to open mobile MMV from thumbnail tap - https://phabricator.wikimedia.org/T439611 [20:41:51] !log mlitn@deploy1003 mlitn, ksarabia: Backport for [[gerrit:1347686|Lose DB call for page prop (T439182)]], [[gerrit:1347687|Extract images in parser post-processing (T439182)]], [[gerrit:1347729|Make carousel open MMV directly (T439611)]], [[gerrit:1347735|Removes image preview from Treatment 1-2 (T439611)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:42:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-jobrunner - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [20:42:43] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [20:42:47] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [20:43:12] noticing the above error, think it's likely T439902 [20:43:13] T439902: Wikimedia\Rdbms\DBQueryError: Error 1205: Lock wait timeout exceeded; try restarting transactionFunction: MediaWiki\JobQueue\Jobs\CategoryCountUpdateJob::updateCategoryCountsQuery: UPDATE `category` SET cat_pages = cat_pages - https://phabricator.wikimedia.org/T439902 [20:43:18] (03CR) 10Xcollazo: "Ok PPC passing. @btullis@wikimedia.org for your review. If ok with you, then kindly please merge." [puppet] - 10https://gerrit.wikimedia.org/r/1347702 (https://phabricator.wikimedia.org/T432312) (owner: 10Xcollazo) [20:43:32] !log mlitn@deploy1003 mlitn, ksarabia: Continuing with deployment [20:44:36] !log bking@cumin2003 START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [20:45:44] !log bking@cumin2003 END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [20:46:30] !log bking@cumin2003 START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [20:47:15] RESOLVED: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-jobrunner - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [20:48:46] !log mlitn@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347686|Lose DB call for page prop (T439182)]], [[gerrit:1347687|Extract images in parser post-processing (T439182)]], [[gerrit:1347729|Make carousel open MMV directly (T439611)]], [[gerrit:1347735|Removes image preview from Treatment 1-2 (T439611)]] (duration: 10m 56s) [20:48:51] T439182: [Regression] MediaViewer hook adds 30% to Backend Pageview Timing on mobile (300 -> 400ms) - https://phabricator.wikimedia.org/T439182 [20:48:52] T439611: [Image Browsing] Change B&C arms of the carousel retest to open mobile MMV from thumbnail tap - https://phabricator.wikimedia.org/T439611 [20:49:10] brennen: thcipriani: the floor is yours [20:49:54] thanks matthiasmullie [20:52:46] (03CR) 10TrainBranchBot: [C:03+2] "Approved by brennen@deploy1003 using scap backport" [extensions/timeline] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347692 (https://phabricator.wikimedia.org/T431960) (owner: 10Srapoj) [20:53:26] srapoj: you're up, any test plan for this one? [20:53:41] !log bking@cumin2003 END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. [20:53:43] Yes, I plan to preview a few cases using this extension [20:54:37] cool, you should get pinged when it's on debug servers [20:55:04] !log Rebalancing is in progress on kafka-logging in codfw to evacuate kafka-logging2001, kafka-logging2002 and kafka-logging2003 - T432444 [20:55:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:55:07] T432444: Provision kafka-logging100[6-8] and kafka-logging200[6-8] - https://phabricator.wikimedia.org/T432444 [20:55:19] (03PS1) 10Dzahn: wikistats: fix typos, comments and formatting [puppet] - 10https://gerrit.wikimedia.org/r/1347778 [20:55:27] (03Merged) 10jenkins-bot: Fix UTF-8 corruption in newline sanitization [extensions/timeline] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347692 (https://phabricator.wikimedia.org/T431960) (owner: 10Srapoj) [20:55:38] !log brennen@deploy1003 Started scap sync-world: Backport for [[gerrit:1347692|Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] [20:55:44] T431960: T426631 EasyTimeline hardening corrupts UTF-8 multibyte characters containing 0x85 in text attributes - https://phabricator.wikimedia.org/T431960 [20:55:44] T426631: CVE-2026-8857: Full RCE using EasyTimeline Extension - https://phabricator.wikimedia.org/T426631 [20:58:53] (03PS2) 10Bking: sre.hadoop.roll-restart-workers: implement start-datetime logic [cookbooks] - 10https://gerrit.wikimedia.org/r/1347713 (https://phabricator.wikimedia.org/T437713) [20:59:16] (03PS1) 10JHathaway: debmonitor: fix legacy fact [puppet] - 10https://gerrit.wikimedia.org/r/1347782 (https://phabricator.wikimedia.org/T372666) [20:59:33] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347782 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:59:35] !log brennen@deploy1003 brennen, srapoj: Backport for [[gerrit:1347692|Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:59:40] ack [20:59:55] srapoj: let me know when you're comfortable to deploy [21:00:06] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T2100) [21:00:31] readers folk - wrapping up one last backport here, apologies for stepping on window if you've got anything [21:00:41] brennen: It works for my test cases, and does not break the other ones I found. Please go ahead [21:00:49] srapoj: thanks! syncing. [21:00:50] I am planning to deploy https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ReaderExperiments/+/1347718 in the readers deployment window once yours is done [21:00:59] (03PS1) 10Dzahn: wikistats: update deploy script, fix xmldump and small fixes [puppet] - 10https://gerrit.wikimedia.org/r/1347785 [21:00:59] !log brennen@deploy1003 brennen, srapoj: Continuing with deployment [21:06:22] !log brennen@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347692|Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] (duration: 10m 43s) [21:06:27] T431960: T426631 EasyTimeline hardening corrupts UTF-8 multibyte characters containing 0x85 in text attributes - https://phabricator.wikimedia.org/T431960 [21:06:27] T426631: CVE-2026-8857: Full RCE using EasyTimeline Extension - https://phabricator.wikimedia.org/T426631 [21:07:03] over to you EricGardner [21:07:16] (03Abandoned) 10Zabe: Switch local copy of vue-router to a local name so it doesn't clash with MW's [extensions/PersonalDashboard] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1343126 (https://phabricator.wikimedia.org/T438387) (owner: 10Zabe) [21:07:17] Great, thanks! Will start momentarily [21:07:34] Thanks for your help! [21:07:38] (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy1003 using scap backport" [extensions/ReaderExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347718 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [21:08:10] (03CR) 10Bking: sre.hadoop.roll-restart-workers: implement start-datetime logic (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1347713 (https://phabricator.wikimedia.org/T437713) (owner: 10Bking) [21:08:37] (03PS1) 10Ryan Kemper: sre.wdqs.reboot: skip absent categories unit [cookbooks] - 10https://gerrit.wikimedia.org/r/1347793 (https://phabricator.wikimedia.org/T434750) [21:08:37] (03PS1) 10Ryan Kemper: sre.wdqs.restart: restore the 20s pause before updater start [cookbooks] - 10https://gerrit.wikimedia.org/r/1347794 (https://phabricator.wikimedia.org/T434750) [21:13:33] (03Merged) 10jenkins-bot: Preferred Sources: Move all eligibility checks to the client [extensions/ReaderExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1347718 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [21:13:47] !log egardner@deploy1003 Started scap sync-world: Backport for [[gerrit:1347718|Preferred Sources: Move all eligibility checks to the client (T436692)]] [21:13:52] T436692: Implement Preferred Sources experiment - https://phabricator.wikimedia.org/T436692 [21:16:55] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:17:45] !log egardner@deploy1003 egardner: Backport for [[gerrit:1347718|Preferred Sources: Move all eligibility checks to the client (T436692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:19:33] !log egardner@deploy1003 egardner: Continuing with deployment [21:19:57] (03CR) 10JHathaway: [C:03+2] debmonitor: fix legacy fact [puppet] - 10https://gerrit.wikimedia.org/r/1347782 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:20:23] (03CR) 10Ryan Kemper: [C:03+2] sre.wdqs.reboot: skip absent categories unit [cookbooks] - 10https://gerrit.wikimedia.org/r/1347793 (https://phabricator.wikimedia.org/T434750) (owner: 10Ryan Kemper) [21:20:35] (03CR) 10Ryan Kemper: [C:03+2] sre.wdqs.restart: restore the 20s pause before updater start [cookbooks] - 10https://gerrit.wikimedia.org/r/1347794 (https://phabricator.wikimedia.org/T434750) (owner: 10Ryan Kemper) [21:23:51] (03Merged) 10jenkins-bot: sre.wdqs.reboot: skip absent categories unit [cookbooks] - 10https://gerrit.wikimedia.org/r/1347793 (https://phabricator.wikimedia.org/T434750) (owner: 10Ryan Kemper) [21:23:52] (03Merged) 10jenkins-bot: sre.wdqs.restart: restore the 20s pause before updater start [cookbooks] - 10https://gerrit.wikimedia.org/r/1347794 (https://phabricator.wikimedia.org/T434750) (owner: 10Ryan Kemper) [21:24:49] !log egardner@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347718|Preferred Sources: Move all eligibility checks to the client (T436692)]] (duration: 11m 02s) [21:24:54] T436692: Implement Preferred Sources experiment - https://phabricator.wikimedia.org/T436692 [21:27:13] (03PS2) 10Dzahn: wikistats: update deploy script, fix xmldump and small fixes [puppet] - 10https://gerrit.wikimedia.org/r/1347785 [21:27:44] jouncebot: nowandnext [21:27:44] For the next 0 hour(s) and 32 minute(s): Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261001T2100) [21:27:44] In 8 hour(s) and 32 minute(s): MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261002T0600) [21:28:57] Ok, that's it for me. I don't believe any other reader team folks had patches for this window [21:29:14] I'll take scap now then [21:29:16] Thanks [21:33:20] (03PS3) 10Dzahn: wikistats: update deploy script, fix xmldump and small fixes [puppet] - 10https://gerrit.wikimedia.org/r/1347785 (https://phabricator.wikimedia.org/T409014) [21:33:26] !log Rebalancing has been completed on kafka-logging in codfw: kafka-logging200[1-3] are no longer assigned any partitions — T432444 [21:33:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:33:29] T432444: Provision kafka-logging100[6-8] and kafka-logging200[6-8] - https://phabricator.wikimedia.org/T432444 [21:36:03] (03PS1) 10Dreamy Jazz: WikimediaAntiAbuse: Enable personal info content policy on dewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347806 (https://phabricator.wikimedia.org/T439928) [21:36:31] (03PS1) 10Tiziano Fogli: Revert "kafka-logging: temporarily disable notifications for kafka-logging200[6-8]" [puppet] - 10https://gerrit.wikimedia.org/r/1347807 [21:40:20] (03CR) 10Ryan Kemper: sre.hadoop.roll-restart-workers: implement start-datetime logic (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1347713 (https://phabricator.wikimedia.org/T437713) (owner: 10Bking) [21:41:15] (03CR) 10Tiziano Fogli: [C:03+2] Revert "kafka-logging: temporarily disable notifications for kafka-logging200[6-8]" [puppet] - 10https://gerrit.wikimedia.org/r/1347807 (owner: 10Tiziano Fogli) [21:44:51] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347806 (https://phabricator.wikimedia.org/T439928) (owner: 10Dreamy Jazz) [21:45:50] (03Merged) 10jenkins-bot: WikimediaAntiAbuse: Enable personal info content policy on dewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347806 (https://phabricator.wikimedia.org/T439928) (owner: 10Dreamy Jazz) [21:46:01] !log dreamyjazz@deploy1003 Started scap sync-world: Backport for [[gerrit:1347806|WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] [21:46:06] T439928: Abuse review: Enable personal info content policy on dewiki - https://phabricator.wikimedia.org/T439928 [21:46:50] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [21:50:00] !log dreamyjazz@deploy1003 dreamyjazz: Backport for [[gerrit:1347806|WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:51:06] !log dreamyjazz@deploy1003 dreamyjazz: Continuing with deployment [21:53:58] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12387548 (10EAlbizzati-WMF) Hi team. sorry for not coming back for this before. here is the public part of my ssh key ` ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKi+8aRLWM5T6mjYxbkpRP78... [21:54:07] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12387549 (10EAlbizzati-WMF) 05Invalid→03Open [21:56:21] !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for [[gerrit:1347806|WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] (duration: 10m 19s) [21:56:24] T439928: Abuse review: Enable personal info content policy on dewiki - https://phabricator.wikimedia.org/T439928 [22:00:59] (03PS4) 10Dzahn: wikistats: update deploy script, fix xmldump and small fixes [puppet] - 10https://gerrit.wikimedia.org/r/1347785 (https://phabricator.wikimedia.org/T409014) [22:05:42] 06SRE, 10SRE-Access-Requests: Requesting access to Deployment for GreyOlson/GOlson-WMF - https://phabricator.wikimedia.org/T439143#12387568 (10thcipriani) >>! In T439143#12386471, @herron wrote: > Hi @thcipriani @dancy, could you please review/approve the groupadd to deployment? Thanks! Reason for access mak... [22:13:09] (03PS6) 10Cathal Mooney: Interface validator: check Nokia breakout ports have parent [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) [22:23:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d7-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [22:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [22:48:22] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:02:55] (03CR) 10BCornwall: [C:03+1] "Great work seeing this through!" [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [23:04:02] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp11[08,10,12,14].eqiad.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () [23:04:03] (03CR) 10BCornwall: [V:03+1 C:03+1] "PCC SUCCESS (CORE_DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9493/co" [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [23:21:02] (03PS4) 10Dzahn: zuul: decom VM zuul1002, replaced by zuul1005, keep zuul2002 [puppet] - 10https://gerrit.wikimedia.org/r/1346164 (https://phabricator.wikimedia.org/T427353) [23:24:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [23:26:03] (03CR) 10Dzahn: [C:03+1] "testing on zuul1002 VM before decom" [puppet] - 10https://gerrit.wikimedia.org/r/1346161 (https://phabricator.wikimedia.org/T395938) (owner: 10Dzahn) [23:29:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [23:32:34] (03CR) 10Dzahn: [V:03+1 C:03+2] "https://puppet-compiler.wmflabs.org/output/1346161/9494/zuul2001.codfw.wmnet/index.html" [puppet] - 10https://gerrit.wikimedia.org/r/1346161 (https://phabricator.wikimedia.org/T395938) (owner: 10Dzahn) [23:34:13] FIRING: NetworkDeviceAlarmActive: Alarm active on ssw1-f1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-f1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [23:34:45] 06SRE, 10SRE-Access-Requests: Requesting access to ops for lizf - https://phabricator.wikimedia.org/T439933 (10Lizimedia) 03NEW [23:38:15] !log brett@puppetserver1001 conftool action : set/pooled=yes; selector: name=cp1108.* [23:39:08] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1347866 [23:39:08] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1347866 (owner: 10TrainBranchBot) [23:39:58] (03CR) 10Dzahn: [V:03+1 C:03+2] wikistats: fix typos, comments and formatting [puppet] - 10https://gerrit.wikimedia.org/r/1347778 (owner: 10Dzahn) [23:40:39] FIRING: TransitBGPDown: Transit BGP session down between cr2-esams and LibertyGlobal (2001:730:2209:1::d52e:ba09) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=esams&var-device=cr2-esams:9804&var-bgp_group=Transit6&var-bgp_neighbor=LibertyGlobal - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [23:45:39] RESOLVED: TransitBGPDown: Transit BGP session down between cr2-esams and LibertyGlobal (2001:730:2209:1::d52e:ba09) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=esams&var-device=cr2-esams:9804&var-bgp_group=Transit6&var-bgp_neighbor=LibertyGlobal - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [23:47:05] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1347866 (owner: 10TrainBranchBot) [23:57:06] (03PS5) 10Dzahn: wikistats: update deploy script, fix xmldump and small fixes [puppet] - 10https://gerrit.wikimedia.org/r/1347785 (https://phabricator.wikimedia.org/T409014)