[15:30:53] welcome back wm-bot [15:31:23] (03CR) 10Vgutierrez: [C:03+1] profile:puppetserver: create kapow dir and dummy file [puppet] - 10https://gerrit.wikimedia.org/r/1371766 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [15:32:47] (03PS6) 10CWilliams: deployment_server: moving pampinus to aux [puppet] - 10https://gerrit.wikimedia.org/r/1369307 (https://phabricator.wikimedia.org/T443897) [15:34:48] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-a4-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T445286#12583031 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF Rebalanced power [15:36:18] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P97608 and previous config saved to /var/cache/conftool/dbconfig/20261119-153618-marostegui.json [15:39:09] (03CR) 10Arnaudb: "@hashar@free.fr @dzahn@wikimedia.org I can merge this tomorrow morning if you're OK with it" [puppet] - 10https://gerrit.wikimedia.org/r/1371818 (owner: 10Paladox) [15:40:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.91% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:41:29] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission clouddb1017.eqiad.wmnet - https://phabricator.wikimedia.org/T445177#12583087 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF Ran decom script and removed unit from rack [15:41:59] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission clouddb1017.eqiad.wmnet - https://phabricator.wikimedia.org/T445177#12583091 (10VRiley-WMF) [15:44:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [15:44:14] (03PS1) 10DCausse: ml-serve: bump cpu for semantic-highlighting-staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371871 [15:45:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.96% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:49:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [15:51:20] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2192 (T445307)', diff saved to https://phabricator.wikimedia.org/P97609 and previous config saved to /var/cache/conftool/dbconfig/20261119-155120-marostegui.json [15:51:24] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [15:51:36] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2201.codfw.wmnet with reason: Maintenance [15:52:08] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2342.codfw.wmnet, wikikube-worker2262.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2353.codfw.wmnet, wikikube-work [15:52:08] odfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2190.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2039.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2139.codfw.wmnet, wikikube-worker2297.codfw.wmnet, w [15:52:08] worker2315.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2322.codfw.wmnet, wikikube-worker2314.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2301.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [15:52:20] (03CR) 10CWilliams: [C:03+2] deployment_server: moving pampinus to aux [puppet] - 10https://gerrit.wikimedia.org/r/1369307 (https://phabricator.wikimedia.org/T443897) (owner: 10CWilliams) [15:53:08] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [15:53:43] (03PS1) 10Hnowlan: arclamp: disable protected mode [puppet] - 10https://gerrit.wikimedia.org/r/1371875 [15:55:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:55:57] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2211.codfw.wmnet with reason: Maintenance [15:55:59] !log marostegui@cumin1004 dbctl commit (dc=all): 'Depooling db2211 (T445307)', diff saved to https://phabricator.wikimedia.org/P97610 and previous config saved to /var/cache/conftool/dbconfig/20261119-155558-marostegui.json [15:57:17] (03CR) 10Andrew Bogott: [C:03+1] apt: fix typo in AutocleanInterval setting [puppet] - 10https://gerrit.wikimedia.org/r/1371719 (owner: 10Volans) [15:57:45] (03CR) 10Krinkle: "See also https://gerrit.wikimedia.org/r/c/operations/puppet/+/1333288" [puppet] - 10https://gerrit.wikimedia.org/r/1371875 (owner: 10Hnowlan) [15:58:49] (03PS2) 10Krinkle: logging: Disable logstash intake for 'post' channel [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1370698 (https://phabricator.wikimedia.org/T445147) [15:59:34] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1370698 (https://phabricator.wikimedia.org/T445147) (owner: 10Krinkle) [15:59:54] (03PS1) 10Btullis: wmnet: Move the s2, s3, s4 and x4 analytics replicas to dbstore1010 [dns] - 10https://gerrit.wikimedia.org/r/1371876 (https://phabricator.wikimedia.org/T424003) [16:00:51] (03Merged) 10jenkins-bot: logging: Disable logstash intake for 'post' channel [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1370698 (https://phabricator.wikimedia.org/T445147) (owner: 10Krinkle) [16:01:09] (03PS1) 10Btullis: jupyterhub: Run JupyterHub from jupyter-analytics by default [puppet] - 10https://gerrit.wikimedia.org/r/1371877 (https://phabricator.wikimedia.org/T321512) [16:01:19] !log krinkle@deploy2003 Started scap sync-world: Backport for [[gerrit:1370698|logging: Disable logstash intake for 'post' channel (T445147)]] [16:01:23] T445147: Introduce unsampled post.log - https://phabricator.wikimedia.org/T445147 [16:01:36] (03CR) 10Hnowlan: [C:03+1] "redis unexpectedly restarted in prod today and we went into protected mode - I am kinda baffled that we didn't hit this immediately post-u" [puppet] - 10https://gerrit.wikimedia.org/r/1333288 (https://phabricator.wikimedia.org/T436464) (owner: 10Southparkfan) [16:02:01] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2211 (T445307)', diff saved to https://phabricator.wikimedia.org/P97611 and previous config saved to /var/cache/conftool/dbconfig/20261119-160201-marostegui.json [16:02:05] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [16:02:15] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [16:02:47] (03Abandoned) 10Hnowlan: arclamp: disable protected mode [puppet] - 10https://gerrit.wikimedia.org/r/1371875 (owner: 10Hnowlan) [16:03:35] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1371877 (https://phabricator.wikimedia.org/T321512) (owner: 10Btullis) [16:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [16:05:59] (03CR) 10Volans: [C:03+2] apt: run unattended upgrades hourly [puppet] - 10https://gerrit.wikimedia.org/r/1371718 (https://phabricator.wikimedia.org/T445398) (owner: 10Volans) [16:06:16] (03CR) 10Volans: [C:03+2] apt: fix typo in AutocleanInterval setting [puppet] - 10https://gerrit.wikimedia.org/r/1371719 (owner: 10Volans) [16:08:02] !log krinkle@deploy2003 krinkle: Backport for [[gerrit:1370698|logging: Disable logstash intake for 'post' channel (T445147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:08:06] T445147: Introduce unsampled post.log - https://phabricator.wikimedia.org/T445147 [16:09:42] !log krinkle@deploy2003 krinkle: Continuing with deployment [16:11:10] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:13:58] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2364.codfw.wmnet, wikikube-worker2187.codfw.wmnet, wikikube-worker2243.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2282.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2017.codfw.wmnet, wikikube-worker2172.codfw.wmnet, wikikube-work [16:13:58] odfw.wmnet, wikikube-worker2091.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2108.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2110.codfw.wmnet, wikikube-worker2340.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2092.codfw.wmnet, wikikube-worker2345.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2213.codfw.wmnet, wikikube-worker2198.codfw.wmnet, w [16:13:58] worker2315.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2041.codfw.wmnet, wikikube-worker2255.codfw.wmnet, wikikube-worker2002.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [16:14:11] 06SRE, 10DNS, 06Traffic, 13Patch-For-Review: Cleanup unused DNS subdomains - https://phabricator.wikimedia.org/T444019#12583462 (10ssingh) @Dzahn : Thanks for taking care of these. Is this something you would like to handle yourself or you prefer Traffic would do it? (I am asking to be clear.) [16:14:58] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:16:12] (03CR) 10Marostegui: "Can you please run a PCC to make sure this won't break anything?" [puppet] - 10https://gerrit.wikimedia.org/r/1371837 (owner: 10Ladsgroup) [16:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:17:03] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P97612 and previous config saved to /var/cache/conftool/dbconfig/20261119-161702-marostegui.json [16:17:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:18:43] !log krinkle@deploy2003 Finished scap sync-world: Backport for [[gerrit:1370698|logging: Disable logstash intake for 'post' channel (T445147)]] (duration: 17m 24s) [16:18:46] T445147: Introduce unsampled post.log - https://phabricator.wikimedia.org/T445147 [16:20:06] PROBLEM - piwik.wikimedia.org tls expiry on matomo1003 is CRITICAL: connect to address 10.64.32.25 and port 443: Connection refused https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [16:20:06] PROBLEM - piwik.wikimedia.org requires authentication on matomo1003 is CRITICAL: connect to address 10.64.32.25 and port 443: Connection refused https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [16:21:06] RECOVERY - piwik.wikimedia.org tls expiry on matomo1003 is OK: OK - Certificate piwik.wikimedia.org will expire on Thu 17 Dec 2026 03:45:00 AM GMT +0000. https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [16:21:06] RECOVERY - piwik.wikimedia.org requires authentication on matomo1003 is OK: HTTP OK: Status line output matched HTTP/1.1 302 - 582 bytes in 0.059 second response time https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [16:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:22:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:24:03] FIRING: PuppetFailure: Puppet has failed on maps1011:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:24:58] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2262.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2155.codfw.wmnet, wikikube-worker2185.codfw.wmnet, wikikube-worker2158.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2157.codfw.wmnet, wikikube-work [16:24:58] odfw.wmnet, wikikube-worker2198.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2060.codfw.wmnet, wikikube-worker2281.codfw.wmnet, wikikube-worker2125.codfw.wmnet, wikikube-worker2301.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2355.codfw.wmnet, wikikube-worker2058.codfw.wmnet, wikikube-worker2263.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2339.codfw.wmnet, w [16:24:58] worker2309.codfw.wmnet, wikikube-worker2324.codfw.wmnet, wikikube-worker2319.codfw.wmnet, wikikube-worker2056.codfw.wmnet, wikikube-worker2370.codfw.wmnet, wikikube-worker2351.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [16:25:58] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:29:25] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for backup1010.mgmt:22 - https://phabricator.wikimedia.org/T445166#12583536 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF Reseated cable again. I will have to investigate this more if it comes back. It seems like it's up and active [16:29:55] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-worker1181.mgmt:22 - https://phabricator.wikimedia.org/T445176#12583540 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF Reseated cable again. I will have to investigate this more if it comes back. It seems like it's up and active [16:32:05] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P97613 and previous config saved to /var/cache/conftool/dbconfig/20261119-163204-marostegui.json [16:34:12] 06SRE, 10Infrastructure Security, 06Product Safety and Integrity, 06Security-Team, and 4 others: CVE-2026-119618 Apache2 HTTP Server mod_http2 - https://phabricator.wikimedia.org/T444614#12583601 (10sbassett) 05Open→03Declined [16:34:16] 06SRE, 10Infrastructure Security, 06Security-Team, 07SecTeam-Processed, and 3 others: CVE-2026-119618 Apache2 HTTP Server mod_http2 - https://phabricator.wikimedia.org/T444614#12583605 (10sbassett) [16:35:48] !log brouberol@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host is being re-racked to row A [16:36:15] (03PS1) 10Lerickson: Release wdqs-proxy v1.0.1 with the wdqs-jena update including better parse error messages. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371881 (https://phabricator.wikimedia.org/T445363) [16:36:21] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 3 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12583651 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=d57efe4a-2d5d-4660-9f93-1cf8daa79915) set by brouberol@cumin1004 for 4:... [16:39:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12583682 (10phaultfinder) [16:41:15] jouncebot: nowandnext [16:41:15] No deployments scheduled for the next 0 hour(s) and 18 minute(s) [16:41:15] In 0 hour(s) and 18 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261119T1700) [16:43:29] Going to deploy a security patch [16:44:03] FIRING: KafkaUnderReplicatedPartitions: Under replicated partitions for Kafka cluster jumbo-eqiad in eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration - https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-kafka_cluster=jumbo-eqiad - https://alerts.wikimedia.org/?q=alertname%3DKafkaUnderReplicatedPartitions [16:46:40] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 07SecTeam-Processed: Request IRC #mediawiki_security access for Novem Linguae - https://phabricator.wikimedia.org/T442747#12583724 (10sbassett) p:05Triage→03Medium [16:47:07] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2211 (T445307)', diff saved to https://phabricator.wikimedia.org/P97614 and previous config saved to /var/cache/conftool/dbconfig/20261119-164706-marostegui.json [16:47:11] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [16:47:22] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2213.codfw.wmnet with reason: Maintenance [16:47:24] !log marostegui@cumin1004 dbctl commit (dc=all): 'Depooling db2213 (T445307)', diff saved to https://phabricator.wikimedia.org/P97615 and previous config saved to /var/cache/conftool/dbconfig/20261119-164724-marostegui.json [16:50:30] FIRING: KubernetesDeploymentUnavailableReplicas: ... [16:50:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [16:50:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [16:50:58] (03CR) 10Brouberol: Move kafka-jumbo1016 from row E to A [puppet] - 10https://gerrit.wikimedia.org/r/1371048 (https://phabricator.wikimedia.org/T435775) (owner: 10Brouberol) [16:52:05] dancy: Do you need scap during the puppet window? [16:52:59] Dreamy_Jazz: No. You're good to continue [16:53:05] Thanks [16:53:08] (03CR) 10Brouberol: [C:03+2] Move kafka-jumbo1016 from row E to A [puppet] - 10https://gerrit.wikimedia.org/r/1371048 (https://phabricator.wikimedia.org/T435775) (owner: 10Brouberol) [16:53:27] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2213 (T445307)', diff saved to https://phabricator.wikimedia.org/P97616 and previous config saved to /var/cache/conftool/dbconfig/20261119-165326-marostegui.json [16:53:31] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [16:57:17] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [16:59:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12583787 (10phaultfinder) [17:00:05] jhathaway and rzl: How many deployers does it take to do Puppet request window deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261119T1700). [17:00:05] dancy: A patch you scheduled for Puppet request window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [17:00:13] o/ [17:01:29] o/ [17:02:39] dancy: happy to merge, but in the future I would like a +1 before the window, these patch seems reasonable, but I don't have any context [17:02:44] (03CR) 10JHathaway: [C:03+2] docker::gc: Upgrade docker-gc to 1.4.0 [puppet] - 10https://gerrit.wikimedia.org/r/1371108 (https://phabricator.wikimedia.org/T443833) (owner: 10Ahmon Dancy) [17:02:51] cmooney@cumin1004 netbox (PID 3533627) is awaiting input [17:03:01] (03CR) 10Trueg: [C:03+1] Release wdqs-proxy v1.0.1 with the wdqs-jena update including better parse error messages. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371881 (https://phabricator.wikimedia.org/T445363) (owner: 10Lerickson) [17:03:08] dancy: merged [17:03:31] jhathaway: Thanks! [17:05:58] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2141.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2283.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2291.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2215.codfw.wmnet, wikikube-work [17:05:58] odfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2315.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2322.codfw.wmnet, wikikube-worker2293.codfw.wmnet, wikikube-worker2313.codfw.wmnet, wikikube-worker2305.codfw.wmnet, wikikube-worker2164.codfw.wmnet, wikikube-worker2307.codfw.wmnet, wikikube-worker2276.codfw.wmnet, wikikube-worker2296.codfw.wmnet, wikikube-worker2254.codfw.wmnet, wikikube-worker2105.codfw.wmnet, w [17:05:58] worker2300.codfw.wmnet, wikikube-worker2285.codfw.wmnet, wikikube-worker2133.codfw.wmnet, wikikube-worker2252.codfw.wmnet, wikikube-worker2260.codfw.wmnet, wikikube-worker2073.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [17:06:58] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [17:07:55] (03PS3) 10Cwhite: mobileapps: remove statsd-exporter [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371128 (https://phabricator.wikimedia.org/T444937) [17:07:58] PROBLEM - Blazegraph Port for wdqs-blazegraph on wdqs1012 is CRITICAL: connect to address 127.0.0.1 and port 9999: Connection refused https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [17:08:29] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P97617 and previous config saved to /var/cache/conftool/dbconfig/20261119-170828-marostegui.json [17:08:58] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs1012 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [17:09:17] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for kafka-jumbo1016 - cmooney@cumin1004" [17:09:18] Errors with LiquidThreads means my security patch rolled back [17:09:24] Errors seem unrelated, re-trying [17:09:30] (03CR) 10Fabfur: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1371766 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [17:09:49] (03PS6) 10JHathaway: Add a script to set the in_service_date [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) [17:10:04] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: power-on failure for druid-internal1004.eqiad.wmnet - https://phabricator.wikimedia.org/T441866#12583817 (10BTullis) Hi @VRiley-WMF - did you hear back from Supermicro about this case yet? Thanks. [17:11:11] (03CR) 10JHathaway: Add a script to set the in_service_date (032 comments) [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) (owner: 10JHathaway) [17:11:12] (03CR) 10Cwhite: mobileapps: remove statsd-exporter (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371128 (https://phabricator.wikimedia.org/T444937) (owner: 10Cwhite) [17:12:13] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for toyofuku - https://phabricator.wikimedia.org/T445548#12583823 (10Marostegui) At the moment you do not have any existing ssh key, what would you need the new ones for? You may want to double check https://wikitech.wikimedia.org/wik... [17:12:20] cmooney@cumin1004 netbox (PID 3533627) is awaiting input [17:12:20] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for toyofuku - https://phabricator.wikimedia.org/T445548#12583824 (10Marostegui) 05Open→03Stalled [17:14:50] FIRING: [3x] HelmReleaseBadStatus: Helm release mw-videoscaler/main on k8s@codfw in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:16:18] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for kafka-jumbo1016 - cmooney@cumin1004" [17:16:18] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [17:17:00] !log cmooney@cumin1004 START - Cookbook sre.dns.wipe-cache kafka-jumbo1016.eqiad.wmnet on all recursors [17:17:03] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-jumbo1016.eqiad.wmnet on all recursors [17:17:11] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: hw troubleshooting: power-on for wikikube-worker2337.codfw.wmnet - https://phabricator.wikimedia.org/T442714#12583849 (10Jhancock.wm) I still can't get this thing to boot. there's been some progress but still not reaching POST. opened a case with SM. CS-00226723 [17:17:33] !log brouberol@cumin1004 START - Cookbook sre.hosts.reimage for host kafka-jumbo1016.eqiad.wmnet with OS trixie [17:17:49] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 3 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12583851 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1004 for host kafka-jumbo1016.eqiad.wmnet with OS... [17:18:18] !log brouberol@cumin1004 START - Cookbook sre.hosts.dhcp for host kafka-jumbo1016.eqiad.wmnet (push config only) [17:18:22] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host kafka-jumbo1016.eqiad.wmnet (push config only) [17:22:34] 10ops-codfw, 06SRE, 10database-backups, 06DBA, 06DC-Ops: Install a 10G NIC on dbproxy2008 - https://phabricator.wikimedia.org/T445327#12583870 (10Jhancock.wm) this server is already using a 10G card. [17:23:31] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P97618 and previous config saved to /var/cache/conftool/dbconfig/20261119-172330-marostegui.json [17:24:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12583881 (10phaultfinder) [17:24:56] !log joal@deploy2003 Started deploy [analytics/refinery@f1f49a0] (hadoop-test): HOTFIX -- TEST [analytics/refinery@f1f49a03] [17:24:58] 10ops-codfw, 06SRE, 10database-backups, 06DBA, 06DC-Ops: Install a 10G NIC on dbproxy2008 - https://phabricator.wikimedia.org/T445327#12583882 (10Marostegui) 05Open→03Invalid Indeed, and it is in fact syncing at 10G already: ` Speed: 10000Mb/s ` Thank you Jenn [17:25:41] !log joal@deploy2003 Finished deploy [analytics/refinery@f1f49a0] (hadoop-test): HOTFIX -- TEST [analytics/refinery@f1f49a03] (duration: 00m 45s) [17:26:03] 10ops-codfw, 06SRE, 10database-backups, 06DBA, 06DC-Ops: Install a 10G NIC on dbproxy2008 - https://phabricator.wikimedia.org/T445327#12583892 (10Marostegui) All of them in fact: ` 4 hosts will be targeted: dbproxy[2005-2008].codfw.wmnet OK to proceed on 4 hosts? Enter the number of affected hosts to... [17:26:08] !log joal@deploy2003 Started deploy [analytics/refinery@f1f49a0]: HOTFIX [analytics/refinery@f1f49a03] [17:26:25] (03CR) 10Btullis: [C:03+2] presto: Add execution time limits to the resource groups [puppet] - 10https://gerrit.wikimedia.org/r/1371084 (https://phabricator.wikimedia.org/T445374) (owner: 10Aleksandar Mastilovic) [17:27:03] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2219 is not coming back online after reboot - https://phabricator.wikimedia.org/T445479#12583897 (10Jhancock.wm) doing a firmware update to resolve the backplane error. then reseating the cables. will post again when completed. [17:28:29] (03PS1) 10Cwhite: arclamp: compress logs older than 24 hours [puppet] - 10https://gerrit.wikimedia.org/r/1371888 (owner: 10Herron) [17:28:30] (03CR) 10Cwhite: [C:03+2] "PCC OK: https://puppet-compiler.wmflabs.org/output/1371888/9688/" [puppet] - 10https://gerrit.wikimedia.org/r/1371888 (owner: 10Herron) [17:28:55] jouncebot: nowandnext [17:28:55] For the next 0 hour(s) and 31 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261119T1700) [17:28:55] In 0 hour(s) and 31 minute(s): Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261119T1800) [17:28:55] In 0 hour(s) and 31 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261119T1800) [17:29:03] Still going with security patch deployment [17:29:49] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2219 is not coming back online after reboot - https://phabricator.wikimedia.org/T445479#12583919 (10Marostegui) p:05Triage→03Medium [17:29:50] FIRING: [3x] HelmReleaseBadStatus: Helm release mw-videoscaler/main on k8s@codfw in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:29:50] !log brouberol@cumin1004 START - Cookbook sre.hosts.dhcp for host kafka-jumbo1016.eqiad.wmnet (remove config only) [17:29:52] !log dreamyjazz Deployed security patch for T444739 [17:29:54] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host kafka-jumbo1016.eqiad.wmnet (remove config only) [17:31:14] !log joal@deploy2003 Finished deploy [analytics/refinery@f1f49a0]: HOTFIX [analytics/refinery@f1f49a03] (duration: 05m 05s) [17:31:19] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [17:31:40] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [17:31:59] !log brouberol@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage [17:32:09] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [17:32:24] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [17:32:56] (03PS1) 10Tchanders: admin: Add second FIDO SSH key for tchanders [puppet] - 10https://gerrit.wikimedia.org/r/1371891 (https://phabricator.wikimedia.org/T442758) [17:33:48] RESOLVED: PuppetFailure: Puppet has failed on maps1011:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:37:18] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [17:37:30] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [17:37:37] (03PS1) 10Eevans: data-gateway: Upgrade (staging) to v1.0.17 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371893 (https://phabricator.wikimedia.org/T444597) [17:38:13] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1016.eqiad.wmnet with reason: host reimage [17:38:21] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [17:38:32] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2213 (T445307)', diff saved to https://phabricator.wikimedia.org/P97619 and previous config saved to /var/cache/conftool/dbconfig/20261119-173832-marostegui.json [17:38:36] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [17:38:48] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2223.codfw.wmnet with reason: Maintenance [17:38:50] !log marostegui@cumin1004 dbctl commit (dc=all): 'Depooling db2223 (T445307)', diff saved to https://phabricator.wikimedia.org/P97620 and previous config saved to /var/cache/conftool/dbconfig/20261119-173849-marostegui.json [17:39:18] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12584011 (10Jhancock.wm) attempted to balance. waiting to see if new alert happens [17:40:23] !log dreamyjazz Deployed security patch for T444739 [17:40:57] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2017.codfw.wmnet, wikikube-worker2363.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2327.codfw.wmnet, wikikube-worker2197.codfw.wmnet, wikikube-worker2142.codfw.wmnet, wikikube-worker2006.codfw.wmnet, wikikube-worker2093.codfw.wmnet, wikikube-worker2323.codfw.wmnet are marked dow [17:40:57] oled https://wikitech.wikimedia.org/wiki/PyBal [17:41:06] (03PS7) 10JHathaway: Add a script to set the in_service_date [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) [17:41:26] (03CR) 10Isabelle Hurbain-Palatin: [C:03+1] mobileapps: remove statsd-exporter [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371128 (https://phabricator.wikimedia.org/T444937) (owner: 10Cwhite) [17:41:27] !log Deployed security patch for T439894 [17:41:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:41:35] I'm done with security deploys for now [17:41:57] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [17:43:39] (03CR) 10CI reject: [V:04-1] Add a script to set the in_service_date [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) (owner: 10JHathaway) [17:44:00] (03CR) 10Eevans: [C:03+2] data-gateway: Upgrade (staging) to v1.0.17 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371893 (https://phabricator.wikimedia.org/T444597) (owner: 10Eevans) [17:44:36] (03PS8) 10JHathaway: Add a script to set the in_service_date [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) [17:44:59] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2223 (T445307)', diff saved to https://phabricator.wikimedia.org/P97621 and previous config saved to /var/cache/conftool/dbconfig/20261119-174458-marostegui.json [17:45:02] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [17:45:52] (03Merged) 10jenkins-bot: data-gateway: Upgrade (staging) to v1.0.17 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371893 (https://phabricator.wikimedia.org/T444597) (owner: 10Eevans) [17:46:19] !log eevans@deploy2003 helmfile [staging] START helmfile.d/services/data-gateway: apply [17:46:36] !log eevans@deploy2003 helmfile [staging] DONE helmfile.d/services/data-gateway: apply [17:49:22] (03PS1) 10Dreamy Jazz: Enable CheckUser AbuseFilter variables on all wikis but metawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1371897 (https://phabricator.wikimedia.org/T234155) [17:53:31] jouncebot: nowandnext [17:53:31] For the next 0 hour(s) and 6 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261119T1700) [17:53:32] In 0 hour(s) and 6 minute(s): Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261119T1800) [17:53:32] In 0 hour(s) and 6 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261119T1800) [17:54:31] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1371897 (https://phabricator.wikimedia.org/T234155) (owner: 10Dreamy Jazz) [17:54:46] Running scap again for a config patch [17:54:50] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:55:11] Also allows me to check that scap is picking up the security patches correctly, as the rollback during one means I needed to do some steps manually [17:55:35] (03Merged) 10jenkins-bot: Enable CheckUser AbuseFilter variables on all wikis but metawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1371897 (https://phabricator.wikimedia.org/T234155) (owner: 10Dreamy Jazz) [17:56:03] !log dreamyjazz@deploy2003 Started scap sync-world: Backport for [[gerrit:1371897|Enable CheckUser AbuseFilter variables on all wikis but metawiki (T234155)]] [17:56:07] T234155: Create CheckUser-level abuse filters - https://phabricator.wikimedia.org/T234155 [17:57:52] 10ops-codfw, 10ops-drmrs, 10ops-eqdfw, 10ops-eqiad, and 6 others: Fix cables with placeholders names and planned status - https://phabricator.wikimedia.org/T432317#12584098 (10Jhancock.wm) [17:59:20] 10ops-codfw, 10ops-drmrs, 10ops-eqdfw, 10ops-eqiad, and 6 others: Fix cables with placeholders names and planned status - https://phabricator.wikimedia.org/T432317#12584099 (10Jhancock.wm) [18:00:01] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P97622 and previous config saved to /var/cache/conftool/dbconfig/20261119-180000-marostegui.json [18:00:05] bd808: #bothumor My software never has bugs. It just develops random features. Rise for Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261119T1800). [18:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261119T1800) [18:00:38] !log dreamyjazz@deploy2003 dreamyjazz: Backport for [[gerrit:1371897|Enable CheckUser AbuseFilter variables on all wikis but metawiki (T234155)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:00:51] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 97.91% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [18:01:08] (03PS1) 10Dduvall: zuul: Update image refs for 14.3.0-1 release [puppet] - 10https://gerrit.wikimedia.org/r/1371904 (https://phabricator.wikimedia.org/T445248) [18:01:11] !log dreamyjazz@deploy2003 dreamyjazz: Continuing with deployment [18:01:17] Dreamy_Jazz: Please ping me when you're done, because I have an urgent security patch I need to deploy [18:01:22] Sure [18:03:05] (03PS2) 10Brouberol: Move kafka-jumbo1017 from row F to A [puppet] - 10https://gerrit.wikimedia.org/r/1371049 (https://phabricator.wikimedia.org/T435775) [18:03:05] (03PS1) 10Brouberol: kafka-jumbo: update the kafka-jumbo1016 ips after re-racking to row A [puppet] - 10https://gerrit.wikimedia.org/r/1371905 (https://phabricator.wikimedia.org/T435775) [18:03:42] (03CR) 10Cathal Mooney: [C:03+1] kafka-jumbo: update the kafka-jumbo1016 ips after re-racking to row A [puppet] - 10https://gerrit.wikimedia.org/r/1371905 (https://phabricator.wikimedia.org/T435775) (owner: 10Brouberol) [18:04:07] (03PS3) 10MusikAnimal: InitialiseSettings: Enable ModeratorToolkit on 8 more Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1371090 (https://phabricator.wikimedia.org/T445385) [18:04:18] (03CR) 10Brouberol: "`" [puppet] - 10https://gerrit.wikimedia.org/r/1371905 (https://phabricator.wikimedia.org/T435775) (owner: 10Brouberol) [18:04:24] (03CR) 10Brouberol: [C:03+2] kafka-jumbo: update the kafka-jumbo1016 ips after re-racking to row A [puppet] - 10https://gerrit.wikimedia.org/r/1371905 (https://phabricator.wikimedia.org/T435775) (owner: 10Brouberol) [18:06:26] nothing for my window today [18:07:27] !log dreamyjazz@deploy2003 Finished scap sync-world: Backport for [[gerrit:1371897|Enable CheckUser AbuseFilter variables on all wikis but metawiki (T234155)]] (duration: 11m 24s) [18:07:31] T234155: Create CheckUser-level abuse filters - https://phabricator.wikimedia.org/T234155 [18:08:12] RoanKattouw: Done [18:08:52] (03PS1) 10Aleksandar Mastilovic: presto: Add resource groups execution time limits to prod [puppet] - 10https://gerrit.wikimedia.org/r/1371908 (https://phabricator.wikimedia.org/T445374) [18:09:27] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1016.eqiad.wmnet with OS trixie [18:09:38] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 3 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12584168 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1004 for host kafka-jumbo1016.eqiad.wmnet with OS tri... [18:11:09] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2187.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2274.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2044.codfw.wmnet, wikikube-worker2311.codfw.wmnet, wikikube-work [18:11:09] odfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2297.codfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2065.codfw.wmnet, wikikube-worker2060.codfw.wmnet, wikikube-worker2125.codfw.wmnet, wikikube-worker2196.codfw.wmnet, wikikube-worker2255.codfw.wmnet, wikikube-worker2089.codfw.wmnet, wikikube-worker2164.codfw.wmnet, wikikube-worker2149.codfw.wmnet, wikikube-worker2346.codfw.wmnet, wikikube-worker2105.codfw.wmnet, w [18:11:09] worker2114.codfw.wmnet, wikikube-worker2300.codfw.wmnet, wikikube-worker2285.codfw.wmnet, wikikube-worker2252.codfw.wmnet, wikikube-worker2073.codfw.wmnet, wikikube-worker2056.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [18:13:09] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:13:58] (03CR) 10Aleksandar Mastilovic: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1371908 (https://phabricator.wikimedia.org/T445374) (owner: 10Aleksandar Mastilovic) [18:15:03] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P97623 and previous config saved to /var/cache/conftool/dbconfig/20261119-181502-marostegui.json [18:15:52] !log Deploying security patch for T445505 [18:15:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:23:03] (03CR) 10Ssingh: [C:03+2] profile:puppetserver: create kapow dir and dummy file [puppet] - 10https://gerrit.wikimedia.org/r/1371766 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [18:24:56] !log brouberol@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host is being re-racked to row A [18:25:10] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 3 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12584204 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=d97accb6-cfe7-4902-bb24-638e57412dd5) set by brouberol@cumin1004 for 4:... [18:29:20] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for toyofuku - https://phabricator.wikimedia.org/T445548#12584215 (10SToyofuku-WMF) DMed Manuel back, but I had an old one but missed the due date and it was removed ([[ https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppe... [18:30:05] !log marostegui@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2223 (T445307)', diff saved to https://phabricator.wikimedia.org/P97624 and previous config saved to /var/cache/conftool/dbconfig/20261119-183004-marostegui.json [18:30:09] T445307: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307 [18:35:39] (03CR) 10Brouberol: Move kafka-jumbo1017 from row F to A [puppet] - 10https://gerrit.wikimedia.org/r/1371049 (https://phabricator.wikimedia.org/T435775) (owner: 10Brouberol) [18:36:00] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2219 is not coming back online after reboot - https://phabricator.wikimedia.org/T445479#12584241 (10Jhancock.wm) stalled on the bios update. working on it... [18:36:15] (03PS1) 10Milimetric: sqoop: Move wbc_entity_usage to the production not-history job [puppet] - 10https://gerrit.wikimedia.org/r/1371909 (https://phabricator.wikimedia.org/T431477) [18:37:07] (03CR) 10CI reject: [V:04-1] sqoop: Move wbc_entity_usage to the production not-history job [puppet] - 10https://gerrit.wikimedia.org/r/1371909 (https://phabricator.wikimedia.org/T431477) (owner: 10Milimetric) [18:37:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:38:02] (03PS1) 10Lizimedia: Add namespace for sbom-scanner [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371910 (https://phabricator.wikimedia.org/T445130) [18:44:13] (03PS1) 10Lizimedia: Add sbom-scanner usernabes to deployment server [puppet] - 10https://gerrit.wikimedia.org/r/1371912 (https://phabricator.wikimedia.org/T445130) [18:44:35] !log cmooney@cumin1004 START - Cookbook sre.dns.wipe-cache kafka-jumbo1017.eqiad.wmnet on all recursors [18:44:38] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-jumbo1017.eqiad.wmnet on all recursors [18:46:11] (03CR) 10Xcollazo: [C:03+1] presto: Add resource groups execution time limits to prod [puppet] - 10https://gerrit.wikimedia.org/r/1371908 (https://phabricator.wikimedia.org/T445374) (owner: 10Aleksandar Mastilovic) [18:47:20] (03PS3) 10Brouberol: Move kafka-jumbo1017 from row F to A [puppet] - 10https://gerrit.wikimedia.org/r/1371049 (https://phabricator.wikimedia.org/T435775) [18:47:21] (03PS1) 10Brouberol: kafka-jumbo: update the kafka-jumbo1017 ips after re-racking to row A [puppet] - 10https://gerrit.wikimedia.org/r/1371914 (https://phabricator.wikimedia.org/T435775) [18:47:53] (03PS1) 10TChin: [EventBus] Enable tags_change rc0 streams [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1371915 (https://phabricator.wikimedia.org/T434143) [18:49:05] (03CR) 10Brouberol: [C:03+2] Move kafka-jumbo1017 from row F to A [puppet] - 10https://gerrit.wikimedia.org/r/1371049 (https://phabricator.wikimedia.org/T435775) (owner: 10Brouberol) [18:49:45] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [18:50:04] (03PS2) 10Brouberol: kafka-jumbo: update the kafka-jumbo1017 ips after re-racking to row A [puppet] - 10https://gerrit.wikimedia.org/r/1371914 (https://phabricator.wikimedia.org/T435775) [18:50:18] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:51:46] (03CR) 10Cathal Mooney: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1371914 (https://phabricator.wikimedia.org/T435775) (owner: 10Brouberol) [18:52:01] (03CR) 10Brouberol: [C:03+2] kafka-jumbo: update the kafka-jumbo1017 ips after re-racking to row A [puppet] - 10https://gerrit.wikimedia.org/r/1371914 (https://phabricator.wikimedia.org/T435775) (owner: 10Brouberol) [18:54:57] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for kafka-jumbo1017 - cmooney@cumin1004" [18:55:11] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for kafka-jumbo1017 - cmooney@cumin1004" [18:55:11] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [18:57:41] !log cmooney@cumin1004 START - Cookbook sre.dns.wipe-cache kafka-jumbo1017.eqiad.wmnet on all recursors [18:57:44] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kafka-jumbo1017.eqiad.wmnet on all recursors [18:58:45] !log brouberol@cumin1004 START - Cookbook sre.hosts.reimage for host kafka-jumbo1017.eqiad.wmnet with OS trixie [18:58:59] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 3 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12584307 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1004 for host kafka-jumbo1017.eqiad.wmnet with OS... [18:59:28] !log brouberol@cumin1004 START - Cookbook sre.hosts.dhcp for host kafka-jumbo1017.eqiad.wmnet (push config only) [18:59:32] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host kafka-jumbo1017.eqiad.wmnet (push config only) [19:00:05] jeena and jnuche: It is that lovely time of the day again! You are hereby commanded to deploy MediaWiki train - Utc-7 Version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261119T1900). [19:08:30] o/ i'll be running train today [19:11:07] !log brouberol@cumin1004 START - Cookbook sre.hosts.dhcp for host kafka-jumbo1017.eqiad.wmnet (remove config only) [19:11:11] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host kafka-jumbo1017.eqiad.wmnet (remove config only) [19:11:24] (03PS1) 10TrainBranchBot: group2 to 1.48.0-wmf.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1371932 (https://phabricator.wikimedia.org/T442223) [19:11:27] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by dduvall@deploy2003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1371932 (https://phabricator.wikimedia.org/T442223) (owner: 10TrainBranchBot) [19:12:13] (03Merged) 10jenkins-bot: group2 to 1.48.0-wmf.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1371932 (https://phabricator.wikimedia.org/T442223) (owner: 10TrainBranchBot) [19:13:05] !log brouberol@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage [19:14:50] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 22d 23h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [19:15:26] !log jforrester@deploy2003 Finished scap deploy-service wikifunctions: Weekly service deploy / 1371123 (duration: 1265m 04s) [19:16:57] PROBLEM - Host kafka-jumbo1017 is DOWN: PING CRITICAL - Packet loss = 100% [19:19:34] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-jumbo1017.eqiad.wmnet with reason: host reimage [19:19:40] RECOVERY - Host kafka-jumbo1017 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [19:22:56] !log dduvall@deploy2003 rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.5 refs T442223 [19:22:59] T442223: 1.48.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T442223 [19:25:19] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [19:28:10] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [19:29:50] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [19:34:29] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [19:35:15] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-jumbo1017.eqiad.wmnet with OS trixie [19:35:25] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12584448 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1004 for host kafka-jumbo1017.eqiad.wmnet with OS tri... [19:36:14] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2141.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2280.codfw.wmnet, wikikube-worker2171.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2177.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-work [19:36:14] odfw.wmnet, wikikube-worker2372.codfw.wmnet, wikikube-worker2322.codfw.wmnet, wikikube-worker2329.codfw.wmnet, wikikube-worker2339.codfw.wmnet, wikikube-worker2276.codfw.wmnet, wikikube-worker2300.codfw.wmnet, wikikube-worker2073.codfw.wmnet, wikikube-worker2208.codfw.wmnet, wikikube-worker2321.codfw.wmnet, wikikube-worker2292.codfw.wmnet, wikikube-worker2138.codfw.wmnet, wikikube-worker2275.codfw.wmnet, wikikube-worker2318.codfw.wmnet, w [19:36:14] worker2187.codfw.wmnet, wikikube-worker2261.codfw.wmnet, wikikube-worker2291.codfw.wmnet, wikikube-worker2154.codfw.wmnet, wikikube-worker2142.codfw.wmnet, wikikube-worker2147.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [19:37:14] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:37:47] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [19:38:27] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [19:39:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12584451 (10phaultfinder) [19:41:17] (03CR) 10Ladsgroup: "There is no way to run a pcc when the role doesn't match anything 😄" [puppet] - 10https://gerrit.wikimedia.org/r/1371837 (owner: 10Ladsgroup) [19:41:43] (03CR) 10Lerickson: [C:03+2] Release wdqs-proxy v1.0.1 with the wdqs-jena update including better parse error messages. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371881 (https://phabricator.wikimedia.org/T445363) (owner: 10Lerickson) [19:42:08] (03PS1) 10Eevans: data-gateway: Upgrade (production) to v1.0.17 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371966 (https://phabricator.wikimedia.org/T444597) [19:43:05] (03Merged) 10jenkins-bot: Release wdqs-proxy v1.0.1 with the wdqs-jena update including better parse error messages. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371881 (https://phabricator.wikimedia.org/T445363) (owner: 10Lerickson) [19:43:42] (03PS1) 10TChin: [Eventgate] Look up destination_event_service and event_service_name when querying streams [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371967 (https://phabricator.wikimedia.org/T321557) [19:44:04] (03CR) 10Eevans: [C:03+2] data-gateway: Upgrade (production) to v1.0.17 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371966 (https://phabricator.wikimedia.org/T444597) (owner: 10Eevans) [19:45:29] (03Merged) 10jenkins-bot: data-gateway: Upgrade (production) to v1.0.17 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1371966 (https://phabricator.wikimedia.org/T444597) (owner: 10Eevans) [19:47:17] !log eevans@deploy2003 helmfile [eqiad] START helmfile.d/services/data-gateway: apply [19:47:36] !log eevans@deploy2003 helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply [19:48:15] !log eevans@deploy2003 helmfile [codfw] START helmfile.d/services/data-gateway: apply [19:48:29] !log eevans@deploy2003 helmfile [codfw] DONE helmfile.d/services/data-gateway: apply [19:48:38] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [19:49:06] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [19:50:16] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [19:50:39] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [19:51:19] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12584477 (10brouberol) ` brouberol@cumin1004:~$ sudo cumin A:kafka-jumbo-eqiad 'grep rack /etc/kafka/server.properties' 9 hosts will be targeted: ka... [19:52:33] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [19:53:34] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [19:53:43] 10ops-eqiad, 06SRE, 06DC-Ops, 10Kafka-Infrastructure, and 2 others: Heterogeneous kafka-jumbo-eqiad rack placement - https://phabricator.wikimedia.org/T435775#12584480 (10brouberol) `admin_ng` diffs applied in both dse clusters: ` [. dse-k8s-eqiad] external-services, kafka-jumbo-eqiad, Endpoints (v1) has c... [19:54:25] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [19:54:58] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [19:56:11] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1367566 (https://phabricator.wikimedia.org/T441507) (owner: 10JHathaway) [19:56:36] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1367567 (https://phabricator.wikimedia.org/T441507) (owner: 10JHathaway) [19:56:40] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [19:57:03] 10ops-eqiad, 06DC-Ops: Unresponsive management for backup1010.mgmt:22 - https://phabricator.wikimedia.org/T445587 (10phaultfinder) 03NEW [19:57:04] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [19:57:25] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [19:57:46] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [19:58:21] (03CR) 10JHathaway: [C:03+2] Rename role::insetup_noferm to role::insetup::noferm [puppet] - 10https://gerrit.wikimedia.org/r/1367566 (https://phabricator.wikimedia.org/T441507) (owner: 10JHathaway) [19:58:26] (03CR) 10JHathaway: [C:03+2] Rename role::ml_k8s::insetup_gpu to role::insetup::ml_k8s_gpu [puppet] - 10https://gerrit.wikimedia.org/r/1367567 (https://phabricator.wikimedia.org/T441507) (owner: 10JHathaway) [20:02:01] (03PS1) 10Bking: Revert "P:idp remove broken client" [puppet] - 10https://gerrit.wikimedia.org/r/1371969 [20:02:15] FIRING: SwiftObjectCountSiteDisparity: MediaWiki swift object counts site diffs - https://wikitech.wikimedia.org/wiki/Swift/How_To - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift - https://alerts.wikimedia.org/?q=alertname%3DSwiftObjectCountSiteDisparity [20:04:03] (03CR) 10Bking: [C:03+2] Revert "P:idp remove broken client" [puppet] - 10https://gerrit.wikimedia.org/r/1371969 (owner: 10Bking) [20:04:15] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [20:04:21] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [20:05:55] FIRING: GanetiCACertificateAboutToExpire: Ganeti CA certificate ganeti01.svc.drmrs.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Ganeti#Renew_cluster_certificates - TODO - https://alerts.wikimedia.org/?q=alertname%3DGanetiCACertificateAboutToExpire [20:11:10] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:19:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12584540 (10phaultfinder) [20:41:11] (03PS1) 10Bking: matomo: use 'latest' tag for matomo image in helm chart, use real image ID for deploy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372005 (https://phabricator.wikimedia.org/T445553) [20:42:43] (03PS2) 10Bking: matomo: use 'latest' tag for matomo image in helm chart, use real image ID for deploy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372005 (https://phabricator.wikimedia.org/T445553) [20:50:13] (03CR) 10JHathaway: Add a script to set the in_service_date (031 comment) [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) (owner: 10JHathaway) [20:50:29] FIRING: KubernetesDeploymentUnavailableReplicas: ... [20:50:30] Deployment editcheck-headless-main in editcheck-headless at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=editcheck-headless&var-deployment=editcheck-headless-main - ... [20:50:30] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [20:52:25] (03CR) 10JHathaway: "@ayounsi@wikimedia.org Any chance you can review tomorrow? If not @rcoccioli@wikimedia.org said he could review Monday." [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) (owner: 10JHathaway) [20:56:45] (03PS4) 10Medelius: Update suggestion onboarding experiment to include desktop users [extensions/WikimediaEvents] (wmf/1.48.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1371981 (https://phabricator.wikimedia.org/T443854) [20:57:12] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 19 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [extensions/WikimediaEvents] (wmf/1.48.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1371981 (https://phabricator.wikimedia.org/T443854) (owner: 10Medelius) [21:00:04] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on db2219.codfw.wmnet with reason: hw issues [21:00:05] RoanKattouw, urbanecm, TheresNoTime, and cjming: #bothumor My software never has bugs. It just develops random features. Rise for UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261119T2100). [21:00:05] cmede: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [21:00:25] (03CR) 10Herron: [C:03+2] udp2log: Enable hourly logrotate for post.log [puppet] - 10https://gerrit.wikimedia.org/r/1370695 (https://phabricator.wikimedia.org/T445147) (owner: 10Krinkle) [21:00:59] o/ [21:01:16] would someone be able to deploy for me? [21:04:42] (03PS2) 10Ahmon Dancy: buildkit: Bump v0.35.1 [puppet] - 10https://gerrit.wikimedia.org/r/1371072 (https://phabricator.wikimedia.org/T445370) [21:06:25] (03PS1) 10Ahmon Dancy: docker::gc: Upgrade docker-gc to 1.5.0 [puppet] - 10https://gerrit.wikimedia.org/r/1372039 (https://phabricator.wikimedia.org/T443833) [21:07:47] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy2003 using scap backport" [extensions/WikimediaEvents] (wmf/1.48.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1371981 (https://phabricator.wikimedia.org/T443854) (owner: 10Medelius) [21:09:25] (03Merged) 10jenkins-bot: Update suggestion onboarding experiment to include desktop users [extensions/WikimediaEvents] (wmf/1.48.0-wmf.5) - 10https://gerrit.wikimedia.org/r/1371981 (https://phabricator.wikimedia.org/T443854) (owner: 10Medelius) [21:09:44] !log kemayo@deploy2003 Started scap sync-world: Backport for [[gerrit:1371981|Update suggestion onboarding experiment to include desktop users (T443854)]] (1.48.0-wmf.5) [21:09:48] T443854: Instrument experiment Suggestion Mode Onboarding - https://phabricator.wikimedia.org/T443854 [21:11:08] 10ops-codfw, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: wikikube-worker2280 unreachable - https://phabricator.wikimedia.org/T423395#12584724 (10Jhancock.wm) Hey sorry this took me so long to get back to. Supermicro is asking for a few things on this case that i can't provide. can someone with root a... [21:13:39] !log kemayo@deploy2003 caro, kemayo: Backport for [[gerrit:1371981|Update suggestion onboarding experiment to include desktop users (T443854)]] (1.48.0-wmf.5) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:13:44] checking [21:15:22] looks good! [21:15:27] !log kemayo@deploy2003 caro, kemayo: Continuing with deployment [21:21:31] !log kemayo@deploy2003 Finished scap sync-world: Backport for [[gerrit:1371981|Update suggestion onboarding experiment to include desktop users (T443854)]] (1.48.0-wmf.5) (duration: 11m 47s) [21:21:34] T443854: Instrument experiment Suggestion Mode Onboarding - https://phabricator.wikimedia.org/T443854 [21:28:09] (03CR) 10Btullis: "Needs a version bump in Chart.yaml as well, or it won't actually be published." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372005 (https://phabricator.wikimedia.org/T445553) (owner: 10Bking) [21:28:54] (03PS3) 10Bking: matomo: use 'latest' tag for matomo image in helm chart, use real image ID for deploy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372005 (https://phabricator.wikimedia.org/T445553) [21:29:44] (03CR) 10Bking: "ACK, I've updated accordingly." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372005 (https://phabricator.wikimedia.org/T445553) (owner: 10Bking) [21:37:48] (03CR) 10Bking: [C:03+2] matomo: use 'latest' tag for matomo image in helm chart, use real image ID for deploy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372005 (https://phabricator.wikimedia.org/T445553) (owner: 10Bking) [21:39:05] !log bking@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [21:39:36] !log bking@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [21:50:51] RESOLVED: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 90.59% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [21:53:07] (03PS1) 10Bking: matomo-next: Fix container image override [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372113 (https://phabricator.wikimedia.org/T445553) [22:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261119T2200) [22:02:56] (03CR) 10Bking: [C:03+2] "self-merging, as this is a pre-production service." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372113 (https://phabricator.wikimedia.org/T445553) (owner: 10Bking) [22:13:47] (03CR) 10Btullis: [C:03+1] matomo-next: Fix container image override [deployment-charts] - 10https://gerrit.wikimedia.org/r/1372113 (https://phabricator.wikimedia.org/T445553) (owner: 10Bking) [22:21:19] (03CR) 10JHathaway: [C:03+1] "looks good, feel free to merge, one question and a suggestion." [puppet] - 10https://gerrit.wikimedia.org/r/1370979 (https://phabricator.wikimedia.org/T445332) (owner: 10Effie Mouzeli) [22:26:22] (03CR) 10JHathaway: [C:03+1] upgrade-firmware: Allow updates of NonRAID controllers [cookbooks] - 10https://gerrit.wikimedia.org/r/1354829 (https://phabricator.wikimedia.org/T438999) (owner: 10JMeybohm) [22:29:10] (03CR) 10JHathaway: [C:03+1] "yup, my bad, sorry" [puppet] - 10https://gerrit.wikimedia.org/r/1348935 (owner: 10Hashar) [22:29:13] (03CR) 10JHathaway: [C:03+2] rake: pass --ff-only when doing git pull [puppet] - 10https://gerrit.wikimedia.org/r/1348935 (owner: 10Hashar) [22:31:25] !log Scan ran in T439783 has finished [22:31:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:31:29] T439783: MediaModerationImageContentsLookup uses original file mime type not the thumbnail mime type - https://phabricator.wikimedia.org/T439783 [22:37:14] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - linkrecommendation-external_4006: Servers wikikube-worker2251.codfw.wmnet, wikikube-worker2364.codfw.wmnet, wikikube-worker2141.codfw.wmnet, wikikube-worker2341.codfw.wmnet, wikikube-worker2174.codfw.wmnet, wikikube-worker2202.codfw.wmnet, wikikube-worker2102.codfw.wmnet, wikikube-worker2321.codfw.wmnet, wikikube-worker2361.codfw.wmnet, wikikube-work [22:37:14] odfw.wmnet, wikikube-worker2036.codfw.wmnet, wikikube-worker2328.codfw.wmnet, wikikube-worker2150.codfw.wmnet, wikikube-worker2113.codfw.wmnet, wikikube-worker2136.codfw.wmnet, wikikube-worker2161.codfw.wmnet, wikikube-worker2076.codfw.wmnet, wikikube-worker2320.codfw.wmnet, wikikube-worker2261.codfw.wmnet, wikikube-worker2132.codfw.wmnet, wikikube-worker2291.codfw.wmnet, wikikube-worker2165.codfw.wmnet, wikikube-worker2044.codfw.wmnet, w [22:37:14] worker2190.codfw.wmnet, wikikube-worker2278.codfw.wmnet, wikikube-worker2287.codfw.wmnet, wikikube-worker2248.codfw.wmnet, wikikube-worker2273.codfw.wmnet, wikikube-worker2198.codfw.wmn https://wikitech.wikimedia.org/wiki/PyBal [22:37:40] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_codfw-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:38:14] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:42:30] (03CR) 10Dillon: "Production part looks good! I just have one question for Growth about the beta config values removed in PS3 below." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1371090 (https://phabricator.wikimedia.org/T445385) (owner: 10MusikAnimal) [22:50:18] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [22:55:20] !log bking@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [22:55:50] !log bking@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [22:55:52] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [23:14:50] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for cloud_wmnet_ca expires in 22d 19h 41m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [23:25:51] (03PS1) 10Southparkfan: profile::webperf::processors: Allow other Kafka clusters [puppet] - 10https://gerrit.wikimedia.org/r/1372225 (https://phabricator.wikimedia.org/T445011) [23:28:24] (03CR) 10CI reject: [V:04-1] profile::webperf::processors: Allow other Kafka clusters [puppet] - 10https://gerrit.wikimedia.org/r/1372225 (https://phabricator.wikimedia.org/T445011) (owner: 10Southparkfan) [23:30:05] FIRING: HelmReleaseBadStatus: Helm release wdqs/main-internal on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [23:40:11] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for dhardy to restore deployment access - https://phabricator.wikimedia.org/T445598 (10Dillon) 03NEW [23:40:20] (03PS2) 10Southparkfan: profile::webperf::processors: Allow other Kafka clusters [puppet] - 10https://gerrit.wikimedia.org/r/1372225 (https://phabricator.wikimedia.org/T445011) [23:41:00] (03CR) 10Southparkfan: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1372225 (https://phabricator.wikimedia.org/T445011) (owner: 10Southparkfan) [23:45:49] (03PS3) 10Southparkfan: profile::webperf::processors: Allow other Kafka clusters [puppet] - 10https://gerrit.wikimedia.org/r/1372225 (https://phabricator.wikimedia.org/T445011) [23:45:57] (03CR) 10Southparkfan: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1372225 (https://phabricator.wikimedia.org/T445011) (owner: 10Southparkfan) [23:55:21] (03PS4) 10Southparkfan: profile::webperf::processors: Allow other Kafka clusters [puppet] - 10https://gerrit.wikimedia.org/r/1372225 (https://phabricator.wikimedia.org/T445011) [23:58:04] (03CR) 10Southparkfan: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1372225 (https://phabricator.wikimedia.org/T445011) (owner: 10Southparkfan)