[00:02:25] RESOLVED: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:12:13] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1317757 [01:12:13] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1317757 (owner: 10TrainBranchBot) [01:25:12] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1317757 (owner: 10TrainBranchBot) [01:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [02:00:53] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:07:30] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 36s) [02:08:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:13:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:40] PROBLEM - SSH on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [02:21:40] PROBLEM - librenms.wikimedia.org tls expiry on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [02:21:58] PROBLEM - librenms.wikimedia.org requires authentication on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [02:23:57] FIRING: [4x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:36:30] RECOVERY - librenms.wikimedia.org tls expiry on netmon2002 is OK: OK - Certificate librenms.wikimedia.org will expire on Thu 10 Sep 2026 02:24:49 AM GMT +0000. https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [02:36:30] RECOVERY - SSH on netmon2002 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u10 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [02:36:48] RECOVERY - librenms.wikimedia.org requires authentication on netmon2002 is OK: HTTP OK: Status line output matched HTTP/1.1 302 - 701 bytes in 0.132 second response time https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [02:43:57] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:54:27] FIRING: [15x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [03:12:40] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [03:13:24] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [03:16:14] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [03:16:31] FIRING: [2x] RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [03:18:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:21:31] FIRING: [2x] RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [03:49:30] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [03:51:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [03:53:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [04:48:11] !log ryankemper@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host [04:48:15] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [04:48:17] !log ryankemper@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host (duration: 00m 05s) [04:48:21] !log ryankemper@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host [04:48:26] !log ryankemper@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 deploy to fresh wdqs host (duration: 00m 05s) [04:51:42] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs1018.eqiad.wmnet, repooling source-only afterwards [04:51:52] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs1019.eqiad.wmnet, repooling source-only afterwards [05:15:17] (03PS1) 10Phuedx: sessionTick: Use Test Kitchen to send action=feature_not_available events [extensions/WikimediaEvents] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1317814 (https://phabricator.wikimedia.org/T413296) [05:15:28] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 27 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1317814 (https://phabricator.wikimedia.org/T413296) (owner: 10Phuedx) [05:15:50] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 27 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313923 (https://phabricator.wikimedia.org/T415370) (owner: 10Phuedx) [05:24:27] FIRING: [15x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:35:24] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [05:38:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [05:38:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:43:16] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [05:43:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [05:43:57] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:48:57] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:00:30] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs1018.eqiad.wmnet, repooling source-only afterwards [06:00:34] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs1019.eqiad.wmnet, repooling source-only afterwards [06:00:34] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [06:03:54] (03PS1) 10Marostegui: check_private_data_report: Add clouddb1032 [puppet] - 10https://gerrit.wikimedia.org/r/1317825 (https://phabricator.wikimedia.org/T409557) [06:07:24] (03CR) 10Marostegui: [C:03+2] check_private_data_report: Add clouddb1032 [puppet] - 10https://gerrit.wikimedia.org/r/1317825 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [06:08:40] (03PS1) 10Marostegui: Revert "pontoon: add rkemper-kafka stack" [puppet] - 10https://gerrit.wikimedia.org/r/1317829 [06:09:23] (03CR) 10Marostegui: [C:03+2] Revert "pontoon: add rkemper-kafka stack" [puppet] - 10https://gerrit.wikimedia.org/r/1317829 (owner: 10Marostegui) [06:17:02] (03PS1) 10Marostegui: clouddb1032: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1317830 [06:18:56] (03CR) 10Marostegui: [C:03+2] clouddb1032: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1317830 (owner: 10Marostegui) [06:22:29] (03PS1) 10Marostegui: eqiad.yaml: Add clouddb1032 [puppet] - 10https://gerrit.wikimedia.org/r/1317831 (https://phabricator.wikimedia.org/T409557) [06:23:26] (03CR) 10Marostegui: [C:03+2] eqiad.yaml: Add clouddb1032 [puppet] - 10https://gerrit.wikimedia.org/r/1317831 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [06:25:21] !log marostegui@cumin1003 conftool action : set/weight=100; selector: name=clouddb1032.eqiad.wmnet,service=s4 [06:25:25] !log marostegui@cumin1003 conftool action : set/weight=100; selector: name=clouddb1032.eqiad.wmnet,service=s6 [06:25:34] !log marostegui@cumin1003 conftool action : set/pooled=no; selector: name=clouddb1032.eqiad.wmnet,service=s4 [06:25:38] !log marostegui@cumin1003 conftool action : set/pooled=no; selector: name=clouddb1032.eqiad.wmnet,service=s6 [06:25:46] !log marostegui@cumin1003 conftool action : set/weight=50; selector: name=clouddb1032.eqiad.wmnet,service=s6 [06:25:49] !log marostegui@cumin1003 conftool action : set/weight=50; selector: name=clouddb1032.eqiad.wmnet,service=s4 [06:25:53] !log marostegui@cumin1003 conftool action : set/pooled=yes; selector: name=clouddb1032.eqiad.wmnet,service=s6 [06:25:56] !log marostegui@cumin1003 conftool action : set/pooled=yes; selector: name=clouddb1032.eqiad.wmnet,service=s4 [06:29:15] !log marostegui@cumin1003 conftool action : set/weight=100; selector: name=clouddb1032.eqiad.wmnet,service=s6 [06:29:18] !log marostegui@cumin1003 conftool action : set/weight=100; selector: name=clouddb1032.eqiad.wmnet,service=s4 [06:35:29] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1228.eqiad.wmnet with reason: Rebooting [06:37:34] (03CR) 10Brouberol: [C:03+1] wdqs: restore Blazegraph exporters [cookbooks] - 10https://gerrit.wikimedia.org/r/1315665 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [06:38:31] (03CR) 10Brouberol: [C:03+1] wdqs: bound missing updater metrics [cookbooks] - 10https://gerrit.wikimedia.org/r/1315666 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [06:39:16] (03CR) 10Brouberol: [C:03+1] sre.wdqs.data-transfer: abort on failed transfer (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1317127 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [06:40:29] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1020.eqiad.wmnet with reason: Cloning [06:41:56] (03PS2) 10Hashar: python2-build-bullseye: Fix pip in Python 2 virtualenvs [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1312574 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [06:42:51] (03CR) 10Hashar: [C:03+1] "Rebased to resolve the conflict in Debian changelog following the weekly rebuild of images." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1312574 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [06:44:24] (03PS1) 10Marostegui: mariadb: Productionize clouddb1033 [puppet] - 10https://gerrit.wikimedia.org/r/1317834 (https://phabricator.wikimedia.org/T409557) [06:46:47] (03CR) 10Marostegui: [C:03+2] mariadb: Productionize clouddb1033 [puppet] - 10https://gerrit.wikimedia.org/r/1317834 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [06:48:58] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:53:57] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:54:26] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [06:56:46] !log ryankemper@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs1018.eqiad.wmnet [06:57:36] !log ryankemper@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs1019.eqiad.wmnet [06:58:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [06:58:57] (03CR) 10Slyngshede: [C:03+1] signup: Do not use a valid, unrelated domain as a placeholder [software/bitu] - 10https://gerrit.wikimedia.org/r/1307422 (owner: 10Majavah) [06:58:57] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:59:03] (03CR) 10Slyngshede: [C:03+2] signup: Do not use a valid, unrelated domain as a placeholder [software/bitu] - 10https://gerrit.wikimedia.org/r/1307422 (owner: 10Majavah) [06:59:15] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [06:59:27] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:00:04] Amir1, urbanecm, and awight: That opportune time for a UTC morning backport window deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T0700). [07:00:04] phuedx: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:53] o/ I can deploy these two patches with SpiderPig. I'll do the instrumentation first and then monitor it. The config change is a NOOP [07:01:04] (03CR) 10Marostegui: "Let me know when you want me to test again." [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) (owner: 10Federico Ceratto) [07:01:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by phuedx@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1317814 (https://phabricator.wikimedia.org/T413296) (owner: 10Phuedx) [07:02:16] (03Merged) 10jenkins-bot: signup: Do not use a valid, unrelated domain as a placeholder [software/bitu] - 10https://gerrit.wikimedia.org/r/1307422 (owner: 10Majavah) [07:03:25] (03Merged) 10jenkins-bot: sessionTick: Use Test Kitchen to send action=feature_not_available events [extensions/WikimediaEvents] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1317814 (https://phabricator.wikimedia.org/T413296) (owner: 10Phuedx) [07:03:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [07:04:43] !log phuedx@deploy1003 Started scap sync-world: Backport for [[gerrit:1317814|sessionTick: Use Test Kitchen to send action=feature_not_available events (T413296)]] [07:04:47] T413296: Improve session tick instrument's handling of lack of localStorage - https://phabricator.wikimedia.org/T413296 [07:13:41] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [07:14:31] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [07:16:47] !log T430880 [WDQS] Reimaged `wdqs1018` and `wdqs1019` to Bookworm, restored data using test-cookbook change 1317128, and repooled both; 25/36 hosts complete [07:16:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:16:51] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [07:20:42] !log phuedx@deploy1003 phuedx: Backport for [[gerrit:1317814|sessionTick: Use Test Kitchen to send action=feature_not_available events (T413296)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:20:45] T413296: Improve session tick instrument's handling of lack of localStorage - https://phabricator.wikimedia.org/T413296 [07:20:46] (03PS7) 10Trueg: WDQSv2-next: Staging deployment of WDQSv2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313858 (https://phabricator.wikimedia.org/T432348) [07:22:27] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1179.eqiad.wmnet with reason: Maintenance [07:22:35] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1179 (T431660)', diff saved to https://phabricator.wikimedia.org/P95149 and previous config saved to /var/cache/conftool/dbconfig/20260727-072234-cwilliams.json [07:22:45] !log Drop tables in akwiki nawiki pihwiki - growthexperiments_* T428885 [07:22:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:22:51] T428885: Drop growthexperiments_* tables from closed wikis - https://phabricator.wikimedia.org/T428885 [07:23:36] (03PS1) 10Trueg: WDQSv2: Use a PVC template spec instead of a single PVC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317845 (https://phabricator.wikimedia.org/T432029) [07:24:50] No errors in the browser console. Continuing with sync. I'll monitor the client errors dashboard [07:25:00] !log phuedx@deploy1003 phuedx: Continuing with deployment [07:26:58] !log Rename tables on s3 T426341 [07:27:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:27:02] T426341: Drop DiscussionTools database tables from wikis which don't need them - https://phabricator.wikimedia.org/T426341 [07:28:30] (03PS1) 10Jelto: Revert "hieradata: Add Wikimania address to extra_trust" [puppet] - 10https://gerrit.wikimedia.org/r/1317847 (https://phabricator.wikimedia.org/T432377) [07:28:32] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1179: Maintenance [07:31:18] The sync is taking longer than I expected. I'm not going to deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1313923 and instead focus on monitor the sessionTick change [07:31:55] (03PS2) 10Trueg: WDQSv2: scale resource limits to max with respect to the dedicated wdqs nodes. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313874 (https://phabricator.wikimedia.org/T431395) [07:31:58] I've updated the deployment calendar [07:34:47] (03CR) 10Marostegui: "What's pending here? Another round of code review?" [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [07:37:16] !log phuedx@deploy1003 Finished scap sync-world: Backport for [[gerrit:1317814|sessionTick: Use Test Kitchen to send action=feature_not_available events (T413296)]] (duration: 32m 33s) [07:37:20] T413296: Improve session tick instrument's handling of lack of localStorage - https://phabricator.wikimedia.org/T413296 [07:43:03] (03CR) 10Hashar: [C:03+1] "The diff looks good against `stable-3.10` Thank you!" [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [07:44:05] There has been no obvious spike in server- or client-side errors. Events from the sessionTick instrument with action=feature_not_available has settled to nothing [07:44:21] !log UTC morning backport window done [07:44:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:44:31] (03CR) 10Hashar: [C:03+1] "@jwodstrcil@wikimedia.org can you puppet-merge this change please? If I remember well the new Soy template will then be taken in account" [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [07:45:19] (03PS8) 10Trueg: WDQSv2-next: Staging deployment of WDQSv2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313858 (https://phabricator.wikimedia.org/T432348) [07:45:40] marostegui: hi! there's some failures in clouddb systemd units (wmf-pt-kill), and one of them is silenced by you, are the others also expected? (https://alerts.wikimedia.org/?q=team%3Dwmcs&q=alertname%3DSystemdUnitFailed) [07:46:01] dcaro: that's expected [07:46:52] ack, thanks! I'll silence them out then 👍 (/me oncall this week) [07:47:48] thank you! [07:50:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [07:57:59] Stepping away from the keyboard for a while [07:58:15] (03CR) 10CWilliams: [C:03+1] "LGTM, thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1315894 (owner: 10Elukey) [08:00:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [08:01:12] (03CR) 10Jelto: [C:03+2] deployment_server/k8s: set kubeconfig files for etherpad [puppet] - 10https://gerrit.wikimedia.org/r/1315749 (owner: 10Jelto) [08:15:25] (03CR) 10Ayounsi: [C:03+2] Add GanetiBGPNoInPrefixes alert [alerts] - 10https://gerrit.wikimedia.org/r/1315778 (owner: 10Ayounsi) [08:16:01] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1179: Maintenance [08:18:03] (03Merged) 10jenkins-bot: Add GanetiBGPNoInPrefixes alert [alerts] - 10https://gerrit.wikimedia.org/r/1315778 (owner: 10Ayounsi) [08:25:35] (03PS1) 10JMeybohm: admin/data: Remove lost yubikey [puppet] - 10https://gerrit.wikimedia.org/r/1317859 [08:27:42] (03CR) 10JMeybohm: [C:03+2] admin/data: Remove lost yubikey [puppet] - 10https://gerrit.wikimedia.org/r/1317859 (owner: 10JMeybohm) [08:34:15] (03CR) 10Effie Mouzeli: [C:03+2] mcrouter_wancache: re-add mc-gp2006 to the gutter-pool [puppet] - 10https://gerrit.wikimedia.org/r/1315946 (https://phabricator.wikimedia.org/T432716) (owner: 10Effie Mouzeli) [08:36:25] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [08:36:56] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance [08:37:04] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1203 (T431660)', diff saved to https://phabricator.wikimedia.org/P95154 and previous config saved to /var/cache/conftool/dbconfig/20260727-083703-cwilliams.json [08:39:27] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:39:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [08:43:01] !log jiji@cumin1003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1136.eqiad.wmnet [08:43:05] !log jiji@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1136.eqiad.wmnet [08:43:07] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1203: Maintenance [08:43:40] !log jiji@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1136.eqiad.wmnet [08:44:50] !log Rename tables on s3 T425066 [08:44:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:44:54] T425066: Drop AbuseFilter tables from closed wikis - https://phabricator.wikimedia.org/T425066 [08:45:15] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [08:46:40] jiji@cumin1003 renumber-node (PID 2814278) is awaiting input [08:46:55] !log jiji@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-worker1136.eqiad.wmnet with OS trixie [08:47:24] !log jiji@cumin1003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1136 [08:48:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:49:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [08:50:10] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [08:50:26] jiji@cumin1003 renumber-node (PID 2814278) is awaiting input [08:50:46] !log jiji@cumin1003 START - Cookbook sre.dns.netbox [08:51:42] FIRING: [2x] ProbeDown: Service urldownloader1005:8080 has failed probes (http_url_downloader_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Url-downloader - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:51:55] !log jiji@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply [08:52:00] !log jiji@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply [08:52:09] !log jiji@deploy1003 helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply [08:52:20] !log jiji@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply [08:54:20] (03CR) 10Jelto: [V:03+1] "PCC SUCCESS (NOOP 3): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9059/console" [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [08:54:55] PROBLEM - SSH on urldownloader1005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [08:56:04] (03PS1) 10Brouberol: airflow: don't automatically mount worker.extra_files into all kubernetespodoperator tasks [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317880 (https://phabricator.wikimedia.org/T416625) [08:56:06] (03PS1) 10Brouberol: dse-k8s-eqiad/airflow: only define the eventate listeners into analytics instances [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317881 (https://phabricator.wikimedia.org/T416625) [08:56:07] (03CR) 10Btullis: [C:03+2] dse-k8s: Enable the k8s-ingress-dse-postgresql service [puppet] - 10https://gerrit.wikimedia.org/r/1315900 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [08:56:57] jiji@cumin1003 renumber-node (PID 2814278) is awaiting input [08:59:48] (03CR) 10Jelto: [V:03+1 C:03+2] gerrit: Add RegisterNewEmailHtml.soy site override [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [08:59:53] (03CR) 10CI reject: [V:04-1] dse-k8s-eqiad/airflow: only define the eventate listeners into analytics instances [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317881 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [09:00:00] (03CR) 10CI reject: [V:04-1] airflow: don't automatically mount worker.extra_files into all kubernetespodoperator tasks [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317880 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [09:02:54] (03PS1) 10Cathal Mooney: Temp: Increase OSPF cost for Arelion esams<->eqiad transport [homer/public] - 10https://gerrit.wikimedia.org/r/1317883 [09:02:59] PROBLEM - PyBal connections to etcd on lvs1019 is CRITICAL: CRITICAL: 78 connections established with conf1007.eqiad.wmnet:4001 (min=79) https://wikitech.wikimedia.org/wiki/PyBal [09:04:07] !log jiji@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1136 - jiji@cumin1003" [09:04:11] !log jiji@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1136 - jiji@cumin1003" [09:04:11] !log jiji@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:04:11] !log jiji@cumin1003 START - Cookbook sre.dns.wipe-cache wikikube-worker1136.eqiad.wmnet 191.32.64.10.in-addr.arpa 1.9.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [09:04:15] !log jiji@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1136.eqiad.wmnet 191.32.64.10.in-addr.arpa 1.9.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [09:04:16] !log jiji@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1136 [09:05:08] (03PS2) 10Brouberol: airflow: don't automatically mount worker.extra_files into all kubernetespodoperator tasks [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317880 (https://phabricator.wikimedia.org/T416625) [09:05:08] (03PS2) 10Brouberol: dse-k8s-eqiad/airflow: only define the eventate listeners into analytics instances [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317881 (https://phabricator.wikimedia.org/T416625) [09:05:23] (03CR) 10Jelto: [V:03+1 C:03+2] "@hashar@free.fr merged and deployed to all Gerrit hosts!" [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [09:05:55] PROBLEM - PyBal IPVS diff check on lvs1019 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [09:06:57] PROBLEM - PyBal connections to etcd on lvs1020 is CRITICAL: CRITICAL: 114 connections established with conf1007.eqiad.wmnet:4001 (min=115) https://wikitech.wikimedia.org/wiki/PyBal [09:07:03] PROBLEM - PyBal IPVS diff check on lvs1020 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [09:09:16] (03CR) 10Ayounsi: [C:03+1] Temp: Increase OSPF cost for Arelion esams<->eqiad transport [homer/public] - 10https://gerrit.wikimedia.org/r/1317883 (owner: 10Cathal Mooney) [09:10:44] (03CR) 10AikoChou: "nit (wording): This service is a dataset-lookup service — it serves a pre-generated dataset via API for the frontend, with no inference or" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315748 (https://phabricator.wikimedia.org/T430030) (owner: 10Ozge) [09:15:23] (03CR) 10Cathal Mooney: [C:03+2] Temp: Increase OSPF cost for Arelion esams<->eqiad transport [homer/public] - 10https://gerrit.wikimedia.org/r/1317883 (owner: 10Cathal Mooney) [09:16:59] (03Merged) 10jenkins-bot: Temp: Increase OSPF cost for Arelion esams<->eqiad transport [homer/public] - 10https://gerrit.wikimedia.org/r/1317883 (owner: 10Cathal Mooney) [09:17:32] !log jiji@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1136 [09:17:33] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1136 [09:18:22] (03CR) 10AikoChou: [C:03+1] ml-services: Deploy latest edit-check model version on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1314809 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [09:22:35] (03PS2) 10Ozge: ml-services: Bump editing-suggestions artifact to updated MoS copy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315748 (https://phabricator.wikimedia.org/T430030) [09:24:27] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [09:26:14] (03CR) 10Ozge: "updated!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315748 (https://phabricator.wikimedia.org/T430030) (owner: 10Ozge) [09:27:11] (03CR) 10Hnowlan: [C:03+2] rpkivalidator: migrate TCP check to blackbox check [puppet] - 10https://gerrit.wikimedia.org/r/1307182 (https://phabricator.wikimedia.org/T407117) (owner: 10Hnowlan) [09:28:17] (03PS2) 10Hnowlan: rpkivalidator: remove disabled nrpe check [puppet] - 10https://gerrit.wikimedia.org/r/1307183 (https://phabricator.wikimedia.org/T407117) [09:30:12] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1203: Maintenance [09:30:46] (03CR) 10David Caro: paws scan-and-shrink: fix logic error (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1314850 (owner: 10Andrew Bogott) [09:32:46] !log jiji@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1136.eqiad.wmnet with reason: host reimage [09:33:21] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1237.eqiad.wmnet with reason: Maintenance [09:33:28] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1237 (T431660)', diff saved to https://phabricator.wikimedia.org/P95159 and previous config saved to /var/cache/conftool/dbconfig/20260727-093328-cwilliams.json [09:33:53] RECOVERY - SSH on urldownloader1005 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:36:37] PROBLEM - Host wikikube-worker1136 is DOWN: PING CRITICAL - Packet loss = 100% [09:36:42] RESOLVED: [2x] ProbeDown: Service urldownloader1005:8080 has failed probes (http_url_downloader_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Url-downloader - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:37:27] (03PS1) 10Slyngshede: data.yaml offboarding for suecarmol [puppet] - 10https://gerrit.wikimedia.org/r/1318055 [09:38:04] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1136.eqiad.wmnet with reason: host reimage [09:39:04] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1237: Maintenance [09:39:05] PROBLEM - SSH on urldownloader1005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:39:24] !log restart burrow-main-eqiad.service on kafkamon1003 to see if a recurrent kafka error on kafka-main1008 goes away [09:39:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:39:42] FIRING: [2x] ProbeDown: Service urldownloader1005:8080 has failed probes (http_url_downloader_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Url-downloader - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:40:08] (03CR) 10Btullis: [C:03+1] "nice, thanks." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317880 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [09:40:32] (03PS3) 10Ayounsi: Add BGP sharding support [homer/public] - 10https://gerrit.wikimedia.org/r/1311448 (https://phabricator.wikimedia.org/T320264) [09:40:36] (03CR) 10Btullis: [C:03+1] dse-k8s-eqiad/airflow: only define the eventate listeners into analytics instances [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317881 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [09:40:48] (03CR) 10AikoChou: [C:03+1] "LGTM!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315748 (https://phabricator.wikimedia.org/T430030) (owner: 10Ozge) [09:41:39] RECOVERY - Host wikikube-worker1136 is UP: PING OK - Packet loss = 0%, RTA = 0.38 ms [09:43:02] (03Abandoned) 10Slyngshede: Update email for shell user "derick" [puppet] - 10https://gerrit.wikimedia.org/r/1165526 (owner: 10D3r1ck01) [09:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [09:44:27] (03Abandoned) 10Slyngshede: admin: also remove the old ed25519 key for the time being [puppet] - 10https://gerrit.wikimedia.org/r/635497 (owner: 10Giuseppe Lavagetto) [09:45:54] RECOVERY - PyBal IPVS diff check on lvs1019 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [09:46:26] (03CR) 10Ozge: [C:03+2] ml-services: Bump editing-suggestions artifact to updated MoS copy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315748 (https://phabricator.wikimedia.org/T430030) (owner: 10Ozge) [09:46:56] RECOVERY - PyBal connections to etcd on lvs1020 is OK: OK: 115 connections established with conf1007.eqiad.wmnet:4001 (min=115) https://wikitech.wikimedia.org/wiki/PyBal [09:47:06] RECOVERY - PyBal IPVS diff check on lvs1020 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [09:48:00] RECOVERY - PyBal connections to etcd on lvs1019 is OK: OK: 79 connections established with conf1007.eqiad.wmnet:4001 (min=79) https://wikitech.wikimedia.org/wiki/PyBal [09:48:18] (03PS3) 10Slyngshede: P:idp allow selective mfa enablement [puppet] - 10https://gerrit.wikimedia.org/r/1304784 (https://phabricator.wikimedia.org/T277841) [09:48:20] (03PS2) 10Arthur taylor: wikidata-query-gui: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315685 (https://phabricator.wikimedia.org/T433051) [09:48:50] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - k8s-ingress-dse-postgresql_31432: Servers dse-k8s-worker1012.eqiad.wmnet, dse-k8s-worker1014.eqiad.wmnet, dse-k8s-worker1008.eqiad.wmnet, dse-k8s-worker1001.eqiad.wmnet, dse-k8s-worker1028.eqiad.wmnet, dse-k8s-worker1007.eqiad.wmnet, dse-k8s-worker1011.eqiad.wmnet, dse-k8s-worker1020.eqiad.wmnet, dse-k8s-worker1022.eqiad.wmnet, dse-k8s-wdqs1002.eqiad [09:48:50] dse-k8s-wdqs-test1001.eqiad.wmnet, dse-k8s-worker1018.eqiad.wmnet, dse-k8s-worker1004.eqiad.wmnet, dse-k8s-worker1021.eqiad.wmnet, dse-k8s-worker1023.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [09:48:56] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - k8s-ingress-dse-postgresql_31432: Servers dse-k8s-worker1012.eqiad.wmnet, dse-k8s-worker1014.eqiad.wmnet, dse-k8s-worker1008.eqiad.wmnet, dse-k8s-worker1001.eqiad.wmnet, dse-k8s-worker1028.eqiad.wmnet, dse-k8s-worker1007.eqiad.wmnet, dse-k8s-worker1011.eqiad.wmnet, dse-k8s-worker1020.eqiad.wmnet, dse-k8s-worker1022.eqiad.wmnet, dse-k8s-wdqs1002.eqiad [09:48:56] dse-k8s-wdqs-test1001.eqiad.wmnet, dse-k8s-worker1018.eqiad.wmnet, dse-k8s-worker1004.eqiad.wmnet, dse-k8s-worker1021.eqiad.wmnet, dse-k8s-worker1023.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [09:49:13] (03Merged) 10jenkins-bot: ml-services: Bump editing-suggestions artifact to updated MoS copy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315748 (https://phabricator.wikimedia.org/T430030) (owner: 10Ozge) [09:53:30] (03PS1) 10Slyngshede: data.yaml Incorrect expiry date for account jniren-ctr [puppet] - 10https://gerrit.wikimedia.org/r/1318057 [09:55:53] 10SRE-tools, 10Cumin, 06Infrastructure-Foundations, 10Spicerack: Add shell completion to 'cookbook' - https://phabricator.wikimedia.org/T433228 (10fgiunchedi) 03NEW [09:58:25] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1136.eqiad.wmnet with OS trixie [09:59:38] (03PS2) 10Effie Mouzeli: admin_ng: remove api-gateway. #3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315834 (https://phabricator.wikimedia.org/T432460) (owner: 10Blake) [10:00:04] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T1000) [10:01:11] (03CR) 10Elukey: [C:03+2] profile::spicerack::test_cookbook: allow to set PYTHONPATH (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1315894 (owner: 10Elukey) [10:04:45] (03CR) 10Brouberol: [C:03+2] airflow: don't automatically mount worker.extra_files into all kubernetespodoperator tasks [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317880 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [10:04:50] (03CR) 10Brouberol: [C:03+2] dse-k8s-eqiad/airflow: only define the eventate listeners into analytics instances [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317881 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [10:04:56] !log restart burrow main-eqiad on kafkamon2003 to clear some errors on kafka-main1008 [10:04:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:05:59] (03PS2) 10Blake: api-gateway: absent kubernetes service. #1 [puppet] - 10https://gerrit.wikimedia.org/r/1315800 (https://phabricator.wikimedia.org/T432460) [10:06:04] (03PS1) 10Blake: api-gateway: final cleanup. #2 [puppet] - 10https://gerrit.wikimedia.org/r/1315807 (https://phabricator.wikimedia.org/T432460) [10:06:07] RECOVERY - SSH on urldownloader1005 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [10:07:27] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [10:08:22] (03Merged) 10jenkins-bot: airflow: don't automatically mount worker.extra_files into all kubernetespodoperator tasks [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317880 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [10:08:26] (03Merged) 10jenkins-bot: dse-k8s-eqiad/airflow: only define the eventate listeners into analytics instances [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317881 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [10:09:09] PROBLEM - SSH on urldownloader1005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [10:09:17] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [10:11:23] (03PS1) 10Brouberol: airflow/dev: remove refinery related extra config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318060 (https://phabricator.wikimedia.org/T416625) [10:14:00] (03PS3) 10Arthur taylor: wikidata-query-gui: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315685 (https://phabricator.wikimedia.org/T433051) [10:18:58] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:19:27] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:21:16] (03CR) 10Hnowlan: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9062/co" [puppet] - 10https://gerrit.wikimedia.org/r/1311028 (https://phabricator.wikimedia.org/T432226) (owner: 10Hnowlan) [10:21:54] (03CR) 10Hnowlan: [V:03+1 C:03+2] profile::alertmanager: extend default login session length [puppet] - 10https://gerrit.wikimedia.org/r/1311028 (https://phabricator.wikimedia.org/T432226) (owner: 10Hnowlan) [10:22:36] (03Abandoned) 10Zaidusyy: Add clickable link to email verification message [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305218 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [10:23:58] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:24:30] 10SRE-tools, 10Cumin, 06Infrastructure-Foundations, 10Spicerack: Add shell completion to 'cookbook' - https://phabricator.wikimedia.org/T433228#12158108 (10fgiunchedi) →14Duplicate dup:03T367230 [10:24:31] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Tab completion for cookbook names - https://phabricator.wikimedia.org/T367230#12158110 (10fgiunchedi) [10:24:47] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1237: Maintenance [10:27:01] RECOVERY - SSH on urldownloader1005 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [10:27:07] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [10:27:39] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [10:28:50] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply [10:29:23] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply [10:29:40] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [10:30:09] PROBLEM - SSH on urldownloader1005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [10:30:18] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [10:31:41] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [10:31:56] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1264.eqiad.wmnet with reason: Maintenance [10:32:04] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1264 (T431660)', diff saved to https://phabricator.wikimedia.org/P95164 and previous config saved to /var/cache/conftool/dbconfig/20260727-103204-cwilliams.json [10:32:15] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [10:32:23] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply [10:32:56] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply [10:32:59] RECOVERY - SSH on urldownloader1005 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [10:34:07] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply [10:34:39] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply [10:34:42] RESOLVED: [2x] ProbeDown: Service urldownloader1005:8080 has failed probes (http_url_downloader_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Url-downloader - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:34:46] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply [10:35:12] FIRING: ProbeDown: Service urldownloader1005:8080 has failed probes (http_url_downloader_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Url-downloader - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:35:16] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply [10:35:23] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply [10:35:58] (03PS1) 10Btullis: admin_ng: allow traffic to the ingressgateway PostgreSQL port on dse-k8s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318075 (https://phabricator.wikimedia.org/T432104) [10:36:03] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply [10:36:09] PROBLEM - SSH on urldownloader1005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [10:36:12] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply [10:36:42] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply [10:36:54] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply [10:37:24] (03CR) 10Blake: [C:03+2] api-gateway: absent kubernetes service. #1 [puppet] - 10https://gerrit.wikimedia.org/r/1315800 (https://phabricator.wikimedia.org/T432460) (owner: 10Blake) [10:37:31] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply [10:37:39] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply [10:38:13] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply [10:38:21] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply [10:38:52] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply [10:38:59] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply [10:39:07] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1264: Maintenance [10:39:31] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply [10:40:03] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] wikidata-query-gui: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315685 (https://phabricator.wikimedia.org/T433051) (owner: 10Arthur taylor) [10:40:12] FIRING: [2x] ProbeDown: Service urldownloader1005:8080 has failed probes (http_url_downloader_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Url-downloader - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:40:47] (03CR) 10Brouberol: [C:03+1] "Nice" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318075 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [10:43:10] jiji@cumin1003 renumber-node (PID 2814278) is awaiting input [10:43:12] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [10:47:08] (03CR) 10Arthur taylor: [C:03+2] wikidata-query-gui: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315685 (https://phabricator.wikimedia.org/T433051) (owner: 10Arthur taylor) [10:47:42] will deploy a change to the wikidata-query-gui shortly [10:49:49] (03Merged) 10jenkins-bot: wikidata-query-gui: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315685 (https://phabricator.wikimedia.org/T433051) (owner: 10Arthur taylor) [10:50:27] !log arthurtaylor@deploy1003 helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply [10:50:46] (03CR) 10Majavah: [C:03+1] Revert "hieradata: Add Wikimania address to extra_trust" [puppet] - 10https://gerrit.wikimedia.org/r/1317847 (https://phabricator.wikimedia.org/T432377) (owner: 10Jelto) [10:51:31] (03PS2) 10Blake: api-gateway: final cleanup. #2 [puppet] - 10https://gerrit.wikimedia.org/r/1315807 (https://phabricator.wikimedia.org/T432460) [10:51:53] (03CR) 10Btullis: [C:03+2] admin_ng: allow traffic to the ingressgateway PostgreSQL port on dse-k8s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318075 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [10:53:10] (03CR) 10Blake: [C:03+2] api-gateway: final cleanup. #2 [puppet] - 10https://gerrit.wikimedia.org/r/1315807 (https://phabricator.wikimedia.org/T432460) (owner: 10Blake) [10:54:21] (03CR) 10Blake: [C:03+2] admin_ng: remove api-gateway. #3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315834 (https://phabricator.wikimedia.org/T432460) (owner: 10Blake) [10:55:53] (03CR) 10Majavah: [C:03+2] Revert "hieradata: Add Wikimania address to extra_trust" [puppet] - 10https://gerrit.wikimedia.org/r/1317847 (https://phabricator.wikimedia.org/T432377) (owner: 10Jelto) [10:57:00] (03CR) 10Jforrester: "@dreamyjazzwikipedia@gmail.com Do you want to deploy this at a time of your choosing, or should I? I imagine you'd be best-placed to tripl" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316832 (owner: 10Jforrester) [11:00:34] (03Merged) 10jenkins-bot: admin_ng: allow traffic to the ingressgateway PostgreSQL port on dse-k8s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318075 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [11:02:57] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [11:03:14] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [11:03:31] (03Merged) 10jenkins-bot: admin_ng: remove api-gateway. #3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315834 (https://phabricator.wikimedia.org/T432460) (owner: 10Blake) [11:05:25] !log blake@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [11:05:54] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [11:06:58] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [11:08:28] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [11:08:34] !log blake@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [11:09:29] !log btullis@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [11:09:43] !log btullis@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [11:09:58] !log blake@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [11:11:35] !log blake@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [11:12:29] !log blake@deploy1003 helmfile [codfw] START helmfile.d/admin 'apply'. [11:12:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [11:13:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:15:19] (03CR) 10Federico Ceratto: "I updated the updates sent to Phab in https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1311405/13..14 , do they make sense to you? " [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) (owner: 10Federico Ceratto) [11:16:12] (03PS1) 10Slyngshede: Account blocking: Only unset email if requested [software/bitu] - 10https://gerrit.wikimedia.org/r/1318083 (https://phabricator.wikimedia.org/T431114) [11:16:18] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [11:17:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [11:18:19] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [11:18:50] !log blake@deploy1003 helmfile [eqiad] START helmfile.d/admin 'apply'. [11:18:56] (03CR) 10CI reject: [V:04-1] Account blocking: Only unset email if requested [software/bitu] - 10https://gerrit.wikimedia.org/r/1318083 (https://phabricator.wikimedia.org/T431114) (owner: 10Slyngshede) [11:18:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:20:02] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [11:20:28] !log blake@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [11:21:27] (03CR) 10Marostegui: "Yes the logic was fine, it was just the logged messages what needed fixing." [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) (owner: 10Federico Ceratto) [11:22:09] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool es1050: testing [11:22:18] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1050: testing [11:23:03] !log marostegui@cumin1003 dbctl commit (dc=all): 'Repool es1050', diff saved to https://phabricator.wikimedia.org/P95169 and previous config saved to /var/cache/conftool/dbconfig/20260727-112302-marostegui.json [11:23:27] !log marostegui@cumin1003 dbctl commit (dc=all): 'Repool es1050 as master', diff saved to https://phabricator.wikimedia.org/P95170 and previous config saved to /var/cache/conftool/dbconfig/20260727-112326-marostegui.json [11:24:01] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool es1038: testing [11:24:45] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1264: Maintenance [11:26:01] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1038: testing [11:26:40] !log ayounsi@cumin1003 START - Cookbook sre.dns.admin DNS admin: depool ulsfo [reason: router upgrade, T431752] [11:26:44] T431752: cr4-ulsfo: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431752 [11:26:55] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool es1038: es1038 repool [11:26:57] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool ulsfo [reason: router upgrade, T431752] [11:27:16] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool es1035: testing [11:27:25] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1035: testing [11:28:06] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool es1035: testing [11:28:21] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1035: testing [11:28:51] (03CR) 10Marostegui: [C:03+1] "https://phabricator.wikimedia.org/T430769#12158270" [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) (owner: 10Federico Ceratto) [11:29:16] !log arthurtaylor@deploy1003 helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply [11:29:20] (03CR) 10Marostegui: "Actually:" [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) (owner: 10Federico Ceratto) [11:29:40] (03PS2) 10Urbanecm: [Growth] Deploy automated mentor list cleaner to all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311832 (https://phabricator.wikimedia.org/T431804) [11:29:42] !log arthurtaylor@deploy1003 helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply [11:29:46] jouncebot: nowandnext [11:29:46] No deployments scheduled for the next 1 hour(s) and 30 minute(s) [11:29:46] In 1 hour(s) and 30 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T1300) [11:29:50] !log start draining cr4-ulsfo - T431752 [11:29:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:29:53] (03CR) 10Urbanecm: [C:03+2] [Growth] Deploy automated mentor list cleaner to all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311832 (https://phabricator.wikimedia.org/T431804) (owner: 10Urbanecm) [11:30:38] (03CR) 10Marostegui: [C:03+1] "nevermind, we do: https://phabricator.wikimedia.org/T430769#12158267" [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) (owner: 10Federico Ceratto) [11:35:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr2-eqsin and cr4-ulsfo (198.35.26.129) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:35:40] !log arthurtaylor@deploy1003 helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply [11:36:35] !log arthurtaylor@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply [11:36:43] !log arthurtaylor@deploy1003 helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply [11:37:12] !log arthurtaylor@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply [11:37:58] (03CR) 10Majavah: [C:03+2] P:toolforge: k8s: haproxy: Add sticky silent-drop for very high rate IPs [puppet] - 10https://gerrit.wikimedia.org/r/1314938 (owner: 10Majavah) [11:38:06] (03Merged) 10jenkins-bot: [Growth] Deploy automated mentor list cleaner to all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311832 (https://phabricator.wikimedia.org/T431804) (owner: 10Urbanecm) [11:39:02] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1311832|[Growth] Deploy automated mentor list cleaner to all wikis (T431804)]] [11:39:06] T431804: Deploy automated mentor list cleanup to all Wikimedia wikis - https://phabricator.wikimedia.org/T431804 [11:42:46] !log urbanecm@deploy1003 urbanecm: Backport for [[gerrit:1311832|[Growth] Deploy automated mentor list cleaner to all wikis (T431804)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:43:31] !log urbanecm@deploy1003 urbanecm: Continuing with deployment [11:48:37] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance [11:48:45] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1187 (T431660)', diff saved to https://phabricator.wikimedia.org/P95178 and previous config saved to /var/cache/conftool/dbconfig/20260727-114844-cwilliams.json [11:50:09] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1311832|[Growth] Deploy automated mentor list cleaner to all wikis (T431804)]] (duration: 11m 07s) [11:50:13] T431804: Deploy automated mentor list cleanup to all Wikimedia wikis - https://phabricator.wikimedia.org/T431804 [11:50:39] !log ayounsi@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr4-ulsfo,cr4-ulsfo IPv6,cr4-ulsfo.mgmt with reason: router upgrade [11:50:50] 06SRE, 06Infrastructure-Foundations, 10netops: cr4-ulsfo: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431752#12158344 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=fe3278b9-2acb-46d1-af9d-66d64acae08f) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their servi... [11:51:40] (03PS1) 10Cparle: Add key to X-Analaytics if a thumbnail was generated during the request [puppet] - 10https://gerrit.wikimedia.org/r/1318088 (https://phabricator.wikimedia.org/T433075) [11:53:32] !log urbanecm@deploy1003 mwscript-k8s job started: foreachwikiindblist growthexperiments GrowthExperiments:cleanMentorList # T431804 [11:55:14] (03CR) 10Hnowlan: [C:03+2] rpkivalidator: remove disabled nrpe check [puppet] - 10https://gerrit.wikimedia.org/r/1307183 (https://phabricator.wikimedia.org/T407117) (owner: 10Hnowlan) [11:55:39] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1187: Maintenance [11:57:31] (03PS1) 10Brouberol: airflow-analytic-test: remove deprecated configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318091 (https://phabricator.wikimedia.org/T416625) [11:58:42] jouncebot: nowandnext [11:58:42] No deployments scheduled for the next 1 hour(s) and 1 minute(s) [11:58:42] In 1 hour(s) and 1 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T1300) [12:06:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:07:29] (03PS2) 10Kamila Součková: aptrepo: add php85 component [puppet] - 10https://gerrit.wikimedia.org/r/1315940 (https://phabricator.wikimedia.org/T432983) [12:07:29] (03PS3) 10Kamila Součková: package_builder: add pbuilder hook for component/php85 [puppet] - 10https://gerrit.wikimedia.org/r/1315948 (https://phabricator.wikimedia.org/T432983) [12:08:49] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 38 hosts [12:09:29] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team (Q1 FY2026-27): eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12158427 (10isarantopoulos) [12:09:46] jiji@cumin1003 renumber-node (PID 2814278) is awaiting input [12:10:56] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 38 hosts [12:11:44] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [12:12:24] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1038: es1038 repool [12:12:54] (03PS2) 10Brouberol: airflow-analytic-test: remove deprecated configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318091 (https://phabricator.wikimedia.org/T416625) [12:12:58] jouncebot: nowandnext [12:12:58] No deployments scheduled for the next 0 hour(s) and 47 minute(s) [12:12:58] In 0 hour(s) and 47 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T1300) [12:13:02] (03PS1) 10Dreamy Jazz: Enable CheckUser SI special page on dewiki and ukwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318098 (https://phabricator.wikimedia.org/T432835) [12:13:51] !log rebooting cr4-ulsfo for upgrade - T431752 [12:13:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:13:55] T431752: cr4-ulsfo: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431752 [12:14:04] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318098 (https://phabricator.wikimedia.org/T432835) (owner: 10Dreamy Jazz) [12:15:40] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 10/11 UP : OSPFv3: 10/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:16:31] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [12:16:40] (03CR) 10Gkyziridis: [C:03+2] ml-services: Deploy latest edit-check model version on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1314809 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [12:17:20] PROBLEM - OSPF status on cr2-eqdfw is CRITICAL: OSPFv2: 6/7 UP : OSPFv3: 6/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:17:20] PROBLEM - OSPF status on cr3-ulsfo is CRITICAL: OSPFv2: 2/4 UP : OSPFv3: 2/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:17:20] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 2 hosts [12:17:22] PROBLEM - OSPF status on cr2-eqsin is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:17:28] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 2 hosts [12:17:51] FIRING: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-22-ulsfo:ethernet-1/56 (Core: cr4-ulsfo:et-0/0/1 {#G24090478750000399}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [12:18:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:18:58] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [12:19:10] FIRING: [2x] BFDdown: BFD session down between cr3-ulsfo and 198.35.26.129 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr3-ulsfo:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:19:47] (03Merged) 10jenkins-bot: ml-services: Deploy latest edit-check model version on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1314809 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [12:20:02] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [12:20:16] (03Merged) 10jenkins-bot: Enable CheckUser SI special page on dewiki and ukwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318098 (https://phabricator.wikimedia.org/T432835) (owner: 10Dreamy Jazz) [12:20:27] !log dreamyjazz@deploy1003 Started scap sync-world: Backport for [[gerrit:1318098|Enable CheckUser SI special page on dewiki and ukwiki (T432835 T433226)]] [12:20:33] T432835: Enable suggested investigations on dewiki - https://phabricator.wikimedia.org/T432835 [12:20:33] T433226: Enable suggested investigations on ukwiki - https://phabricator.wikimedia.org/T433226 [12:20:39] FIRING: [5x] CoreBGPDown: Core BGP session down between cr1-codfw and cr4-ulsfo (198.35.26.129) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [12:21:09] (03CR) 10Slyngshede: [C:03+1] Remove the acmechief config for mirrors.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1307378 (https://phabricator.wikimedia.org/T416707) (owner: 10Muehlenhoff) [12:22:11] !log dreamyjazz@deploy1003 dreamyjazz: Backport for [[gerrit:1318098|Enable CheckUser SI special page on dewiki and ukwiki (T432835 T433226)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:23:15] (03CR) 10Kamila Součková: aptrepo: add php85 component (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1315940 (https://phabricator.wikimedia.org/T432983) (owner: 10Kamila Součková) [12:23:54] (03CR) 10Aqu: [C:03+1] airflow/dev: remove refinery related extra config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318060 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [12:24:20] RECOVERY - OSPF status on cr2-eqdfw is OK: OSPFv2: 7/7 UP : OSPFv3: 7/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:24:20] interfaces are back up [12:24:22] RECOVERY - OSPF status on cr2-eqsin is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:24:40] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 11/11 UP : OSPFv3: 11/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:25:24] RECOVERY - OSPF status on cr3-ulsfo is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:25:39] FIRING: [5x] CoreBGPDown: Core BGP session down between cr1-codfw and cr4-ulsfo (198.35.26.129) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [12:25:46] (03CR) 10Aqu: [C:03+1] airflow-analytic-test: remove deprecated configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318091 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [12:25:51] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1318103 (owner: 10L10n-bot) [12:25:51] !log dreamyjazz@deploy1003 dreamyjazz: Continuing with deployment [12:27:19] (03PS1) 10Ssingh: wikimedia.org: move urldownloader in eqiad to 1006 [dns] - 10https://gerrit.wikimedia.org/r/1318106 [12:27:51] RESOLVED: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-22-ulsfo:ethernet-1/56 (Core: cr4-ulsfo:et-0/0/1 {#G24090478750000399}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [12:28:40] (03CR) 10Ssingh: [C:03+2] wikimedia.org: move urldownloader in eqiad to 1006 [dns] - 10https://gerrit.wikimedia.org/r/1318106 (owner: 10Ssingh) [12:28:43] !log sukhe@dns1004 START - running authdns-update [12:28:58] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [12:29:10] RESOLVED: [2x] BFDdown: BFD session down between cr3-ulsfo and 198.35.26.129 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr3-ulsfo:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:29:28] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [12:30:00] !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for [[gerrit:1318098|Enable CheckUser SI special page on dewiki and ukwiki (T432835 T433226)]] (duration: 09m 32s) [12:30:05] T432835: Enable suggested investigations on dewiki - https://phabricator.wikimedia.org/T432835 [12:30:06] T433226: Enable suggested investigations on ukwiki - https://phabricator.wikimedia.org/T433226 [12:30:39] FIRING: [5x] CoreBGPDown: Core BGP session down between cr1-codfw and cr4-ulsfo (198.35.26.129) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [12:30:46] !log sukhe@dns1004 END - running authdns-update [12:30:49] !log sukhe@cumin1003 START - Cookbook sre.dns.wipe-cache url-downloader.eqiad.wikimedia.org on all recursors [12:30:53] !log sukhe@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) url-downloader.eqiad.wikimedia.org on all recursors [12:32:18] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [12:32:30] (03CR) 10Federico Ceratto: "Done" [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) (owner: 10Federico Ceratto) [12:32:37] (03CR) 10Federico Ceratto: [C:03+2] sre.mysql.pool: support esX sections [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) (owner: 10Federico Ceratto) [12:33:26] (03PS4) 10Ayounsi: Add BGP sharding support [homer/public] - 10https://gerrit.wikimedia.org/r/1311448 (https://phabricator.wikimedia.org/T320264) [12:33:58] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:34:12] (03Abandoned) 10Filippo Giunchedi: dumps: do not remove legacy mountpoint directory on cloud vps [puppet] - 10https://gerrit.wikimedia.org/r/1310975 (https://phabricator.wikimedia.org/T411248) (owner: 10Filippo Giunchedi) [12:36:29] (03CR) 10Ayounsi: [C:03+2] Add BGP sharding support [homer/public] - 10https://gerrit.wikimedia.org/r/1311448 (https://phabricator.wikimedia.org/T320264) (owner: 10Ayounsi) [12:36:31] FIRING: [2x] RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [12:37:01] 06SRE, 06Infrastructure-Foundations, 10netops: cr4-ulsfo: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431752#12158517 (10ayounsi) 05Open→03Resolved a:03ayounsi upgraded! [12:37:27] (03Merged) 10jenkins-bot: sre.mysql.pool: support esX sections [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) (owner: 10Federico Ceratto) [12:37:41] (03PS1) 10Brouberol: global_config: register the webproxy external service [puppet] - 10https://gerrit.wikimedia.org/r/1318115 (https://phabricator.wikimedia.org/T429338) [12:38:24] (03CR) 10CI reject: [V:04-1] global_config: register the webproxy external service [puppet] - 10https://gerrit.wikimedia.org/r/1318115 (https://phabricator.wikimedia.org/T429338) (owner: 10Brouberol) [12:38:28] (03Merged) 10jenkins-bot: Add BGP sharding support [homer/public] - 10https://gerrit.wikimedia.org/r/1311448 (https://phabricator.wikimedia.org/T320264) (owner: 10Ayounsi) [12:39:27] (03PS2) 10Brouberol: global_config: register the webproxy external service [puppet] - 10https://gerrit.wikimedia.org/r/1318115 (https://phabricator.wikimedia.org/T429338) [12:40:34] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [12:41:31] RESOLVED: [2x] RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [12:43:37] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1187: Maintenance [12:43:57] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance [12:43:58] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:44:05] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1201 (T431660)', diff saved to https://phabricator.wikimedia.org/P95186 and previous config saved to /var/cache/conftool/dbconfig/20260727-124404-cwilliams.json [12:44:25] (03CR) 10Brouberol: [C:03+2] airflow/dev: remove refinery related extra config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318060 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [12:44:29] (03CR) 10Brouberol: [C:03+2] airflow-analytic-test: remove deprecated configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318091 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [12:46:10] FIRING: [10x] BFDdown: BFD session down between cr1-codfw and 198.35.26.202 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:46:41] (03Merged) 10jenkins-bot: airflow/dev: remove refinery related extra config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318060 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [12:47:17] (03Merged) 10jenkins-bot: airflow-analytic-test: remove deprecated configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318091 (https://phabricator.wikimedia.org/T416625) (owner: 10Brouberol) [12:48:08] !log gkyziridis@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . [12:50:25] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [12:51:07] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1201: Maintenance [12:51:10] RESOLVED: [10x] BFDdown: BFD session down between cr1-codfw and 198.35.26.202 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:58:44] (03PS1) 10Gkyziridis: ml-services: Deploy latest edit-check model version on production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318126 (https://phabricator.wikimedia.org/T429675) [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: #bothumor My software never has bugs. It just develops random features. Rise for UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T1300). [13:00:05] Silvan_WMDE: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:22] o/ [13:01:14] I can deploy! [13:01:26] Hi Lucas, I'm here o/ [13:02:19] Silvan_WMDE: should your changes be deployed together or separately [13:02:20] ? [13:02:34] oh, also, do you want to do the deployment yourself? ^^ [13:02:57] the first is a no-op which only removes an unused config parameter, so they can be deployed together [13:02:59] !log repool cr4-ulsfo - T431752 [13:03:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:03:03] T431752: cr4-ulsfo: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431752 [13:03:32] no thanks - I'm excited enough already to be sitting here watching you :-) [13:03:45] ok ^^ [13:04:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315843 (https://phabricator.wikimedia.org/T430943) (owner: 10Silvan Heintze) [13:04:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315842 (https://phabricator.wikimedia.org/T302959) (owner: 10Silvan Heintze) [13:04:19] (https://spiderpig.wikimedia.org/jobs/2531 if you want to watch) [13:05:50] random question if someone else is here – how can I tell if someone else has Spiderpig-access permission, or one of the other permissions at https://idm.wikimedia.org/permissions/ ? (given that AFAICT this is neither in LDAP nor in puppet.git) [13:06:15] (03Merged) 10jenkins-bot: Remove obsolete wmgWikibaseRestApiEnabled setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315842 (https://phabricator.wikimedia.org/T302959) (owner: 10Silvan Heintze) [13:06:21] (03Merged) 10jenkins-bot: Enable WikibaseLexeme REST API on beta wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315843 (https://phabricator.wikimedia.org/T430943) (owner: 10Silvan Heintze) [13:06:34] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1315843|Enable WikibaseLexeme REST API on beta wikidata (T430943)]], [[gerrit:1315842|Remove obsolete wmgWikibaseRestApiEnabled setting (T302959 T324999 T383774)]] [13:06:44] T430943: 🦦 Enable lexeme dev routes on beta wikidata - https://phabricator.wikimedia.org/T430943 [13:06:45] T302959: Create a test/validation system for the Wikibase REST API - https://phabricator.wikimedia.org/T302959 [13:06:45] T324999: configure Wikibase REST API on Wikidata - https://phabricator.wikimedia.org/T324999 [13:06:45] T383774: Remove v0 routes and the corresponding test - https://phabricator.wikimedia.org/T383774 [13:07:02] !log ayounsi@cumin1003 START - Cookbook sre.dns.admin DNS admin: pool ulsfo [reason: router upgrade finished, T431752] [13:07:05] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool ulsfo [reason: router upgrade finished, T431752] [13:08:21] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, sihe: Backport for [[gerrit:1315843|Enable WikibaseLexeme REST API on beta wikidata (T430943)]], [[gerrit:1315842|Remove obsolete wmgWikibaseRestApiEnabled setting (T302959 T324999 T383774)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:08:41] Silvan_WMDE: please test ^^ [13:08:57] (03CR) 10Clément Goubert: [C:03+1] rest-gateway: Move api-gateway to rest-gateway. [alerts] - 10https://gerrit.wikimedia.org/r/1313887 (owner: 10Blake) [13:10:38] Lucas_WMDE: the GET lexeme endpoint doesn't seem to work on beta: https://www.wikidata.beta.wmcloud.org/w/rest.php/wikibase/v0/entities/lexemes/L1 [13:10:39] RESOLVED: CoreBGPDown: Core BGP session down between cr2-eqsin and cr4-ulsfo (198.35.26.129) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqsin&var-device=cr2-eqsin:9804&var-bgp_group=Confed_ulsfo&var-bgp_neighbor=cr4-ulsfo - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:11:06] oh, right, it’s beta only [13:11:07] (03PS1) 10Majavah: add fake opensearch passwords [labs/private] - 10https://gerrit.wikimedia.org/r/1318131 [13:11:18] you can’t test beta changes with WikimediaDebug, they’ll be deployed within 10 minutes after the change is merged [13:11:29] Silvan_WMDE: please just test that nothing is broken on Wikidata or Test Wikidata, then :) [13:13:27] (03CR) 10Blake: [C:03+2] rest-gateway: Move api-gateway to rest-gateway. [alerts] - 10https://gerrit.wikimedia.org/r/1313887 (owner: 10Blake) [13:13:31] Lucas_WMDE: ok, wikidata and test wikidata do look ok: nothing changed there [13:13:36] (03CR) 10Majavah: [V:03+2 C:03+2] add fake opensearch passwords [labs/private] - 10https://gerrit.wikimedia.org/r/1318131 (owner: 10Majavah) [13:14:15] yay, thanks [13:14:21] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, sihe: Continuing with deployment [13:14:42] Lukas_WMDE: now Lexemes actually work on beta - thanks \o/ [13:16:30] (03Merged) 10jenkins-bot: rest-gateway: Move api-gateway to rest-gateway. [alerts] - 10https://gerrit.wikimedia.org/r/1313887 (owner: 10Blake) [13:16:42] \o/ [13:18:18] Lucas_WMDE: thank you - bye :wave: [13:18:24] (03PS1) 10Majavah: P:toolforge::opensearch: Add HAProxy for client access [puppet] - 10https://gerrit.wikimedia.org/r/1318136 (https://phabricator.wikimedia.org/T401818) [13:18:31] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1315843|Enable WikibaseLexeme REST API on beta wikidata (T430943)]], [[gerrit:1315842|Remove obsolete wmgWikibaseRestApiEnabled setting (T302959 T324999 T383774)]] (duration: 11m 57s) [13:18:48] T430943: 🦦 Enable lexeme dev routes on beta wikidata - https://phabricator.wikimedia.org/T430943 [13:18:48] T302959: Create a test/validation system for the Wikibase REST API - https://phabricator.wikimedia.org/T302959 [13:18:49] T324999: configure Wikibase REST API on Wikidata - https://phabricator.wikimedia.org/T324999 [13:18:49] T383774: Remove v0 routes and the corresponding test - https://phabricator.wikimedia.org/T383774 [13:19:07] nothing else to deploy? [13:19:10] * Lucas_WMDE waits a few minutes [13:20:09] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9070/co" [puppet] - 10https://gerrit.wikimedia.org/r/1318136 (https://phabricator.wikimedia.org/T401818) (owner: 10Majavah) [13:20:52] (03PS1) 10Filippo Giunchedi: wmcs: add clouddumps compat symlinks [puppet] - 10https://gerrit.wikimedia.org/r/1318139 (https://phabricator.wikimedia.org/T411248) [13:21:46] (03PS1) 10Majavah: P:toolforge::prometheus: Add opensearch metrics [puppet] - 10https://gerrit.wikimedia.org/r/1318141 (https://phabricator.wikimedia.org/T401818) [13:27:34] !log UTC afternoon backport+config window doen [13:27:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:27:38] *done, grmbl [13:28:14] (wow, SAL search suggests I’m the first person to make this typo ^^) [13:30:34] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12158671 (10ssingh) Thanks @BCornwall. @RobH: Is there a preference for the day of the week from your end? I am assuming we can't do tomorrow Sin... [13:31:01] (03PS1) 10Slyngshede: data.yaml convert hahmed to staff [puppet] - 10https://gerrit.wikimedia.org/r/1318146 [13:32:28] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [13:33:04] (03Abandoned) 10Federico Ceratto: setup.py: limit ruff version [cookbooks] - 10https://gerrit.wikimedia.org/r/1315858 (owner: 10Federico Ceratto) [13:35:11] (03CR) 10Filippo Giunchedi: [C:03+1] P:toolforge::opensearch: Add HAProxy for client access [puppet] - 10https://gerrit.wikimedia.org/r/1318136 (https://phabricator.wikimedia.org/T401818) (owner: 10Majavah) [13:36:08] (03CR) 10Majavah: [V:03+1 C:03+2] P:toolforge::opensearch: Add HAProxy for client access [puppet] - 10https://gerrit.wikimedia.org/r/1318136 (https://phabricator.wikimedia.org/T401818) (owner: 10Majavah) [13:36:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [13:37:18] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [13:38:08] 10SRE-swift-storage, 06MediaWiki-Media-Platform-Team, 10MediaWiki-Uploading, 05MW-1.47-notes (1.47.0-wmf.11; 2026-07-14), 07Wikimedia-production-error: MediaWiki\Upload\Exception\UploadChunkFileException: Error storing file in '{chunkPath}': backend-fai... - https://phabricator.wikimedia.org/T430986#12158716 [13:39:48] (03CR) 10Bking: [C:03+2] prometheus: start exporter with Blazegraph [puppet] - 10https://gerrit.wikimedia.org/r/1315664 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [13:41:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [13:42:11] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1201: Maintenance [13:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:43:20] (03CR) 10Jelto: [C:03+2] create a namespace for etherpad on k8s-aux cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/1314806 (owner: 10Jelto) [13:43:53] (03PS1) 10Sbisson: Article Guidance: clean up old wikidata config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318149 (https://phabricator.wikimedia.org/T421250) [13:46:49] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 34 hosts [13:49:55] (03CR) 10Majavah: [C:03+2] P:toolforge::prometheus: Add opensearch metrics [puppet] - 10https://gerrit.wikimedia.org/r/1318141 (https://phabricator.wikimedia.org/T401818) (owner: 10Majavah) [13:50:12] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 34 hosts [13:50:53] (03CR) 10JHathaway: [C:03+1] data.yaml convert hahmed to staff [puppet] - 10https://gerrit.wikimedia.org/r/1318146 (owner: 10Slyngshede) [13:52:24] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs1020.eqiad.wmnet with OS bookworm [13:52:33] (03Merged) 10jenkins-bot: create a namespace for etherpad on k8s-aux cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/1314806 (owner: 10Jelto) [13:52:53] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance [13:52:59] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs2010.codfw.wmnet with OS bookworm [13:53:01] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2175 (T431660)', diff saved to https://phabricator.wikimedia.org/P95191 and previous config saved to /var/cache/conftool/dbconfig/20260727-135300-cwilliams.json [13:54:06] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2156.codfw.wmnet with reason: Maintenance [13:54:14] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2156 (T431660)', diff saved to https://phabricator.wikimedia.org/P95192 and previous config saved to /var/cache/conftool/dbconfig/20260727-135413-cwilliams.json [13:55:24] (03CR) 10Slyngshede: [C:03+2] data.yaml convert hahmed to staff [puppet] - 10https://gerrit.wikimedia.org/r/1318146 (owner: 10Slyngshede) [13:55:45] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs1020 [13:55:45] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1020 [13:56:06] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance [13:56:14] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2155 (T431660)', diff saved to https://phabricator.wikimedia.org/P95193 and previous config saved to /var/cache/conftool/dbconfig/20260727-135613-cwilliams.json [13:56:29] (03PS1) 10Jgiannelos: kartotherian: Pin staging to latest image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318152 [13:57:28] (03CR) 10Majavah: [C:03+1] Puppet 8: Replace legacy facts in profile toolforge [puppet] - 10https://gerrit.wikimedia.org/r/1315007 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [13:57:50] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs2010 [13:58:21] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [13:58:24] (03PS2) 10Jgiannelos: kartotherian: Pin staging to latest image for testing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318152 [13:58:52] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [13:58:56] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12158773 (10RobH) Right now it is 13:55 GMT = 21:55 Singapore, so if we schedule now it won't be confirmed until Tuesday, for possibly a Wednesda... [13:59:15] (03CR) 10Cathal Mooney: [C:03+1] Don't fetch the cable label [software/homer] - 10https://gerrit.wikimedia.org/r/1313177 (https://phabricator.wikimedia.org/T432689) (owner: 10Ayounsi) [13:59:22] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2175: Maintenance [13:59:36] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance [13:59:44] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2157 (T431660)', diff saved to https://phabricator.wikimedia.org/P95194 and previous config saved to /var/cache/conftool/dbconfig/20260727-135943-cwilliams.json [14:00:12] FIRING: [5x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:00:34] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2156: Maintenance [14:00:53] bking@cumin2003 reimage (PID 849144) is awaiting input [14:01:10] !log jiji@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1136.eqiad.wmnet [14:01:11] !log jiji@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1136.eqiad.wmnet [14:01:13] !log jiji@cumin1003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1136.eqiad.wmnet [14:01:46] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 11 hosts [14:02:51] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2155: Maintenance [14:03:04] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 11 hosts [14:04:57] !log root@cumin1003 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2175: Maintenance [14:05:26] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2157: Maintenance [14:07:21] (03CR) 10Eevans: [C:03+2] restbase: Canary Cassandra 5.0.8 to restbase[1031,2024] [puppet] - 10https://gerrit.wikimedia.org/r/1315182 (https://phabricator.wikimedia.org/T433028) (owner: 10Eevans) [14:07:54] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12158818 (10ssingh) >>! In T433097#12158773, @RobH wrote: > > Proposed window: 2026-07-30 @ 01:00 GMT (9AM Singapore). Depool to take place a fe... [14:08:28] !log root@cumin1003 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2155: Maintenance [14:13:19] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [14:13:28] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [14:13:59] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [14:14:10] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [14:14:24] !log jelto@deploy1003 helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. [14:14:40] !log jelto@deploy1003 helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. [14:14:58] !log jelto@deploy1003 helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. [14:15:06] !log jelto@deploy1003 helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. [14:16:02] (03CR) 10Jgiannelos: [C:03+2] kartotherian: Pin staging to latest image for testing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318152 (owner: 10Jgiannelos) [14:16:13] !log bking@cumin2003 START - Cookbook sre.dns.netbox [14:17:03] !log sudo gnt-instance reboot urldownloader1005.wikimedia.org [14:17:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:17:51] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1020.eqiad.wmnet with reason: host reimage [14:18:43] (03Merged) 10jenkins-bot: kartotherian: Pin staging to latest image for testing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318152 (owner: 10Jgiannelos) [14:19:59] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for iberker - https://phabricator.wikimedia.org/T433258 (10IBerker-WMF) 03NEW [14:20:00] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/kartotherian: apply [14:20:05] RECOVERY - SSH on urldownloader1005 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [14:20:12] RESOLVED: [2x] ProbeDown: Service urldownloader1005:8080 has failed probes (http_url_downloader_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Url-downloader - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:20:22] (03CR) 10Bking: [C:03+2] wdqs: bound missing updater metrics [cookbooks] - 10https://gerrit.wikimedia.org/r/1315666 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [14:20:37] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/kartotherian: apply [14:21:08] (03PS1) 10Effie Mouzeli: dsh: scap should stop syncing to parsoidtest1001 [puppet] - 10https://gerrit.wikimedia.org/r/1318156 (https://phabricator.wikimedia.org/T433239) [14:21:39] (03CR) 10Bking: [C:03+2] wdqs: restore Blazegraph exporters [cookbooks] - 10https://gerrit.wikimedia.org/r/1315665 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [14:21:51] (03PS2) 10Ryan Kemper: wdqs: bound missing updater metrics [cookbooks] - 10https://gerrit.wikimedia.org/r/1315666 (https://phabricator.wikimedia.org/T430880) [14:21:54] (03PS26) 10Federico Ceratto: mysql: update replication source [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) [14:22:06] bking@cumin2003 reimage (PID 849144) is awaiting input [14:22:11] (03CR) 10Bking: [V:03+2] wdqs: bound missing updater metrics [cookbooks] - 10https://gerrit.wikimedia.org/r/1315666 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [14:22:16] (03CR) 10Majavah: "does this need to manually restart the service?" [puppet] - 10https://gerrit.wikimedia.org/r/1314020 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [14:22:21] (03CR) 10Bking: [C:03+2] "recheck" [cookbooks] - 10https://gerrit.wikimedia.org/r/1315666 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [14:23:31] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[1031,2024]*: Upgrade Cassandra to 5.0.8 (canary) - eevans@cumin1003 [14:23:52] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1020.eqiad.wmnet with reason: host reimage [14:24:10] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2010 - bking@cumin2003" [14:24:15] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2010 - bking@cumin2003" [14:24:15] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:24:16] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache wdqs2010.codfw.wmnet 94.16.192.10.in-addr.arpa 4.9.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [14:24:19] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2010.codfw.wmnet 94.16.192.10.in-addr.arpa 4.9.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [14:24:20] !log bking@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wdqs2010 [14:24:27] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:24:30] !log bking@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2010 [14:24:31] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2010 [14:25:20] (03Merged) 10jenkins-bot: wdqs: bound missing updater metrics [cookbooks] - 10https://gerrit.wikimedia.org/r/1315666 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [14:25:40] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12158992 (10RobH) I'm going to estimate the work starting at 9AM Singapore will take up to 4 hours to complete, so we'd like the site depooled fr... [14:26:36] (03CR) 10Federico Ceratto: "@marostegui@wikimedia.org can you shed some light on the query to delete stale heartbeat data? E.g. is `DELETE FROM` OK?" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [14:26:57] (03CR) 10Federico Ceratto: mysql: update replication source (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [14:27:17] (03PS7) 10Bking: cirrussearch: update Elasticsearch verbiage to OpenSearch [alerts] - 10https://gerrit.wikimedia.org/r/1312570 (https://phabricator.wikimedia.org/T431304) [14:28:03] (03CR) 10Bking: cirrussearch: update Elasticsearch verbiage to OpenSearch (033 comments) [alerts] - 10https://gerrit.wikimedia.org/r/1312570 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [14:29:09] (03CR) 10Scott French: [C:03+1] aptrepo: add php85 component [puppet] - 10https://gerrit.wikimedia.org/r/1315940 (https://phabricator.wikimedia.org/T432983) (owner: 10Kamila Součková) [14:30:01] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12159006 (10ssingh) >>! In T433097#12158992, @RobH wrote: > I'm going to estimate the work starting at 9AM Singapore will take up to 4 hours to c... [14:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T1430) [14:33:33] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2175: Maintenance [14:34:29] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2155: Maintenance [14:36:53] (03CR) 10Bking: "I reviewed the table, LGTM!" [alerts] - 10https://gerrit.wikimedia.org/r/1312570 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [14:37:04] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12159020 (10RobH) I've sent an email to Jin to confirm the date window of 2026-07-29 @ 23:00 GMT to 2026-07-30 @ 03:00 GMT, which is 2026-07-30 @... [14:37:23] (03CR) 10Bking: [C:03+2] cirrussearch: update Elasticsearch verbiage to OpenSearch [alerts] - 10https://gerrit.wikimedia.org/r/1312570 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [14:41:48] !log eevans@cumin1003 END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase[1031,2024]*: Upgrade Cassandra to 5.0.8 (canary) - eevans@cumin1003 [14:42:00] (03CR) 10Ozge: [C:03+1] ml-services: Deploy latest version of revise-tone-task-generator on staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1314825 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [14:42:06] (03CR) 10Ozge: [C:03+1] ml-services: Deploy latest edit-check model version on production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318126 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [14:42:42] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2010.codfw.wmnet with reason: host reimage [14:44:44] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations: Move Docker images under the /v2/releng prefix to S3 - https://phabricator.wikimedia.org/T432829#12159044 (10elukey) p:05Triage→03Medium [14:46:19] PROBLEM - Host wdqs2010 is DOWN: PING CRITICAL - Packet loss = 100% [14:47:29] (03PS1) 10Andrew Bogott: Ceph: support setting bdev_enable_discard true [puppet] - 10https://gerrit.wikimedia.org/r/1318163 (https://phabricator.wikimedia.org/T429387) [14:48:11] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2010.codfw.wmnet with reason: host reimage [14:50:25] (03CR) 10Ladsgroup: [C:03+1] "I'll deploy it tomorrow." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1313088 (https://phabricator.wikimedia.org/T432686) (owner: 10Samwilson) [14:51:21] RECOVERY - Host wdqs2010 is UP: PING OK - Packet loss = 0%, RTA = 31.52 ms [14:51:34] (03CR) 10Scott French: [C:03+1] "Thanks, Effie! From a quick scan of the relevant scap code, I believe this (i.e., leaving the file intact, but containing no hosts) should" [puppet] - 10https://gerrit.wikimedia.org/r/1318156 (https://phabricator.wikimedia.org/T433239) (owner: 10Effie Mouzeli) [14:51:37] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2156: Maintenance [14:51:49] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1020.eqiad.wmnet with OS bookworm [14:51:59] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance [14:52:07] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2157: Maintenance [14:52:07] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2177 (T431660)', diff saved to https://phabricator.wikimedia.org/P95208 and previous config saved to /var/cache/conftool/dbconfig/20260727-145206-cwilliams.json [14:52:29] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance [14:52:37] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2171 (T431660)', diff saved to https://phabricator.wikimedia.org/P95209 and previous config saved to /var/cache/conftool/dbconfig/20260727-145236-cwilliams.json [14:53:23] (03CR) 10Hashar: "The actual problem is that Zuul web does not support receiving parameters that had special characters encoded. The Zuul web frontend does " [puppet] - 10https://gerrit.wikimedia.org/r/1314120 (https://phabricator.wikimedia.org/T431003) (owner: 10Dzahn) [14:53:28] (03CR) 10Hashar: [C:04-1] zuul: fix API 404s for encoded project slashes [puppet] - 10https://gerrit.wikimedia.org/r/1314120 (https://phabricator.wikimedia.org/T431003) (owner: 10Dzahn) [14:53:54] (03CR) 10Jelto: [C:03+2] Add profile::server_depool instructions for gitlab-runners [puppet] - 10https://gerrit.wikimedia.org/r/1314798 (https://phabricator.wikimedia.org/T327300) (owner: 10Jelto) [14:54:42] (03CR) 10Ladsgroup: Remove $wmg hacks for $wgBabelCentralDb and $wgPFEnableStringFunctions (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1314716 (https://phabricator.wikimedia.org/T119117) (owner: 10SomeRandomDeveloper) [14:56:53] (03PS27) 10Federico Ceratto: mysql: update replication source [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) [14:58:27] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2171: Maintenance [14:58:31] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2177: Maintenance [14:58:42] (03CR) 10Ahmon Dancy: "Yay! Congratulations" [puppet] - 10https://gerrit.wikimedia.org/r/1318156 (https://phabricator.wikimedia.org/T433239) (owner: 10Effie Mouzeli) [15:01:02] (03CR) 10Gkyziridis: [C:03+2] ml-services: Deploy latest version of revise-tone-task-generator on staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1314825 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [15:01:03] (03PS7) 10Andrew Bogott: ceph.conf: support adjusting slow ops health messages [puppet] - 10https://gerrit.wikimedia.org/r/1313966 (https://phabricator.wikimedia.org/T429387) [15:01:25] (03PS8) 10Andrew Bogott: ceph.conf: support adjusting slow ops health messages [puppet] - 10https://gerrit.wikimedia.org/r/1313966 (https://phabricator.wikimedia.org/T429387) [15:02:19] jouncebot: nowandnext [15:02:19] No deployments scheduled for the next 0 hour(s) and 27 minute(s) [15:02:19] In 0 hour(s) and 27 minute(s): Wikimedia Portals Update (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T1530) [15:03:29] (03Merged) 10jenkins-bot: ml-services: Deploy latest version of revise-tone-task-generator on staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1314825 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [15:04:06] (03PS1) 10Zabe: SpecialWantedFiles: Simplify query plan [core] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318166 (https://phabricator.wikimedia.org/T431518) [15:04:08] !log root@cumin1003 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2177: Maintenance [15:04:14] (03CR) 10Zabe: [C:03+2] SpecialWantedFiles: Simplify query plan [core] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318166 (https://phabricator.wikimedia.org/T431518) (owner: 10Zabe) [15:04:52] (03CR) 10Ahmon Dancy: [C:03+2] wmf-config/logging.php: Fix typo in comment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316039 (owner: 10Ahmon Dancy) [15:05:40] (03CR) 10Btullis: [C:03+1] "Looks good to me." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317845 (https://phabricator.wikimedia.org/T432029) (owner: 10Trueg) [15:06:11] (03Merged) 10jenkins-bot: wmf-config/logging.php: Fix typo in comment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316039 (owner: 10Ahmon Dancy) [15:06:29] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [15:07:22] (03PS1) 10Btullis: cloudnative-pg-cluster: allow issuing the server certificate from the WMF PKI [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318167 (https://phabricator.wikimedia.org/T432104) [15:07:24] (03PS1) 10Btullis: postgresql-superset-metrics: issue the server certificate from the WMF PKI [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318168 (https://phabricator.wikimedia.org/T432104) [15:08:22] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2177: Maintenance [15:09:01] (03CR) 10TrainBranchBot: [C:03+2] "Approved by zabe@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318166 (https://phabricator.wikimedia.org/T431518) (owner: 10Zabe) [15:10:09] (03CR) 10Andrew Bogott: cloud-vps: override SYSTEMD_NETLINK_DEFAULT_TIMEOUT on all instances (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1314020 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [15:10:16] (03PS10) 10Andrew Bogott: cloud-vps: override SYSTEMD_NETLINK_DEFAULT_TIMEOUT on all instances [puppet] - 10https://gerrit.wikimedia.org/r/1314020 (https://phabricator.wikimedia.org/T432426) [15:10:29] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [15:10:30] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318163 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [15:10:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [15:12:32] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2010.codfw.wmnet with OS bookworm [15:15:17] (03Merged) 10jenkins-bot: SpecialWantedFiles: Simplify query plan [core] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318166 (https://phabricator.wikimedia.org/T431518) (owner: 10Zabe) [15:15:24] !log bking@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [15:15:25] (03CR) 10Ahmon Dancy: "I like the simplification. I have a minor suggestion." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316171 (owner: 10Krinkle) [15:15:28] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [15:15:29] !log bking@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 06s) [15:15:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [15:15:31] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1318166|SpecialWantedFiles: Simplify query plan (T431518)]] [15:15:38] T431518: MediaWiki periodic job update-special-pages-s5 failed - https://phabricator.wikimedia.org/T431518 [15:15:43] !log bking@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [15:15:51] !log bking@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 12s) [15:16:22] (03PS2) 10Ryan Kemper: sre.wdqs.data-transfer: abort on failed transfer [cookbooks] - 10https://gerrit.wikimedia.org/r/1317127 (https://phabricator.wikimedia.org/T430880) [15:16:28] (03CR) 10Bking: [C:03+2] sre.wdqs.data-transfer: abort on failed transfer [cookbooks] - 10https://gerrit.wikimedia.org/r/1317127 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [15:16:57] (03CR) 10Ahmon Dancy: [C:03+1] logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316171 (owner: 10Krinkle) [15:17:20] !log zabe@deploy1003 zabe: Backport for [[gerrit:1318166|SpecialWantedFiles: Simplify query plan (T431518)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:18:15] !log zabe@deploy1003 zabe: Continuing with deployment [15:18:48] (03CR) 10Ahmon Dancy: [C:03+1] logging: Remove $wmgUdp2logDest duplicate in favor of $wmgLocalServices [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316172 (owner: 10Krinkle) [15:19:03] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2175: Maintenance [15:19:20] (03PS2) 10Andrew Bogott: Ceph: support setting bdev_enable_discard true [puppet] - 10https://gerrit.wikimedia.org/r/1318163 (https://phabricator.wikimedia.org/T429387) [15:19:20] (03PS9) 10Andrew Bogott: ceph.conf: support adjusting slow ops health messages [puppet] - 10https://gerrit.wikimedia.org/r/1313966 (https://phabricator.wikimedia.org/T429387) [15:19:56] (03CR) 10Ahmon Dancy: wmf-config/logging.php: Adjustments for WMF_MAINTENANCE_OFFLINE (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315968 (https://phabricator.wikimedia.org/T433104) (owner: 10Ahmon Dancy) [15:19:58] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2155: Maintenance [15:20:44] (03PS3) 10Andrew Bogott: Ceph: support setting bdev_enable_discard [puppet] - 10https://gerrit.wikimedia.org/r/1318163 (https://phabricator.wikimedia.org/T429387) [15:20:45] (03PS10) 10Andrew Bogott: ceph.conf: support adjusting slow ops health messages [puppet] - 10https://gerrit.wikimedia.org/r/1313966 (https://phabricator.wikimedia.org/T429387) [15:21:10] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318163 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [15:21:16] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards [15:21:20] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [15:21:37] jouncebot: next [15:21:38] In 0 hour(s) and 8 minute(s): Wikimedia Portals Update (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T1530) [15:22:31] (03Merged) 10jenkins-bot: sre.wdqs.data-transfer: abort on failed transfer [cookbooks] - 10https://gerrit.wikimedia.org/r/1317127 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [15:22:32] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1318166|SpecialWantedFiles: Simplify query plan (T431518)]] (duration: 07m 00s) [15:22:36] T431518: MediaWiki periodic job update-special-pages-s5 failed - https://phabricator.wikimedia.org/T431518 [15:23:15] (03PS2) 10Ryan Kemper: sre.wdqs: stop transferpy burying cookbook output [cookbooks] - 10https://gerrit.wikimedia.org/r/1317128 (https://phabricator.wikimedia.org/T430880) [15:23:33] (03CR) 10Bking: [C:03+2] sre.wdqs: stop transferpy burying cookbook output [cookbooks] - 10https://gerrit.wikimedia.org/r/1317128 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [15:26:06] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling source-only afterwards [15:26:26] (03Merged) 10jenkins-bot: sre.wdqs: stop transferpy burying cookbook output [cookbooks] - 10https://gerrit.wikimedia.org/r/1317128 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [15:27:54] (03PS4) 10Andrew Bogott: Ceph: support setting bdev_enable_discard [puppet] - 10https://gerrit.wikimedia.org/r/1318163 (https://phabricator.wikimedia.org/T429387) [15:27:54] (03PS11) 10Andrew Bogott: ceph.conf: support adjusting slow ops health messages [puppet] - 10https://gerrit.wikimedia.org/r/1313966 (https://phabricator.wikimedia.org/T429387) [15:29:04] !log bking@cumin2003 END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards [15:29:10] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [15:30:04] jan_drewniak: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for Wikimedia Portals Update. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T1530). [15:31:37] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards [15:32:05] !log bking@cumin2003 END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling source-only afterwards [15:32:35] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling source-only afterwards [15:34:30] !log bking@cumin2003 END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards [15:34:35] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [15:37:10] FIRING: BFDdown: BFD session down between cr4-ulsfo and 198.35.26.203 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr4-ulsfo:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:38:24] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1314020 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [15:38:25] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance [15:38:34] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2189 (T431660)', diff saved to https://phabricator.wikimedia.org/P95220 and previous config saved to /var/cache/conftool/dbconfig/20260727-153833-cwilliams.json [15:38:35] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards [15:39:20] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance [15:39:28] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2172 (T431660)', diff saved to https://phabricator.wikimedia.org/P95222 and previous config saved to /var/cache/conftool/dbconfig/20260727-153927-cwilliams.json [15:41:12] !log bking@cumin2003 END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards [15:41:17] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [15:42:10] RESOLVED: BFDdown: BFD session down between cr4-ulsfo and 198.35.26.203 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr4-ulsfo:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:43:19] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards [15:44:36] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2189: Maintenance [15:45:31] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2171: Maintenance [15:45:52] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance [15:45:59] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2172: Maintenance [15:46:00] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2178 (T431660)', diff saved to https://phabricator.wikimedia.org/P95224 and previous config saved to /var/cache/conftool/dbconfig/20260727-154559-cwilliams.json [15:46:36] (03CR) 10Majavah: cloud-vps: override SYSTEMD_NETLINK_DEFAULT_TIMEOUT on all instances (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1314020 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [15:47:43] (03CR) 10David Caro: Ceph: support setting bdev_enable_discard (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1318163 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [15:49:30] (03PS11) 10Andrew Bogott: cloud-vps: override SYSTEMD_NETLINK_DEFAULT_TIMEOUT on all instances [puppet] - 10https://gerrit.wikimedia.org/r/1314020 (https://phabricator.wikimedia.org/T432426) [15:51:38] !log root@cumin1003 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2172: Maintenance [15:51:39] FIRING: CoreBGPDown: Core BGP session down between cr2-drmrs and cr1-eqiad (2a02:ec80:600:fe04::2) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=drmrs&var-device=cr2-drmrs:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:51:46] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2178: Maintenance [15:53:46] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2172: Maintenance [15:53:50] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2177: Maintenance [15:56:39] RESOLVED: CoreBGPDown: Core BGP session down between cr2-drmrs and cr1-eqiad (2a02:ec80:600:fe04::2) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=drmrs&var-device=cr2-drmrs:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:56:51] (03CR) 10Dzahn: "Could we maybe take a step back and focus on the immediate question: does it work? Because to me it does." [puppet] - 10https://gerrit.wikimedia.org/r/1314120 (https://phabricator.wikimedia.org/T431003) (owner: 10Dzahn) [15:58:58] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:59:44] (03CR) 10Dzahn: "> Meanwhile I don't see a need for this change given requests from the Zuul frontend, the OpenAPI web interface or the `zuul-client encryp" [puppet] - 10https://gerrit.wikimedia.org/r/1314120 (https://phabricator.wikimedia.org/T431003) (owner: 10Dzahn) [16:01:01] (03PS5) 10Andrew Bogott: Ceph: support setting bdev_enable_discard [puppet] - 10https://gerrit.wikimedia.org/r/1318163 (https://phabricator.wikimedia.org/T429387) [16:01:01] (03PS12) 10Andrew Bogott: ceph.conf: support adjusting slow ops health messages [puppet] - 10https://gerrit.wikimedia.org/r/1313966 (https://phabricator.wikimedia.org/T429387) [16:03:02] (03CR) 10Andrew Bogott: Ceph: support setting bdev_enable_discard (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1318163 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [16:03:50] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318163 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [16:03:54] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1313966 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [16:05:55] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2190.codfw.wmnet with reason: Maintenance [16:06:03] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2190 (T431660)', diff saved to https://phabricator.wikimedia.org/P95230 and previous config saved to /var/cache/conftool/dbconfig/20260727-160602-cwilliams.json [16:06:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:08:25] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12159656 (10BCornwall) Confirming I can do those times (2026-07-29 @ 16:00-20:00 PDT) [16:08:29] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [16:08:43] (03CR) 10Trueg: [C:03+2] WDQSv2: Use a PVC template spec instead of a single PVC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317845 (https://phabricator.wikimedia.org/T432029) (owner: 10Trueg) [16:08:58] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:09:19] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [16:10:02] (03CR) 10Dzahn: "There seems to be a general misunderstanding here about this ticket. It describes a test URL getting a 404 as a problem. Then in a patch t" [puppet] - 10https://gerrit.wikimedia.org/r/1314120 (https://phabricator.wikimedia.org/T431003) (owner: 10Dzahn) [16:11:15] (03Merged) 10jenkins-bot: WDQSv2: Use a PVC template spec instead of a single PVC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317845 (https://phabricator.wikimedia.org/T432029) (owner: 10Trueg) [16:11:33] (03CR) 10Dzahn: "so regarding https://phabricator.wikimedia.org/T431003 - are we really just waiting for upstream? and your comment that you will eventual" [puppet] - 10https://gerrit.wikimedia.org/r/1314120 (https://phabricator.wikimedia.org/T431003) (owner: 10Dzahn) [16:13:31] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2190: Maintenance [16:13:58] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:13:58] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [16:14:27] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:31:19] (03PS1) 10Btullis: Add component/libmysql-java to bookworm [puppet] - 10https://gerrit.wikimedia.org/r/1318190 (https://phabricator.wikimedia.org/T406746) [16:32:29] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be209[7,8] - https://phabricator.wikimedia.org/T424892#12159791 (10Jhancock.wm) a:03Jhancock.wm [16:32:33] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2189: Maintenance [16:33:00] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance [16:33:08] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2225 (T431660)', diff saved to https://phabricator.wikimedia.org/P95238 and previous config saved to /var/cache/conftool/dbconfig/20260727-163307-cwilliams.json [16:33:10] (03PS2) 10Btullis: Add component/libmysql-java to bookworm [puppet] - 10https://gerrit.wikimedia.org/r/1318190 (https://phabricator.wikimedia.org/T406746) [16:33:51] 10ops-codfw, 06SRE, 06DC-Ops: Check wikikube-worker2088's BMC - https://phabricator.wikimedia.org/T432857#12159807 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [16:34:21] (03CR) 10Btullis: [C:03+2] Add component/libmysql-java to bookworm [puppet] - 10https://gerrit.wikimedia.org/r/1318190 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [16:36:25] (03PS1) 10Cory Massaro: Bump up the maximum concurrent calls (32 -> 512 in orchestrator, 100 -> 512 in evaluators). [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318191 [16:37:07] (03PS2) 10Cory Massaro: Bump up the maximum concurrent calls (32 -> 512 in orchestrator, 100 -> 512 in evaluators). [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318191 (https://phabricator.wikimedia.org/T433280) [16:37:24] (03PS3) 10Cory Massaro: Bump up the maximum concurrent calls (32 -> 512 in orchestrator, 100 -> 512 in evaluators). [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318191 (https://phabricator.wikimedia.org/T433280) [16:38:45] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts [16:38:47] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts [16:38:58] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [16:39:14] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2172: Maintenance [16:39:23] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2225: Maintenance [16:39:44] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2178: Maintenance [16:40:08] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2211.codfw.wmnet with reason: Maintenance [16:40:16] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2211 (T431660)', diff saved to https://phabricator.wikimedia.org/P95242 and previous config saved to /var/cache/conftool/dbconfig/20260727-164015-cwilliams.json [16:42:49] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12159857 (10RobH) Jin confirmed via email just now for the window: 2026-07-29 @ 23:00 GMT to 2026-07-30 @ 03:00 GMT I'll update our directions... [16:43:05] 10ops-codfw, 06SRE, 06DC-Ops: Check wikikube-worker2088's BMC - https://phabricator.wikimedia.org/T432857#12159858 (10elukey) Sorry for the late reply, there is a weirdness in Redfish that seemed related to the BMC's health, but you are right it seems working. Thanks! [16:43:40] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12159861 (10RobH) [16:46:07] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2211: Maintenance [16:50:25] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [16:52:02] (03CR) 10Andrew Bogott: cloud-vps: override SYSTEMD_NETLINK_DEFAULT_TIMEOUT on all instances (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1314020 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [16:52:13] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1020.eqiad.wmnet, repooling source-only afterwards [16:52:19] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T1700) [17:00:05] ryankemper: gettimeofday() says it's time for Wikidata Query Service weekly deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T1700) [17:03:58] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [17:04:35] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2190: Maintenance [17:04:45] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2194.codfw.wmnet with reason: Maintenance [17:04:54] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2194 (T431660)', diff saved to https://phabricator.wikimedia.org/P95248 and previous config saved to /var/cache/conftool/dbconfig/20260727-170453-cwilliams.json [17:06:25] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:08:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:11:24] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2194: Maintenance [17:12:00] (03PS1) 10Majavah: Undeploy WP25EasterEggs (I) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318202 (https://phabricator.wikimedia.org/T418134) [17:12:02] (03PS1) 10Majavah: Undeploy WP25EasterEggs (II) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318203 (https://phabricator.wikimedia.org/T418134) [17:13:19] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling source-only afterwards [17:13:23] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [17:15:17] (03CR) 10Jforrester: [C:03+1] Undeploy WP25EasterEggs (I) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318202 (https://phabricator.wikimedia.org/T418134) (owner: 10Majavah) [17:16:28] is anyone using the mw infrastructure window? [17:16:38] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9071/co" [puppet] - 10https://gerrit.wikimedia.org/r/1314020 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [17:17:05] (03CR) 10Majavah: [V:03+1 C:03+1] "thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1314020 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [17:21:02] (03CR) 10Scott French: [V:03+2 C:03+1] "Great, thanks Antoine." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1312574 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [17:21:46] 10ops-eqiad, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp1001 ram upgrade - https://phabricator.wikimedia.org/T433284 (10RobH) 03NEW p:05Triage→03Medium [17:21:49] 10ops-codfw, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp2001 ram upgrade - https://phabricator.wikimedia.org/T433285 (10RobH) 03NEW p:05Triage→03Medium [17:21:54] 10ops-eqiad, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp1001 ram upgrade - https://phabricator.wikimedia.org/T433284#12160052 (10RobH) [17:21:59] i will take the empty calendar and silence both here and on -sre as a no [17:22:00] 06SRE, 10Observability-Metrics, 07Wikimedia-production-error: PHP Warning: RedisException: read error on connection (arclamp1001.eqiad.wmnet) - https://phabricator.wikimedia.org/T432730#12160053 (10RobH) [17:22:02] 10ops-codfw, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp2001 ram upgrade - https://phabricator.wikimedia.org/T433285#12160054 (10RobH) [17:22:03] 06SRE, 10Observability-Metrics, 07Wikimedia-production-error: PHP Warning: RedisException: read error on connection (arclamp1001.eqiad.wmnet) - https://phabricator.wikimedia.org/T432730#12160055 (10RobH) [17:23:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by taavi@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318202 (https://phabricator.wikimedia.org/T418134) (owner: 10Majavah) [17:23:29] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [17:23:58] 10ops-eqiad, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp1001 ram upgrade - https://phabricator.wikimedia.org/T433284#12160057 (10RobH) [17:24:01] 10ops-codfw, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp2001 ram upgrade - https://phabricator.wikimedia.org/T433285#12160061 (10RobH) [17:25:11] (03Merged) 10jenkins-bot: Undeploy WP25EasterEggs (I) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318202 (https://phabricator.wikimedia.org/T418134) (owner: 10Majavah) [17:25:19] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [17:25:21] !log taavi@deploy1003 Started scap sync-world: Backport for [[gerrit:1318202|Undeploy WP25EasterEggs (I) (T418134)]] [17:25:25] T418134: [EPIC] Undeploy WP25EasterEggs extension & related WP25 birthday features - https://phabricator.wikimedia.org/T418134 [17:26:06] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2225: Maintenance [17:26:12] 10ops-codfw, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp2001 ram upgrade - https://phabricator.wikimedia.org/T433285#12160073 (10colewhite) Feel free to offline the hosts whenever you're ready. There's no paging associated and all I would do to depool is add host and services downtime in Icinga. [17:26:16] 10ops-eqiad, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp1001 ram upgrade - https://phabricator.wikimedia.org/T433284#12160074 (10colewhite) Feel free to offline the hosts whenever you're ready. There's no paging associated and all I would do to depool is add host and services downtime in Icinga. [17:26:28] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance [17:26:36] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2226 (T431660)', diff saved to https://phabricator.wikimedia.org/P95256 and previous config saved to /var/cache/conftool/dbconfig/20260727-172636-cwilliams.json [17:27:10] !log taavi@deploy1003 taavi: Backport for [[gerrit:1318202|Undeploy WP25EasterEggs (I) (T418134)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:27:44] !log taavi@deploy1003 taavi: Continuing with deployment [17:27:47] (03PS1) 10MusikAnimal: CodeMirrorWikiEditor: don't autofocus from live preview when RTP is open [extensions/CodeMirror] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318211 [17:28:26] 10ops-codfw, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp2001 ram upgrade - https://phabricator.wikimedia.org/T433285#12160079 (10RobH) [17:28:58] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [17:29:05] (03PS1) 10Ammarpad: admin: Add new SSH public key for Ammarpad [puppet] - 10https://gerrit.wikimedia.org/r/1318214 [17:29:41] 10ops-codfw, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp2001 ram upgrade - https://phabricator.wikimedia.org/T433285#12160081 (10RobH) [17:31:55] !log taavi@deploy1003 Finished scap sync-world: Backport for [[gerrit:1318202|Undeploy WP25EasterEggs (I) (T418134)]] (duration: 06m 33s) [17:31:59] T418134: [EPIC] Undeploy WP25EasterEggs extension & related WP25 birthday features - https://phabricator.wikimedia.org/T418134 [17:32:17] 10ops-eqiad, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp1001 ram upgrade - https://phabricator.wikimedia.org/T433284#12160115 (10VRiley-WMF) [17:32:34] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2226: Maintenance [17:32:53] 10ops-eqiad, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp1001 ram upgrade - https://phabricator.wikimedia.org/T433284#12160129 (10VRiley-WMF) 05Open→03In progress Proceeding with this as @colewhite said we can take this unit down today. [17:34:02] (03CR) 10Ammarpad: "I have added the new key here https://www.mediawiki.org/wiki/User:Ammarpad" [puppet] - 10https://gerrit.wikimedia.org/r/1318214 (owner: 10Ammarpad) [17:34:10] (03CR) 10Scott French: [V:03+2 C:03+2] python2-build-bullseye: Fix pip in Python 2 virtualenvs [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1312574 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [17:35:53] PROBLEM - Host arclamp1001 is DOWN: PING CRITICAL - Packet loss = 100% [17:36:54] !log bking@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs1020\.eqiad\.wmnet,dc=eqiad,cluster=wdqs\-main,service=wdqs\-main [17:37:11] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2211: Maintenance [17:37:33] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance [17:37:41] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2213 (T431660)', diff saved to https://phabricator.wikimedia.org/P95260 and previous config saved to /var/cache/conftool/dbconfig/20260727-173740-cwilliams.json [17:38:42] !log bking@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs2010\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main [17:40:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [17:41:02] (03CR) 10Scott French: [V:03+2 C:03+2] "The 0.3.1 image is now built / live." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1312574 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [17:41:59] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 27 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318149 (https://phabricator.wikimedia.org/T421250) (owner: 10Sbisson) [17:42:22] (03PS1) 10Sbisson: ArticleGuidance: Enable wgArticleGuidanceWikidataConnectEnabled in prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318217 (https://phabricator.wikimedia.org/T421250) [17:42:50] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 27 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318217 (https://phabricator.wikimedia.org/T421250) (owner: 10Sbisson) [17:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:43:29] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2213: Maintenance [17:43:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:44:25] RECOVERY - Host arclamp1001 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [17:44:36] 10ops-eqiad, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp1001 ram upgrade - https://phabricator.wikimedia.org/T433284#12160163 (10VRiley-WMF) [17:45:06] 10ops-eqiad, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp1001 ram upgrade - https://phabricator.wikimedia.org/T433284#12160164 (10VRiley-WMF) 05In progress→03Resolved Added requested DIMM into arclamp1001. [17:46:44] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs1021.eqiad.wmnet with OS bookworm [17:47:11] (03PS1) 10Scott French: hieradata: Explicitly enable the v2 API on conf2004 [puppet] - 10https://gerrit.wikimedia.org/r/1314015 (https://phabricator.wikimedia.org/T428495) [17:47:12] (03PS1) 10Scott French: hieradata: Temporarily move etcd replication to conf2004 [puppet] - 10https://gerrit.wikimedia.org/r/1314016 (https://phabricator.wikimedia.org/T428495) [17:47:13] (03PS5) 10Scott French: hieradata: Explicitly enable the v2 API in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1309317 (https://phabricator.wikimedia.org/T428495) [17:48:05] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs2011.codfw.wmnet with OS bookworm [17:48:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:50:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [17:51:05] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs1021 [17:51:05] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1021 [17:51:36] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs2011 [17:52:21] !log bking@cumin2003 START - Cookbook sre.dns.netbox [17:56:36] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2011 - bking@cumin2003" [17:56:40] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2011 - bking@cumin2003" [17:56:40] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [17:56:41] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache wdqs2011.codfw.wmnet 37.32.192.10.in-addr.arpa 7.3.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [17:56:44] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2011.codfw.wmnet 37.32.192.10.in-addr.arpa 7.3.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [17:56:45] !log bking@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wdqs2011 [17:56:59] !log bking@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2011 [17:56:59] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2011 [18:02:25] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2194: Maintenance [18:02:49] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: Maintenance [18:02:57] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2227 (T431660)', diff saved to https://phabricator.wikimedia.org/P95265 and previous config saved to /var/cache/conftool/dbconfig/20260727-180256-cwilliams.json [18:07:10] (03PS1) 10Majavah: P:toolforge::bastion: Drop grid engine script shims [puppet] - 10https://gerrit.wikimedia.org/r/1318226 [18:07:10] (03PS1) 10Majavah: P:toolforge::bastion: Minor cleanup [puppet] - 10https://gerrit.wikimedia.org/r/1318227 [18:07:10] (03PS1) 10Majavah: P:toolforge::bastion: Add script to migrate OpenSearch indexes [puppet] - 10https://gerrit.wikimedia.org/r/1318228 (https://phabricator.wikimedia.org/T401818) [18:09:26] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2227: Maintenance [18:12:33] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1021.eqiad.wmnet with reason: host reimage [18:13:32] (03PS6) 10Bking: cirrussearch: add cluster health and shard ratio alerts [alerts] - 10https://gerrit.wikimedia.org/r/1312571 (https://phabricator.wikimedia.org/T431304) [18:13:58] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [18:14:52] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2011.codfw.wmnet with reason: host reimage [18:17:53] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1021.eqiad.wmnet with reason: host reimage [18:19:04] (03CR) 10Krinkle: logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316171 (owner: 10Krinkle) [18:19:13] (03PS3) 10Krinkle: logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316171 [18:19:15] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2226: Maintenance [18:19:37] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance [18:19:45] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2238 (T431660)', diff saved to https://phabricator.wikimedia.org/P95271 and previous config saved to /var/cache/conftool/dbconfig/20260727-181944-cwilliams.json [18:19:51] (03CR) 10Krinkle: logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316171 (owner: 10Krinkle) [18:19:59] (03PS3) 10Krinkle: logging: Remove $wmgUdp2logDest duplicate in favor of $wmgLocalServices [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316172 [18:21:50] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2011.codfw.wmnet with reason: host reimage [18:22:28] (03PS4) 10Krinkle: logging: Remove $wmgUdp2logDest duplicate in favor of $wmgLocalServices [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316172 [18:24:27] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [18:26:00] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2238: Maintenance [18:27:36] (03PS7) 10Bking: cirrussearch: add cluster health and shard ratio alerts [alerts] - 10https://gerrit.wikimedia.org/r/1312571 (https://phabricator.wikimedia.org/T431304) [18:29:12] (03CR) 10TrainBranchBot: [C:03+2] "Approved by musikanimal@deploy1003 using scap backport" [extensions/CodeMirror] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318211 (owner: 10MusikAnimal) [18:31:28] (03CR) 10Ahmon Dancy: [C:03+1] logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316171 (owner: 10Krinkle) [18:31:46] (03Merged) 10jenkins-bot: CodeMirrorWikiEditor: don't autofocus from live preview when RTP is open [extensions/CodeMirror] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318211 (owner: 10MusikAnimal) [18:32:01] !log musikanimal@deploy1003 Started scap sync-world: Backport for [[gerrit:1318211|CodeMirrorWikiEditor: don't autofocus from live preview when RTP is open]] [18:32:38] (03CR) 10Ahmon Dancy: [C:03+1] logging: Remove $wmgUdp2logDest duplicate in favor of $wmgLocalServices [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316172 (owner: 10Krinkle) [18:33:31] 06SRE, 10LDAP-Access-Requests: Grant Access to for - https://phabricator.wikimedia.org/T433296 (10Rosewolfe41) 03NEW [18:33:45] !log musikanimal@deploy1003 musikanimal: Backport for [[gerrit:1318211|CodeMirrorWikiEditor: don't autofocus from live preview when RTP is open]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:34:31] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2213: Maintenance [18:34:51] !log musikanimal@deploy1003 musikanimal: Continuing with deployment [18:34:52] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance [18:35:00] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2223 (T431660)', diff saved to https://phabricator.wikimedia.org/P95275 and previous config saved to /var/cache/conftool/dbconfig/20260727-183500-cwilliams.json [18:39:06] !log musikanimal@deploy1003 Finished scap sync-world: Backport for [[gerrit:1318211|CodeMirrorWikiEditor: don't autofocus from live preview when RTP is open]] (duration: 07m 05s) [18:39:15] musikanimal: I'll do a config deploy after you [18:40:48] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2223: Maintenance [18:42:21] (03CR) 10Ahmon Dancy: "Thanks!" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1312574 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [18:42:59] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2011.codfw.wmnet with OS bookworm [18:44:25] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: DIMM module in slot A7 for db1208.eqiad.wmnet - https://phabricator.wikimedia.org/T432116#12160367 (10VRiley-WMF) 05Open→03In progress moving forward with this... [18:44:54] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1021.eqiad.wmnet with OS bookworm [18:45:51] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: DIMM module in slot A7 for db1208.eqiad.wmnet - https://phabricator.wikimedia.org/T432116#12160371 (10VRiley-WMF) [18:47:36] Hi, could someone merge this change (1314816) for the Beta Cluster? [18:49:53] PROBLEM - Host db1208 is DOWN: PING CRITICAL - Packet loss = 100% [18:50:30] (03CR) 10Mforns: [C:03+1] "LGTM overall! +1 on my side, although I don't know much of VCL!" [puppet] - 10https://gerrit.wikimedia.org/r/1318088 (https://phabricator.wikimedia.org/T433075) (owner: 10Cparle) [18:51:53] (03CR) 10Ssingh: "@ffurnari@wikimedia.org can help review from Traffic's end, thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1318088 (https://phabricator.wikimedia.org/T433075) (owner: 10Cparle) [18:55:56] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/services/mw-experimental: apply [18:56:06] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: DIMM module in slot A7 for db1208.eqiad.wmnet - https://phabricator.wikimedia.org/T432116#12160405 (10VRiley-WMF) 05In progress→03Open Swapped out DIMM and matched what it has in there. @BTullis please see if there are any issues, it should be good to go. [18:56:09] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1314020 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [18:57:18] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply [18:57:23] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2227: Maintenance [18:57:26] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-experimental: apply [18:57:57] RECOVERY - Host db1208 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [18:58:05] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply [18:59:00] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: DIMM module in slot A7 for db1208.eqiad.wmnet - https://phabricator.wikimedia.org/T432116#12160415 (10VRiley-WMF) [19:01:02] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: DIMM module in slot A7 for db1208.eqiad.wmnet - https://phabricator.wikimedia.org/T432116#12160417 (10VRiley-WMF) 05Open→03Resolved a:05BTullis→03VRiley-WMF [19:01:43] (03CR) 10Btullis: [C:03+1] "Looks good to me. Thanks." [alerts] - 10https://gerrit.wikimedia.org/r/1312571 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [19:11:45] (03PS1) 10Btullis: hadoop-test: Temporarily bootstrap the new namenodes [puppet] - 10https://gerrit.wikimedia.org/r/1318252 (https://phabricator.wikimedia.org/T406746) [19:12:17] jouncebot: next [19:12:17] In 0 hour(s) and 47 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T2000) [19:12:21] (03CR) 10CI reject: [V:04-1] hadoop-test: Temporarily bootstrap the new namenodes [puppet] - 10https://gerrit.wikimedia.org/r/1318252 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [19:12:46] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2238: Maintenance [19:12:51] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316171 (owner: 10Krinkle) [19:12:52] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316172 (owner: 10Krinkle) [19:13:15] (03CR) 10Ladsgroup: [C:03+1] "I can deploy it today or later if you want me to." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307506 (https://phabricator.wikimedia.org/T397070) (owner: 10Jforrester) [19:13:22] (03PS2) 10Btullis: hadoop-test: Temporarily bootstrap the new namenodes [puppet] - 10https://gerrit.wikimedia.org/r/1318252 (https://phabricator.wikimedia.org/T406746) [19:14:49] (03Merged) 10jenkins-bot: logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316171 (owner: 10Krinkle) [19:14:52] (03Merged) 10jenkins-bot: logging: Remove $wmgUdp2logDest duplicate in favor of $wmgLocalServices [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316172 (owner: 10Krinkle) [19:15:04] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1316171|logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile]], [[gerrit:1316172|logging: Remove $wmgUdp2logDest duplicate in favor of $wmgLocalServices]] [19:15:59] (03CR) 10Btullis: [C:03+2] hadoop-test: Temporarily bootstrap the new namenodes [puppet] - 10https://gerrit.wikimedia.org/r/1318252 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [19:16:49] !log krinkle@deploy1003 krinkle: Backport for [[gerrit:1316171|logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile]], [[gerrit:1316172|logging: Remove $wmgUdp2logDest duplicate in favor of $wmgLocalServices]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:16:59] (03CR) 10Andrew Bogott: [C:03+2] cloud-vps: override SYSTEMD_NETLINK_DEFAULT_TIMEOUT on all instances [puppet] - 10https://gerrit.wikimedia.org/r/1314020 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [19:20:23] (03PS1) 10Andrew Bogott: cloud-vps: add OS version guard around systemd-networkd override [puppet] - 10https://gerrit.wikimedia.org/r/1318256 (https://phabricator.wikimedia.org/T432426) [19:20:36] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318256 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [19:23:08] (03CR) 10Andrew Bogott: [C:03+2] cloud-vps: add OS version guard around systemd-networkd override [puppet] - 10https://gerrit.wikimedia.org/r/1318256 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [19:23:19] !log krinkle@deploy1003 krinkle: Continuing with deployment [19:26:43] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2223: Maintenance [19:26:51] 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302 (10CDiggs-WMF) 03NEW [19:27:04] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance [19:27:12] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2228 (T431660)', diff saved to https://phabricator.wikimedia.org/P95285 and previous config saved to /var/cache/conftool/dbconfig/20260727-192711-cwilliams.json [19:27:16] (03CR) 10Ladsgroup: [C:03+2] Simplify Docker images for tests [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1315693 (https://phabricator.wikimedia.org/T432686) (owner: 10Samwilson) [19:27:30] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1316171|logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile]], [[gerrit:1316172|logging: Remove $wmgUdp2logDest duplicate in favor of $wmgLocalServices]] (duration: 12m 26s) [19:30:05] (03CR) 10Lerickson: [C:03+1] WDQSv2: Use a PVC template spec instead of a single PVC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1317845 (https://phabricator.wikimedia.org/T432029) (owner: 10Trueg) [19:30:51] (03CR) 10Lerickson: [C:03+1] WDQSv2: scale resource limits to max with respect to the dedicated wdqs nodes. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313874 (https://phabricator.wikimedia.org/T431395) (owner: 10Trueg) [19:33:02] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2228: Maintenance [19:33:15] (03PS1) 10Scott French: mediawiki: Allow configurable msmtp timeout and bump to 5s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318259 (https://phabricator.wikimedia.org/T383047) [19:36:10] (03CR) 10JHathaway: [C:03+1] mediawiki: Allow configurable msmtp timeout and bump to 5s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318259 (https://phabricator.wikimedia.org/T383047) (owner: 10Scott French) [19:37:57] (03PS1) 10Ahmon Dancy: Merge remote-tracking branch 'origin/master' into train-dev [mediawiki-config] (train-dev) - 10https://gerrit.wikimedia.org/r/1318262 [19:38:11] (03CR) 10RLazarus: [C:03+1] mediawiki: Allow configurable msmtp timeout and bump to 5s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318259 (https://phabricator.wikimedia.org/T383047) (owner: 10Scott French) [19:38:12] (03CR) 10Ahmon Dancy: [C:03+2] Merge remote-tracking branch 'origin/master' into train-dev [mediawiki-config] (train-dev) - 10https://gerrit.wikimedia.org/r/1318262 (owner: 10Ahmon Dancy) [19:38:57] (03CR) 10Bking: [C:03+2] cirrussearch: add cluster health and shard ratio alerts [alerts] - 10https://gerrit.wikimedia.org/r/1312571 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [19:39:58] (03Merged) 10jenkins-bot: Simplify Docker images for tests [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1315693 (https://phabricator.wikimedia.org/T432686) (owner: 10Samwilson) [19:40:00] hi folks - we have a potential mitigation incoming for what's happening in T383047, which will require a mediawiki deployment. anyone about to start a deployment? [19:40:01] T383047: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047 [19:40:16] (03Merged) 10jenkins-bot: Merge remote-tracking branch 'origin/master' into train-dev [mediawiki-config] (train-dev) - 10https://gerrit.wikimedia.org/r/1318262 (owner: 10Ahmon Dancy) [19:40:24] Krinkle: it looks like you're done with your config deployment? [19:40:51] (03CR) 10Btullis: [C:03+2] hadoop-test: Fix typo in namenodes [puppet] - 10https://gerrit.wikimedia.org/r/1318261 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [19:41:45] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [19:42:02] (03CR) 10Scott French: "Thanks folks!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318259 (https://phabricator.wikimedia.org/T383047) (owner: 10Scott French) [19:42:03] (03CR) 10Scott French: [C:03+2] mediawiki: Allow configurable msmtp timeout and bump to 5s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318259 (https://phabricator.wikimedia.org/T383047) (owner: 10Scott French) [19:43:16] swfrench-wmf: correct [19:43:26] awesome, thanks [19:45:31] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [19:45:59] (03Merged) 10jenkins-bot: mediawiki: Allow configurable msmtp timeout and bump to 5s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318259 (https://phabricator.wikimedia.org/T383047) (owner: 10Scott French) [19:47:19] now that this is merged ^ I'll be starting a deployment shortly (once it's picked up by chart museum) [19:47:41] !log bking@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [19:47:45] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [19:47:50] !log bking@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 46s) [19:48:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:49:22] !log bking@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [19:49:28] !log bking@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 10s) [19:50:37] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-experimental: apply [19:51:09] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply [19:51:26] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs2011.codfw.wmnet, repooling source-only afterwards [19:52:30] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1021.eqiad.wmnet, repooling source-only afterwards [19:53:34] !log swfrench@deploy1003 Started scap sync-world: Deploy helmfile-only msmtp timeout change - T383047 [19:54:19] !log swfrench@deploy1003 swfrench: Deploy helmfile-only msmtp timeout change - T383047 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:55:25] !log swfrench@deploy1003 swfrench: Continuing with deployment [19:55:39] can no longer repro from mw-debug [19:57:15] !log swfrench@deploy1003 Finished scap sync-world: Deploy helmfile-only msmtp timeout change - T383047 (duration: 03m 40s) [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: #bothumor My software never has bugs. It just develops random features. Rise for UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T2000). [20:00:05] VadymTS1 and stephanebisson: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:13] o/ hello [20:00:52] VadymTS1 are you around? [20:01:27] I'll go ahead and deploy my patches. Should be relatively quick. [20:02:53] (03PS2) 10Btullis: cloudnative-pg-cluster: allow issuing the server certificate from the WMF PKI [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318167 (https://phabricator.wikimedia.org/T432104) [20:02:53] (03PS2) 10Btullis: postgresql-superset-metrics: issue the server certificate from the WMF PKI [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318168 (https://phabricator.wikimedia.org/T432104) [20:03:45] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbisson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318149 (https://phabricator.wikimedia.org/T421250) (owner: 10Sbisson) [20:03:45] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbisson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318217 (https://phabricator.wikimedia.org/T421250) (owner: 10Sbisson) [20:06:20] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12160709 (10Dzahn) a:05Aklapper→03None [20:07:03] (03Merged) 10jenkins-bot: Article Guidance: clean up old wikidata config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318149 (https://phabricator.wikimedia.org/T421250) (owner: 10Sbisson) [20:07:10] (03Merged) 10jenkins-bot: ArticleGuidance: Enable wgArticleGuidanceWikidataConnectEnabled in prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318217 (https://phabricator.wikimedia.org/T421250) (owner: 10Sbisson) [20:07:19] !log sbisson@deploy1003 Started scap sync-world: Backport for [[gerrit:1318149|Article Guidance: clean up old wikidata config (T421250)]], [[gerrit:1318217|ArticleGuidance: Enable wgArticleGuidanceWikidataConnectEnabled in prod (T421250)]] [20:07:23] T421250: Article Guidance: Automatically connect new articles to Wikidata items - https://phabricator.wikimedia.org/T421250 [20:07:41] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12160716 (10Dzahn) Hello and welcome @CDiggs-WMF . The process to get membership in the "wmf" LDAP group has changed to a self-service model. https://wikitech.wikimedia.o... [20:09:05] !log sbisson@deploy1003 sbisson: Backport for [[gerrit:1318149|Article Guidance: clean up old wikidata config (T421250)]], [[gerrit:1318217|ArticleGuidance: Enable wgArticleGuidanceWikidataConnectEnabled in prod (T421250)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:09:43] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [20:11:13] !log sbisson@deploy1003 sbisson: Continuing with deployment [20:12:18] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12160722 (10Dzahn) I see now this looks like a duplicate of T433112. Please let us know if there was some specific follow-up outside the ticket or a blocker you are runnin... [20:13:29] 06SRE, 10LDAP-Access-Requests: Grant access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433112#12160727 (10Dzahn) T433302 has been created. Should we just reopen here and close it as a duplicate? [20:13:30] (03PS1) 10Btullis: hadoop-test: Allow for a second hive-metastore [puppet] - 10https://gerrit.wikimedia.org/r/1318267 (https://phabricator.wikimedia.org/T406746) [20:13:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:15:22] !log sbisson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1318149|Article Guidance: clean up old wikidata config (T421250)]], [[gerrit:1318217|ArticleGuidance: Enable wgArticleGuidanceWikidataConnectEnabled in prod (T421250)]] (duration: 08m 03s) [20:15:29] T421250: Article Guidance: Automatically connect new articles to Wikidata items - https://phabricator.wikimedia.org/T421250 [20:15:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [20:17:18] (03CR) 10Btullis: [C:03+2] hadoop-test: Allow for a second hive-metastore [puppet] - 10https://gerrit.wikimedia.org/r/1318267 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [20:18:49] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: Maintenance [20:21:20] I'm all done. Over to you VadymTS1 (if you're around) [20:25:37] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12160751 (10VRiley-WMF) Dell called me today with an update. They are still having trouble obtaining the mainboard. However, they did assure me that a tech will be onsite Wednesday the 29th i... [20:30:11] FIRING: Temperature: GPU Temp issue on ml-serve1013:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1013 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [20:32:36] (03PS1) 10Bking: cirrus: Remove Cirrus frozen write monitors [puppet] - 10https://gerrit.wikimedia.org/r/1318274 (https://phabricator.wikimedia.org/T384998) [20:32:50] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318274 (https://phabricator.wikimedia.org/T384998) (owner: 10Bking) [20:35:11] RESOLVED: Temperature: GPU Temp issue on ml-serve1013:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1013 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [20:36:47] (03CR) 10Bking: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1318274 (https://phabricator.wikimedia.org/T384998) (owner: 10Bking) [20:41:58] (03CR) 10SomeRandomDeveloper: Remove $wmg hacks for $wgBabelCentralDb and $wgPFEnableStringFunctions (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1314716 (https://phabricator.wikimedia.org/T119117) (owner: 10SomeRandomDeveloper) [20:46:12] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12160850 (10Aklapper) Per T433112#12154474 pointing to the instructions at https://phabricator.wikimedia.org/project/profile/1564/ I'm not sure why this ticket was created? [20:50:25] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [20:50:29] (03CR) 10Dzahn: "I re-enabled puppet on zuul1001 where this change was applied, to showcase it. So if your test worked and now does not anymore that is why" [puppet] - 10https://gerrit.wikimedia.org/r/1314120 (https://phabricator.wikimedia.org/T431003) (owner: 10Dzahn) [20:51:24] !log zuul1001 - re-enabled puppet - revert "cherry-picked" gerrit:1314120 - T431003 [20:51:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:51:29] T431003: zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003 [20:53:52] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1021.eqiad.wmnet, repooling source-only afterwards [20:53:56] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [20:58:10] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs1022.eqiad.wmnet with OS bookworm [20:59:14] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12160886 (10Dzahn) Ready to setup zuul1004/2004 on our end. Not particularly urgent but just for planning purposes and because I am going on... [21:00:04] alexsanford, Reedy, sbassett, Maryum, and manfredi: May I have your attention please! Weekly Security deployment window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T2100) [21:00:23] (03CR) 10Dzahn: "Antoine, how about removing old jenkins from contint now? I would love to do it before going on vacation." [puppet] - 10https://gerrit.wikimedia.org/r/1308249 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [21:00:29] (03PS4) 10Ahmon Dancy: wmf-config/logging.php: Adjustments for WMF_MAINTENANCE_OFFLINE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315968 (https://phabricator.wikimedia.org/T433104) [21:01:05] (03CR) 10Ahmon Dancy: wmf-config/logging.php: Adjustments for WMF_MAINTENANCE_OFFLINE (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315968 (https://phabricator.wikimedia.org/T433104) (owner: 10Ahmon Dancy) [21:01:28] (03CR) 10Dzahn: "Arnaudb, just wondering what you think here." [puppet] - 10https://gerrit.wikimedia.org/r/1314095 (owner: 10Dzahn) [21:01:40] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs1022 [21:02:04] (03PS2) 10Dzahn: site/installserver: add codesearch with insetup role [puppet] - 10https://gerrit.wikimedia.org/r/1316215 [21:02:33] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, July 28 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306988 (https://phabricator.wikimedia.org/T428375) (owner: 10BPirkle) [21:02:43] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, July 28 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1312694 (https://phabricator.wikimedia.org/T422405) (owner: 10Aaron Schulz) [21:03:28] (03PS3) 10Dzahn: site/installserver: add codesearch with insetup role [puppet] - 10https://gerrit.wikimedia.org/r/1316215 (https://phabricator.wikimedia.org/T268199) [21:04:19] (03CR) 10Dzahn: [C:04-1] "NOT nftables - docker!" [puppet] - 10https://gerrit.wikimedia.org/r/1316215 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [21:04:43] bking@cumin2003 reimage (PID 941808) is awaiting input [21:05:27] !log bking@cumin2003 START - Cookbook sre.dns.netbox [21:06:00] (03PS4) 10Dzahn: site/installserver: add codesearch with insetup role [puppet] - 10https://gerrit.wikimedia.org/r/1316215 (https://phabricator.wikimedia.org/T268199) [21:08:19] (03CR) 10CI reject: [V:04-1] site/installserver: add codesearch with insetup role [puppet] - 10https://gerrit.wikimedia.org/r/1316215 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [21:08:20] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313 (10Bethany) 03NEW [21:08:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:09:19] (03PS5) 10Aaron Schulz: REST: use RestExternalModules config variable [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306988 (https://phabricator.wikimedia.org/T433314) (owner: 10BPirkle) [21:10:34] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs2011.codfw.wmnet, repooling source-only afterwards [21:10:35] (03PS1) 10Krinkle: php: Implement php_apcu_cache_entries gauge [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1318289 (https://phabricator.wikimedia.org/T433312) [21:10:39] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [21:10:53] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12160970 (10VRiley-WMF) Hey @Dzahn thanks for the ping on this. I've recently been freed up to work on this a bit more as I was having a lot... [21:11:14] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [21:11:36] bking@cumin2003 reimage (PID 941808) is awaiting input [21:11:36] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313#12160985 (10Bethany) @leila or @DKumar-WMF, could one of you please review and comment with your approval of this Level 3 analytics-privatedata-users access request? T... [21:13:11] 06SRE, 06Content-Platform-Team, 06ServiceOps new, 10Wikipedia-Android-App, 06WikipediaApp: Investigate Code 414 error when selecting zh-classical (lzh) language from article toolbar - https://phabricator.wikimedia.org/T425545#12160989 (10Phreelance) The same problem happens for `zh-min-nan`. The API call... [21:13:33] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12160991 (10Dzahn) Thanks @VRiley-WMF it's alright either way. It was just about who to assign it to on our end. [21:15:44] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1004] - vriley@cumin1003" [21:15:49] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1004] - vriley@cumin1003" [21:15:49] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [21:16:28] Howdy! We're about to start some security deployments. Just checking in case anyone is doing anything. [21:16:52] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host zuul1004 [21:17:51] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host zuul1004 [21:19:31] jouncebot: nowandnext [21:19:31] For the next 1 hour(s) and 40 minute(s): Weekly Security deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T2100) [21:19:31] In 1 hour(s) and 40 minute(s): Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T2300) [21:19:32] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [21:19:50] happy deployment! [21:21:03] hashar thanks!! [21:21:34] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [21:21:57] (03PS1) 10Scott French: hieradata: Temporarily point codfw PyBals at eqiad etcd [puppet] - 10https://gerrit.wikimedia.org/r/1314076 (https://phabricator.wikimedia.org/T428495) [21:22:00] (03PS2) 10Scott French: wmnet: Temporarily direct codfw, eqsin, ulsfo etcd clients to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1314077 (https://phabricator.wikimedia.org/T428495) [21:22:06] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [21:26:38] (03CR) 10BBlack: Add key to X-Analaytics if a thumbnail was generated during the request (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1318088 (https://phabricator.wikimedia.org/T433075) (owner: 10Cparle) [21:27:09] (03PS1) 10Bking: cirrussearch: Replace icinga check with prometheus blackbox check [puppet] - 10https://gerrit.wikimedia.org/r/1318301 (https://phabricator.wikimedia.org/T384998) [21:27:50] (03PS5) 10Dzahn: site/installserver: add codesearch with insetup role [puppet] - 10https://gerrit.wikimedia.org/r/1316215 (https://phabricator.wikimedia.org/T268199) [21:29:40] (03PS2) 10Bking: cirrussearch: Replace icinga check with prometheus blackbox check [puppet] - 10https://gerrit.wikimedia.org/r/1318301 (https://phabricator.wikimedia.org/T384998) [21:29:52] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318301 (https://phabricator.wikimedia.org/T384998) (owner: 10Bking) [21:30:42] (03CR) 10Dzahn: [C:03+2] site/installserver: add codesearch with insetup role [puppet] - 10https://gerrit.wikimedia.org/r/1316215 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [21:31:54] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [21:32:43] (03PS1) 10Ahmon Dancy: scap.cfg.erb: delay_messageblobstore_purge: true [puppet] - 10https://gerrit.wikimedia.org/r/1318302 (https://phabricator.wikimedia.org/T263872) [21:34:41] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS bookworm [21:34:55] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12161058 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host zuul1004.eqiad.wmnet with OS b... [21:35:13] (03PS3) 10Bking: cirrussearch: Replace icinga check with prometheus blackbox check [puppet] - 10https://gerrit.wikimedia.org/r/1318301 (https://phabricator.wikimedia.org/T384998) [21:35:19] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318301 (https://phabricator.wikimedia.org/T384998) (owner: 10Bking) [21:41:06] !log bking@cumin2003 END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [21:41:11] !log bking@cumin2003 START - Cookbook sre.dns.netbox [21:41:45] 06SRE, 06Infrastructure-Foundations, 10vm-requests: Site: 2 VM %request for codesearch - https://phabricator.wikimedia.org/T433316 (10Dzahn) 03NEW [21:42:46] 06SRE, 06Infrastructure-Foundations, 10vm-requests: Site: 2 VM %request for codesearch - https://phabricator.wikimedia.org/T433316#12161076 (10Dzahn) [21:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [21:44:23] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [21:44:24] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache wdqs1022.eqiad.wmnet 239.48.64.10.in-addr.arpa 9.3.2.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [21:44:27] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1022.eqiad.wmnet 239.48.64.10.in-addr.arpa 9.3.2.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [21:44:28] !log bking@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wdqs1022 [21:45:42] !log bking@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1022 [21:45:43] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1022 [21:53:08] (03PS2) 10Dzahn: Revert^2 "gitlab: lower TTL for CNAMEs" [dns] - 10https://gerrit.wikimedia.org/r/1310135 [21:54:11] (03CR) 10Dzahn: "this explains how one can be owner but not author btw" [dns] - 10https://gerrit.wikimedia.org/r/1310135 (owner: 10Dzahn) [21:55:10] !log bking@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs2011\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main [21:57:04] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs2012.codfw.wmnet with OS bookworm [22:00:34] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs2012 [22:01:46] !log Deployed security fix for T431819 [22:01:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:03:37] bking@cumin2003 reimage (PID 953155) is awaiting input [22:05:27] (03CR) 10Hashar: "contint1002 has role::ci, to drop Jenkins from it I think you simply drop the related Jenkins profile from the role:" [puppet] - 10https://gerrit.wikimedia.org/r/1308249 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [22:07:27] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1022.eqiad.wmnet with reason: host reimage [22:08:49] (03CR) 10Dzahn: "I was mostly just asking if the general idea of removing it at this point in time seems ok to you." [puppet] - 10https://gerrit.wikimedia.org/r/1308249 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [22:13:37] (03CR) 10Hashar: "Yes for sure Jenkins can be removed from contint1002. But I am not a fan of having a profile Jenkins with a hiera setting that disables i" [puppet] - 10https://gerrit.wikimedia.org/r/1308249 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [22:14:14] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1022.eqiad.wmnet with reason: host reimage [22:22:17] !log Deployed security patch for T431819 [22:22:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:24:02] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#12161199 (10Papaul) [22:24:27] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [22:32:47] !log Deployed security fix for T432789 [22:32:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:37:24] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS bullseye [22:37:39] (03CR) 10Dzahn: "thanks for confirming that it can be remove (one way or another) - that is what was important to me." [puppet] - 10https://gerrit.wikimedia.org/r/1308249 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [22:37:39] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12161242 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host zuul1004.eqiad.wmnet with OS b... [22:37:47] (03CR) 10RLazarus: [C:03+1] hieradata: Explicitly enable the v2 API on conf2004 [puppet] - 10https://gerrit.wikimedia.org/r/1314015 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [22:37:51] (03CR) 10RLazarus: [C:03+1] hieradata: Temporarily move etcd replication to conf2004 [puppet] - 10https://gerrit.wikimedia.org/r/1314016 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [22:37:55] (03CR) 10RLazarus: [C:03+1] hieradata: Explicitly enable the v2 API in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1309317 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [22:38:06] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1022.eqiad.wmnet with OS bookworm [22:39:46] !log Deploy security fix for T432877 [22:39:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:39:52] and that's the end of the security deploy for today [22:44:16] (03CR) 10JHathaway: [C:03+1] sre.hosts.reimage: pass dhcp_filename_ipxe to http_boot_once [cookbooks] - 10https://gerrit.wikimedia.org/r/1315820 (https://phabricator.wikimedia.org/T394357) (owner: 10Elukey) [22:45:56] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12161260 (10VRiley-WMF) Hey @MoritzMuehlenhoff is there a recommended OS for these devices? [22:45:59] (03CR) 10JHathaway: [C:03+1] redfish: expand force_http_boot_once for RedfishSupermicro [software/spicerack] - 10https://gerrit.wikimedia.org/r/1315812 (https://phabricator.wikimedia.org/T394357) (owner: 10Elukey) [22:46:58] (03PS3) 10Samwilson: Update Makefile to Compose v2, and improve documentation [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1313088 (https://phabricator.wikimedia.org/T432686) [22:55:43] 06SRE, 10Observability-Metrics, 07Wikimedia-production-error: PHP Warning: RedisException: read error on connection (arclamp1001.eqiad.wmnet) - https://phabricator.wikimedia.org/T432730#12161272 (10colewhite) Memory was upgraded on arclamp1001 today and I'm seeing used memory max out at around 32GB. I'll ke... [23:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260727T2300) [23:25:50] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [23:25:57] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [23:26:22] vriley@cumin1003 reimage (PID 2997863) is awaiting input [23:26:46] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [23:27:06] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1004.eqiad.wmnet with OS bullseye [23:27:19] (03CR) 10Dzahn: "This would not actually remove things though. Then the options are to either carefully read it all and manually remove it all or to leave " [puppet] - 10https://gerrit.wikimedia.org/r/1308249 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [23:27:24] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12161319 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host zuul1004.eqiad.wmnet with OS bulls... [23:28:02] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [23:42:12] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1318318 [23:42:13] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1318318 (owner: 10TrainBranchBot) [23:50:07] !log mass deleting vp8 transcodes [23:50:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:57:49] 06SRE, 06ServiceOps new, 10VisualEditor, 10VisualEditor Suggestion Mode, and 3 others: New Service Request: Headless VE - https://phabricator.wikimedia.org/T431497#12161367 (10Scott_French) >>! In T431497#12151902, @ppelberg wrote: > > @Scott_French, hi! An update and a resulting question for you (I think)... [23:57:55] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1318318 (owner: 10TrainBranchBot)