[12:11:27] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [12:15:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [12:18:13] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [12:18:13] (03PS3) 10Federico Ceratto: mysql: Fix ruff 0.16.0 lints in test files [cookbooks] - 10https://gerrit.wikimedia.org/r/1315864 [12:18:57] FIRING: [4x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:20:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [12:23:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:35:50] (03CR) 10Jakob: [C:03+1] "thanks!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315842 (https://phabricator.wikimedia.org/T302959) (owner: 10Silvan Heintze) [12:39:42] (03CR) 10Jakob: [C:03+1] Enable WikibaseLexeme REST API on beta wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315843 (https://phabricator.wikimedia.org/T430943) (owner: 10Silvan Heintze) [12:43:19] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, July 28 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309727 (https://phabricator.wikimedia.org/T431858) (owner: 10Esanders) [12:43:19] (03CR) 10Cathal Mooney: [C:03+1] "LGTM! Verified on Slack with Federico." [puppet] - 10https://gerrit.wikimedia.org/r/1307070 (owner: 10Federico Ceratto) [12:44:03] (03CR) 10Federico Ceratto: [C:03+2] admin: Rotate yubico pubkey [puppet] - 10https://gerrit.wikimedia.org/r/1307070 (owner: 10Federico Ceratto) [12:58:06] 06SRE, 06collaboration-services, 10Wikimedia-Mailing-lists: Further improve DMARC compatibility on lists.wikimedia.org - https://phabricator.wikimedia.org/T379517#12153106 (10LSobanski) Sounds good, let's do that. [13:03:57] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:06:02] (03CR) 10Elukey: [C:03+1] mysql: Fix ruff 0.16.0 lints in test files [cookbooks] - 10https://gerrit.wikimedia.org/r/1315864 (owner: 10Federico Ceratto) [13:06:56] (03CR) 10Federico Ceratto: [C:03+2] mysql: Fix ruff 0.16.0 lints in test files [cookbooks] - 10https://gerrit.wikimedia.org/r/1315864 (owner: 10Federico Ceratto) [13:09:35] (03Merged) 10jenkins-bot: mysql: Fix ruff 0.16.0 lints in test files [cookbooks] - 10https://gerrit.wikimedia.org/r/1315864 (owner: 10Federico Ceratto) [13:10:53] !log fnegri@cumin1003 START - Cookbook sre.mysql.multiinstance_reboot for clouddb[1029-1031].eqiad.wmnet [13:12:03] (03CR) 10Aghirelli: [C:03+1] REST: use RestExternalModules config variable [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306988 (https://phabricator.wikimedia.org/T428375) (owner: 10BPirkle) [13:24:26] !log fnegri@cumin1003 END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for clouddb[1029-1031].eqiad.wmnet [13:24:29] (03Abandoned) 10Btullis: cirrus: ship server JSON logs on production clusters [puppet] - 10https://gerrit.wikimedia.org/r/1311425 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [13:25:10] (03Abandoned) 10Btullis: rsyslog: support imfile readTimeout in rsyslog::input::file [puppet] - 10https://gerrit.wikimedia.org/r/1311423 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [13:25:48] (03Abandoned) 10Btullis: logstash: convert opensearch server logs to ECS [puppet] - 10https://gerrit.wikimedia.org/r/1311422 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [13:30:37] !log reboot mr1-eqsin for maintenance [13:30:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:31:46] (03PS1) 10Elukey: profile::spicerack::test_cookbook: allow to set PYTHONPATH [puppet] - 10https://gerrit.wikimedia.org/r/1315894 [13:33:31] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 27 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315843 (https://phabricator.wikimedia.org/T430943) (owner: 10Silvan Heintze) [13:34:21] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [13:34:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr2-eqsin and mr1-eqsin (103.102.166.143) - group Management - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqsin&var-device=cr2-eqsin:9804&var-bgp_group=Management&var-bgp_neighbor=mr1-eqsin - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:34:52] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 27 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315842 (https://phabricator.wikimedia.org/T302959) (owner: 10Silvan Heintze) [13:39:23] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 217.67 ms [13:41:13] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [13:42:25] !log jiji@cumin1003 START - Cookbook sre.hosts.reimage for host mc-gp2006.codfw.wmnet with OS bookworm [13:42:35] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps new: Rerack mc-gp2006 and mc2046 - https://phabricator.wikimedia.org/T432716#12153256 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host mc-gp2006.codfw.wmnet with OS bookworm [13:44:30] !log cmooney@cumin1003 START - Cookbook sre.dns.wipe-cache mc-gp2006.codfw.wmnet on all recursors [13:44:33] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) mc-gp2006.codfw.wmnet on all recursors [13:44:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr2-eqsin and mr1-eqsin (103.102.166.143) - group Management - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqsin&var-device=cr2-eqsin:9804&var-bgp_group=Management&var-bgp_neighbor=mr1-eqsin - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:44:49] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [extensions/3D] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1315130 (https://phabricator.wikimedia.org/T432911) (owner: 10Krinkle) [13:45:34] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: push new IPs for mc-gp2006 - cmooney@cumin1003" [13:45:38] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: push new IPs for mc-gp2006 - cmooney@cumin1003" [13:45:39] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:48:20] (03PS3) 10Elukey: sre.hosts.reimage: pass dhcp_filename_ipxe to http_boot_once [cookbooks] - 10https://gerrit.wikimedia.org/r/1315820 [13:49:31] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12153283 (10Papaul) I put in a remote hands order 1-262642171482 we are working on this [13:52:22] (03Merged) 10jenkins-bot: Add missing normaliseParams() call to ThreeDHandler::doTransform [extensions/3D] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1315130 (https://phabricator.wikimedia.org/T432911) (owner: 10Krinkle) [13:53:12] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1315130|Add missing normaliseParams() call to ThreeDHandler::doTransform (T432911)]] [13:53:16] T432911: PHP Deprecated: round(): Passing null to parameter #1 ($num) of type int|float is deprecated - https://phabricator.wikimedia.org/T432911 [13:54:56] topranks: yes [13:55:19] !log krinkle@deploy1003 krinkle: Backport for [[gerrit:1315130|Add missing normaliseParams() call to ThreeDHandler::doTransform (T432911)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:57:36] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [13:58:01] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [13:58:02] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [13:58:31] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [14:03:44] (03PS5) 10FNegri: sre.mysql.multiinstance_reboot: fix host type detection [cookbooks] - 10https://gerrit.wikimedia.org/r/1315819 (https://phabricator.wikimedia.org/T420203) [14:04:00] (03PS6) 10FNegri: sre.mysql.multiinstance_reboot: fix host type detection [cookbooks] - 10https://gerrit.wikimedia.org/r/1315819 (https://phabricator.wikimedia.org/T420203) [14:06:24] (03PS1) 10Btullis: dse-k8s: Enable the k8s-ingress-dse-postgresql service [puppet] - 10https://gerrit.wikimedia.org/r/1315900 (https://phabricator.wikimedia.org/T432104) [14:06:38] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1315900 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [14:08:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:09:18] !log jiji@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1071.eqiad.wmnet [14:09:18] !log jiji@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1071.eqiad.wmnet [14:09:30] !log jiji@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1071.eqiad.wmnet [14:09:31] !log jiji@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1071.eqiad.wmnet [14:10:07] !log jiji@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1072.eqiad.wmnet [14:10:08] !log jiji@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1072.eqiad.wmnet [14:10:18] !log jiji@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1072.eqiad.wmnet [14:10:19] !log jiji@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1072.eqiad.wmnet [14:10:47] (03PS3) 10Elukey: redfish: expand force_http_boot_once for RedfishSupermicro [software/spicerack] - 10https://gerrit.wikimedia.org/r/1315812 [14:10:47] (03PS1) 10Elukey: ruff: allow `None` not at the end of the type union [software/spicerack] - 10https://gerrit.wikimedia.org/r/1315903 [14:13:32] !log jiji@cumin1003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1135.eqiad.wmnet [14:13:36] !log jiji@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1135.eqiad.wmnet [14:14:09] !log jiji@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1135.eqiad.wmnet [14:14:46] !log jiji@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-worker1135.eqiad.wmnet with OS trixie [14:15:14] !log jiji@cumin1003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1135 [14:18:17] jiji@cumin1003 renumber-node (PID 2377574) is awaiting input [14:19:38] (03CR) 10Gmodena: [C:04-1] WDQSv2-next: Staging deployment of WDQSv2 (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313858 (https://phabricator.wikimedia.org/T432348) (owner: 10Trueg) [14:22:36] (03PS4) 10Elukey: redfish: expand force_http_boot_once for RedfishSupermicro [software/spicerack] - 10https://gerrit.wikimedia.org/r/1315812 (https://phabricator.wikimedia.org/T394357) [14:24:36] (03PS4) 10Elukey: sre.hosts.reimage: pass dhcp_filename_ipxe to http_boot_once [cookbooks] - 10https://gerrit.wikimedia.org/r/1315820 (https://phabricator.wikimedia.org/T394357) [14:26:03] (03CR) 10Brouberol: [C:03+1] dse-k8s: Enable the k8s-ingress-dse-postgresql service [puppet] - 10https://gerrit.wikimedia.org/r/1315900 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [14:26:37] !log jiji@cumin1003 START - Cookbook sre.dns.netbox [14:26:55] (03PS6) 10Trueg: WDQSv2-next: Staging deployment of WDQSv2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313858 (https://phabricator.wikimedia.org/T432348) [14:26:55] (03CR) 10Trueg: WDQSv2-next: Staging deployment of WDQSv2 (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313858 (https://phabricator.wikimedia.org/T432348) (owner: 10Trueg) [14:27:24] !log jiji@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host mc-gp2006.codfw.wmnet with OS bookworm [14:27:37] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps new: Rerack mc-gp2006 and mc2046 - https://phabricator.wikimedia.org/T432716#12153391 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host mc-gp2006.codfw.wmnet with OS bookworm executed with errors: - mc-gp2006 (**FAIL**)... [14:28:24] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12153394 (10cmooney) @papaul resolved the issue by shutting down ge-0/0/2 (connected to msw2) on mr1-eqsin. In brief the mgmt connectivity is working... [14:29:22] !log jiji@cumin1003 START - Cookbook sre.hosts.reimage for host mc-gp2006.codfw.wmnet with OS bookworm [14:29:32] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps new: Rerack mc-gp2006 and mc2046 - https://phabricator.wikimedia.org/T432716#12153397 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host mc-gp2006.codfw.wmnet with OS bookworm [14:29:51] !log krinkle@deploy1003 krinkle: Continuing with deployment [14:32:25] !log jiji@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1135 - jiji@cumin1003" [14:32:30] !log jiji@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1135 - jiji@cumin1003" [14:32:30] !log jiji@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:32:30] !log jiji@cumin1003 START - Cookbook sre.dns.wipe-cache wikikube-worker1135.eqiad.wmnet 177.32.64.10.in-addr.arpa 7.7.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [14:32:33] !log jiji@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1135.eqiad.wmnet 177.32.64.10.in-addr.arpa 7.7.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [14:32:34] !log jiji@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1135 [14:33:57] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:34:23] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1315130|Add missing normaliseParams() call to ThreeDHandler::doTransform (T432911)]] (duration: 41m 12s) [14:34:28] T432911: PHP Deprecated: round(): Passing null to parameter #1 ($num) of type int|float is deprecated - https://phabricator.wikimedia.org/T432911 [14:37:57] (03CR) 10Federico Ceratto: profile::spicerack::test_cookbook: allow to set PYTHONPATH (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1315894 (owner: 10Elukey) [14:38:57] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:42:45] (03PS2) 10Elukey: profile::spicerack::test_cookbook: allow to set PYTHONPATH [puppet] - 10https://gerrit.wikimedia.org/r/1315894 [14:43:27] (03CR) 10Elukey: [C:03+2] ruff: allow `None` not at the end of the type union [software/spicerack] - 10https://gerrit.wikimedia.org/r/1315903 (owner: 10Elukey) [14:43:57] FIRING: [7x] CertAlmostExpired: gNMI TLS certificate for lsw1-c5-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:46:22] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [14:48:24] !log jiji@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc-gp2006.codfw.wmnet with OS bookworm [14:48:36] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps new: Rerack mc-gp2006 and mc2046 - https://phabricator.wikimedia.org/T432716#12153443 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host mc-gp2006.codfw.wmnet with OS bookworm executed with errors: - mc-gp2006 (**FAIL**)... [14:48:57] FIRING: [11x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:49:24] !log jiji@cumin1003 START - Cookbook sre.hosts.reimage for host mc-gp2006.codfw.wmnet with OS bookworm [14:49:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [14:49:34] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps new: Rerack mc-gp2006 and mc2046 - https://phabricator.wikimedia.org/T432716#12153445 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1003 for host mc-gp2006.codfw.wmnet with OS bookworm [14:53:03] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12153467 (10elukey) After a chat with Supermicro it seems that new firmwares (we don't know it if it is for all models, and the timeline) need something like... [14:53:57] FIRING: [12x] CertAlmostExpired: gNMI TLS certificate for lsw1-c3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:53:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:58:57] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:58:57] FIRING: [13x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:59:12] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [14:59:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [14:59:51] (03PS14) 10Federico Ceratto: sre.mysql.pool: support esX sections [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) [15:03:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:08:22] !log jiji@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc-gp2006.codfw.wmnet with reason: host reimage [15:10:13] (03CR) 10BBlack: [C:03+1] "LGTM! :)" [puppet] - 10https://gerrit.wikimedia.org/r/1315900 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [15:13:15] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc-gp2006.codfw.wmnet with reason: host reimage [15:15:51] !log jiji@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1135 [15:15:51] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1135 [15:23:47] (03PS1) 10Kamila Součková: aptrepo: add php85 component [puppet] - 10https://gerrit.wikimedia.org/r/1315940 (https://phabricator.wikimedia.org/T432983) [15:26:02] (03CR) 10Kamila Součková: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9055/console" [puppet] - 10https://gerrit.wikimedia.org/r/1315940 (https://phabricator.wikimedia.org/T432983) (owner: 10Kamila Součková) [15:26:11] FIRING: Temperature: GPU Temp issue on ml-serve1013:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1013 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [15:30:29] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc-gp2006.codfw.wmnet with OS bookworm [15:30:38] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps new: Rerack mc-gp2006 and mc2046 - https://phabricator.wikimedia.org/T432716#12153876 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1003 for host mc-gp2006.codfw.wmnet with OS bookworm completed: - mc-gp2006 (**PASS**) - Remove... [15:31:11] RESOLVED: Temperature: GPU Temp issue on ml-serve1013:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1013 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [15:31:18] (03CR) 10Kamila Součková: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9056/console" [puppet] - 10https://gerrit.wikimedia.org/r/1315940 (https://phabricator.wikimedia.org/T432983) (owner: 10Kamila Součková) [15:32:23] !log jiji@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1135.eqiad.wmnet with reason: host reimage [15:33:07] !log cmooney@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: upgrade lswtest-d8-eqiad [15:36:45] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1135.eqiad.wmnet with reason: host reimage [15:37:13] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097 (10RobH) 03NEW p:05Triage→03High [15:37:15] !log upgrade SR-Linux OS on lswtest-d8-eqiad [15:37:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:38:57] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12153909 (10RobH) [15:39:08] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12153910 (10RobH) a:03ssingh Sukhbir, Please review the task description and provide feedback on when we could depool eqsin for the proposed w... [15:39:37] (03PS2) 10Ryan Kemper: prometheus: start exporter with Blazegraph [puppet] - 10https://gerrit.wikimedia.org/r/1315664 (https://phabricator.wikimedia.org/T430880) [15:40:25] (03CR) 10CI reject: [V:04-1] prometheus: start exporter with Blazegraph [puppet] - 10https://gerrit.wikimedia.org/r/1315664 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [15:41:22] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps new: Rerack mc-gp2006 and mc2046 - https://phabricator.wikimedia.org/T432716#12153925 (10jijiki) @Jhancock.wm mc-gp2006 is alive an well (tx @cmooney for sorting out the networking bits). Is it alright if we re-rack mc2046 sometime next week? [15:43:08] (03PS1) 10Effie Mouzeli: mcrouter_wancache: re-add mc-gp2006 to the gutter-pool [puppet] - 10https://gerrit.wikimedia.org/r/1315946 (https://phabricator.wikimedia.org/T432716) [15:47:08] (03PS1) 10Kamila Součková: package_builder: add pbuilder hook for component/php85 [puppet] - 10https://gerrit.wikimedia.org/r/1315948 (https://phabricator.wikimedia.org/T432983) [15:48:48] (03CR) 10Kamila Součková: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9057/co" [puppet] - 10https://gerrit.wikimedia.org/r/1315948 (https://phabricator.wikimedia.org/T432983) (owner: 10Kamila Součková) [15:51:34] (03PS3) 10Ryan Kemper: prometheus: start exporter with Blazegraph [puppet] - 10https://gerrit.wikimedia.org/r/1315664 (https://phabricator.wikimedia.org/T430880) [15:52:12] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 40 hosts [15:53:04] (03CR) 10Ryan Kemper: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1315664 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [15:54:13] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 40 hosts [15:55:32] (03PS2) 10Kamila Součková: package_builder: add pbuilder hook for component/php85 [puppet] - 10https://gerrit.wikimedia.org/r/1315948 (https://phabricator.wikimedia.org/T432983) [15:56:26] (03CR) 10Ryan Kemper: "recheck" [cookbooks] - 10https://gerrit.wikimedia.org/r/1315665 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [15:56:35] (03CR) 10Ryan Kemper: "recheck" [cookbooks] - 10https://gerrit.wikimedia.org/r/1315666 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [15:56:37] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1135.eqiad.wmnet with OS trixie [16:03:15] 06SRE, 06serviceops-deprecated, 05MediaWiki-backport-deployments, 05Train Deployments: MW script "eval.php" failing during scap operations - https://phabricator.wikimedia.org/T379044#12153996 (10dancy) 05In progress→03Resolved Here's what's happening today: ` dancy@deploy1003:~$ scap mwscript -- ev... [16:03:35] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1011.eqiad.wmnet -> wdqs1014.eqiad.wmnet, repooling source-only afterwards [16:03:38] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [16:03:52] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2007.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling source-only afterwards [16:08:21] (03PS1) 10Elukey: docker_registry: remove unused file [puppet] - 10https://gerrit.wikimedia.org/r/1315960 [16:08:54] (03CR) 10Elukey: [C:03+2] docker_registry: remove support for the nginx blob cache (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1304512 (https://phabricator.wikimedia.org/T427175) (owner: 10Elukey) [16:08:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:09:46] (03CR) 10FNegri: "After rebasing, the CI is now passing." [cookbooks] - 10https://gerrit.wikimedia.org/r/1315819 (https://phabricator.wikimedia.org/T420203) (owner: 10FNegri) [16:09:50] 06SRE, 06Infrastructure-Foundations, 10netops: Nokia SR-Linux: Update Homer automation to support v26 - https://phabricator.wikimedia.org/T433105 (10cmooney) 03NEW p:05Triage→03Medium [16:11:23] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [16:11:26] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [16:11:27] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [16:11:30] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [16:11:54] (03CR) 10Kamila Součková: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1315948 (https://phabricator.wikimedia.org/T432983) (owner: 10Kamila Součková) [16:11:54] 06SRE, 06Infrastructure-Foundations, 10netops: Nokia SR-Linux: Update Homer automation to support v26 - https://phabricator.wikimedia.org/T433105#12154065 (10cmooney) [16:13:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:14:27] 06SRE, 06Infrastructure-Foundations, 10netops: Nokia SR-Linux: Update Homer automation to support v26 - https://phabricator.wikimedia.org/T433105#12154069 (10cmooney) Actually I double-checked the ACL one. If you display the config `as json` it is identical in v25 and v26. Seems they just changed how it is... [16:18:57] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:19:23] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [16:22:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [16:22:55] (03PS1) 10BryanDavis: developer-portal: Bump container to 2026-07-24-161534-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315964 (https://phabricator.wikimedia.org/T432285) [16:23:15] jouncebot: nowandnext [16:23:15] For the next 14 hour(s) and 36 minute(s): No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260724T0700) [16:23:15] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [16:23:15] In 14 hour(s) and 36 minute(s): No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260725T0700) [16:23:57] FIRING: [4x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:25:07] (03CR) 10Elukey: [C:03+2] docker_registry: remove unused file [puppet] - 10https://gerrit.wikimedia.org/r/1315960 (owner: 10Elukey) [16:25:45] federico3: o/ ok to puppet-merge? [16:26:05] looking [16:26:37] (03CR) 10BryanDavis: [C:03+2] developer-portal: Bump container to 2026-07-24-161534-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315964 (https://phabricator.wikimedia.org/T432285) (owner: 10BryanDavis) [16:26:59] federico3: admin: Rotate yubico pubkey [16:27:05] yes go ahead [16:27:17] thanks for asking @elukey [16:27:23] np! Done! [16:27:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [16:27:33] I have a fix for developer.wikimedia.org's CSP policy ready. federico3 has kindly given me approval to do a Friday deploy to get the fix out. I will proceed with deployment soon. [16:28:33] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [16:28:35] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [16:28:36] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [16:28:40] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [16:28:57] FIRING: [4x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:29:15] (03Merged) 10jenkins-bot: developer-portal: Bump container to 2026-07-24-161534-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1315964 (https://phabricator.wikimedia.org/T432285) (owner: 10BryanDavis) [16:33:15] !log bd808@deploy1003 helmfile [staging] START helmfile.d/services/developer-portal: apply [16:33:44] !log bd808@deploy1003 helmfile [staging] DONE helmfile.d/services/developer-portal: apply [16:34:04] !log bd808@deploy1003 helmfile [codfw] START helmfile.d/services/developer-portal: apply [16:34:20] !log bd808@deploy1003 helmfile [codfw] DONE helmfile.d/services/developer-portal: apply [16:34:36] !log bd808@deploy1003 helmfile [eqiad] START helmfile.d/services/developer-portal: apply [16:34:52] !log bd808@deploy1003 helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply [16:35:08] (03PS1) 10Ahmon Dancy: wmf-config/logging.php: Adjustments for WMF_MAINTENANCE_OFFLINE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315968 (https://phabricator.wikimedia.org/T433104) [16:35:52] (03PS2) 10Ahmon Dancy: wmf-config/logging.php: Adjustments for WMF_MAINTENANCE_OFFLINE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315968 (https://phabricator.wikimedia.org/T433104) [16:36:45] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [16:36:45] (03CR) 10Ahmon Dancy: "https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1226 must be deployed first." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315968 (https://phabricator.wikimedia.org/T433104) (owner: 10Ahmon Dancy) [16:37:43] all done federico3. The deploy was smooth; the CSP change works when I cache bust. We can let the natural CDN edge expiration roll the fix out to everyone. [16:37:56] wow, thanks [16:39:20] FIRING: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 1930.9970730555558 times in the last 10 minutes due to memory usage (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [16:45:02] (03PS1) 10Btullis: Fix an issue with disabling the jobs on an-test-coord1002 [puppet] - 10https://gerrit.wikimedia.org/r/1315972 (https://phabricator.wikimedia.org/T406746) [16:47:14] (03CR) 10Btullis: [C:03+2] Fix an issue with disabling the jobs on an-test-coord1002 [puppet] - 10https://gerrit.wikimedia.org/r/1315972 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [16:57:46] 10ops-eqsin, 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12154233 (10BCornwall) @ssingh I am willing to be available on my Monday evening/Tuesday Singapore time. [17:09:21] RESOLVED: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 159.76847777777778 times in the last 10 minutes due to memory usage (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [17:11:45] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [17:14:33] 10ops-eqiad, 06SRE, 06DC-Ops: Check lvs10[13,15] BMCs - https://phabricator.wikimedia.org/T432856#12154280 (10BCornwall) These are just testing hosts that haven't been in production for a while. lvs1017-1020 are the current production hosts. I've made a note for traffic to discuss whether we can decomm these... [17:15:00] 10ops-eqiad, 06SRE, 06DC-Ops: Check lvs10[13,15] BMCs - https://phabricator.wikimedia.org/T432856#12154283 (10VRiley-WMF) a:03VRiley-WMF [17:15:41] 10ops-eqiad, 06SRE, 06DC-Ops: Check lvs10[13,15] BMCs - https://phabricator.wikimedia.org/T432856#12154285 (10VRiley-WMF) 05Open→03Resolved Thanks @BCornwall I will proceed to close this. Feel free to reopen if needed. [17:21:17] !log jiji@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1135.eqiad.wmnet [17:21:18] !log jiji@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1135.eqiad.wmnet [17:21:20] !log jiji@cumin1003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1135.eqiad.wmnet [17:26:52] (03CR) 10Zaidusyy: "Done" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305218 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [17:42:56] (03Abandoned) 10Btullis: WIP: airflow-wikidata: add qlever index PVCs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1305854 (https://phabricator.wikimedia.org/T428235) (owner: 10Gmodena) [17:42:57] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:43:43] (03Abandoned) 10Btullis: WIP: admin_ng: add wdqs local-storage resources for qlever indexer [deployment-charts] - 10https://gerrit.wikimedia.org/r/1305852 (https://phabricator.wikimedia.org/T428235) (owner: 10Gmodena) [17:43:57] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:44:50] (03Abandoned) 10Btullis: admin_ng: define the topolvm CSI releases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1305977 (https://phabricator.wikimedia.org/T429331) (owner: 10Btullis) [17:58:07] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-presto1013 - https://phabricator.wikimedia.org/T433030#12154419 (10VRiley-WMF) @BTullis it looks like this disk has failed that was predicted in ticket https://phabricator.wikimedia.org/T408065 As previously mentioned, @RobH and @wiki_willy, we would need to... [17:59:12] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Degraded RAID on an-presto1013 - https://phabricator.wikimedia.org/T433030#12154422 (10VRiley-WMF) a:03VRiley-WMF [18:13:59] (03PS1) 10Ahmon Dancy: wmf-config/logging.php: Fix typo in comment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316039 [18:19:23] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [18:21:13] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [18:23:57] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549) {#12267}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [18:25:20] 06SRE, 10LDAP-Access-Requests: Grant access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433112#12154474 (10Aklapper) 05Open→03Invalid a:05Jelto→03None Hi and welcome! > The username of your existing account on wikitech.wikimedia.org This data is outdated. ^ If you followed some... [18:33:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [18:39:20] FIRING: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 81.67131666666668 times in the last 10 minutes due to memory usage (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [18:41:00] (03CR) 10Dzahn: [C:03+1] deployment_server/k8s: set kubeconfig files for etherpad [puppet] - 10https://gerrit.wikimedia.org/r/1315749 (owner: 10Jelto) [18:42:47] (03CR) 10Dzahn: [C:03+1] "I think it's a good thing. Ff it already supports it and saves a user from rebuilding, sure let's do that." [puppet] - 10https://gerrit.wikimedia.org/r/1314798 (https://phabricator.wikimedia.org/T327300) (owner: 10Jelto) [18:44:21] RESOLVED: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 72.28010277777778 times in the last 10 minutes due to memory usage (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [18:45:45] (03PS6) 10BCornwall: varnish: Split out media functions from upload [puppet] - 10https://gerrit.wikimedia.org/r/1311110 (https://phabricator.wikimedia.org/T427465) [18:48:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [18:52:55] (03PS12) 10BCornwall: varnish: Configure text for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [19:16:49] (03PS13) 10BCornwall: varnish: Configure text for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [19:18:05] (03CR) 10BCornwall: [C:03+1] wmnet: add disc-urldownloader A/A record [dns] - 10https://gerrit.wikimedia.org/r/1313950 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [19:18:53] 10ops-eqiad, 06DC-Ops: Cardboard off data center floor eqiad - https://phabricator.wikimedia.org/T433119 (10VRiley-WMF) 03NEW [19:21:11] 10ops-eqiad, 06DC-Ops: Cardboard off data center floor eqiad - https://phabricator.wikimedia.org/T433119#12154601 (10VRiley-WMF) 05Open→03Resolved Cleared out those locations. [19:21:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [19:36:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [19:37:38] (03PS1) 10Ahmon Dancy: wmf-config/logging.php: Adjustments for WMF_MAINTENANCE_OFFLINE [mediawiki-config] (train-dev) - 10https://gerrit.wikimedia.org/r/1316090 (https://phabricator.wikimedia.org/T433104) [19:38:30] (03PS3) 10Ahmon Dancy: wmf-config/logging.php: Adjustments for WMF_MAINTENANCE_OFFLINE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315968 (https://phabricator.wikimedia.org/T433104) [19:54:15] (03CR) 10BBlack: [C:03+1] "Seems to be a true functional no-op now in PS6." [puppet] - 10https://gerrit.wikimedia.org/r/1311110 (https://phabricator.wikimedia.org/T427465) (owner: 10BCornwall) [20:01:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [20:06:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [20:08:08] 10ops-codfw, 10ops-drmrs, 10ops-eqdfw, 10ops-eqiad, and 6 others: Fix cables with placeholders names and planned status - https://phabricator.wikimedia.org/T432317#12154803 (10VRiley-WMF) [20:08:20] (03CR) 10BBlack: [C:04-1] varnish: Configure text for thumbnails (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [20:21:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [20:29:26] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:31:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [20:32:23] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [20:36:31] FIRING: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [20:38:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:41:03] 10ops-codfw, 10ops-drmrs, 10ops-eqdfw, 10ops-eqiad, and 6 others: Fix cables with placeholders names and planned status - https://phabricator.wikimedia.org/T432317#12154898 (10VRiley-WMF) [20:43:13] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [20:46:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp1001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=eqiad&var-job=redis_arclamp&var-instance=arclamp1001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [20:48:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:50:36] (03CR) 10BCornwall: varnish: Configure text for thumbnails (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [20:57:26] (03PS7) 10BCornwall: varnish: Split out media functions from upload [puppet] - 10https://gerrit.wikimedia.org/r/1311110 (https://phabricator.wikimedia.org/T427465) [20:57:27] (03PS14) 10BCornwall: varnish: Configure text for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [21:00:50] (03PS1) 10Zaidusyy: gerrit: Add RegisterNewEmailHtml.soy site override [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) [21:01:17] (03PS2) 10Zaidusyy: gerrit: Add RegisterNewEmailHtml.soy site override [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) [21:01:36] 10ops-codfw, 10ops-drmrs, 10ops-eqdfw, 10ops-eqiad, and 6 others: Fix cables with placeholders names and planned status - https://phabricator.wikimedia.org/T432317#12154965 (10VRiley-WMF) [21:13:37] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [21:14:27] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [21:21:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [21:26:25] (03CR) 10Krinkle: [C:03+1] wmf-config/logging.php: Adjustments for WMF_MAINTENANCE_OFFLINE (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315968 (https://phabricator.wikimedia.org/T433104) (owner: 10Ahmon Dancy) [21:30:54] (03CR) 10Krinkle: [C:03+1] wmf-config/logging.php: Adjustments for WMF_MAINTENANCE_OFFLINE (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315968 (https://phabricator.wikimedia.org/T433104) (owner: 10Ahmon Dancy) [21:31:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST secrets) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [21:37:18] jhathaway@cumin1003 reimage (PID 2437319) is awaiting input [21:38:21] !log jhathaway@cumin1003 START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie [21:39:22] !log jhathaway@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie [21:39:37] 10ops-codfw, 10ops-drmrs, 10ops-eqdfw, 10ops-eqiad, and 6 others: Fix cables with placeholders names and planned status - https://phabricator.wikimedia.org/T432317#12155097 (10VRiley-WMF) [21:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [21:43:22] !log jhathaway@cumin1003 START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie [21:44:26] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [21:46:31] (03CR) 10Paladox: [C:03+1] gerrit: Add RegisterNewEmailHtml.soy site override [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [21:47:17] (03CR) 10CI reject: [V:04-1] gerrit: Add RegisterNewEmailHtml.soy site override [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [21:47:38] !log jhathaway@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie [21:48:19] 10ops-codfw, 10ops-drmrs, 10ops-eqdfw, 10ops-eqiad, and 6 others: Fix cables with placeholders names and planned status - https://phabricator.wikimedia.org/T432317#12155104 (10VRiley-WMF) [21:48:57] (03PS1) 10Krinkle: logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316171 [21:48:57] (03PS1) 10Krinkle: logging: Remove $wmgUdp2logDest duplicate in favor of $wmgLocalServices [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316172 [21:49:22] (03CR) 10Krinkle: [C:03+1] wmf-config/logging.php: Fix typo in comment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316039 (owner: 10Ahmon Dancy) [21:50:02] (03PS2) 10Krinkle: logging: Simplify $wmgEnableExtraLogFile without $wmgExtraLogFile [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316171 [21:50:02] (03PS2) 10Krinkle: logging: Remove $wmgUdp2logDest duplicate in favor of $wmgLocalServices [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316172 [21:50:30] (03PS3) 10Zaidusyy: gerrit: Add RegisterNewEmailHtml.soy site override [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) [21:50:52] 10ops-codfw, 10ops-drmrs, 10ops-eqdfw, 10ops-eqiad, and 6 others: Fix cables with placeholders names and planned status - https://phabricator.wikimedia.org/T432317#12155109 (10VRiley-WMF) Updated several cables, will need to continue to update a few more. [21:51:06] !log jhathaway@cumin1003 START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie [21:53:13] !log jhathaway@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2010.codfw.wmnet with OS trixie [21:53:25] (03CR) 10Zaidusyy: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [21:53:27] !log jhathaway@cumin1003 START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie [22:00:12] jhathaway@cumin1003 reimage (PID 2440512) is awaiting input [22:00:32] !log jhathaway@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2010.codfw.wmnet with OS trixie [22:04:10] 06SRE, 06MediaWiki-Core-Platform-Team, 05Front-end Modernization: Introduce a Front-end Build Step for MediaWiki Skins and Extensions - https://phabricator.wikimedia.org/T279108#12155159 (10MGoncalves-WMF) [22:11:37] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [22:13:28] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [22:13:31] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [22:13:32] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [22:13:35] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [22:15:31] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [22:18:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:24:26] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549) {#12267}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [22:26:45] (03PS1) 10Dzahn: site/installserver: add codesearch with insetup role [puppet] - 10https://gerrit.wikimedia.org/r/1316215 [22:33:56] !log jhathaway@cumin1003 START - Cookbook sre.hosts.reimage for host sretest2010.codfw.wmnet with OS trixie [22:34:52] (03CR) 10Paladox: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [22:35:57] (03CR) 10Paladox: [C:03+1] gerrit: Add RegisterNewEmailHtml.soy site override [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [22:36:13] (03CR) 10Paladox: [C:03+1] "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1316125 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [22:39:48] 06SRE, 10SRE-swift-storage, 10MediaWiki-extensions-Score: Add cache key information to metadata json - https://phabricator.wikimedia.org/T257093#12155183 (10TheDJ) now that we have svg... should this maybe be run again ? [22:40:27] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [22:43:57] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:45:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [22:53:24] (03CR) 10BBlack: [C:03+1] varnish: Split out media functions from upload [puppet] - 10https://gerrit.wikimedia.org/r/1311110 (https://phabricator.wikimedia.org/T427465) (owner: 10BCornwall) [23:03:35] !log jhathaway@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage [23:08:52] !log jhathaway@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2010.codfw.wmnet with reason: host reimage [23:11:31] 06SRE, 10SRE-swift-storage, 10MediaWiki-extensions-Score: Add cache key information to metadata json - https://phabricator.wikimedia.org/T257093#12155271 (10TheDJ) See also: {T427866} [23:28:24] PROBLEM - SSH on arclamp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [23:30:14] RECOVERY - SSH on arclamp1001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [23:42:32] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1316279 [23:42:33] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1316279 (owner: 10TrainBranchBot) [23:43:28] !log jhathaway@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2010.codfw.wmnet with OS trixie [23:53:04] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1316279 (owner: 10TrainBranchBot) [23:54:41] !log ryankemper@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs2008.codfw.wmnet [23:54:42] !log ryankemper@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs1014.eqiad.wmnet