[00:13:58] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:21:02] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [00:24:26] !log dzahn@cumin1003 START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie [00:26:25] !log attempting reimage with trixie on zuul1004 re-purposed physical hardware - dcops reported install issue - host was in busybox shell (T427353) [00:26:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:26:29] T427353: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353 [00:33:31] (03CR) 10Ssingh: [C:03+1] hieradata: Temporarily point codfw PyBals at eqiad etcd [puppet] - 10https://gerrit.wikimedia.org/r/1314076 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [00:34:16] (03CR) 10Ssingh: [C:03+1] wmnet: Temporarily direct codfw, eqsin, ulsfo etcd clients to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1314077 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [00:43:07] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12161452 (10Dzahn) I tried to run reimage as well, with trixie. Just getting BusyBox shell. Looks like this machine did not get an IP via... [00:47:25] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [00:47:45] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [00:47:57] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply [00:48:14] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply [00:50:25] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [01:08:06] !log pt1979@cumin2003 START - Cookbook sre.dns.netbox [01:08:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:11:15] (03PS1) 10TrainBranchBot: Branch commit for wmf/1.47.0-wmf.13 [core] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318352 (https://phabricator.wikimedia.org/T430832) [01:11:18] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/1.47.0-wmf.13 [core] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318352 (https://phabricator.wikimedia.org/T430832) (owner: 10TrainBranchBot) [01:12:01] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12161475 (10Papaul) [01:12:14] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1318353 [01:12:14] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1318353 (owner: 10TrainBranchBot) [01:12:44] !log pt1979@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add mgmt for new switches in eqsin - pt1979@cumin2003" [01:12:48] !log pt1979@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add mgmt for new switches in eqsin - pt1979@cumin2003" [01:12:48] !log pt1979@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [01:13:48] dzahn@cumin1003 reimage (PID 3010677) is awaiting input [01:21:02] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [01:22:31] (03PS1) 10Dzahn: installserver: switch zuul[12]004 to raid10-4dev partman recipe [puppet] - 10https://gerrit.wikimedia.org/r/1318354 (https://phabricator.wikimedia.org/T427353) [01:24:18] (03CR) 10Dzahn: [C:03+2] installserver: switch zuul[12]004 to raid10-4dev partman recipe [puppet] - 10https://gerrit.wikimedia.org/r/1318354 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [01:24:20] !log pt1979@cumin1003 START - Cookbook sre.network.tls for network device asw1-603-eqsin [01:25:26] !log pt1979@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-603-eqsin [01:27:37] (03Merged) 10jenkins-bot: Branch commit for wmf/1.47.0-wmf.13 [core] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318352 (https://phabricator.wikimedia.org/T430832) (owner: 10TrainBranchBot) [01:27:45] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1318353 (owner: 10TrainBranchBot) [01:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:43:28] !log dzahn@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1004.eqiad.wmnet with OS trixie [01:43:45] !log dzahn@cumin1003 START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie [01:53:58] FIRING: [14x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:57:38] (03PS1) 10Dzahn: installserver: switch zuul[12]004 to -efi partman recipes [puppet] - 10https://gerrit.wikimedia.org/r/1318362 (https://phabricator.wikimedia.org/T427353) [01:57:51] (03PS2) 10Dzahn: installserver: switch zuul[12]004 to -efi partman recipes [puppet] - 10https://gerrit.wikimedia.org/r/1318362 (https://phabricator.wikimedia.org/T427353) [01:59:40] (03PS3) 10Dzahn: installserver: switch zuul[12]004 to -efi partman recipes [puppet] - 10https://gerrit.wikimedia.org/r/1318362 (https://phabricator.wikimedia.org/T427353) [01:59:53] (03PS4) 10Dzahn: installserver: switch zuul[12]004 to -efi partman recipes [puppet] - 10https://gerrit.wikimedia.org/r/1318362 (https://phabricator.wikimedia.org/T427353) [02:00:04] Deploy window Automatic branching of MediaWiki, extensions, skins, and vendor – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T0200) [02:06:06] (03CR) 10Dzahn: [C:03+2] installserver: switch zuul[12]004 to -efi partman recipes [puppet] - 10https://gerrit.wikimedia.org/r/1318362 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [02:08:25] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:08:58] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:10:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:13:58] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:14:27] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:15:50] !log dzahn@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host zuul1004.eqiad.wmnet with OS trixie [02:16:05] !log dzahn@cumin1003 START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie [02:30:32] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12161510 (10Dzahn) @VRiley-WMF Looks like I got it working after using different partman recipes. That was on me! I will take care of the s... [02:31:16] !log dzahn@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage [02:37:57] !log dzahn@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage [02:45:18] (03PS1) 10Dzahn: ci: remove jenkins profiles from role::ci [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) [02:45:52] (03CR) 10CI reject: [V:04-1] ci: remove jenkins profiles from role::ci [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [02:46:06] (03CR) 10Dzahn: "https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318370" [puppet] - 10https://gerrit.wikimedia.org/r/1308249 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [02:49:53] (03CR) 10Dzahn: "Here is the way you have been suggesting. But now if I look at https://puppet-compiler.wmflabs.org/output/1318370/9072/contint1002.wikimed" [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [02:50:07] (03PS2) 10Dzahn: ci: remove jenkins profiles from role::ci [puppet] - 10https://gerrit.wikimedia.org/r/1318370 (https://phabricator.wikimedia.org/T418521) [02:55:39] !log dzahn@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - dzahn@cumin1003" [02:57:44] !log dzahn@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - dzahn@cumin1003" [02:57:44] !log dzahn@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1004.eqiad.wmnet with OS trixie [03:00:05] Deploy window Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T0300) [03:01:55] (03PS1) 10TrainBranchBot: testwikis to 1.47.0-wmf.13 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318371 (https://phabricator.wikimedia.org/T430832) [03:01:58] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by mwpresync@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318371 (https://phabricator.wikimedia.org/T430832) (owner: 10TrainBranchBot) [03:03:01] (03Merged) 10jenkins-bot: testwikis to 1.47.0-wmf.13 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318371 (https://phabricator.wikimedia.org/T430832) (owner: 10TrainBranchBot) [03:03:18] !log mwpresync@deploy1003 Started scap sync-world: testwikis to 1.47.0-wmf.13 refs T430832 [03:03:26] T430832: 1.47.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T430832 [03:19:08] 10ops-eqiad, 10Ceph, 06cloud-services-team, 10Cloud-VPS, and 2 others: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267#12161522 (10Andrew) Dc-ops folks: Please have a look at the backplane issue on this server. It is drained and out of service so you can power it down as needed. [03:39:24] !log mwpresync@deploy1003 Finished scap sync-world: testwikis to 1.47.0-wmf.13 refs T430832 (duration: 36m 06s) [03:39:29] T430832: 1.47.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T430832 [03:59:17] FIRING: [2x] ProbeDown: Service wdqs2012:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs2012:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:00:05] Deploy window Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T0400) [04:02:40] !log mwpresync@deploy1003 Pruned MediaWiki: 1.47.0-wmf.10 (duration: 02m 34s) [04:50:25] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [05:33:58] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave) {#11372}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [05:37:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr2-eqord and cr3-ulsfo (198.35.26.128) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [05:38:58] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:xe-0/0/1:1 (Transport: cr2-eqord:xe-0/1/0 (Arelion, IC-314534 29ms 10Gbps wave) {#10694_12249-2}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [05:41:44] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [05:42:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr2-eqord and cr3-ulsfo (198.35.26.128) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [05:43:13] FIRING: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:46:31] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [05:48:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:53:36] (03PS2) 10KineticPelagic: rest: Add test server option to REST Sandbox for Wikipedia projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313827 (https://phabricator.wikimedia.org/T408816) [05:53:58] RESOLVED: [4x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:xe-0/0/1:1 (Transport: cr2-eqord:xe-0/1/0 (Arelion, IC-314534 29ms 10Gbps wave) {#10694_12249-2}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [05:54:27] FIRING: [13x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T0600) [06:00:05] marostegui, Amir1, and federico3: How many deployers does it take to do Primary database switchover deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T0600). [06:10:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:11:42] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [06:13:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:16:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [06:28:32] (03CR) 10Slyngshede: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1318214 (owner: 10Ammarpad) [06:29:34] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 36133760 and 4 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [06:31:34] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 0 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [06:34:23] (03CR) 10Slyngshede: [C:03+2] data.yaml Incorrect expiry date for account jniren-ctr [puppet] - 10https://gerrit.wikimedia.org/r/1318057 (owner: 10Slyngshede) [06:38:43] (03CR) 10Brouberol: [C:03+1] cloudnative-pg-cluster: allow issuing the server certificate from the WMF PKI [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318167 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [06:39:30] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2235].codfw.wmnet,db[1164,1217,1228].eqiad.wmnet with reason: m5 master switch T432967 [06:39:34] T432967: Switchover m5 master db1164 -> db1228 - https://phabricator.wikimedia.org/T432967 [06:40:00] (03CR) 10Brouberol: "I think we also need to include `postgresql-superset-metrics.discovery.wmnet` as part of the certificate DNSNames?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318168 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [06:41:23] (03PS1) 10Marostegui: mariadb: Promote db1228 to m5 master [puppet] - 10https://gerrit.wikimedia.org/r/1318563 (https://phabricator.wikimedia.org/T432967) [06:42:43] (03CR) 10Marostegui: [C:03+2] mariadb: Promote db1228 to m5 master [puppet] - 10https://gerrit.wikimedia.org/r/1318563 (https://phabricator.wikimedia.org/T432967) (owner: 10Marostegui) [06:44:02] !log Failover m5 from db1164 to db1228 - T432967 [06:44:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:51:49] (03PS1) 10Marostegui: installserver: Do not format clouddb1033 [puppet] - 10https://gerrit.wikimedia.org/r/1318564 (https://phabricator.wikimedia.org/T409557) [06:58:42] (03CR) 10Marostegui: [C:03+2] installserver: Do not format clouddb1033 [puppet] - 10https://gerrit.wikimedia.org/r/1318564 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [07:00:05] Amir1, urbanecm, and awight: Time to do the UTC morning backport window deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T0700). [07:00:05] No Gerrit patches in the queue for this window AFAICS. [07:00:46] (03PS1) 10Marostegui: mariadb: Enable notifications on m5 master [puppet] - 10https://gerrit.wikimedia.org/r/1318566 (https://phabricator.wikimedia.org/T432967) [07:02:12] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance [07:02:20] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2206 (T431660)', diff saved to https://phabricator.wikimedia.org/P95292 and previous config saved to /var/cache/conftool/dbconfig/20260728-070219-cwilliams.json [07:06:19] (03CR) 10Marostegui: [C:03+2] mariadb: Enable notifications on m5 master [puppet] - 10https://gerrit.wikimedia.org/r/1318566 (https://phabricator.wikimedia.org/T432967) (owner: 10Marostegui) [07:08:48] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2206: Maintenance [07:13:26] (03CR) 10Trueg: [C:03+2] WDQSv2-next: Staging deployment of WDQSv2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313858 (https://phabricator.wikimedia.org/T432348) (owner: 10Trueg) [07:13:27] (03PS1) 10Marostegui: db1228: Remove note [puppet] - 10https://gerrit.wikimedia.org/r/1318567 [07:14:16] (03CR) 10Marostegui: [C:03+2] db1228: Remove note [puppet] - 10https://gerrit.wikimedia.org/r/1318567 (owner: 10Marostegui) [07:15:57] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance [07:16:05] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2168 (T431660)', diff saved to https://phabricator.wikimedia.org/P95294 and previous config saved to /var/cache/conftool/dbconfig/20260728-071604-cwilliams.json [07:16:21] (03Merged) 10jenkins-bot: WDQSv2-next: Staging deployment of WDQSv2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313858 (https://phabricator.wikimedia.org/T432348) (owner: 10Trueg) [07:16:33] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance [07:16:41] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2154 (T431660)', diff saved to https://phabricator.wikimedia.org/P95295 and previous config saved to /var/cache/conftool/dbconfig/20260728-071640-cwilliams.json [07:21:14] (03CR) 10Filippo Giunchedi: [C:03+2] networktests: use dumps canonical paths [puppet] - 10https://gerrit.wikimedia.org/r/1313830 (https://phabricator.wikimedia.org/T432587) (owner: 10Filippo Giunchedi) [07:22:11] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2168: Maintenance [07:22:46] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2154: Maintenance [07:24:05] (03PS1) 10Jelto: helmfile.d/* use helm3.17 and helm3.19 helmBinary [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318569 (https://phabricator.wikimedia.org/T388390) [07:29:37] (03CR) 10Jelto: "I guess the rake file should be updated as well? Should I create a dedicated change for https://gerrit.wikimedia.org/r/plugins/gitiles/ope" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318569 (https://phabricator.wikimedia.org/T388390) (owner: 10Jelto) [07:31:11] (03CR) 10MVernon: [C:03+1] "LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/1313071 (https://phabricator.wikimedia.org/T411617) (owner: 10Ayounsi) [07:31:24] (03CR) 10MVernon: [C:03+1] "LGTM, thanks." [dns] - 10https://gerrit.wikimedia.org/r/1313072 (https://phabricator.wikimedia.org/T411617) (owner: 10Ayounsi) [07:43:36] (03CR) 10Gkyziridis: [C:03+2] ml-services: Deploy latest edit-check model version on production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318126 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [07:45:55] (03PS1) 10Kevin Bazira: ml-services: deploy tts isvc in prod [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318573 (https://phabricator.wikimedia.org/T430536) [07:46:25] (03Merged) 10jenkins-bot: ml-services: Deploy latest edit-check model version on production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318126 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [07:50:09] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' . [07:50:25] !log gkyziridis@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . [07:59:32] FIRING: [2x] ProbeDown: Service wdqs2012:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs2012:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:59:51] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2206: Maintenance [08:00:01] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2210.codfw.wmnet with reason: Maintenance [08:00:09] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2210 (T431660)', diff saved to https://phabricator.wikimedia.org/P95305 and previous config saved to /var/cache/conftool/dbconfig/20260728-080008-cwilliams.json [08:03:02] (03PS1) 10Ayounsi: cr2-drmrs: Enable BGP RIB Sharding [homer/public] - 10https://gerrit.wikimedia.org/r/1318659 (https://phabricator.wikimedia.org/T320264) [08:05:17] (03PS1) 10Gkyziridis: ml-services: Deploy latest version of revise-tone-task-generator on ml-serve-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318662 (https://phabricator.wikimedia.org/T429675) [08:05:34] !log ayounsi@cumin1003 START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: router upgrade, T431749] [08:05:38] T431749: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749 [08:05:39] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool drmrs [reason: router upgrade, T431749] [08:06:26] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2210: Maintenance [08:06:47] !log ayounsi@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-drmrs,cr1-drmrs IPv6,cr1-drmrs.mgmt with reason: router upgrade [08:06:54] 06SRE, 06Infrastructure-Foundations, 10netops: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749#12161827 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8089257c-9548-4d05-a03b-bdb3c1224c47) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their servi... [08:08:26] (03PS2) 10Slyngshede: Account blocking: Only unset email if requested [software/bitu] - 10https://gerrit.wikimedia.org/r/1318083 (https://phabricator.wikimedia.org/T431114) [08:09:17] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Maintenance [08:09:40] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance [08:09:48] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2182 (T431660)', diff saved to https://phabricator.wikimedia.org/P95307 and previous config saved to /var/cache/conftool/dbconfig/20260728-080947-cwilliams.json [08:10:14] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2154: Maintenance [08:10:37] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance [08:10:45] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2161 (T431660)', diff saved to https://phabricator.wikimedia.org/P95309 and previous config saved to /var/cache/conftool/dbconfig/20260728-081044-cwilliams.json [08:12:02] !log root@cumin1003 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2210: Maintenance [08:12:06] (03CR) 10Gkyziridis: "There is this comment in the values file:" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318662 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [08:16:06] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2182: Maintenance [08:17:06] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2161: Maintenance [08:21:43] !log root@cumin1003 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2182: Maintenance [08:28:46] (03CR) 10Kevin Bazira: [C:03+2] ml-services: deploy tts isvc in prod [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318573 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [08:29:42] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2182: Maintenance [08:31:01] (03Merged) 10jenkins-bot: ml-services: deploy tts isvc in prod [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318573 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [08:31:02] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2210: Maintenance [08:31:37] (03CR) 10Marostegui: "I'd prefer if we limit it, to the host we are deleting it from. Similar to what we do with master switches eg:" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [08:33:14] !log draining cr1-drmrs - T431749 [08:33:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:33:18] T431749: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749 [08:35:54] !log rebooting cr1-drmrs - T431749 [08:35:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:36:35] 06SRE, 10Observability-Metrics, 07Wikimedia-production-error: PHP Warning: RedisException: read error on connection (arclamp1001.eqiad.wmnet) - https://phabricator.wikimedia.org/T432730#12161966 (10MLechvien-WMF) >>! In T432730#12141549, @Dreamy_Jazz wrote: > Adding #sre for triage (not sure who maintains th... [08:38:30] 10ops-eqiad, 06DC-Ops: Unresponsive management for an-worker1147.mgmt:22 - https://phabricator.wikimedia.org/T433348 (10phaultfinder) 03NEW [08:38:43] PROBLEM - OSPF status on cr2-drmrs is CRITICAL: OSPFv2: 3/5 UP : OSPFv3: 3/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [08:38:45] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 9/11 UP : OSPFv3: 9/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [08:39:39] FIRING: [2x] CoreBGPDown: Core BGP session down between asw1-b12-drmrs and cr1-drmrs (185.15.58.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=drmrs&var-device=asw1-b12-drmrs:9804&var-bgp_group=core&var-bgp_neighbor=cr1-drmrs - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [08:40:10] FIRING: [2x] BFDdown: BFD session down between cr2-drmrs and 185.15.58.128 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [08:40:47] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [08:40:51] FIRING: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b12-drmrs:et-0/0/48 (Core: cr1-drmrs:et-0/0/1 {#D0100}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [08:42:35] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [08:44:37] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [08:44:39] FIRING: [6x] CoreBGPDown: Core BGP session down between asw1-b12-drmrs and cr1-drmrs (185.15.58.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [08:45:10] FIRING: [6x] BFDdown: BFD session down between cr2-drmrs and 185.15.58.128 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [08:45:30] router is back up waiting for the interfaces to initialize [08:46:43] RECOVERY - OSPF status on cr2-drmrs is OK: OSPFv2: 5/5 UP : OSPFv3: 5/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [08:46:45] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 11/11 UP : OSPFv3: 11/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [08:47:20] !log klausman@cumin1003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-staging-worker [08:47:21] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet [08:48:09] !log un-drain cr1-drmrs - T431749 [08:48:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:48:12] T431749: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749 [08:49:39] RESOLVED: [6x] CoreBGPDown: Core BGP session down between asw1-b12-drmrs and cr1-drmrs (185.15.58.142) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [08:50:10] RESOLVED: [6x] BFDdown: BFD session down between cr2-drmrs and 185.15.58.128 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [08:50:25] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [08:50:51] RESOLVED: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b12-drmrs:et-0/0/48 (Core: cr1-drmrs:et-0/0/1 {#D0100}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [08:52:22] all done for cr1-drmrs, pushing the sofware update to cr2-drmrs [08:52:52] (03CR) 10JMeybohm: "Good find! I've re-opened https://phabricator.wikimedia.org/T388390 for the asset.rb change since the workaround was introduced at part of" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318569 (https://phabricator.wikimedia.org/T388390) (owner: 10Jelto) [08:55:55] (03CR) 10Ladsgroup: [C:03+1] "Will deploy later" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1314716 (https://phabricator.wikimedia.org/T119117) (owner: 10SomeRandomDeveloper) [08:57:29] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet [09:01:31] (03PS2) 10Cparle: Add key to X-Analytics if a thumbnail was generated during the request [puppet] - 10https://gerrit.wikimedia.org/r/1318088 (https://phabricator.wikimedia.org/T433075) [09:04:08] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet [09:04:10] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet [09:04:14] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2002.codfw.wmnet [09:06:08] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2161: Maintenance [09:06:31] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance [09:06:39] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2163 (T431660)', diff saved to https://phabricator.wikimedia.org/P95320 and previous config saved to /var/cache/conftool/dbconfig/20260728-090638-cwilliams.json [09:06:56] !log draining cr2-drmrs - T431749 [09:07:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:07:00] T431749: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749 [09:09:38] (03PS3) 10Btullis: postgresql-superset-metrics: issue the server certificate from the WMF PKI [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318168 (https://phabricator.wikimedia.org/T432104) [09:10:45] !log ayounsi@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-drmrs,cr2-drmrs IPv6,cr2-drmrs.mgmt with reason: router upgrade [09:10:59] 06SRE, 06Infrastructure-Foundations, 10netops: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749#12162043 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=5c71b24d-45b0-4c80-bf42-231dd7bab42a) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their servi... [09:11:04] !log rebooting cr2-drmrs - T431749 [09:11:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:12:45] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2163: Maintenance [09:14:23] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2002.codfw.wmnet [09:14:43] PROBLEM - OSPF status on cr2-eqdfw is CRITICAL: OSPFv2: 6/7 UP : OSPFv3: 6/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:14:45] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 10/11 UP : OSPFv3: 10/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:14:49] (03CR) 10Ayounsi: [C:03+2] "self merging to deploy while drmrs is depooled and cr2-drmrs is being upgraded." [homer/public] - 10https://gerrit.wikimedia.org/r/1318659 (https://phabricator.wikimedia.org/T320264) (owner: 10Ayounsi) [09:14:59] (03PS1) 10Kevin Bazira: ml-services: deploy tts-section-generator in prod [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318668 (https://phabricator.wikimedia.org/T430536) [09:15:09] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2182: Maintenance [09:15:19] PROBLEM - OSPF status on cr1-eqiad is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:15:39] FIRING: [3x] CoreBGPDown: Core BGP session down between asw1-b12-drmrs and cr2-drmrs (185.15.58.140) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:15:50] (03PS1) 10Ozge: ml-services: Update jina-embeddings-staging to transformers image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318669 (https://phabricator.wikimedia.org/T432717) [09:16:10] FIRING: [6x] BFDdown: BFD session down between cr1-eqiad and 185.15.58.146 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:16:24] !log cmooney@cumin1003 START - Cookbook sre.network.tls for network device lsw1-f4-codfw [09:16:30] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2210: Maintenance [09:16:51] FIRING: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b12-drmrs:et-0/0/50 (Core: cr2-drmrs:et-0/0/2 {#D0103}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [09:17:25] !log cmooney@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f4-codfw [09:17:28] !log cmooney@cumin1003 START - Cookbook sre.network.tls for network device lsw1-e5-codfw [09:17:34] !log cmooney@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e5-codfw [09:17:37] !log cmooney@cumin1003 START - Cookbook sre.network.tls for network device lsw1-e2-codfw [09:17:49] !log cmooney@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e2-codfw [09:17:51] !log cmooney@cumin1003 START - Cookbook sre.network.tls for network device lsw1-e4-codfw [09:18:16] (03Merged) 10jenkins-bot: cr2-drmrs: Enable BGP RIB Sharding [homer/public] - 10https://gerrit.wikimedia.org/r/1318659 (https://phabricator.wikimedia.org/T320264) (owner: 10Ayounsi) [09:18:22] !log root@cumin1003 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2163: Maintenance [09:18:30] !log cmooney@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e4-codfw [09:18:33] !log cmooney@cumin1003 START - Cookbook sre.network.tls for network device lsw1-f2-codfw [09:19:23] !log cmooney@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f2-codfw [09:19:51] FIRING: CoreOutboundSaturation: Core link outbound traffic above 90% capacity - asw1-by27-esams:ae0 (Core: ... [09:19:51] cr1-esams) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#Primary_outbound_port_utilization_over_90% - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=esams+prometheus%2Fops&var-device=asw1-by27-esams:9804&var-interface=ae0 - https://alerts.wikimedia.org/?q=alertname%3DCoreOutboundSaturation [09:20:15] !ack [09:20:16] All incidents are already acked. [09:20:16] !ack [09:20:16] All incidents are already acked. [09:20:24] XioNoX: can that be related to your operation? ^ [09:20:39] FIRING: [6x] CoreBGPDown: Core BGP session down between asw1-b12-drmrs and cr2-drmrs (185.15.58.140) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:20:54] Like esams getting all the traffic from the depooled dc? [09:20:54] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2002.codfw.wmnet [09:20:56] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2002.codfw.wmnet [09:21:00] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet [09:22:04] marostegui: yeah but last time I checked esams could handle it well [09:22:10] looking [09:22:15] thanks [09:22:18] timing lines up really well on the outage front door graph [09:22:19] RECOVERY - OSPF status on cr1-eqiad is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:22:27] eqsin collapsing as esams takes over [09:22:28] !log cmooney@cumin1003 START - Cookbook sre.network.tls for network device lsw1-d8-eqiad [09:22:41] !log cmooney@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d8-eqiad [09:22:43] RECOVERY - OSPF status on cr2-eqdfw is OK: OSPFv2: 7/7 UP : OSPFv3: 7/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:22:45] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 11/11 UP : OSPFv3: 11/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:22:54] !log klausman@cumin1003 START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2001.codfw.wmnet [09:22:55] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Update jina-embeddings-staging to transformers image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318669 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [09:22:55] marostegui: repooling drmrs very soon [09:22:56] !log klausman@cumin1003 END (ERROR) - Cookbook sre.ganeti.reboot-vm (exit_code=97) for VM ml-serve-ctrl2001.codfw.wmnet [09:23:01] XioNoX: ok thanks [09:23:49] (03CR) 10Kevin Bazira: [C:03+2] ml-services: deploy tts-section-generator in prod [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318668 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [09:23:58] FIRING: [13x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [09:23:58] *drmrs falls, not eqsin [09:24:51] RESOLVED: CoreOutboundSaturation: Core link outbound traffic above 90% capacity - asw1-by27-esams:ae0 (Core: ... [09:24:51] cr1-esams) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#Primary_outbound_port_utilization_over_90% - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=esams+prometheus%2Fops&var-device=asw1-by27-esams:9804&var-interface=ae0 - https://alerts.wikimedia.org/?q=alertname%3DCoreOutboundSaturation [09:25:05] the recovery arrived [09:25:39] RESOLVED: [6x] CoreBGPDown: Core BGP session down between asw1-b12-drmrs and cr2-drmrs (185.15.58.140) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:26:08] (03Merged) 10jenkins-bot: ml-services: deploy tts-section-generator in prod [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318668 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [09:26:10] RESOLVED: [6x] BFDdown: BFD session down between cr1-eqiad and 185.15.58.146 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:26:51] RESOLVED: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b12-drmrs:et-0/0/50 (Core: cr2-drmrs:et-0/0/2 {#D0103}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [09:26:53] (03PS4) 10Marostegui: sre.mysql.test-cluster: adding new cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1315796 (https://phabricator.wikimedia.org/T433044) (owner: 10CWilliams) [09:26:53] (03CR) 10Marostegui: "As discussed yesterday in team meeting and today with Ceri, "ignoring" the __init__ I do find this readable and overall clear." [cookbooks] - 10https://gerrit.wikimedia.org/r/1315796 (https://phabricator.wikimedia.org/T433044) (owner: 10CWilliams) [09:27:03] (03CR) 10Ozge: [C:03+2] ml-services: Update jina-embeddings-staging to transformers image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318669 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [09:27:23] marostegui: from a quick look I don't see any a real saturation, I'll have a closer look after the maintenance, drmrs is ready to be repooled, just doing some cleanup to let it sit for a few min [09:27:43] (03CR) 10Ozge: [V:03+2 C:03+2] ml-services: Update jina-embeddings-staging to transformers image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318669 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [09:27:45] XioNoX: got it thanks. We got a recovery a a couple of mins ago [09:27:52] thx [09:27:58] RESOLVED: CertAlmostExpired: Certificate for service lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#lsw1-d8-eqiad.mgmt.eqiad.wmnet:57400 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [09:28:10] FIRING: BFDdown: BFD session down between cr1-eqiad and fe80::a6e1:1a00:106f:d3a3 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:29:10] 06SRE, 06Infrastructure-Foundations, 10netops: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749#12162103 (10ayounsi) [09:29:24] 06SRE, 06Infrastructure-Foundations, 10netops: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749#12162106 (10ayounsi) 05Open→03Resolved a:03ayounsi All done. [09:29:39] (03PS17) 10Federico Ceratto: cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) [09:30:14] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [09:30:26] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . [09:30:36] !log ayounsi@cumin1003 START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: router upgrade, T431749] [09:30:38] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2163: Maintenance [09:30:40] T431749: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749 [09:30:41] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: router upgrade, T431749] [09:30:52] (03CR) 10Btullis: [C:03+2] cloudnative-pg-cluster: allow issuing the server certificate from the WMF PKI [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318167 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [09:31:08] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet [09:31:16] (03CR) 10Btullis: [C:03+2] postgresql-superset-metrics: issue the server certificate from the WMF PKI [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318168 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [09:32:25] !log kevinbazira@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [09:33:10] RESOLVED: BFDdown: BFD session down between cr1-eqiad and fe80::a6e1:1a00:106f:d3a3 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:34:11] (03Merged) 10jenkins-bot: cloudnative-pg-cluster: allow issuing the server certificate from the WMF PKI [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318167 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [09:34:20] (03Merged) 10jenkins-bot: postgresql-superset-metrics: issue the server certificate from the WMF PKI [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318168 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [09:37:07] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet [09:37:08] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet [09:37:08] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-staging-worker [09:38:49] (03PS1) 10Jgiannelos: kartotherian: Pin staging to latest image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318672 [09:46:02] (03CR) 10Federico Ceratto: "Done" [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [09:47:16] (03CR) 10Federico Ceratto: "As discussed on Monday's team meeting I updated the CR after rebase and cleanups, now it's ready for the final review." [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [09:50:16] (03PS1) 10CWilliams: profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1318675 (https://phabricator.wikimedia.org/T433344) [09:50:27] (03PS1) 10Ladsgroup: thumbor: Move more operations to asyncio [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318677 (https://phabricator.wikimedia.org/T431767) [09:50:53] (03CR) 10CI reject: [V:04-1] profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1318675 (https://phabricator.wikimedia.org/T433344) (owner: 10CWilliams) [09:51:39] (03PS2) 10CWilliams: profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1318675 (https://phabricator.wikimedia.org/T433344) [09:52:18] (03CR) 10CI reject: [V:04-1] profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1318675 (https://phabricator.wikimedia.org/T433344) (owner: 10CWilliams) [10:00:04] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T1000) [10:03:06] (03CR) 10Jgiannelos: [C:03+2] kartotherian: Pin staging to latest image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318672 (owner: 10Jgiannelos) [10:03:07] (03CR) 10Marostegui: "From my side it is all fine, I also tested it in May, I assume functionality hasn't changed." [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [10:03:49] (03PS2) 10Jelto: helmfile.d/* use helm3.17 and helm3.19 helmBinary [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318569 (https://phabricator.wikimedia.org/T388390) [10:04:18] (03CR) 10CI reject: [V:04-1] helmfile.d/* use helm3.17 and helm3.19 helmBinary [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318569 (https://phabricator.wikimedia.org/T388390) (owner: 10Jelto) [10:05:27] (03Merged) 10jenkins-bot: kartotherian: Pin staging to latest image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318672 (owner: 10Jgiannelos) [10:08:58] FIRING: [12x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:10:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:13:39] (03CR) 10Jelto: "I think this needs Idd8b363553912393a8628ec792248a2e251ddf77 to make CI happy" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318569 (https://phabricator.wikimedia.org/T388390) (owner: 10Jelto) [10:13:58] FIRING: [12x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:14:27] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:16:08] (03PS1) 10Marostegui: check_private_data_report: Move clouddb1026 [puppet] - 10https://gerrit.wikimedia.org/r/1318681 (https://phabricator.wikimedia.org/T432745) [10:16:08] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2163: Maintenance [10:17:15] (03CR) 10Marostegui: [C:03+2] check_private_data_report: Move clouddb1026 [puppet] - 10https://gerrit.wikimedia.org/r/1318681 (https://phabricator.wikimedia.org/T432745) (owner: 10Marostegui) [10:17:29] (03PS2) 10Jforrester: logging: Switch the wmfconfig processor to Monolog 3's type (LogRecord) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307506 (https://phabricator.wikimedia.org/T397070) [10:17:31] Is the MW window being used? If not I have a MW prod logging patch to deploy. [10:19:24] Taking silence as assent. ;-) [10:19:28] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jforrester@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307506 (https://phabricator.wikimedia.org/T397070) (owner: 10Jforrester) [10:20:03] !log jiji@cumin1003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1137.eqiad.wmnet [10:20:07] !log jiji@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1137.eqiad.wmnet [10:20:33] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/kartotherian: apply [10:20:40] !log jiji@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1137.eqiad.wmnet [10:21:11] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/kartotherian: apply [10:21:22] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply [10:21:26] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-superset-metrics: apply [10:23:00] (03PS2) 10Blake: mw-pretrain: Add a canary-debug values.yaml. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318671 (https://phabricator.wikimedia.org/T427668) [10:23:40] jiji@cumin1003 renumber-node (PID 3081527) is awaiting input [10:24:02] (03Merged) 10jenkins-bot: logging: Switch the wmfconfig processor to Monolog 3's type (LogRecord) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307506 (https://phabricator.wikimedia.org/T397070) (owner: 10Jforrester) [10:25:21] !log jforrester@deploy1003 Started scap sync-world: Backport for [[gerrit:1307506|logging: Switch the wmfconfig processor to Monolog 3's type (LogRecord) (T397070)]] [10:25:26] T397070: Upgrade monolog/monolog to 3.9.0 or later - https://phabricator.wikimedia.org/T397070 [10:25:37] (03CR) 10Elukey: [C:03+2] redfish: expand force_http_boot_once for RedfishSupermicro [software/spicerack] - 10https://gerrit.wikimedia.org/r/1315812 (https://phabricator.wikimedia.org/T394357) (owner: 10Elukey) [10:27:34] !log jforrester@deploy1003 jforrester: Backport for [[gerrit:1307506|logging: Switch the wmfconfig processor to Monolog 3's type (LogRecord) (T397070)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [10:28:33] !log jforrester@deploy1003 jforrester: Continuing with deployment [10:30:52] 06SRE, 10Maps, 07affects-Kiwix-and-openZIM, 06Content-Platform-Team (Work In Progress), 07Essential-Work: Wikipedia wikis have broken maps URLs in infobox: "Bad GeoJSON - unknown \"type\" property \"ExternalData\"" - https://phabricator.wikimedia.org/T424046#12162317 (10Jgiannelos) Ι tried the workaround... [10:34:24] !log jiji@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-worker1137.eqiad.wmnet with OS trixie [10:34:36] !log klausman@cumin1003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-codfw [10:34:41] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2001.codfw.wmnet [10:34:52] !log jiji@cumin1003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1137 [10:34:53] !log jforrester@deploy1003 Finished scap sync-world: Backport for [[gerrit:1307506|logging: Switch the wmfconfig processor to Monolog 3's type (LogRecord) (T397070)]] (duration: 09m 31s) [10:34:58] T397070: Upgrade monolog/monolog to 3.9.0 or later - https://phabricator.wikimedia.org/T397070 [10:35:03] !log jiji@cumin1003 START - Cookbook sre.dns.netbox [10:38:46] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313#12162341 (10Aklapper) > Wikimedia developer account username: BGerdemann_(WMF) That might be incorrect per https://ldap.toolforge.org/user/bgwiki ? [10:38:51] (03PS1) 10Kevin Bazira: Add tts-section-generator CNAMEs to k8s-ingress-ml-serve [dns] - 10https://gerrit.wikimedia.org/r/1318685 (https://phabricator.wikimedia.org/T430536) [10:39:21] (03PS1) 10Blake: httpbb: Add a --request-header argument. [software/httpbb] - 10https://gerrit.wikimedia.org/r/1318682 (https://phabricator.wikimedia.org/T428972) [10:39:48] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2001.codfw.wmnet [10:41:04] jiji@cumin1003 renumber-node (PID 3081527) is awaiting input [10:43:13] (03PS1) 10Btullis: dse-k8s: Sort the discovery records alphabetically [dns] - 10https://gerrit.wikimedia.org/r/1318686 (https://phabricator.wikimedia.org/T432104) [10:43:16] (03PS1) 10Btullis: dse-k8s: Add a discovery record for postgresql-superset-metrics [dns] - 10https://gerrit.wikimedia.org/r/1318687 (https://phabricator.wikimedia.org/T432104) [10:45:55] (03CR) 10Kevin Bazira: "For more context, we can access this service in staging using:" [dns] - 10https://gerrit.wikimedia.org/r/1318685 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [10:47:10] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2001.codfw.wmnet [10:47:12] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2001.codfw.wmnet [10:47:18] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2002.codfw.wmnet [10:50:10] (03PS1) 10Jgiannelos: kartotherian: Bump prod image to latest [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318688 [10:52:26] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2002.codfw.wmnet [10:53:14] (03CR) 10Klausman: [C:03+1] dse-k8s: Add a discovery record for postgresql-superset-metrics [dns] - 10https://gerrit.wikimedia.org/r/1318687 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [10:53:23] (03PS4) 10CWilliams: profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1318675 (https://phabricator.wikimedia.org/T433344) [10:53:23] (03CR) 10CWilliams: "This seems to be the least disruptive way to disable the check whilst leaving everything in place should we wish to enable it later, e.g. " [puppet] - 10https://gerrit.wikimedia.org/r/1318675 (https://phabricator.wikimedia.org/T433344) (owner: 10CWilliams) [10:53:47] (03CR) 10CI reject: [V:04-1] profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1318675 (https://phabricator.wikimedia.org/T433344) (owner: 10CWilliams) [10:53:53] !log jiji@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1137 - jiji@cumin1003" [10:54:46] (03CR) 10Klausman: [C:03+1] dse-k8s: Sort the discovery records alphabetically [dns] - 10https://gerrit.wikimedia.org/r/1318686 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [10:56:45] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance [10:56:53] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2219 (T431660)', diff saved to https://phabricator.wikimedia.org/P95330 and previous config saved to /var/cache/conftool/dbconfig/20260728-105652-cwilliams.json [10:56:58] jiji@cumin1003 renumber-node (PID 3081527) is awaiting input [10:57:04] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance [10:57:12] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2208 (T431660)', diff saved to https://phabricator.wikimedia.org/P95331 and previous config saved to /var/cache/conftool/dbconfig/20260728-105711-cwilliams.json [10:57:41] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance [10:57:49] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2164 (T431660)', diff saved to https://phabricator.wikimedia.org/P95332 and previous config saved to /var/cache/conftool/dbconfig/20260728-105749-cwilliams.json [10:58:35] (03CR) 10Marostegui: "I'd push this change, let it spread for a day so it gets removed from icinga via "absent" and then simply delete the whole block of code i" [puppet] - 10https://gerrit.wikimedia.org/r/1318675 (https://phabricator.wikimedia.org/T433344) (owner: 10CWilliams) [10:59:15] (03CR) 10Marostegui: [C:03+1] "+1ing as the test failure is due to commit message and not any of the functionality." [puppet] - 10https://gerrit.wikimedia.org/r/1318675 (https://phabricator.wikimedia.org/T433344) (owner: 10CWilliams) [11:03:29] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2208: Maintenance [11:03:56] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2002.codfw.wmnet [11:03:58] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2002.codfw.wmnet [11:04:01] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2164: Maintenance [11:04:04] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2003.codfw.wmnet [11:04:24] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2219: Maintenance [11:06:40] (03PS5) 10CWilliams: profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1318675 (https://phabricator.wikimedia.org/T433344) [11:07:06] (03PS6) 10CWilliams: profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1318675 (https://phabricator.wikimedia.org/T433344) [11:09:02] (03PS7) 10CWilliams: profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1318675 (https://phabricator.wikimedia.org/T433344) [11:09:10] FIRING: BFDdown: BFD session down between cr2-eqdfw and fe80::b6f9:5dff:fe30:e538 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:10:00] !log root@cumin1003 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2219: Maintenance [11:11:24] 06SRE, 10Infrastructure Security, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12162401 (10klausman) [11:11:33] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2219: Maintenance [11:12:12] 06SRE, 10Infrastructure Security, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12162409 (10klausman) LW staging workers (ml-staging2xxx) are all done, proceeding wo... [11:14:10] RESOLVED: BFDdown: BFD session down between cr2-eqdfw and fe80::b6f9:5dff:fe30:e538 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:14:12] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2003.codfw.wmnet [11:15:49] (03CR) 10CWilliams: [C:03+2] profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1318675 (https://phabricator.wikimedia.org/T433344) (owner: 10CWilliams) [11:18:09] (03PS28) 10Federico Ceratto: mysql: update replication source [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) [11:19:46] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2003.codfw.wmnet [11:19:48] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2003.codfw.wmnet [11:19:54] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2004.codfw.wmnet [11:20:40] FIRING: [2x] BFDdown: BFD session down between cr2-eqdfw and fe80::b6f9:5dff:fe30:e538 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:24:33] (03CR) 10Kamila Součková: [C:03+1] "Thank you!" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1318289 (https://phabricator.wikimedia.org/T433312) (owner: 10Krinkle) [11:25:03] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2004.codfw.wmnet [11:25:40] RESOLVED: [2x] BFDdown: BFD session down between cr2-eqdfw and fe80::b6f9:5dff:fe30:e538 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:30:01] !log jiji@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1137 - jiji@cumin1003" [11:30:01] !log jiji@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:30:01] !log jiji@cumin1003 START - Cookbook sre.dns.wipe-cache wikikube-worker1137.eqiad.wmnet 192.32.64.10.in-addr.arpa 2.9.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [11:30:05] !log jiji@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1137.eqiad.wmnet 192.32.64.10.in-addr.arpa 2.9.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [11:30:05] !log jiji@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1137 [11:30:46] !log jiji@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1137 [11:30:46] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1137 [11:33:00] (03CR) 10Krinkle: "Go ahead. (I don't actually have +2 or deploy access for this repo.)" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1318289 (https://phabricator.wikimedia.org/T433312) (owner: 10Krinkle) [11:35:24] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2004.codfw.wmnet [11:35:26] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2004.codfw.wmnet [11:35:31] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet [11:40:09] (03CR) 10Jgiannelos: [C:03+2] kartotherian: Bump prod image to latest [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318688 (owner: 10Jgiannelos) [11:40:38] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet [11:42:27] (03Merged) 10jenkins-bot: kartotherian: Bump prod image to latest [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318688 (owner: 10Jgiannelos) [11:45:47] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet [11:45:48] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet [11:45:54] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2006.codfw.wmnet [11:47:20] !log jiji@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1137.eqiad.wmnet with reason: host reimage [11:50:28] PROBLEM - Host wikikube-worker1137 is DOWN: PING CRITICAL - Packet loss = 100% [11:50:52] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2164: Maintenance [11:51:13] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance [11:51:20] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2166 (T431660)', diff saved to https://phabricator.wikimedia.org/P95342 and previous config saved to /var/cache/conftool/dbconfig/20260728-115119-cwilliams.json [11:51:27] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2208: Maintenance [11:51:48] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance [11:51:56] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2218 (T431660)', diff saved to https://phabricator.wikimedia.org/P95344 and previous config saved to /var/cache/conftool/dbconfig/20260728-115155-cwilliams.json [11:52:27] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1137.eqiad.wmnet with reason: host reimage [11:55:31] RECOVERY - Host wikikube-worker1137 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [11:56:02] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2006.codfw.wmnet [11:57:01] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2219: Maintenance [11:57:24] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2166: Maintenance [11:58:02] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2218: Maintenance [11:59:32] FIRING: [2x] ProbeDown: Service wdqs2012:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs2012:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:00:04] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T1200) [12:00:23] RECOVERY - jenkins_service_running on contint1002 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [12:01:11] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2006.codfw.wmnet [12:01:12] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2006.codfw.wmnet [12:01:18] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2007.codfw.wmnet [12:02:45] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance [12:02:54] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2236 (T431660)', diff saved to https://phabricator.wikimedia.org/P95348 and previous config saved to /var/cache/conftool/dbconfig/20260728-120253-cwilliams.json [12:03:23] PROBLEM - jenkins_service_running on contint1002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [12:06:25] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2007.codfw.wmnet [12:08:44] (03CR) 10Ladsgroup: [C:03+1] "will merge later today" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1311830 (https://phabricator.wikimedia.org/T430528) (owner: 10Samwilson) [12:09:16] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2236: Maintenance [12:11:34] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2007.codfw.wmnet [12:11:36] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2007.codfw.wmnet [12:11:42] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2008.codfw.wmnet [12:12:53] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1137.eqiad.wmnet with OS trixie [12:12:58] (03PS1) 10Ozge: ml-services: Update jina-embeddings model version in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318695 (https://phabricator.wikimedia.org/T432717) [12:14:08] (03CR) 10Ozge: "new model version has all files except for onnx and gguf ones." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318695 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [12:14:53] !log root@cumin1003 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2236: Maintenance [12:14:59] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2236: Maintenance [12:16:07] !log pt1979@cumin1003 START - Cookbook sre.network.tls for network device asw1-604-eqsin [12:16:48] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2008.codfw.wmnet [12:16:53] !log pt1979@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device asw1-604-eqsin [12:18:24] (03PS2) 10Anzx: bolwiki: add logo, sitename, projectnamespace and timezone [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313960 (https://phabricator.wikimedia.org/T429951) [12:18:28] (03PS3) 10Anzx: magwiki: update logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315908 (https://phabricator.wikimedia.org/T433011) [12:18:43] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, July 28 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315908 (https://phabricator.wikimedia.org/T433011) (owner: 10Anzx) [12:18:52] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, July 28 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313960 (https://phabricator.wikimedia.org/T429951) (owner: 10Anzx) [12:21:57] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2008.codfw.wmnet [12:21:59] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2008.codfw.wmnet [12:22:04] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2009.codfw.wmnet [12:27:12] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2009.codfw.wmnet [12:28:39] (03CR) 10Krinkle: wmf-config/logging.php: Adjustments for WMF_MAINTENANCE_OFFLINE (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315968 (https://phabricator.wikimedia.org/T433104) (owner: 10Ahmon Dancy) [12:30:47] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12162617 (10Papaul) [12:31:08] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12162620 (10Papaul) [12:31:52] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2009.codfw.wmnet [12:31:54] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2009.codfw.wmnet [12:31:59] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2010.codfw.wmnet [12:32:06] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/kartotherian: apply [12:32:22] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/kartotherian: apply [12:32:27] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/kartotherian: apply [12:32:30] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/kartotherian: apply [12:32:37] !log jgiannelos@deploy1003 helmfile [codfw] START helmfile.d/services/kartotherian: apply [12:33:32] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12162627 (10Papaul) First homer run is getting me this error ` ERROR:homer:Device asw1-603-eqsin.mgmt.eqsin.wmnet failed to render the template, skipping.... [12:34:54] !log pt1979@cumin2003 START - Cookbook sre.dns.netbox [12:37:05] (03PS1) 10Elukey: install_server: add sretest2010 as swift node [puppet] - 10https://gerrit.wikimedia.org/r/1318701 (https://phabricator.wikimedia.org/T394357) [12:37:07] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2010.codfw.wmnet [12:38:09] /l/ [12:39:16] (03PS2) 10Elukey: install_server: add sretest2010 as swift node [puppet] - 10https://gerrit.wikimedia.org/r/1318701 (https://phabricator.wikimedia.org/T394357) [12:39:25] 10ops-eqiad, 06DC-Ops: Unresponsive management for ms-be1065.mgmt:22 - https://phabricator.wikimedia.org/T433368 (10phaultfinder) 03NEW [12:40:16] (03CR) 10Kamila Součková: [C:03+2] php: Implement php_apcu_cache_entries gauge [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1318289 (https://phabricator.wikimedia.org/T433312) (owner: 10Krinkle) [12:41:01] pt1979@cumin2003 netbox (PID 1115999) is awaiting input [12:41:12] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Update jina-embeddings model version in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318695 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [12:41:37] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2010.codfw.wmnet [12:41:39] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2010.codfw.wmnet [12:41:44] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2011.codfw.wmnet [12:42:53] !log jgiannelos@deploy1003 helmfile [codfw] DONE helmfile.d/services/kartotherian: apply [12:43:15] (03CR) 10Kamila Součková: [V:03+2 C:03+2] php: Implement php_apcu_cache_entries gauge [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1318289 (https://phabricator.wikimedia.org/T433312) (owner: 10Krinkle) [12:43:15] (03PS2) 10Majavah: P:toolforge::bastion: Drop grid engine script shims [puppet] - 10https://gerrit.wikimedia.org/r/1318226 [12:43:16] (03PS2) 10Majavah: P:toolforge::bastion: Minor cleanup [puppet] - 10https://gerrit.wikimedia.org/r/1318227 [12:43:16] (03PS2) 10Majavah: P:toolforge::bastion: Add script to migrate OpenSearch indexes [puppet] - 10https://gerrit.wikimedia.org/r/1318228 (https://phabricator.wikimedia.org/T401818) [12:44:29] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2166: Maintenance [12:44:50] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance [12:44:58] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2167 (T431660)', diff saved to https://phabricator.wikimedia.org/P95354 and previous config saved to /var/cache/conftool/dbconfig/20260728-124457-cwilliams.json [12:45:31] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: Maintenance [12:45:54] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance [12:46:02] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2221 (T431660)', diff saved to https://phabricator.wikimedia.org/P95357 and previous config saved to /var/cache/conftool/dbconfig/20260728-124601-cwilliams.json [12:46:51] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2011.codfw.wmnet [12:47:26] (03PS1) 10Lucas Werkmeister (WMDE): Add i18n/LanguageNames to MessagesDirs [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318704 (https://phabricator.wikimedia.org/T231755) [12:47:28] (03PS1) 10Lucas Werkmeister (WMDE): Load language names from JSON instead of PHP [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318705 (https://phabricator.wikimedia.org/T231755) [12:47:31] (03PS1) 10Lucas Werkmeister (WMDE): Update rebuild.php to also write message JSON files [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318706 (https://phabricator.wikimedia.org/T231755) [12:47:34] (03PS1) 10Lucas Werkmeister (WMDE): Optimize language name loading with fallbacks [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318707 (https://phabricator.wikimedia.org/T231755) [12:47:36] (03PS1) 10Lucas Werkmeister (WMDE): Remove most of LocalNames [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318708 (https://phabricator.wikimedia.org/T231755) [12:47:39] (03PS1) 10Lucas Werkmeister (WMDE): Remove T231755.php again [extensions/cldr] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318709 (https://phabricator.wikimedia.org/T231755) [12:47:57] (03PS1) 10Elukey: admin_ng: bump kartotherian's ns cpu limit to allow deployments [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318710 [12:48:18] (03PS1) 10Papaul: Add srlinux_version for eqsin new swiches [homer/public] - 10https://gerrit.wikimedia.org/r/1318711 (https://phabricator.wikimedia.org/T418439) [12:48:33] !log pt1979@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add loopback for asw1-603 - pt1979@cumin2003" [12:48:43] (03CR) 10MVernon: "Hi," [puppet] - 10https://gerrit.wikimedia.org/r/1318701 (https://phabricator.wikimedia.org/T394357) (owner: 10Elukey) [12:48:59] PROBLEM - Druid historical on an-druid1007 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [12:49:15] !log pt1979@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add loopback for asw1-603 - pt1979@cumin2003" [12:49:15] !log pt1979@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:50:25] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [12:51:20] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2011.codfw.wmnet [12:51:22] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2011.codfw.wmnet [12:51:22] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-codfw [12:51:24] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2167: Maintenance [12:51:36] 06SRE, 10Infrastructure Security, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12162707 (10klausman) LW/ML eqiad workers all done. [12:51:54] (03CR) 10Cathal Mooney: [C:03+1] Add srlinux_version for eqsin new swiches [homer/public] - 10https://gerrit.wikimedia.org/r/1318711 (https://phabricator.wikimedia.org/T418439) (owner: 10Papaul) [12:52:07] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance [12:52:15] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2153 (T431660)', diff saved to https://phabricator.wikimedia.org/P95358 and previous config saved to /var/cache/conftool/dbconfig/20260728-125214-cwilliams.json [12:52:15] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2221: Maintenance [12:52:27] (03CR) 10Papaul: [C:03+2] Add srlinux_version for eqsin new swiches [homer/public] - 10https://gerrit.wikimedia.org/r/1318711 (https://phabricator.wikimedia.org/T418439) (owner: 10Papaul) [12:53:46] (03PS3) 10Elukey: install_server: add sretest2010 as swift node [puppet] - 10https://gerrit.wikimedia.org/r/1318701 (https://phabricator.wikimedia.org/T394357) [12:54:36] (03PS4) 10Elukey: install_server: add sretest2010 as swift node [puppet] - 10https://gerrit.wikimedia.org/r/1318701 (https://phabricator.wikimedia.org/T394357) [12:55:08] (03PS5) 10Elukey: Set sretest2010 as test swift host for Data Persistence [puppet] - 10https://gerrit.wikimedia.org/r/1318701 (https://phabricator.wikimedia.org/T394357) [12:55:33] (03CR) 10Ozge: [V:03+2 C:03+2] ml-services: Update jina-embeddings model version in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318695 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [12:58:10] (03Merged) 10jenkins-bot: ml-services: Update jina-embeddings model version in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318695 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [12:58:11] (03CR) 10Btullis: [C:03+2] WDQSv2: scale resource limits to max with respect to the dedicated wdqs nodes. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313874 (https://phabricator.wikimedia.org/T431395) (owner: 10Trueg) [12:58:49] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2153: Maintenance [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: That opportune time for a UTC afternoon backport window deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T1300). [13:00:05] edsanders, AaronSchulz, and anzx: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:12] o/ can self deploy [13:00:12] o/ [13:00:20] I can deploy too [13:00:31] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9073/co" [puppet] - 10https://gerrit.wikimedia.org/r/1318139 (https://phabricator.wikimedia.org/T411248) (owner: 10Filippo Giunchedi) [13:00:38] I'll begin [13:00:39] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2236: Maintenance [13:00:43] yup, go ahead [13:00:52] (03CR) 10Majavah: [V:03+1 C:03+1] wmcs: add clouddumps compat symlinks [puppet] - 10https://gerrit.wikimedia.org/r/1318139 (https://phabricator.wikimedia.org/T411248) (owner: 10Filippo Giunchedi) [13:01:00] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance [13:01:01] o/ [13:01:04] (03CR) 10TrainBranchBot: [C:03+2] "Approved by esanders@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309727 (https://phabricator.wikimedia.org/T431858) (owner: 10Esanders) [13:01:08] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2237 (T431660)', diff saved to https://phabricator.wikimedia.org/P95362 and previous config saved to /var/cache/conftool/dbconfig/20260728-130107-cwilliams.json [13:01:38] and then I can deploy for anzx and would do that before the deployment(s) for AaronSchulz, because volunteer [13:02:10] (03Merged) 10jenkins-bot: Set wgMFFallbackEditor to 'visual' on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309727 (https://phabricator.wikimedia.org/T431858) (owner: 10Esanders) [13:02:24] (03CR) 10Filippo Giunchedi: [C:03+2] wmcs: add clouddumps compat symlinks [puppet] - 10https://gerrit.wikimedia.org/r/1318139 (https://phabricator.wikimedia.org/T411248) (owner: 10Filippo Giunchedi) [13:02:30] !log esanders@deploy1003 Started scap sync-world: Backport for [[gerrit:1309727|Set wgMFFallbackEditor to 'visual' on enwiki (T431858)]] [13:02:35] T431858: Make VisualEditor the default editor on enwiki's mobile web, especially for newbies and TAs - https://phabricator.wikimedia.org/T431858 [13:02:59] RECOVERY - Druid historical on an-druid1007 is OK: PROCS OK: 1 process with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [13:03:56] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12162771 (10Papaul) [13:04:26] !log root@cumin1003 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2153: Maintenance [13:04:32] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2153: Maintenance [13:04:41] !log esanders@deploy1003 esanders: Backport for [[gerrit:1309727|Set wgMFFallbackEditor to 'visual' on enwiki (T431858)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:05:39] testing... [13:06:22] !log esanders@deploy1003 esanders: Continuing with deployment [13:06:40] (03CR) 10Eevans: [C:03+2] restbase: Upgrade Cassandra to 5.0.8 [puppet] - 10https://gerrit.wikimedia.org/r/1315183 (https://phabricator.wikimedia.org/T433028) (owner: 10Eevans) [13:07:30] (03Merged) 10jenkins-bot: WDQSv2: scale resource limits to max with respect to the dedicated wdqs nodes. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1313874 (https://phabricator.wikimedia.org/T431395) (owner: 10Trueg) [13:07:41] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2237: Maintenance [13:08:15] !log pt1979@cumin2003 START - Cookbook sre.dns.netbox [13:10:42] !log esanders@deploy1003 Finished scap sync-world: Backport for [[gerrit:1309727|Set wgMFFallbackEditor to 'visual' on enwiki (T431858)]] (duration: 08m 11s) [13:10:47] T431858: Make VisualEditor the default editor on enwiki's mobile web, especially for newbies and TAs - https://phabricator.wikimedia.org/T431858 [13:11:36] !log bking@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [13:11:38] edsanders: all done? :) [13:11:40] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [13:11:42] !log bking@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 20s) [13:11:51] done [13:11:54] thank s [13:12:05] anzx: I assume it’s fine to deploy your two changes together? [13:12:16] yes [13:12:26] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315908 (https://phabricator.wikimedia.org/T433011) (owner: 10Anzx) [13:12:26] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313960 (https://phabricator.wikimedia.org/T429951) (owner: 10Anzx) [13:12:49] !log pt1979@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add loopback IPV6 for asw1-604 - pt1979@cumin2003" [13:12:52] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [13:12:53] !log pt1979@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add loopback IPV6 for asw1-604 - pt1979@cumin2003" [13:12:54] !log pt1979@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:13:12] !log bking@cumin2003 START - Cookbook sre.dns.netbox [13:13:19] !log root@cumin1003 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2237: Maintenance [13:13:25] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2237: Maintenance [13:13:40] (03Merged) 10jenkins-bot: magwiki: update logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315908 (https://phabricator.wikimedia.org/T433011) (owner: 10Anzx) [13:13:44] (03Merged) 10jenkins-bot: bolwiki: add logo, sitename, projectnamespace and timezone [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1313960 (https://phabricator.wikimedia.org/T429951) (owner: 10Anzx) [13:14:02] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1315908|magwiki: update logos (T433011)]], [[gerrit:1313960|bolwiki: add logo, sitename, projectnamespace and timezone (T429951)]] [13:14:07] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[2025-2038].codfw.wmnet,restbase[1032-1045].eqiad.wmnet: Upgrade Cassandra to 5.0.8 — T433028 - eevans@cumin1003 [13:14:08] T433011: Requesting logo for mag.wikimedia.org - https://phabricator.wikimedia.org/T433011 [13:14:08] T429951: Post-creation work for bolwiki - https://phabricator.wikimedia.org/T429951 [13:14:12] T433028: Upgrade restbase cluster to Cassandra 5.0.8 - https://phabricator.wikimedia.org/T433028 [13:16:03] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, anzx: Backport for [[gerrit:1315908|magwiki: update logos (T433011)]], [[gerrit:1313960|bolwiki: add logo, sitename, projectnamespace and timezone (T429951)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:16:08] checking [13:16:20] (03CR) 10Scott French: [C:03+2] wmnet: Temporarily direct codfw, eqsin, ulsfo etcd clients to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1314077 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [13:16:22] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:16:23] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache wdqs2012.codfw.wmnet 57.48.192.10.in-addr.arpa 7.5.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [13:16:26] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2012.codfw.wmnet 57.48.192.10.in-addr.arpa 7.5.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [13:16:27] !log bking@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wdqs2012 [13:16:40] !log swfrench@dns1004 START - running authdns-update [13:17:03] Lucas_WMDE: both changes looks good, ok to continue [13:17:07] thanks! [13:17:10] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, anzx: Continuing with deployment [13:18:03] (03CR) 10Effie Mouzeli: [C:03+1] admin_ng: bump kartotherian's ns cpu limit to allow deployments [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318710 (owner: 10Elukey) [13:18:55] !log swfrench@dns1004 END - running authdns-update [13:19:30] bking@cumin2003 reimage (PID 953155) is awaiting input [13:20:07] !log authdns-update to direct codfw, eqsin, ulsfo etcd clients to eqiad - T428495 [13:20:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:20:11] T428495: Migrate conf* hosts away from bullseye - https://phabricator.wikimedia.org/T428495 [13:21:21] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1315908|magwiki: update logos (T433011)]], [[gerrit:1313960|bolwiki: add logo, sitename, projectnamespace and timezone (T429951)]] (duration: 07m 19s) [13:21:32] T433011: Requesting logo for mag.wikimedia.org - https://phabricator.wikimedia.org/T433011 [13:21:33] T429951: Post-creation work for bolwiki - https://phabricator.wikimedia.org/T429951 [13:21:34] (03CR) 10Elukey: [C:03+2] admin_ng: bump kartotherian's ns cpu limit to allow deployments [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318710 (owner: 10Elukey) [13:21:46] !log lucaswerkmeister-wmde@deploy1003 mwscript-k8s job started: cleanupTitles bolwiki # T429951 [13:22:02] (^ just in case, though it turned out not to be needed, 0 of 17 rows updated) [13:22:04] AaronSchulz: over to you [13:22:11] anzx: should be done! [13:23:27] Lucas_WMDE: thanks for deploying [13:24:13] yw :) [13:27:02] !log bking@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2012 [13:27:02] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2012 [13:27:11] !log begin rolling restart of confd in codfw, eqsin, ulsfo - T428495 [13:27:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:27:15] T428495: Migrate conf* hosts away from bullseye - https://phabricator.wikimedia.org/T428495 [13:27:53] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1022.eqiad.wmnet, repooling source-only afterwards [13:27:58] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [13:28:52] PROBLEM - Host wdqs2012 is DOWN: PING CRITICAL - Packet loss = 100% [13:28:57] !log ayounsi@cumin1003 START - Cookbook sre.dns.admin DNS admin: depool esams [reason: router upgrade, T431749] [13:29:01] T431749: drmrs: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431749 [13:29:03] !log ayounsi@cumin1003 END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool esams [reason: router upgrade, T431749] [13:29:17] !log ayounsi@cumin1003 START - Cookbook sre.dns.admin DNS admin: depool esams [reason: router upgrade, T431751] [13:29:21] T431751: cr2-esams: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431751 [13:29:31] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: router upgrade, T431751] [13:30:21] !log restarted navtiming on webperf2003 - T428495 [13:30:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:37:25] (03CR) 10Filippo Giunchedi: [C:03+1] "TIL 'crontab' was overridden, cheeky" [puppet] - 10https://gerrit.wikimedia.org/r/1318226 (owner: 10Majavah) [13:37:35] (03CR) 10Btullis: [C:03+2] dse-k8s: Sort the discovery records alphabetically [dns] - 10https://gerrit.wikimedia.org/r/1318686 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [13:37:46] (03CR) 10Btullis: [C:03+2] dse-k8s: Add a discovery record for postgresql-superset-metrics [dns] - 10https://gerrit.wikimedia.org/r/1318687 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [13:38:02] (03CR) 10Ssingh: [C:03+2] hieradata: Temporarily point codfw PyBals at eqiad etcd [puppet] - 10https://gerrit.wikimedia.org/r/1314076 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [13:38:24] (03CR) 10Majavah: [C:03+2] P:toolforge::bastion: Drop grid engine script shims [puppet] - 10https://gerrit.wikimedia.org/r/1318226 (owner: 10Majavah) [13:38:28] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2167: Maintenance [13:38:50] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance [13:38:58] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2181 (T431660)', diff saved to https://phabricator.wikimedia.org/P95374 and previous config saved to /var/cache/conftool/dbconfig/20260728-133857-cwilliams.json [13:39:19] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2221: Maintenance [13:39:41] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance [13:39:49] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2222 (T431660)', diff saved to https://phabricator.wikimedia.org/P95376 and previous config saved to /var/cache/conftool/dbconfig/20260728-133948-cwilliams.json [13:42:09] !log btullis@dns1004 START - running authdns-update [13:42:20] !log restart pybal on lvs2014 [13:42:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:42:56] (03PS1) 10Andrew Bogott: scan-and-shrink.py: expand some variable names [puppet] - 10https://gerrit.wikimedia.org/r/1318720 [13:43:12] PROBLEM - PyBal connections to etcd on lvs2011 is CRITICAL: CRITICAL: 0 connections established with conf1008.eqiad.wmnet:4001 (min=12) https://wikitech.wikimedia.org/wiki/PyBal [13:43:31] uh interesting [13:43:32] PROBLEM - PyBal connections to etcd on lvs2012 is CRITICAL: CRITICAL: 0 connections established with conf1008.eqiad.wmnet:4001 (min=6) https://wikitech.wikimedia.org/wiki/PyBal [13:43:42] (03PS29) 10Federico Ceratto: mysql: update replication source [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) [13:43:54] (03CR) 10Federico Ceratto: "The value depends on the previous replication configuration. In some cases there will be obsoleted heartbeat rows from multiple hosts e.g." [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [13:44:19] !log btullis@dns1004 END - running authdns-update [13:44:25] (03CR) 10Andrew Bogott: [C:03+2] paws scan-and-shrink: fix logic error (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1314850 (owner: 10Andrew Bogott) [13:45:06] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2181: Maintenance [13:45:07] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2012.codfw.wmnet with reason: host reimage [13:45:21] !log restart pybal on A:lvs-codfw [13:45:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:46:09] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2222: Maintenance [13:46:47] (03CR) 10Filippo Giunchedi: [C:03+1] P:toolforge::bastion: Minor cleanup [puppet] - 10https://gerrit.wikimedia.org/r/1318227 (owner: 10Majavah) [13:47:27] (03CR) 10Fabfur: [C:03+1] wmnet/10.in-addr: add urldownloader service IPs [dns] - 10https://gerrit.wikimedia.org/r/1313290 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [13:48:12] RECOVERY - PyBal connections to etcd on lvs2011 is OK: OK: 12 connections established with conf1008.eqiad.wmnet:4001 (min=12) https://wikitech.wikimedia.org/wiki/PyBal [13:48:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:48:32] RECOVERY - PyBal connections to etcd on lvs2012 is OK: OK: 6 connections established with conf1008.eqiad.wmnet:4001 (min=6) https://wikitech.wikimedia.org/wiki/PyBal [13:49:04] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [13:49:17] RESOLVED: [2x] ProbeDown: Service wdqs2012:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs2012:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:49:52] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [13:50:03] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2012.codfw.wmnet with reason: host reimage [13:50:13] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2153: Maintenance [13:50:15] !log drain cr2-esams - T431751 [13:50:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:50:19] T431751: cr2-esams: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431751 [13:50:36] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance [13:50:44] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2170 (T431660)', diff saved to https://phabricator.wikimedia.org/P95381 and previous config saved to /var/cache/conftool/dbconfig/20260728-135043-cwilliams.json [13:51:00] !log sukhe@cumin1003 START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo (T428495) [13:51:04] T428495: Migrate conf* hosts away from bullseye - https://phabricator.wikimedia.org/T428495 [13:51:45] !log ayounsi@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr2-esams,cr2-esams IPv6,cr2-esams.mgmt with reason: router upgrade [13:51:58] 06SRE, 06Infrastructure-Foundations, 10netops: cr2-esams: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431751#12163043 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a2dbde41-3fc7-4120-8a34-b7696ca56502) set by ayounsi@cumin1003 for 1:00:00 on 3 host(s) and their servi... [13:52:52] !log sukhe@cumin1003 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo (T428495) [13:53:31] PROBLEM - Ensure trafficserver_exporter is running for instance backend on cp4048 is CRITICAL: PROCS CRITICAL: 2 processes with args /usr/bin/python3 /usr/bin/prometheus-trafficserver-exporter --no-procstats --no-ssl-verification --endpoint http://127.0.0.1:3128/_stats --port 9122 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [13:54:06] (03CR) 10Fabfur: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1313285 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [13:54:31] RECOVERY - Ensure trafficserver_exporter is running for instance backend on cp4048 is OK: PROCS OK: 1 process with args /usr/bin/python3 /usr/bin/prometheus-trafficserver-exporter --no-procstats --no-ssl-verification --endpoint http://127.0.0.1:3128/_stats --port 9122 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [13:55:37] !log reboot cr2-esams - T431751 [13:55:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:55:42] T431751: cr2-esams: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431751 [13:56:56] !log sukhe@cumin1003 START - Cookbook sre.loadbalancer.upgrade restart A:liberica-eqsin (T428495) [13:57:00] T428495: Migrate conf* hosts away from bullseye - https://phabricator.wikimedia.org/T428495 [13:57:20] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2170: Maintenance [13:58:12] (03PS1) 10Ayounsi: cr2-esams: enable BGP RIB sharding [homer/public] - 10https://gerrit.wikimedia.org/r/1318723 (https://phabricator.wikimedia.org/T320264) [13:58:27] PROBLEM - OSPF status on cr1-eqiad is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [13:58:49] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 10/11 UP : OSPFv3: 10/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [13:58:49] PROBLEM - OSPF status on cr2-eqdfw is CRITICAL: OSPFv2: 6/7 UP : OSPFv3: 6/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [13:58:58] (expected) [13:59:10] FIRING: [3x] BFDdown: BFD session down between cr1-esams and 185.15.59.129 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:59:39] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2237: Maintenance [13:59:39] FIRING: [3x] CoreBGPDown: Core BGP session down between asw1-by27-esams and cr2-esams (185.15.59.150) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:00:03] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2240.codfw.wmnet with reason: Maintenance [14:00:05] Deploy window Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T1400) [14:00:12] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2240 (T431660)', diff saved to https://phabricator.wikimedia.org/P95384 and previous config saved to /var/cache/conftool/dbconfig/20260728-140011-cwilliams.json [14:00:39] (03CR) 10Ayounsi: [C:03+2] "self merging to deploy while esams is depooled" [homer/public] - 10https://gerrit.wikimedia.org/r/1318723 (https://phabricator.wikimedia.org/T320264) (owner: 10Ayounsi) [14:00:51] FIRING: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-bw27-esams:et-0/0/50 (Core: cr2-esams:et-0/0/2 {#30369}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [14:01:07] !log sukhe@cumin1003 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-eqsin (T428495) [14:01:27] jiji@cumin1003 renumber-node (PID 3081527) is awaiting input [14:02:56] !log klausman@cumin1003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-worker-eqiad [14:02:59] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1001.eqiad.wmnet [14:03:01] PROBLEM - Router interfaces on mr1-eqsin is CRITICAL: CRITICAL: No response from remote host 103.102.166.128 for 1.3.6.1.2.1.2.2.1.8 with snmp version 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [14:03:33] (03Merged) 10jenkins-bot: cr2-esams: enable BGP RIB sharding [homer/public] - 10https://gerrit.wikimedia.org/r/1318723 (https://phabricator.wikimedia.org/T320264) (owner: 10Ayounsi) [14:03:34] router is up, waiting for the interfaces to initialize [14:03:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:04:10] FIRING: [6x] BFDdown: BFD session down between cr1-eqiad and 185.15.59.145 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:04:39] FIRING: [5x] CoreBGPDown: Core BGP session down between asw1-bw27-esams and cr2-esams (185.15.59.158) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:04:55] RECOVERY - Router interfaces on mr1-eqsin is OK: OK: host 103.102.166.128, interfaces up: 37, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [14:05:48] !log elukey@deploy1003 helmfile [codfw] START helmfile.d/admin 'sync'. [14:06:15] !log un-drain cr2-esams - T431751 [14:06:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:06:19] T431751: cr2-esams: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431751 [14:06:27] RECOVERY - OSPF status on cr1-eqiad is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:06:44] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2240: Maintenance [14:06:45] !log elukey@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'sync'. [14:06:49] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 11/11 UP : OSPFv3: 11/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:06:49] RECOVERY - OSPF status on cr2-eqdfw is OK: OSPFv2: 7/7 UP : OSPFv3: 7/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [14:07:31] !log elukey@deploy1003 helmfile [eqiad] START helmfile.d/admin 'sync'. [14:07:54] !log elukey@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'sync'. [14:08:01] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1001.eqiad.wmnet [14:08:13] (03CR) 10Scott French: [C:03+2] hieradata: Explicitly enable the v2 API on conf2004 [puppet] - 10https://gerrit.wikimedia.org/r/1314015 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [14:08:36] 06SRE, 06App Experience, 06Content-Platform-Team, 06ServiceOps new, and 2 others: Investigate Code 414 error when selecting zh-classical (lzh) language from article toolbar - https://phabricator.wikimedia.org/T425545#12163097 (10Seddon) [14:08:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:09:10] RESOLVED: [6x] BFDdown: BFD session down between cr1-eqiad and 185.15.59.145 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:09:39] RESOLVED: [5x] CoreBGPDown: Core BGP session down between asw1-bw27-esams and cr2-esams (185.15.59.158) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:10:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:10:51] RESOLVED: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-bw27-esams:et-0/0/50 (Core: cr2-esams:et-0/0/2 {#30369}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [14:11:16] !log ayounsi@cumin1003 START - Cookbook sre.dns.admin DNS admin: pool esams [reason: router upgrade, T431751] [14:11:18] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: router upgrade, T431751] [14:11:20] T431751: cr2-esams: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431751 [14:11:34] (03PS2) 10Hnowlan: ncredir: migrate nrpe check to alertmanager [puppet] - 10https://gerrit.wikimedia.org/r/1307419 (https://phabricator.wikimedia.org/T407117) [14:11:34] (03PS1) 10Hnowlan: ncredir: remove absented icinga check [puppet] - 10https://gerrit.wikimedia.org/r/1318724 (https://phabricator.wikimedia.org/T407117) [14:11:45] (03CR) 10Marostegui: "Maybe what we can do is , by default we don't do anything and if we want to clean it up we simply do a --delete-heartbeat and we issue a d" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [14:12:11] 06SRE, 06Infrastructure-Foundations, 10netops: cr2-esams: upgrade to Junos 23.4R2-S8 - https://phabricator.wikimedia.org/T431751#12163147 (10ayounsi) 05Open→03Resolved a:03ayounsi Upgraded. [14:13:21] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1001.eqiad.wmnet [14:13:22] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1001.eqiad.wmnet [14:13:28] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1002.eqiad.wmnet [14:13:35] 10SRE-SLO: Add alerts to detect missing sli error ratio rate metrics - https://phabricator.wikimedia.org/T432140#12163176 (10hnowlan) 05Open→03Resolved a:03tappof I think this is done, please reopen if not. [14:13:58] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:14:27] (03CR) 10MVernon: [C:03+1] "I think this will work :)" [puppet] - 10https://gerrit.wikimedia.org/r/1318701 (https://phabricator.wikimedia.org/T394357) (owner: 10Elukey) [14:16:52] (03PS1) 10Scott French: hieradata: Explicitly set cluster_bootstrap to false on conf2004 [puppet] - 10https://gerrit.wikimedia.org/r/1318725 (https://phabricator.wikimedia.org/T428495) [14:17:06] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2012.codfw.wmnet with OS bookworm [14:17:10] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318725 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [14:18:30] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1002.eqiad.wmnet [14:23:33] (03PS3) 10Majavah: P:toolforge::bastion: Minor cleanup [puppet] - 10https://gerrit.wikimedia.org/r/1318227 [14:23:34] (03PS3) 10Majavah: P:toolforge::bastion: Add script to migrate OpenSearch indexes [puppet] - 10https://gerrit.wikimedia.org/r/1318228 (https://phabricator.wikimedia.org/T401818) [14:23:36] !log bking@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [14:23:40] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [14:23:45] !log bking@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 19s) [14:23:50] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1002.eqiad.wmnet [14:23:51] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1002.eqiad.wmnet [14:23:56] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1003.eqiad.wmnet [14:25:28] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/kartotherian: apply [14:25:32] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/kartotherian: apply [14:25:33] (03PS1) 10Elukey: sre.hosts.bmc-user.mgmt: set proper IPMI permissions for wmfroot [cookbooks] - 10https://gerrit.wikimedia.org/r/1318727 (https://phabricator.wikimedia.org/T426180) [14:25:38] !log jgiannelos@deploy1003 helmfile [codfw] START helmfile.d/services/kartotherian: apply [14:25:40] (03CR) 10Elukey: [C:03+2] Set sretest2010 as test swift host for Data Persistence [puppet] - 10https://gerrit.wikimedia.org/r/1318701 (https://phabricator.wikimedia.org/T394357) (owner: 10Elukey) [14:26:34] (03CR) 10Majavah: [C:03+2] P:toolforge::bastion: Minor cleanup [puppet] - 10https://gerrit.wikimedia.org/r/1318227 (owner: 10Majavah) [14:27:06] (03PS2) 10Scott French: hieradata: Explicitly set cluster_bootstrap to false on conf2004 [puppet] - 10https://gerrit.wikimedia.org/r/1318725 (https://phabricator.wikimedia.org/T428495) [14:27:13] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318725 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [14:27:48] (03CR) 10CI reject: [V:04-1] hieradata: Explicitly set cluster_bootstrap to false on conf2004 [puppet] - 10https://gerrit.wikimedia.org/r/1318725 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [14:28:09] (03PS3) 10Scott French: hieradata: Explicitly set cluster_bootstrap to false on conf2004 [puppet] - 10https://gerrit.wikimedia.org/r/1318725 (https://phabricator.wikimedia.org/T428495) [14:28:35] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318725 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T1430) [14:30:53] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts [14:31:08] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts [14:31:34] (03CR) 10Blake: [C:03+1] hieradata: Explicitly set cluster_bootstrap to false on conf2004 [puppet] - 10https://gerrit.wikimedia.org/r/1318725 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [14:31:49] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2181: Maintenance [14:32:11] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance [14:32:19] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2195 (T431660)', diff saved to https://phabricator.wikimedia.org/P95394 and previous config saved to /var/cache/conftool/dbconfig/20260728-143218-cwilliams.json [14:32:28] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12163280 (10CDiggs-WMF) @Dzahn Could anyone help me here to get my account set up on https://idm.wikimedia.org/? Or point me in the right direction? Many thanks! [14:32:41] (03CR) 10Scott French: [C:03+2] hieradata: Explicitly set cluster_bootstrap to false on conf2004 [puppet] - 10https://gerrit.wikimedia.org/r/1318725 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [14:32:44] (03PS2) 10Elukey: sre.hosts.bmc-user.mgmt: set proper IPMI permissions for wmfroot [cookbooks] - 10https://gerrit.wikimedia.org/r/1318727 (https://phabricator.wikimedia.org/T426180) [14:33:14] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2222: Maintenance [14:33:59] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1003.eqiad.wmnet [14:35:10] (03PS3) 10Elukey: sre.hosts.bmc-user.mgmt: set proper IPMI permissions for wmfroot [cookbooks] - 10https://gerrit.wikimedia.org/r/1318727 (https://phabricator.wikimedia.org/T426180) [14:36:30] (03CR) 10Elukey: "Tested with ml-serve1002, worked nicely." [cookbooks] - 10https://gerrit.wikimedia.org/r/1318727 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [14:36:45] !log jgiannelos@deploy1003 helmfile [codfw] DONE helmfile.d/services/kartotherian: apply [14:36:45] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1026.eqiad.wmnet -> wdqs1022.eqiad.wmnet, repooling source-only afterwards [14:36:51] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [14:38:38] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2195: Maintenance [14:38:52] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1003.eqiad.wmnet [14:38:54] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1003.eqiad.wmnet [14:38:59] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1004.eqiad.wmnet [14:39:03] !log swfrench@cumin2002 START - Cookbook sre.hosts.reboot-single for host conf2004.codfw.wmnet [14:39:34] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12163336 (10Aklapper) @CDiggs-WMF Hi, it's hard to help without knowing what the problem is / where you are stuck. Please follow https://wikitech.wikimedia.org/wiki/SRE/LDA... [14:43:15] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12163349 (10Dzahn) @CDiggs-WMF As far as I know you would just have to fill out this form: https://idm.wikimedia.org/signup/ Are you running into a problem or error? [14:44:01] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1004.eqiad.wmnet [14:44:24] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2170: Maintenance [14:44:46] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance [14:44:54] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2173 (T431660)', diff saved to https://phabricator.wikimedia.org/P95399 and previous config saved to /var/cache/conftool/dbconfig/20260728-144453-cwilliams.json [14:45:23] !log swfrench@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host conf2004.codfw.wmnet [14:49:44] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 13 hosts [14:50:33] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1004.eqiad.wmnet [14:50:34] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1004.eqiad.wmnet [14:50:39] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1005.eqiad.wmnet [14:51:31] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 13 hosts [14:51:32] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2173: Maintenance [14:51:41] (03PS1) 10Trueg: WDQSv2: Set resourcequota for the wdqs namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318730 (https://phabricator.wikimedia.org/T431395) [14:54:43] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2240: Maintenance [14:55:06] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2248.codfw.wmnet with reason: Maintenance [14:55:24] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2245-2247].codfw.wmnet with reason: Maintenance [14:55:32] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2248 (T431660)', diff saved to https://phabricator.wikimedia.org/P95403 and previous config saved to /var/cache/conftool/dbconfig/20260728-145532-cwilliams.json [14:55:42] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1005.eqiad.wmnet [14:56:11] PROBLEM - Etcd cluster health on conf2004 is CRITICAL: The etcd server is unhealthy https://wikitech.wikimedia.org/wiki/Etcd [14:57:17] ^ expected - will be silenced shortly [14:59:16] !log swfrench@cumin2002 START - Cookbook sre.hosts.reimage for host conf2004.codfw.wmnet with OS bookworm [14:59:51] !log dzahn@cumin2002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on phab1004.eqiad.wmnet with reason: deployment [14:59:52] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2248: Maintenance [15:00:05] jelto, arnoldokoth, mutante, and arnaudb: #bothumor I � Unicode. All rise for SRE Collaboration Services office hours deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T1500). [15:00:48] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1005.eqiad.wmnet [15:00:49] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1005.eqiad.wmnet [15:00:54] !log dzahn@cumin2002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on phab1006.eqiad.wmnet with reason: deployment [15:00:55] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1006.eqiad.wmnet [15:01:26] !log dzahn@cumin2002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on phab1005.eqiad.wmnet with reason: deployment [15:01:58] !log dzahn@cumin2002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on phab2003.codfw.wmnet with reason: deployment [15:05:18] !log bking@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs1022\.eqiad\.wmnet,dc=eqiad,cluster=wdqs\-main,service=wdqs\-main [15:05:24] (03CR) 10Effie Mouzeli: [C:03+2] dsh: scap should stop syncing to parsoidtest1001 [puppet] - 10https://gerrit.wikimedia.org/r/1318156 (https://phabricator.wikimedia.org/T433239) (owner: 10Effie Mouzeli) [15:05:57] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1006.eqiad.wmnet [15:06:19] !log jiji@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1137.eqiad.wmnet [15:06:20] !log jiji@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1137.eqiad.wmnet [15:06:23] !log jiji@cumin1003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1137.eqiad.wmnet [15:06:31] (03PS4) 10Elukey: sre.hosts.bmc-user.mgmt: set proper IPMI permissions for wmfroot [cookbooks] - 10https://gerrit.wikimedia.org/r/1318727 (https://phabricator.wikimedia.org/T426180) [15:07:12] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 13 hosts [15:07:53] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2012.codfw.wmnet, repooling source-only afterwards [15:07:57] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [15:08:16] !log brennen@deploy1003 Started deploy [phabricator/deployment@f8b349f]: deploy phab2003 for T433382 [15:08:20] T433382: Deploy Phab/Phorge 2026-07-28 - https://phabricator.wikimedia.org/T433382 [15:09:06] (03CR) 10Btullis: [C:03+1] "Looks good. Maybe add some comments to explain that the limitranges are defined in the common file, but the resourcequotas are different p" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318730 (https://phabricator.wikimedia.org/T431395) (owner: 10Trueg) [15:09:11] !log brennen@deploy1003 Finished deploy [phabricator/deployment@f8b349f]: deploy phab2003 for T433382 (duration: 00m 55s) [15:09:26] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 13 hosts [15:10:04] !log brennen@deploy1003 Started deploy [phabricator/deployment@f8b349f]: deploy phab1004 for T433382 [15:10:47] !log brennen@deploy1003 Finished deploy [phabricator/deployment@f8b349f]: deploy phab1004 for T433382 (duration: 00m 43s) [15:11:01] !log jiji@cumin1003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1138.eqiad.wmnet [15:11:04] !log jiji@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1138.eqiad.wmnet [15:11:20] (03CR) 10BBlack: [C:03+1] "LGTM, thank you!" [puppet] - 10https://gerrit.wikimedia.org/r/1318088 (https://phabricator.wikimedia.org/T433075) (owner: 10Cparle) [15:11:33] !log eevans@cumin1003 END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase[2025-2038].codfw.wmnet,restbase[1032-1045].eqiad.wmnet: Upgrade Cassandra to 5.0.8 — T433028 - eevans@cumin1003 [15:11:37] T433028: Upgrade restbase cluster to Cassandra 5.0.8 - https://phabricator.wikimedia.org/T433028 [15:11:40] !log jiji@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1138.eqiad.wmnet [15:12:38] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1006.eqiad.wmnet [15:12:39] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1006.eqiad.wmnet [15:12:45] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1007.eqiad.wmnet [15:14:40] jiji@cumin1003 renumber-node (PID 3217686) is awaiting input [15:14:52] 10SRE-swift-storage, 06Data-Persistence, 10MediaViewer, 10Thumbor, and 5 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#12163523 (10Krinkle) [15:16:26] (03CR) 10Brennen Bearnes: [C:03+1] "Discussed this in Collab Services office hours. Acknowledging that this is difficult to test, but we believe it's likely to be fine for al" [puppet] - 10https://gerrit.wikimedia.org/r/1298763 (https://phabricator.wikimedia.org/T405596) (owner: 10Aklapper) [15:17:47] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1007.eqiad.wmnet [15:19:23] (03CR) 10Aklapper: "Also realized in this meeting that I should bring this in T374926 and T424098 first." [puppet] - 10https://gerrit.wikimedia.org/r/1298763 (https://phabricator.wikimedia.org/T405596) (owner: 10Aklapper) [15:19:40] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts [15:19:42] !log swfrench@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on conf2004.codfw.wmnet with reason: host reimage [15:19:43] (03PS2) 10Aklapper: Remove Phabricator Diffusion as canonical repository source [puppet] - 10https://gerrit.wikimedia.org/r/1298763 (https://phabricator.wikimedia.org/T405596) [15:19:59] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts [15:21:53] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs2013.codfw.wmnet with OS bookworm [15:22:31] !log swfrench@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf2004.codfw.wmnet with reason: host reimage [15:22:53] (03CR) 10JHathaway: [C:03+1] sre.hosts.bmc-user.mgmt: set proper IPMI permissions for wmfroot [cookbooks] - 10https://gerrit.wikimedia.org/r/1318727 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [15:23:37] (03PS2) 10Urbanecm: mediawiki::maintenance::growthexperiments: Correct the logs comment [puppet] - 10https://gerrit.wikimedia.org/r/1202203 [15:24:27] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1007.eqiad.wmnet [15:24:28] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1007.eqiad.wmnet [15:24:34] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1008.eqiad.wmnet [15:24:34] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12163571 (10CDiggs-WMF) @Dzahn I went in to create my account, but apparently my email address is already in use so I can't. I don't see a confirmation in my inbox so I ima... [15:25:03] (03CR) 10Elukey: [C:03+2] sre.hosts.bmc-user.mgmt: set proper IPMI permissions for wmfroot [cookbooks] - 10https://gerrit.wikimedia.org/r/1318727 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [15:25:11] jouncebot: nowandnext [15:25:12] For the next 0 hour(s) and 34 minute(s): SRE Collaboration Services office hours (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T1500) [15:25:12] In 0 hour(s) and 34 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T1600) [15:25:19] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2195: Maintenance [15:25:34] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs2013 [15:25:38] (03CR) 10Ladsgroup: [C:03+2] thumbor: Move more operations to asyncio [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318677 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup) [15:27:31] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 290 hosts [15:27:43] (03PS2) 10Trueg: WDQSv2: Set resourcequota for the wdqs namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318730 (https://phabricator.wikimedia.org/T431395) [15:28:25] (03Merged) 10jenkins-bot: thumbor: Move more operations to asyncio [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318677 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup) [15:28:37] bking@cumin2003 reimage (PID 1149286) is awaiting input [15:28:49] (03PS1) 10Lucas Werkmeister (WMDE): wikidata-query-gui: update staging custom-config.json [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318731 [15:29:37] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1008.eqiad.wmnet [15:29:51] (03CR) 10Lucas Werkmeister (WMDE): "I have no idea if this works correctly :D" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318731 (owner: 10Lucas Werkmeister (WMDE)) [15:31:29] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [15:31:49] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [15:34:11] (03CR) 10Lucas Werkmeister (WMDE): "Hm, I’m not sure if the diff in https://integration.wikimedia.org/ci/job/helm-lint/34352/console is correct… there are a bunch of added co" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318731 (owner: 10Lucas Werkmeister (WMDE)) [15:34:43] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1008.eqiad.wmnet [15:34:44] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1008.eqiad.wmnet [15:34:49] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1009.eqiad.wmnet [15:35:25] !log ladsgroup@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [15:37:46] !log ladsgroup@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [15:38:14] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2173: Maintenance [15:38:37] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance [15:38:45] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2174 (T431660)', diff saved to https://phabricator.wikimedia.org/P95413 and previous config saved to /var/cache/conftool/dbconfig/20260728-153844-cwilliams.json [15:39:38] !log robh@cumin2002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on arclamp2001.codfw.wmnet with reason: ram upgrade [15:39:53] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1009.eqiad.wmnet [15:41:02] (03CR) 10Btullis: [C:03+2] WDQSv2: Set resourcequota for the wdqs namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318730 (https://phabricator.wikimedia.org/T431395) (owner: 10Trueg) [15:41:46] !log jiji@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-worker1138.eqiad.wmnet with OS trixie [15:42:15] !log jiji@cumin1003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1138 [15:43:47] (03CR) 10Lucas Werkmeister (WMDE): wikidata-query-gui: update staging custom-config.json (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318731 (owner: 10Lucas Werkmeister (WMDE)) [15:44:33] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1009.eqiad.wmnet [15:44:34] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1009.eqiad.wmnet [15:44:40] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1010.eqiad.wmnet [15:45:18] jiji@cumin1003 renumber-node (PID 3217686) is awaiting input [15:45:22] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2174: Maintenance [15:46:09] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12163764 (10Aklapper) I don't think you need to create a Developer Account account because you already have a Developer Account: https://ldap.toolforge.org/user/Cdiggs-ctr [15:46:35] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2248: Maintenance [15:48:16] !log jiji@cumin1003 START - Cookbook sre.dns.netbox [15:48:21] !log ladsgroup@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [15:48:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:49:38] (03Merged) 10jenkins-bot: WDQSv2: Set resourcequota for the wdqs namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318730 (https://phabricator.wikimedia.org/T431395) (owner: 10Trueg) [15:49:43] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1010.eqiad.wmnet [15:50:55] !log ladsgroup@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [15:53:46] jouncebot: next [15:53:46] In 0 hour(s) and 6 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T1600) [15:54:03] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [15:54:09] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1010.eqiad.wmnet [15:54:10] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1010.eqiad.wmnet [15:54:15] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1011.eqiad.wmnet [15:54:25] jiji@cumin1003 renumber-node (PID 3217686) is awaiting input [15:54:52] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [15:55:39] (03PS1) 10Trueg: WDQSv2: Scheduling based on resource requests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318733 (https://phabricator.wikimedia.org/T431395) [15:55:51] !log btullis@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [15:56:02] (03CR) 10Ladsgroup: "Hugh: Any concerns?" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1313088 (https://phabricator.wikimedia.org/T432686) (owner: 10Samwilson) [15:56:27] !log btullis@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [15:57:27] (03CR) 10Hnowlan: [C:03+1] "lgtm!" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1313088 (https://phabricator.wikimedia.org/T432686) (owner: 10Samwilson) [15:57:31] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [15:58:58] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5) {#122601}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:58:59] (03CR) 10Ladsgroup: [C:03+2] "Awesome. Merging!" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1313088 (https://phabricator.wikimedia.org/T432686) (owner: 10Samwilson) [15:59:06] PROBLEM - Host arclamp2001 is DOWN: PING CRITICAL - Packet loss = 100% [15:59:18] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1011.eqiad.wmnet [16:00:05] jhathaway and rzl: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for Puppet request window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T1600). [16:00:05] !log swfrench@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf2004.codfw.wmnet with OS bookworm [16:00:05] No Gerrit patches in the queue for this window AFAICS. [16:01:48] RECOVERY - Host arclamp2001 is UP: PING OK - Packet loss = 0%, RTA = 31.59 ms [16:03:45] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1011.eqiad.wmnet [16:03:46] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1011.eqiad.wmnet [16:03:51] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1012.eqiad.wmnet [16:03:58] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:06:46] 10ops-codfw, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp2001 ram upgrade - https://phabricator.wikimedia.org/T433285#12163928 (10Jhancock.wm) [16:07:08] 10ops-codfw, 06SRE, 06DC-Ops, 10Observability-Metrics: arclamp2001 ram upgrade - https://phabricator.wikimedia.org/T433285#12163929 (10Jhancock.wm) 05Open→03Resolved @colewhite this is completed! [16:07:31] RESOLVED: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [16:07:53] 06SRE, 10SRE-tools, 10Cumin, 06DC-Ops, 06Infrastructure-Foundations: add dcops group to run sre.hosts.downtime cookbook - https://phabricator.wikimedia.org/T433409 (10RobH) 03NEW p:05Triage→03Medium [16:08:47] (03Merged) 10jenkins-bot: Update Makefile to Compose v2, and improve documentation [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1313088 (https://phabricator.wikimedia.org/T432686) (owner: 10Samwilson) [16:08:54] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313#12163953 (10Bethany) thanks @Aklapper. I put my wikitech account username but clearly wasn't sure which username was needed. [16:08:58] FIRING: [4x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:10:25] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:11:14] PROBLEM - Host sretest2010 is DOWN: PING CRITICAL - Packet loss = 100% [16:11:27] (03PS2) 10Trueg: WDQSv2: Scheduling based on resource requests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318733 (https://phabricator.wikimedia.org/T431395) [16:11:44] RECOVERY - Host sretest2010 is UP: PING OK - Packet loss = 0%, RTA = 31.71 ms [16:12:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:13:55] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1012.eqiad.wmnet [16:13:58] FIRING: [4x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:14:27] FIRING: [4x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:14:36] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12163980 (10elukey) The host is now ready for a test by data persistence! [16:16:26] (03CR) 10Audrey Penven: "Looks like `values-staging` is used in all "releases". It's added to the default values [1], which are then included for staging versions " [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318731 (owner: 10Lucas Werkmeister (WMDE)) [16:17:02] (03PS3) 10Blake: mw-pretrain: Add a canary values.yaml. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318671 (https://phabricator.wikimedia.org/T427668) [16:20:40] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1012.eqiad.wmnet [16:20:42] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1012.eqiad.wmnet [16:20:47] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1013.eqiad.wmnet [16:25:52] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet [16:27:48] (03PS3) 10Genoveva Galarza: Add CacheAbstractContentFragment to high_traffic_jobs with concurrency=1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1314852 (https://phabricator.wikimedia.org/T430898) [16:28:29] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2012.codfw.wmnet, repooling source-only afterwards [16:28:33] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [16:28:50] (03CR) 10Federico Ceratto: "@marostegui@wikimedia.org you mean an optional `--delete-heartbeat` that executes a `DELETE FROM heartbeat.heartbeat` without filtering by" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [16:29:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [16:30:58] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 96540464 and 13 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [16:32:05] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2174: Maintenance [16:32:11] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet [16:32:12] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet [16:32:18] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet [16:32:28] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance [16:32:36] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2176 (T431660)', diff saved to https://phabricator.wikimedia.org/P95420 and previous config saved to /var/cache/conftool/dbconfig/20260728-163235-cwilliams.json [16:32:50] FIRING: KubernetesCalicoDown: ml-serve1013.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-mlserve&var-instance=ml-serve1013.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:33:58] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3566048 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [16:34:07] (03PS4) 10Btullis: opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) [16:34:07] (03PS3) 10Btullis: cirrus: enable ECS server log shipping on relforge [puppet] - 10https://gerrit.wikimedia.org/r/1311439 (https://phabricator.wikimedia.org/T324335) [16:34:07] (03PS3) 10Btullis: cirrus: enable ECS server log shipping on cloudelastic [puppet] - 10https://gerrit.wikimedia.org/r/1311440 (https://phabricator.wikimedia.org/T324335) [16:34:08] (03PS3) 10Btullis: cirrus: ship ECS server logs by default [puppet] - 10https://gerrit.wikimedia.org/r/1311441 (https://phabricator.wikimedia.org/T324335) [16:34:09] (03PS4) 10Btullis: cirrus: remove the log4j syslog appender log shipping path [puppet] - 10https://gerrit.wikimedia.org/r/1311426 (https://phabricator.wikimedia.org/T324335) [16:34:10] (03PS1) 10Btullis: opensearch: support EcsLayout for the on-disk JSON server logs [puppet] - 10https://gerrit.wikimedia.org/r/1318740 (https://phabricator.wikimedia.org/T324335) [16:34:13] (03PS1) 10Btullis: cirrus: use EcsLayout server logs on relforge [puppet] - 10https://gerrit.wikimedia.org/r/1318741 (https://phabricator.wikimedia.org/T324335) [16:35:30] (03CR) 10CI reject: [V:04-1] cirrus: enable ECS server log shipping on relforge [puppet] - 10https://gerrit.wikimedia.org/r/1311439 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:36:02] (03CR) 10CI reject: [V:04-1] cirrus: enable ECS server log shipping on cloudelastic [puppet] - 10https://gerrit.wikimedia.org/r/1311440 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:36:23] (03CR) 10CI reject: [V:04-1] opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:37:23] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet [16:37:40] !log jiji@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1138 - jiji@cumin1003" [16:37:45] !log jiji@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1138 - jiji@cumin1003" [16:37:45] !log jiji@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [16:37:45] !log jiji@cumin1003 START - Cookbook sre.dns.wipe-cache wikikube-worker1138.eqiad.wmnet 193.32.64.10.in-addr.arpa 3.9.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [16:37:48] !log jiji@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1138.eqiad.wmnet 193.32.64.10.in-addr.arpa 3.9.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [16:37:49] !log jiji@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1138 [16:37:50] RESOLVED: KubernetesCalicoDown: ml-serve1013.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-mlserve&var-instance=ml-serve1013.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:38:18] !log jiji@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1138 [16:38:18] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1138 [16:38:33] (03CR) 10CI reject: [V:04-1] cirrus: ship ECS server logs by default [puppet] - 10https://gerrit.wikimedia.org/r/1311441 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:38:55] (03CR) 10CI reject: [V:04-1] cirrus: use EcsLayout server logs on relforge [puppet] - 10https://gerrit.wikimedia.org/r/1318741 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:38:56] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2176: Maintenance [16:40:21] (03CR) 10CI reject: [V:04-1] cirrus: remove the log4j syslog appender log shipping path [puppet] - 10https://gerrit.wikimedia.org/r/1311426 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:40:29] (03CR) 10CI reject: [V:04-1] opensearch: support EcsLayout for the on-disk JSON server logs [puppet] - 10https://gerrit.wikimedia.org/r/1318740 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:42:05] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 290 hosts [16:43:07] !log sukhe@puppetserver1001 conftool action : set/pooled=no; selector: name=dns3003.wikimedia.org [reason: depooling for reimage to trixie] [16:43:34] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet [16:43:35] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet [16:43:41] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet [16:43:57] !log sukhe@cumin1003 START - Cookbook sre.hosts.reimage for host dns3003.wikimedia.org with OS trixie [16:45:15] (03PS2) 10Btullis: opensearch: support EcsLayout for the on-disk JSON server logs [puppet] - 10https://gerrit.wikimedia.org/r/1318740 (https://phabricator.wikimedia.org/T324335) [16:45:15] (03PS5) 10Btullis: opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) [16:45:15] (03PS2) 10Btullis: cirrus: use EcsLayout server logs on relforge [puppet] - 10https://gerrit.wikimedia.org/r/1318741 (https://phabricator.wikimedia.org/T324335) [16:45:16] (03PS4) 10Btullis: cirrus: enable ECS server log shipping on relforge [puppet] - 10https://gerrit.wikimedia.org/r/1311439 (https://phabricator.wikimedia.org/T324335) [16:45:17] (03PS4) 10Btullis: cirrus: enable ECS server log shipping on cloudelastic [puppet] - 10https://gerrit.wikimedia.org/r/1311440 (https://phabricator.wikimedia.org/T324335) [16:45:18] (03PS4) 10Btullis: cirrus: ship ECS server logs by default [puppet] - 10https://gerrit.wikimedia.org/r/1311441 (https://phabricator.wikimedia.org/T324335) [16:45:21] (03PS5) 10Btullis: cirrus: remove the log4j syslog appender log shipping path [puppet] - 10https://gerrit.wikimedia.org/r/1311426 (https://phabricator.wikimedia.org/T324335) [16:46:37] (03CR) 10CI reject: [V:04-1] cirrus: use EcsLayout server logs on relforge [puppet] - 10https://gerrit.wikimedia.org/r/1318741 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:47:07] (03CR) 10CI reject: [V:04-1] cirrus: enable ECS server log shipping on relforge [puppet] - 10https://gerrit.wikimedia.org/r/1311439 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:47:42] (03CR) 10CI reject: [V:04-1] cirrus: enable ECS server log shipping on cloudelastic [puppet] - 10https://gerrit.wikimedia.org/r/1311440 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:47:53] PROBLEM - BFD status on asw1-by27-esams.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [16:48:10] FIRING: [2x] BFDdown: BFD session down between asw1-by27-esams and 185.15.59.34 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-by27-esams:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [16:48:18] ^expected [16:48:19] (03CR) 10CI reject: [V:04-1] opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:48:22] (03CR) 10CI reject: [V:04-1] opensearch: support EcsLayout for the on-disk JSON server logs [puppet] - 10https://gerrit.wikimedia.org/r/1318740 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:50:10] (03CR) 10CI reject: [V:04-1] cirrus: ship ECS server logs by default [puppet] - 10https://gerrit.wikimedia.org/r/1311441 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:50:25] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [16:50:26] (03CR) 10Eevans: [C:03+2] restbase: Upgrade to JDK 17 [puppet] - 10https://gerrit.wikimedia.org/r/1315184 (https://phabricator.wikimedia.org/T433028) (owner: 10Eevans) [16:51:28] (03CR) 10CI reject: [V:04-1] cirrus: remove the log4j syslog appender log shipping path [puppet] - 10https://gerrit.wikimedia.org/r/1311426 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [16:51:31] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12164169 (10Dzahn) @CDiggs-WMF I see! Seems like this might be related to you having a contractor position and then switching to staff? Try logging in with the existing ac... [16:53:48] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet [16:53:58] FIRING: [3x] JobUnavailable: Reduced availability for job haproxy in ops@esams - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:54:28] !log jiji@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1138.eqiad.wmnet with reason: host reimage [16:55:14] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-codfw: Upgrade Java to 17.0.20 — T433028 - eevans@cumin1003 [16:55:18] T433028: Upgrade restbase cluster to Cassandra 5.0.8 - https://phabricator.wikimedia.org/T433028 [16:55:55] 06SRE, 10LDAP-Access-Requests: Grant Access to for - https://phabricator.wikimedia.org/T433417 (10JAATPH) 03NEW [16:57:37] 06SRE, 10LDAP-Access-Requests: Grant Access to NDA for jaleman-vdr-wmf - https://phabricator.wikimedia.org/T433417#12164224 (10JAATPH) [16:58:00] 10ops-eqsin, 06SRE, 06DC-Ops, 06Traffic: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#12164226 (10RobH) Replacement work scheduled for Tuesday, August 4th @ 00:00 GMT (08:00 Singapore). I'll depool and power down this single host when the engineer arrives on-site and re-pool it once t... [16:59:02] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1138.eqiad.wmnet with reason: host reimage [16:59:31] 06SRE, 10LDAP-Access-Requests: Grant Access to NDA for jaleman-vdr-wmf - https://phabricator.wikimedia.org/T433417#12164238 (10Dzahn) Hi @JAATPH could you please send an email to @KFrancis (see address on https://meta.wikimedia.org/wiki/User:KFrancis_(WMF)) and tell her you made this ticket and would like to s... [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T1700) [17:00:10] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet [17:00:11] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet [17:00:11] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-eqiad [17:01:22] 06SRE, 10Infrastructure Security, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12164262 (10klausman) LW prod workers all done, leaving only the build and lab machines. [17:01:26] 10ops-eqsin, 06SRE, 06DC-Ops, 06Traffic: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#12164263 (10ssingh) >>! In T414411#12164226, @RobH wrote: > Replacement CPU work scheduled for Tuesday, August 4th @ 00:00 GMT (08:00 Singapore). I'll depool and power down this single host when the... [17:01:48] 06SRE, 10Infrastructure Security, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12164275 (10klausman) [17:02:10] 10ops-eqsin, 06SRE, 06DC-Ops, 06Traffic: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#12164276 (10ssingh) Additionally, feel free to leave the repooling to us so that we can check everything is OK on the software side of things. You can verify the hardware for us. Thanks! [17:03:58] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164279 (10Dzahn) @VRiley-WMF So.. zuul1004 is done now. works for me:) I was going to continue with zuul2004 but realized that does not e... [17:05:32] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164281 (10Dzahn) Oh, maybe codfw is not involved at all and I just assumed too much. Are all hosts we are repurposing eqiad and nothing in... [17:06:19] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164286 (10Dzahn) @VRiley-WMF While zuul1004 works fine I got this when trying to do zuul1005 next: ` spicerack.netbox.NetboxError: Serve... [17:07:38] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164291 (10VRiley-WMF) I'm currently working on those as we speak [17:07:42] !log sukhe@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on dns3003.wikimedia.org with reason: host reimage [17:08:35] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164295 (10Dzahn) Oh cool! Stepping back for now and checking in later. Thank you! [17:08:37] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164296 (10VRiley-WMF) So, zuul1004-1007 would be in eqiad. If it starts with a 2, it would be for codfw [17:09:58] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164297 (10Dzahn) ACK! aware of that. Was just thinking the hardware we are repurposing might have existed in both DCs. [17:10:32] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164301 (10VRiley-WMF) Currently, if you wanted to have zuul2004-7 we would need to find servers that are located in codfw. This would be o... [17:12:52] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164302 (10Dzahn) Yea, that makes sense. Either way that would be another ticket. @MoritzMuehlenhoff Has this always been an eqiad-only t... [17:13:40] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns3003.wikimedia.org with reason: host reimage [17:15:20] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [17:15:30] (03CR) 10BCornwall: [C:03+1] conftool-data: add urldownloader to discovery [puppet] - 10https://gerrit.wikimedia.org/r/1313286 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [17:16:53] PROBLEM - Recursive DNS on 185.15.59.34 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [17:17:01] ^ expected [17:17:35] (03PS3) 10Btullis: opensearch: support EcsLayout for the on-disk JSON server logs [puppet] - 10https://gerrit.wikimedia.org/r/1318740 (https://phabricator.wikimedia.org/T324335) [17:17:35] (03PS5) 10Btullis: cirrus: produce ECS server logs on relforge [puppet] - 10https://gerrit.wikimedia.org/r/1311440 (https://phabricator.wikimedia.org/T324335) [17:17:38] (03PS5) 10Btullis: cirrus: produce ECS server logs on cloudelastic [puppet] - 10https://gerrit.wikimedia.org/r/1311441 (https://phabricator.wikimedia.org/T324335) [17:17:40] (03PS5) 10Btullis: cirrus: produce ECS server logs on the production clusters [puppet] - 10https://gerrit.wikimedia.org/r/1311439 (https://phabricator.wikimedia.org/T324335) [17:17:43] (03PS6) 10Btullis: opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) [17:17:44] (03PS6) 10Btullis: cirrus: remove the log4j syslog appender log shipping path [puppet] - 10https://gerrit.wikimedia.org/r/1311426 (https://phabricator.wikimedia.org/T324335) [17:18:23] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [17:18:37] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host zuul1005 [17:18:57] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host zuul1005 [17:19:13] !log jiji@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1138.eqiad.wmnet with OS trixie [17:19:53] (03CR) 10BCornwall: [C:03+1] service: add definition for urldownloader service [puppet] - 10https://gerrit.wikimedia.org/r/1313947 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [17:20:26] (03CR) 10RLazarus: "Thanks for this! High-level I agree with using httpbb for pretrain validation, and this seems like a sensible way to do it." [software/httpbb] - 10https://gerrit.wikimedia.org/r/1318682 (https://phabricator.wikimedia.org/T428972) (owner: 10Blake) [17:21:52] (03PS2) 10Ssingh: hiera: url_downloader: enable LVS pool and IPIP [puppet] - 10https://gerrit.wikimedia.org/r/1313948 (https://phabricator.wikimedia.org/T429175) [17:21:53] PROBLEM - Recursive DNS on 2a02:ec80:300:2:185:15:59:34 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [17:21:56] expected [17:22:38] (03CR) 10Ssingh: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9075/co" [puppet] - 10https://gerrit.wikimedia.org/r/1313948 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [17:22:59] (03CR) 10BCornwall: [C:03+1] "`Error: Evaluation Error: Error while evaluating a Function Call, 'wmflib::service::validate' parameter 'catalog' entry 'urldownloader' " [puppet] - 10https://gerrit.wikimedia.org/r/1313947 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [17:23:06] (03CR) 10BCornwall: [V:04-1 C:03+1] service: add definition for urldownloader service [puppet] - 10https://gerrit.wikimedia.org/r/1313947 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [17:24:25] PROBLEM - Check unit status of etcd-backup on conf2004 is CRITICAL: CRITICAL: Status of the systemd unit etcd-backup https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:24:28] FIRING: [3x] JobUnavailable: Reduced availability for job haproxy in ops@esams - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:25:40] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2176: Maintenance [17:26:01] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance [17:26:05] (03PS2) 10Ssingh: service: add definition for urldownloader service [puppet] - 10https://gerrit.wikimedia.org/r/1313947 (https://phabricator.wikimedia.org/T429175) [17:26:10] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2188 (T431660)', diff saved to https://phabricator.wikimedia.org/P95425 and previous config saved to /var/cache/conftool/dbconfig/20260728-172609-cwilliams.json [17:26:25] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318740 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [17:26:28] (03PS3) 10Ssingh: service: add definition for urldownloader service [puppet] - 10https://gerrit.wikimedia.org/r/1313947 (https://phabricator.wikimedia.org/T429175) [17:27:30] (03CR) 10Ssingh: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9077/console" [puppet] - 10https://gerrit.wikimedia.org/r/1313947 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [17:27:54] (03CR) 10BCornwall: [C:03+1] service: add definition for urldownloader service [puppet] - 10https://gerrit.wikimedia.org/r/1313947 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [17:28:55] (03CR) 10Aaron Schulz: "Well, I guess that might be more the service itself rather than the job that uses the service. Either way, that seems like something to al" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1314852 (https://phabricator.wikimedia.org/T430898) (owner: 10Genoveva Galarza) [17:29:21] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts [17:29:45] (03PS1) 10Arlolra: Bump wikimedia/parsoid to 0.24.0-a17 [vendor] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318746 (https://phabricator.wikimedia.org/T19006) [17:29:46] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [17:29:53] (03CR) 10Aaron Schulz: [C:03+1] Add CacheAbstractContentFragment to high_traffic_jobs with concurrency=1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1314852 (https://phabricator.wikimedia.org/T430898) (owner: 10Genoveva Galarza) [17:32:45] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2188: Maintenance [17:34:00] (03CR) 10Ssingh: [C:03+2] conftool-data: add urldownloader cluster [puppet] - 10https://gerrit.wikimedia.org/r/1313285 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [17:34:04] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1005] - vriley@cumin1003" [17:34:08] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1005] - vriley@cumin1003" [17:34:09] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [17:34:10] (03PS1) 10Arlolra: Bump wikimedia/parsoid to 0.24.0-a17 [core] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318747 (https://phabricator.wikimedia.org/T433018) [17:34:18] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host zuul1005 [17:34:25] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts [17:34:34] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, July 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item" [core] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318747 (https://phabricator.wikimedia.org/T433018) (owner: 10Arlolra) [17:34:49] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host zuul1005 [17:34:57] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts [17:35:44] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts [17:35:56] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: cluster=urldownloader,service=squid [17:36:12] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host zuul1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [17:36:14] !log sukhe@puppetserver1001 conftool action : set/weight=1; selector: cluster=urldownloader,service=squid [17:36:39] (03CR) 10Ssingh: [C:03+2] conftool-data: add urldownloader to discovery [puppet] - 10https://gerrit.wikimedia.org/r/1313286 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [17:38:33] (03PS1) 10Scott French: zookeeper::server: Include slf4j-api.jar in classpath when using 3.4 [puppet] - 10https://gerrit.wikimedia.org/r/1318749 (https://phabricator.wikimedia.org/T428495) [17:38:52] (03CR) 10Subramanya Sastry: [C:03+1] Bump wikimedia/parsoid to 0.24.0-a17 [core] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318747 (https://phabricator.wikimedia.org/T433018) (owner: 10Arlolra) [17:38:57] RECOVERY - BFD status on asw1-by27-esams.mgmt is OK: UP: 2 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [17:39:38] (03CR) 10Ssingh: [C:03+2] wmnet/10.in-addr: add urldownloader service IPs [dns] - 10https://gerrit.wikimedia.org/r/1313290 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [17:39:43] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318749 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [17:39:48] !log sukhe@dns1004 START - running authdns-update [17:41:37] (03PS1) 10Ssingh: hiera/dns3003: use_new_pdns_cfg: true [puppet] - 10https://gerrit.wikimedia.org/r/1318750 [17:41:54] (03Abandoned) 10Ssingh: hiera/dns3003: use_new_pdns_cfg: true [puppet] - 10https://gerrit.wikimedia.org/r/1318750 (owner: 10Ssingh) [17:41:58] !log sukhe@dns1004 END - running authdns-update [17:42:07] (03PS1) 10Ssingh: hiera/dns3003: use_new_pdns_cfg: true [puppet] - 10https://gerrit.wikimedia.org/r/1318751 [17:43:10] RESOLVED: [2x] BFDdown: BFD session down between asw1-by27-esams and 185.15.59.34 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-by27-esams:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:43:31] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311440 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [17:43:52] 10ops-eqsin, 06SRE, 06DC-Ops, 06Traffic: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#12164492 (10RobH) >>! In T414411#12164276, @ssingh wrote: > Additionally, feel free to leave the repooling to us so that we can check everything is OK on the software side of things. You can verify th... [17:45:27] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1005.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [17:45:57] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie [17:46:09] (03PS4) 10Btullis: opensearch: support EcsLayout for the on-disk JSON server logs [puppet] - 10https://gerrit.wikimedia.org/r/1318740 (https://phabricator.wikimedia.org/T324335) [17:46:09] (03PS6) 10Btullis: cirrus: produce ECS server logs on relforge [puppet] - 10https://gerrit.wikimedia.org/r/1311440 (https://phabricator.wikimedia.org/T324335) [17:46:09] (03PS6) 10Btullis: cirrus: produce ECS server logs on cloudelastic [puppet] - 10https://gerrit.wikimedia.org/r/1311441 (https://phabricator.wikimedia.org/T324335) [17:46:10] (03PS6) 10Btullis: cirrus: produce ECS server logs on the production clusters [puppet] - 10https://gerrit.wikimedia.org/r/1311439 (https://phabricator.wikimedia.org/T324335) [17:46:11] (03PS7) 10Btullis: opensearch: ship the cirrus ECS server logs via rsyslog imfile [puppet] - 10https://gerrit.wikimedia.org/r/1311424 (https://phabricator.wikimedia.org/T324335) [17:46:12] (03PS7) 10Btullis: cirrus: remove the log4j syslog appender log shipping path [puppet] - 10https://gerrit.wikimedia.org/r/1311426 (https://phabricator.wikimedia.org/T324335) [17:46:17] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164516 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host zuul1005.eqiad.wmnet with OS t... [17:46:29] (03CR) 10Scott French: "NOTE: I suspect the fact that PCC shows no diff for conf2004 is due to the facts snapshot being stale - i.e., this host is only recently o" [puppet] - 10https://gerrit.wikimedia.org/r/1318749 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [17:47:58] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318740 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [17:48:01] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 416035280 and 43 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [17:48:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:49:50] (03PS1) 10Arlolra: Support older config values in $wgJsonConfigModels [extensions/Kartographer] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318753 (https://phabricator.wikimedia.org/T433008) [17:50:10] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, July 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item" [extensions/Kartographer] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318753 (https://phabricator.wikimedia.org/T433008) (owner: 10Arlolra) [17:51:01] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 832 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [17:52:06] (03CR) 10Subramanya Sastry: [C:03+1] Support older config values in $wgJsonConfigModels [extensions/Kartographer] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318753 (https://phabricator.wikimedia.org/T433008) (owner: 10Arlolra) [17:53:11] (03CR) 10RLazarus: [C:03+1] "In a perfect world I guess we'd build that as an array, insert the element conditionally, and join it with a : delimiter, but man, this is" [puppet] - 10https://gerrit.wikimedia.org/r/1318749 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [17:53:58] (03CR) 10Ssingh: [C:03+2] hiera/dns3003: use_new_pdns_cfg: true [puppet] - 10https://gerrit.wikimedia.org/r/1318751 (owner: 10Ssingh) [17:54:28] (03PS1) 10Arlolra: Remove icon beside reporting link on desktop view [extensions/ParserMigration] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318754 (https://phabricator.wikimedia.org/T433303) [17:54:50] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, July 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item" [extensions/ParserMigration] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318754 (https://phabricator.wikimedia.org/T433303) (owner: 10Arlolra) [17:55:03] (03CR) 10Ssingh: [C:03+2] "Note: I don't like self-merges without review but the host is depooled and I will reimage this again most likely. Please forgive me." [puppet] - 10https://gerrit.wikimedia.org/r/1318751 (owner: 10Ssingh) [17:55:06] (03CR) 10RLazarus: [C:03+2] "Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1202203 (owner: 10Urbanecm) [17:55:46] (03PS1) 10Arlolra: Remove icon beside reporting link on desktop view [extensions/ParserMigration] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318756 (https://phabricator.wikimedia.org/T433303) [17:56:16] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, July 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item" [extensions/ParserMigration] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318756 (https://phabricator.wikimedia.org/T433303) (owner: 10Arlolra) [17:56:48] !log eevans@cumin1003 END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Upgrade Java to 17.0.20 — T433028 - eevans@cumin1003 [17:56:52] T433028: Upgrade restbase cluster to Cassandra 5.0.8 - https://phabricator.wikimedia.org/T433028 [17:57:16] (03CR) 10Scott French: "Thanks for the review!" [puppet] - 10https://gerrit.wikimedia.org/r/1318749 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [17:57:39] (03CR) 10Scott French: [C:03+2] zookeeper::server: Include slf4j-api.jar in classpath when using 3.4 [puppet] - 10https://gerrit.wikimedia.org/r/1318749 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [17:57:51] RECOVERY - Recursive DNS on 185.15.59.34 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [17:57:51] RECOVERY - Recursive DNS on 2a02:ec80:300:2:185:15:59:34 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [17:57:57] nice [17:59:27] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:00:05] dduvall and dancy: How many deployers does it take to do MediaWiki train - Utc-7 Version deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T1800). [18:03:40] !log vriley@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage [18:05:27] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns3003.wikimedia.org with OS trixie [18:08:09] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage [18:10:04] !log jhancock@cumin2002 START - Cookbook sre.dns.netbox [18:13:58] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [18:15:46] !log jhancock@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding ms-be2097-8 to codfw - jhancock@cumin2002" [18:15:51] !log jhancock@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding ms-be2097-8 to codfw - jhancock@cumin2002" [18:15:51] !log jhancock@cumin2002 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:16:41] !log jhancock@cumin2002 START - Cookbook sre.network.configure-switch-interfaces for host ms-be2097 [18:17:09] !log jhancock@cumin2002 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2097 [18:17:14] !log jhancock@cumin2002 START - Cookbook sre.network.configure-switch-interfaces for host ms-be2098 [18:17:58] !log jhancock@cumin2002 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2098 [18:18:45] !log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host ms-be2097.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [18:19:26] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2188: Maintenance [18:19:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [18:19:51] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2203.codfw.wmnet with reason: Maintenance [18:19:59] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2203 (T431660)', diff saved to https://phabricator.wikimedia.org/P95430 and previous config saved to /var/cache/conftool/dbconfig/20260728-181958-cwilliams.json [18:21:30] (03PS1) 10Scott French: P:etcd::tlsproxy: Reintroduce sslcert::dhparam resource [puppet] - 10https://gerrit.wikimedia.org/r/1318760 (https://phabricator.wikimedia.org/T428495) [18:24:14] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318760 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [18:26:28] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2203: Maintenance [18:27:10] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns3003.wikimedia.org,service=authdns-update [reason: pool authdns-update after reimaging] [18:27:12] (03PS1) 10Ryan Kemper: Revert^2 "pontoon: add rkemper-kafka stack" [puppet] - 10https://gerrit.wikimedia.org/r/1318761 [18:27:22] !log sukhe@dns1004 START - running authdns-update [18:27:38] (03PS2) 10Ryan Kemper: pontoon: add rkemper-kafka stack [puppet] - 10https://gerrit.wikimedia.org/r/1318761 (https://phabricator.wikimedia.org/T276088) [18:27:57] (03CR) 10Ryan Kemper: "this was already +1'd in its current form in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308355 , so i will merge this after CI " [puppet] - 10https://gerrit.wikimedia.org/r/1318761 (https://phabricator.wikimedia.org/T276088) (owner: 10Ryan Kemper) [18:28:43] PROBLEM - OSPF status on cr1-eqiad is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:29:01] PROBLEM - OSPF status on cr2-esams is CRITICAL: OSPFv2: 4/5 UP : OSPFv3: 4/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:29:24] !log sukhe@dns1004 END - running authdns-update [18:30:01] RECOVERY - OSPF status on cr2-esams is OK: OSPFv2: 5/5 UP : OSPFv3: 5/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:30:08] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns3003.wikimedia.org [reason: pool for all services after reimaging] [18:30:41] RECOVERY - OSPF status on cr1-eqiad is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:32:23] !log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2097.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [18:32:49] !log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [18:33:31] !log dancy@deploy1003 Installing scap version "4.275.0" for 3 host(s) [18:34:13] dancy: o/ going over the train process with obvious-dust and Krinkle. we'll roll momentarily [18:34:34] OK. I'll deploying the spiderpig changes at the moment. [18:35:28] !log dancy@deploy1003 Installation of scap version "4.275.0" completed for 3 hosts [18:36:03] dduvall: Done. Reload your spiderpig page if you have it open [18:36:56] !log eevans@cumin1003 START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Upgrade Java to 17.0.20 — T433028 - eevans@cumin1003 [18:37:01] T433028: Upgrade restbase cluster to Cassandra 5.0.8 - https://phabricator.wikimedia.org/T433028 [18:39:00] dancy: ack [18:39:12] (03PS1) 10TrainBranchBot: group0 to 1.47.0-wmf.13 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318762 (https://phabricator.wikimedia.org/T430832) [18:39:16] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by dduvall@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318762 (https://phabricator.wikimedia.org/T430832) (owner: 10TrainBranchBot) [18:39:17] dancy: done. looks good! [18:39:27] jhancock@cumin2002 provision (PID 2059696) is awaiting input [18:40:09] !log jhancock@cumin2002 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [18:40:16] (03Merged) 10jenkins-bot: group0 to 1.47.0-wmf.13 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318762 (https://phabricator.wikimedia.org/T430832) (owner: 10TrainBranchBot) [18:42:59] (03CR) 10Ssingh: [V:03+1] "[Taking the liberty of adding a few folks to the review. Please remove yourself if desired.]" [puppet] - 10https://gerrit.wikimedia.org/r/1313948 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [18:43:41] !log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [18:44:25] RECOVERY - Check unit status of etcd-backup on conf2004 is OK: OK: Status of the systemd unit etcd-backup https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:45:35] !log jhancock@cumin2002 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2098.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [18:45:55] !log bking@cumin2003 START - Cookbook sre.dns.netbox [18:46:30] !log dduvall@deploy1003 rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.13 refs T430832 [18:46:34] T430832: 1.47.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T430832 [18:47:41] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be209[7,8] - https://phabricator.wikimedia.org/T424892#12164682 (10Jhancock.wm) [18:49:52] (03PS1) 10Scott French: etcd::backup: Explicitly set ETCDCTL_API to 2 [puppet] - 10https://gerrit.wikimedia.org/r/1318764 (https://phabricator.wikimedia.org/T428495) [18:52:11] (03CR) 10Ladsgroup: [C:03+2] Deny access to some particular format conversions [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1311830 (https://phabricator.wikimedia.org/T430528) (owner: 10Samwilson) [18:52:13] bking@cumin2003 reimage (PID 1149286) is awaiting input [18:52:17] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318764 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [18:53:24] train looks good. all done for today [18:54:51] (03CR) 10BCornwall: [C:03+1] hiera/dns3003: use_new_pdns_cfg: true [puppet] - 10https://gerrit.wikimedia.org/r/1318751 (owner: 10Ssingh) [18:55:32] (03CR) 10Ssingh: [C:03+2] "Post-merge +1 code review, host is running fine and pooled so won't reimage." [puppet] - 10https://gerrit.wikimedia.org/r/1318751 (owner: 10Ssingh) [18:57:39] (03CR) 10CDanis: [C:03+1] P:etcd::tlsproxy: Reintroduce sslcert::dhparam resource [puppet] - 10https://gerrit.wikimedia.org/r/1318760 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [18:58:18] (03CR) 10CDanis: [C:03+1] etcd::backup: Explicitly set ETCDCTL_API to 2 [puppet] - 10https://gerrit.wikimedia.org/r/1318764 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [19:00:47] (03CR) 10Scott French: [C:03+2] etcd::backup: Explicitly set ETCDCTL_API to 2 [puppet] - 10https://gerrit.wikimedia.org/r/1318764 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [19:01:03] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 236766440 and 27 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [19:01:05] (03CR) 10Scott French: [C:03+2] P:etcd::tlsproxy: Reintroduce sslcert::dhparam resource [puppet] - 10https://gerrit.wikimedia.org/r/1318760 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [19:02:01] (03PS1) 10Ahmon Dancy: python-build: Remove py2-bullseye image definition [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1318767 (https://phabricator.wikimedia.org/T432636) [19:03:14] (03Merged) 10jenkins-bot: Deny access to some particular format conversions [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1311830 (https://phabricator.wikimedia.org/T430528) (owner: 10Samwilson) [19:05:03] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 58264 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [19:05:50] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164722 (10VRiley-WMF) @Dzahn would you be able to take a look at Zuul1005? It looks like it was able to finish the installer but it looks... [19:06:18] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [19:06:36] 14SRE-Sprint-Week-Sustainability-March2023, 06Traffic, 13Patch-For-Review, 07Sustainability (Incident Followup): Experiment with single backend CDN nodes - https://phabricator.wikimedia.org/T288106#12164728 (10BCornwall) Here's the results of 7 days of a text node (cp3073): [[ https://grafana-rw.wikimedia... [19:06:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [19:06:55] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003" [19:06:57] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1005.eqiad.wmnet with OS trixie [19:07:14] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164729 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host zuul1005.eqiad.wmnet with OS trixi... [19:07:20] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164730 (10VRiley-WMF) Oh, I could be wrong. It looks like it just finished [19:11:44] FIRING: [3x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [19:13:09] (03PS2) 10Scott French: hieradata: Temporarily move etcd replication to conf2004 [puppet] - 10https://gerrit.wikimedia.org/r/1314016 (https://phabricator.wikimedia.org/T428495) [19:13:09] (03PS6) 10Scott French: hieradata: Enable the v2 API and set cluster_bootstrap false in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1309317 (https://phabricator.wikimedia.org/T428495) [19:13:13] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2203: Maintenance [19:13:35] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: Maintenance [19:13:43] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db2216 (T431660)', diff saved to https://phabricator.wikimedia.org/P95435 and previous config saved to /var/cache/conftool/dbconfig/20260728-191343-cwilliams.json [19:14:14] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1309317 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [19:14:45] (03CR) 10Scott French: "I've updated this to also reflect the cluster_bootstrap change from https://gerrit.wikimedia.org/r/1318725, in order to minimize restart c" [puppet] - 10https://gerrit.wikimedia.org/r/1309317 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [19:16:44] RESOLVED: [3x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [19:17:42] jiji@cumin1003 renumber-node (PID 3217686) is awaiting input [19:18:25] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164746 (10Dzahn) @VRiley-WMF Looks good to me :) I can ssh to zuul1005 and seems done. Also, thanks for using trixie. [19:19:35] PROBLEM - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:20:03] PROBLEM - NTP peers and stratum check on dns3003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/NTP [19:20:17] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db2216: Maintenance [19:21:20] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [19:21:35] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2013 - bking@cumin2003" [19:21:39] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2013 - bking@cumin2003" [19:21:40] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [19:21:40] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache wdqs2013.codfw.wmnet 84.0.192.10.in-addr.arpa 4.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [19:21:44] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2013.codfw.wmnet 84.0.192.10.in-addr.arpa 4.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [19:21:44] !log bking@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wdqs2013 [19:21:55] !log bking@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2013 [19:21:56] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2013 [19:23:27] PROBLEM - Host wdqs2013 is DOWN: PING CRITICAL - Packet loss = 100% [19:23:57] PROBLEM - ps1-d1-eqiad-infeed-load-tower-A-phase-Z on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:23:57] PROBLEM - ps1-d1-eqiad-infeed-load-tower-B-phase-X on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:23:57] PROBLEM - ps1-d1-eqiad-infeed-load-tower-A-phase-Y on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:23:59] PROBLEM - ps1-d1-eqiad-infeed-load-tower-B-phase-Z on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:23:59] PROBLEM - ps1-d1-eqiad-infeed-load-tower-A-phase-X on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:23:59] PROBLEM - ps1-d1-eqiad-infeed-load-tower-B-phase-Y on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:25:57] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1006] - vriley@cumin1003" [19:26:01] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1006] - vriley@cumin1003" [19:26:01] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [19:26:10] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host zuul1006 [19:26:28] !log bking@cumin2003 conftool action : set/pooled=yes; selector: name=wdqs2012\.codfw\.wmnet,dc=codfw,cluster=wdqs\-main,service=wdqs\-main [19:26:37] (03CR) 10RLazarus: [C:03+1] hieradata: Enable the v2 API and set cluster_bootstrap false in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1309317 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [19:27:36] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host zuul1006 [19:28:01] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:28:35] (03PS1) 10Ayounsi: Add missing PTR include for private1-23-ulsfo and private1-604-eqsin [dns] - 10https://gerrit.wikimedia.org/r/1318769 [19:29:54] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs2014.codfw.wmnet with OS bookworm [19:30:17] FIRING: ProbeDown: Service wdqs2013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs2013:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:32:42] vriley@cumin1003 provision (PID 3413675) is awaiting input [19:33:16] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs2014 [19:33:16] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2014 [19:35:17] FIRING: [2x] ProbeDown: Service wdqs2013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs2013:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:38:08] PROBLEM - Host ps1-d1-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [19:38:32] (03CR) 10Ssingh: [C:03+1] Add missing PTR include for private1-23-ulsfo and private1-604-eqsin (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1318769 (owner: 10Ayounsi) [19:38:53] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [19:39:18] !log eevans@cumin1003 END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Upgrade Java to 17.0.20 — T433028 - eevans@cumin1003 [19:39:22] T433028: Upgrade restbase cluster to Cassandra 5.0.8 - https://phabricator.wikimedia.org/T433028 [19:39:24] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [19:39:35] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [19:41:09] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2013.codfw.wmnet with reason: host reimage [19:41:46] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:42:56] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie [19:43:12] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12164805 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host zuul1006.eqiad.wmnet with OS t... [19:44:59] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2013.codfw.wmnet with reason: host reimage [19:48:06] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 235972608 and 31 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [19:48:15] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [19:48:34] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [19:49:38] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:51:44] (03Abandoned) 10Bking: opensearch: allowlist upstream plugins + overwrite [puppet] - 10https://gerrit.wikimedia.org/r/1271947 (https://phabricator.wikimedia.org/T423327) (owner: 10Ryan Kemper) [19:52:06] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 90112 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [19:52:49] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2014.codfw.wmnet with reason: host reimage [19:52:50] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [19:53:30] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [19:53:54] (03CR) 10Bking: [C:03+2] opensearch: support EcsLayout for the on-disk JSON server logs [puppet] - 10https://gerrit.wikimedia.org/r/1318740 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [19:54:05] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [19:54:15] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [19:57:49] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:58:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [19:58:58] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5) {#122601}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:59:09] (03CR) 10Bking: [C:03+2] cirrus: produce ECS server logs on relforge [puppet] - 10https://gerrit.wikimedia.org/r/1311440 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [19:59:28] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2014.codfw.wmnet with reason: host reimage [19:59:39] (03PS1) 10Anzx: bolwiki: set correct version of logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318718 (https://phabricator.wikimedia.org/T429951) [19:59:54] PROBLEM - Check if ntpsec.service has been restarted after /etc/ntpsec/ntp.conf was changed on dns3003 is CRITICAL: CRITICAL: Service ntpsec.service has not been restarted after /etc/ntpsec/ntp.conf was changed (gt 2h). https://wikitech.wikimedia.org/wiki/NTP%23Monitoring [19:59:57] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, July 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318718 (https://phabricator.wikimedia.org/T429951) (owner: 10Anzx) [20:00:04] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: May I have your attention please! UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T2000) [20:00:05] arlolra and anzx: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:20] o/ [20:01:08] o/ [20:01:58] anzx: do you want me to deploy your patch? [20:02:12] yes [20:02:26] ok [20:05:28] (03CR) 10TrainBranchBot: [C:03+2] "Approved by arlolra@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318718 (https://phabricator.wikimedia.org/T429951) (owner: 10Anzx) [20:05:28] (03CR) 10TrainBranchBot: [C:03+2] "Approved by arlolra@deploy1003 using scap backport" [extensions/ParserMigration] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318756 (https://phabricator.wikimedia.org/T433303) (owner: 10Arlolra) [20:05:28] (03CR) 10TrainBranchBot: [C:03+2] "Approved by arlolra@deploy1003 using scap backport" [extensions/ParserMigration] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318754 (https://phabricator.wikimedia.org/T433303) (owner: 10Arlolra) [20:06:38] (03Merged) 10jenkins-bot: bolwiki: set correct version of logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318718 (https://phabricator.wikimedia.org/T429951) (owner: 10Anzx) [20:06:51] (03Merged) 10jenkins-bot: Remove icon beside reporting link on desktop view [extensions/ParserMigration] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318756 (https://phabricator.wikimedia.org/T433303) (owner: 10Arlolra) [20:06:53] (03Merged) 10jenkins-bot: Remove icon beside reporting link on desktop view [extensions/ParserMigration] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318754 (https://phabricator.wikimedia.org/T433303) (owner: 10Arlolra) [20:07:01] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2216: Maintenance [20:07:15] !log arlolra@deploy1003 Started scap sync-world: Backport for [[gerrit:1318718|bolwiki: set correct version of logo (T429951)]], [[gerrit:1318756|Remove icon beside reporting link on desktop view (T433303)]], [[gerrit:1318754|Remove icon beside reporting link on desktop view (T433303)]] [20:07:21] T429951: Post-creation work for bolwiki - https://phabricator.wikimedia.org/T429951 [20:07:22] T433303: Chill out with the intrusive letter i with a circle around it - https://phabricator.wikimedia.org/T433303 [20:09:18] !log arlolra@deploy1003 anzx, arlolra: Backport for [[gerrit:1318718|bolwiki: set correct version of logo (T429951)]], [[gerrit:1318756|Remove icon beside reporting link on desktop view (T433303)]], [[gerrit:1318754|Remove icon beside reporting link on desktop view (T433303)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:10:08] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 126691680 and 17 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [20:10:30] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2013.codfw.wmnet with OS bookworm [20:10:40] arlolra: logo looks good, ok to continue [20:10:56] great, thanks [20:11:22] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [20:11:26] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [20:11:28] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [20:11:34] !log arlolra@deploy1003 anzx, arlolra: Continuing with deployment [20:11:42] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [20:11:51] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [20:12:02] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [20:12:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:12:55] (03CR) 10Scott French: [C:03+2] hieradata: Temporarily move etcd replication to conf2004 [puppet] - 10https://gerrit.wikimedia.org/r/1314016 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [20:15:08] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 89704 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [20:15:42] !log arlolra@deploy1003 Finished scap sync-world: Backport for [[gerrit:1318718|bolwiki: set correct version of logo (T429951)]], [[gerrit:1318756|Remove icon beside reporting link on desktop view (T433303)]], [[gerrit:1318754|Remove icon beside reporting link on desktop view (T433303)]] (duration: 08m 26s) [20:15:47] T429951: Post-creation work for bolwiki - https://phabricator.wikimedia.org/T429951 [20:15:48] T433303: Chill out with the intrusive letter i with a circle around it - https://phabricator.wikimedia.org/T433303 [20:16:10] arlolra: thanks for deploying [20:16:14] no problem [20:17:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by arlolra@deploy1003 using scap backport" [extensions/Kartographer] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318753 (https://phabricator.wikimedia.org/T433008) (owner: 10Arlolra) [20:17:23] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [20:17:31] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [20:19:21] !log switched etcd-mirror replication from conf2005 to conf2004 - T428495 [20:19:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:19:26] T428495: Migrate conf* hosts away from bullseye - https://phabricator.wikimedia.org/T428495 [20:19:33] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [20:19:36] RECOVERY - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [20:20:36] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [20:21:10] RECOVERY - ps1-d1-eqiad-infeed-load-tower-B-phase-X on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-B-phase-X 588 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:21:10] RECOVERY - ps1-d1-eqiad-infeed-load-tower-B-phase-Y on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-B-phase-Y 580 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:21:10] RECOVERY - ps1-d1-eqiad-infeed-load-tower-A-phase-Y on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-A-phase-Y 568 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:21:10] RECOVERY - ps1-d1-eqiad-infeed-load-tower-B-phase-Z on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-B-phase-Z 606 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:21:10] RECOVERY - ps1-d1-eqiad-infeed-load-tower-A-phase-Z on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-A-phase-Z 617 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:21:10] RECOVERY - ps1-d1-eqiad-infeed-load-tower-A-phase-X on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-A-phase-X 568 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:21:12] RECOVERY - Host ps1-d1-eqiad is UP: PING OK - Packet loss = 0%, RTA = 4.04 ms [20:21:48] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [20:21:58] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [20:23:47] (03PS7) 10Scott French: hieradata: Enable the v2 API and set cluster_bootstrap false in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1309317 (https://phabricator.wikimedia.org/T428495) [20:23:58] FIRING: [2x] JobUnavailable: Reduced availability for job etcdmirror in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:23:59] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1309317 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [20:26:27] (03CR) 10Scott French: [C:03+2] hieradata: Enable the v2 API and set cluster_bootstrap false in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1309317 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [20:26:53] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2014.codfw.wmnet with OS bookworm [20:27:13] (03Merged) 10jenkins-bot: Support older config values in $wgJsonConfigModels [extensions/Kartographer] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1318753 (https://phabricator.wikimedia.org/T433008) (owner: 10Arlolra) [20:27:18] (03PS1) 10JHathaway: mx: disable fuzzy_check for trusted networks [puppet] - 10https://gerrit.wikimedia.org/r/1318778 (https://phabricator.wikimedia.org/T383047) [20:27:33] !log arlolra@deploy1003 Started scap sync-world: Backport for [[gerrit:1318753|Support older config values in $wgJsonConfigModels (T433008)]] [20:27:34] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318778 (https://phabricator.wikimedia.org/T383047) (owner: 10JHathaway) [20:27:38] T433008: Wrong geoJSON error message if tag ist used with external data from Wikimedia Commons - https://phabricator.wikimedia.org/T433008 [20:29:32] !log arlolra@deploy1003 arlolra: Backport for [[gerrit:1318753|Support older config values in $wgJsonConfigModels (T433008)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:30:23] !log arlolra@deploy1003 arlolra: Continuing with deployment [20:30:29] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [20:30:39] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [20:31:38] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [20:31:48] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [20:32:41] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [20:32:51] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [20:33:44] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [20:33:54] (03PS2) 10JHathaway: mx: disable fuzzy_check for trusted networks [puppet] - 10https://gerrit.wikimedia.org/r/1318778 (https://phabricator.wikimedia.org/T383047) [20:33:56] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [20:34:04] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318778 (https://phabricator.wikimedia.org/T383047) (owner: 10JHathaway) [20:34:09] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [20:34:22] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [20:34:28] !log arlolra@deploy1003 Finished scap sync-world: Backport for [[gerrit:1318753|Support older config values in $wgJsonConfigModels (T433008)]] (duration: 06m 54s) [20:34:32] T433008: Wrong geoJSON error message if tag ist used with external data from Wikimedia Commons - https://phabricator.wikimedia.org/T433008 [20:35:08] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 126975784 and 18 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [20:35:21] (03CR) 10TrainBranchBot: [C:03+2] "Approved by arlolra@deploy1003 using scap backport" [vendor] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318746 (https://phabricator.wikimedia.org/T19006) (owner: 10Arlolra) [20:35:22] (03CR) 10TrainBranchBot: [C:03+2] "Approved by arlolra@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318747 (https://phabricator.wikimedia.org/T433018) (owner: 10Arlolra) [20:37:08] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2031472 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [20:37:13] (03CR) 10JHathaway: [C:03+1] hiera: url_downloader: enable LVS pool and IPIP [puppet] - 10https://gerrit.wikimedia.org/r/1313948 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [20:39:44] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in role puppetdb [puppet] - 10https://gerrit.wikimedia.org/r/1315097 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:41:14] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile puppetserver [puppet] - 10https://gerrit.wikimedia.org/r/1315095 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:41:30] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile puppet_compiler [puppet] - 10https://gerrit.wikimedia.org/r/1315093 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:41:44] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile pki [puppet] - 10https://gerrit.wikimedia.org/r/1315092 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:42:03] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile sre [puppet] - 10https://gerrit.wikimedia.org/r/1315091 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:42:20] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile postfix [puppet] - 10https://gerrit.wikimedia.org/r/1315090 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:42:37] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile netbox [puppet] - 10https://gerrit.wikimedia.org/r/1315089 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:43:04] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile mail [puppet] - 10https://gerrit.wikimedia.org/r/1315088 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:43:18] (03CR) 10Scott French: [C:03+1] "Very nice find!" [puppet] - 10https://gerrit.wikimedia.org/r/1318778 (https://phabricator.wikimedia.org/T383047) (owner: 10JHathaway) [20:43:23] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile locales [puppet] - 10https://gerrit.wikimedia.org/r/1315087 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:43:35] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie [20:43:40] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile kerberos [puppet] - 10https://gerrit.wikimedia.org/r/1315086 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:43:47] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12165009 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host zuul1006.eqiad.wmnet with OS t... [20:43:55] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile installserver [puppet] - 10https://gerrit.wikimedia.org/r/1315084 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:44:30] (03PS1) 10Andrew Bogott: cloud-vps: kick networkd if network breaks [puppet] - 10https://gerrit.wikimedia.org/r/1318779 (https://phabricator.wikimedia.org/T432426) [20:45:10] (03CR) 10CI reject: [V:04-1] cloud-vps: kick networkd if network breaks [puppet] - 10https://gerrit.wikimedia.org/r/1318779 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [20:45:21] (03PS2) 10Andrew Bogott: cloud-vps: kick networkd if network breaks [puppet] - 10https://gerrit.wikimedia.org/r/1318779 (https://phabricator.wikimedia.org/T432426) [20:45:59] (03CR) 10CI reject: [V:04-1] cloud-vps: kick networkd if network breaks [puppet] - 10https://gerrit.wikimedia.org/r/1318779 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [20:46:23] (03PS3) 10Andrew Bogott: cloud-vps: kick networkd if network breaks [puppet] - 10https://gerrit.wikimedia.org/r/1318779 (https://phabricator.wikimedia.org/T432426) [20:47:01] (03CR) 10CI reject: [V:04-1] cloud-vps: kick networkd if network breaks [puppet] - 10https://gerrit.wikimedia.org/r/1318779 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [20:48:42] (03PS4) 10Andrew Bogott: cloud-vps: kick networkd if network breaks [puppet] - 10https://gerrit.wikimedia.org/r/1318779 (https://phabricator.wikimedia.org/T432426) [20:48:58] FIRING: [2x] JobUnavailable: Reduced availability for job etcdmirror in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:49:12] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [20:49:36] (03Merged) 10jenkins-bot: Bump wikimedia/parsoid to 0.24.0-a17 [vendor] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318746 (https://phabricator.wikimedia.org/T19006) (owner: 10Arlolra) [20:49:48] (03Merged) 10jenkins-bot: Bump wikimedia/parsoid to 0.24.0-a17 [core] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1318747 (https://phabricator.wikimedia.org/T433018) (owner: 10Arlolra) [20:50:11] !log arlolra@deploy1003 Started scap sync-world: Backport for [[gerrit:1318746|Bump wikimedia/parsoid to 0.24.0-a17 (T19006 T234299 T360814 T385317 T432477 T433018)]], [[gerrit:1318747|Bump wikimedia/parsoid to 0.24.0-a17 (T433018)]] [20:50:25] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [20:50:30] T19006: Blank anchor [[#|anchor link]] should not link to the Main Page - https://phabricator.wikimedia.org/T19006 [20:50:31] T234299: Links like "[[:/y/]]" and "[[/y̫/]]" displayed incorrectly in preview - https://phabricator.wikimedia.org/T234299 [20:50:33] T360814: VisualEditor generate wrong label for specific it.wiki template - https://phabricator.wikimedia.org/T360814 [20:50:34] T385317: Section wrapping can break encapsulation continuity so that subsequent passes are confused - https://phabricator.wikimedia.org/T385317 [20:50:34] T432477: Template content encapsulation is incorrect for a large template - https://phabricator.wikimedia.org/T432477 [20:50:35] T433018: CTT tasks week of 2026-07-24 - https://phabricator.wikimedia.org/T433018 [20:50:40] (03CR) 10JHathaway: [C:03+2] mx: disable fuzzy_check for trusted networks [puppet] - 10https://gerrit.wikimedia.org/r/1318778 (https://phabricator.wikimedia.org/T383047) (owner: 10JHathaway) [20:51:15] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [20:51:19] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12165090 (10VRiley-WMF) Sure thing, I'm planning on this tomorrow. Thank you! [20:51:22] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1318779 (https://phabricator.wikimedia.org/T432426) (owner: 10Andrew Bogott) [20:52:03] (03CR) 10JHathaway: [C:03+2] mx: disable fuzzy_check for trusted networks (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1318778 (https://phabricator.wikimedia.org/T383047) (owner: 10JHathaway) [20:52:10] !log arlolra@deploy1003 arlolra: Backport for [[gerrit:1318746|Bump wikimedia/parsoid to 0.24.0-a17 (T19006 T234299 T360814 T385317 T432477 T433018)]], [[gerrit:1318747|Bump wikimedia/parsoid to 0.24.0-a17 (T433018)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:52:58] !log bking@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [20:53:02] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [20:53:06] !log bking@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 30s) [20:54:03] !log bking@deploy1003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [20:54:11] !log bking@deploy1003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 14s) [20:54:27] !log arlolra@deploy1003 arlolra: Continuing with deployment [20:55:18] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2013.codfw.wmnet, repooling source-only afterwards [20:55:47] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2020.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling source-only afterwards [20:58:30] !log arlolra@deploy1003 Finished scap sync-world: Backport for [[gerrit:1318746|Bump wikimedia/parsoid to 0.24.0-a17 (T19006 T234299 T360814 T385317 T432477 T433018)]], [[gerrit:1318747|Bump wikimedia/parsoid to 0.24.0-a17 (T433018)]] (duration: 08m 19s) [20:58:43] T19006: Blank anchor [[#|anchor link]] should not link to the Main Page - https://phabricator.wikimedia.org/T19006 [20:58:43] T234299: Links like "[[:/y/]]" and "[[/y̫/]]" displayed incorrectly in preview - https://phabricator.wikimedia.org/T234299 [20:58:44] T360814: VisualEditor generate wrong label for specific it.wiki template - https://phabricator.wikimedia.org/T360814 [20:58:44] T385317: Section wrapping can break encapsulation continuity so that subsequent passes are confused - https://phabricator.wikimedia.org/T385317 [20:58:45] T432477: Template content encapsulation is incorrect for a large template - https://phabricator.wikimedia.org/T432477 [20:58:45] T433018: CTT tasks week of 2026-07-24 - https://phabricator.wikimedia.org/T433018 [21:00:04] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260728T2100) [21:11:27] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile idp [puppet] - 10https://gerrit.wikimedia.org/r/1315083 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:11:47] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile ganeti [puppet] - 10https://gerrit.wikimedia.org/r/1315082 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:12:04] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in module ssh [puppet] - 10https://gerrit.wikimedia.org/r/1315080 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:12:19] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in module rsync [puppet] - 10https://gerrit.wikimedia.org/r/1315079 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:12:38] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in module puppet_compiler [puppet] - 10https://gerrit.wikimedia.org/r/1315078 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:12:46] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in module puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/1315075 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:13:04] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in module postgresql [puppet] - 10https://gerrit.wikimedia.org/r/1315074 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:13:18] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in module librenms [puppet] - 10https://gerrit.wikimedia.org/r/1315073 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:13:42] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in module aptrepo [puppet] - 10https://gerrit.wikimedia.org/r/1315072 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:14:11] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in module ganeti [puppet] - 10https://gerrit.wikimedia.org/r/1315070 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:14:25] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in module rancid [puppet] - 10https://gerrit.wikimedia.org/r/1315067 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:14:56] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile toolforge [puppet] - 10https://gerrit.wikimedia.org/r/1315007 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:16:59] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile base [puppet] - 10https://gerrit.wikimedia.org/r/1315096 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:17:13] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in misc realm [puppet] - 10https://gerrit.wikimedia.org/r/1315098 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:25:14] (03PS1) 10TChin: [eventstreams] Bump to v0.20.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318784 (https://phabricator.wikimedia.org/T432824) [21:42:08] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 172859288 and 24 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [21:43:08] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2589256 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [21:45:35] (03CR) 10Cwhite: [C:03+2] logstash: add parameters needed for security plugin [puppet] - 10https://gerrit.wikimedia.org/r/1305769 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [21:48:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:53:45] (03Restored) 10D3r1ck01: Update email for shell user "derick" [puppet] - 10https://gerrit.wikimedia.org/r/1165526 (owner: 10D3r1ck01) [21:55:26] (03CR) 10D3r1ck01: "@slyngshede@wikimedia.org, you abandoned the patch with "User no longer active", this is me 😊" [puppet] - 10https://gerrit.wikimedia.org/r/1165526 (owner: 10D3r1ck01) [22:03:50] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie [22:04:03] 10ops-eqiad, 06SRE, 06collaboration-services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12165284 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host zuul1006.eqiad.wmnet with OS trixi... [22:08:06] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2013.codfw.wmnet, repooling source-only afterwards [22:08:10] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [22:11:19] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2020.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling source-only afterwards [22:14:09] jiji@cumin1003 renumber-node (PID 3217686) is awaiting input [22:14:28] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [22:49:58] !log jiji@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1138.eqiad.wmnet [22:49:59] !log jiji@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1138.eqiad.wmnet [22:50:02] !log jiji@cumin1003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1138.eqiad.wmnet [22:50:47] 06SRE, 10LDAP-Access-Requests: Grant Access to NDA for jaleman-vdr-wmf - https://phabricator.wikimedia.org/T433417#12165406 (10Dzahn) That is if it's really about the NDA group and you are not WMF staff. If you are WMF staff then normally the "WMF" group is used that gives the same or more access than the NDA... [22:53:00] 06SRE, 06Infrastructure-Foundations, 10vm-requests: eqiad/codfw: 2 VM request for codesearch - https://phabricator.wikimedia.org/T433316#12165414 (10Dzahn) [22:57:30] (03PS1) 10Cwhite: opensearch: add security-plugin specific configuration to opensearch.yml [puppet] - 10https://gerrit.wikimedia.org/r/1318792 (https://phabricator.wikimedia.org/T350516) [23:02:15] (03PS1) 10Reedy: CommonSettings: Remove $wgOATHAuthEnforce2FAForAll block [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318793 [23:03:22] (03CR) 10Reedy: [C:04-2] "Not yet" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1318793 (owner: 10Reedy) [23:28:41] (03PS9) 10Cwhite: profile: add initial opensearch security plugin config [puppet] - 10https://gerrit.wikimedia.org/r/1306792 (https://phabricator.wikimedia.org/T350516) [23:29:19] (03PS3) 10Cwhite: opensearch: add security-plugin specific configuration to opensearch.yml [puppet] - 10https://gerrit.wikimedia.org/r/1318792 (https://phabricator.wikimedia.org/T350516) [23:29:19] (03CR) 10Cwhite: "PCC OK: https://puppet-compiler.wmflabs.org/output/1306003/9087/" [puppet] - 10https://gerrit.wikimedia.org/r/1318792 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [23:42:23] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1318796 [23:42:23] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1318796 (owner: 10TrainBranchBot) [23:44:28] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [alerts] - 10https://gerrit.wikimedia.org/r/1304769 (https://phabricator.wikimedia.org/T407138) (owner: 10Hnowlan) [23:46:37] (03CR) 10Andrea Denisse: [C:03+1] "I left a non-blocking comment, LGTM!!" [puppet] - 10https://gerrit.wikimedia.org/r/1318792 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [23:50:24] PROBLEM - Host ml-serve2004 is DOWN: PING CRITICAL - Packet loss = 100% [23:53:19] (03CR) 10Andrea Denisse: [C:03+1] "Nice!! LGTM!!" [puppet] - 10https://gerrit.wikimedia.org/r/1306792 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [23:54:01] (03CR) 10Scott French: "Thanks, Blake!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318671 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [23:54:50] FIRING: KubernetesCalicoDown: ml-serve2004.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s-mlserve&var-instance=ml-serve2004.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [23:58:58] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5) {#122601}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [23:59:04] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus