[00:04:56] !log swfrench@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1274.eqiad.wmnet with OS trixie [00:09:24] 06SRE, 06Infrastructure-Foundations, 07Documentation: Update Wikitech documentation about WMF's mirrors of Debian - https://phabricator.wikimedia.org/T432420#12131013 (10Urbanecm_WMF) [00:11:30] 06SRE, 10DNS, 06Infrastructure-Foundations: Remove mirrors.wikimedia.org from public DNS - https://phabricator.wikimedia.org/T432422 (10Urbanecm_WMF) 03NEW [00:11:33] (03PS1) 10Urbanecm: Remove mirrors.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/1311584 (https://phabricator.wikimedia.org/T432422) [00:12:03] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review, 06Release-Engineering-Team (Radar), 07User-notice: Sunsetting mirrors.wikimedia.org - https://phabricator.wikimedia.org/T416707#12131048 (10Urbanecm_WMF) Logged few follow-ups: * {T432422} * {T432420} [00:13:25] (03PS1) 10Ryan Kemper: Constrain WDQS all transfers [cookbooks] - 10https://gerrit.wikimedia.org/r/1311585 (https://phabricator.wikimedia.org/T430880) [00:13:35] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1025.eqiad.wmnet with OS bookworm [00:13:51] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1027.eqiad.wmnet with OS bookworm [00:14:56] !log swfrench@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1274.eqiad.wmnet [00:14:57] !log swfrench@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1274.eqiad.wmnet [00:14:58] !log swfrench@cumin1003 END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1274.eqiad.wmnet [00:23:13] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, Restore wdqs1027 after Bookworm reimage) xfer scholarly_articles from wdqs2026.codfw.wmnet -> wdqs1027.eqiad.wmnet, repooling both afterwards [00:23:17] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [00:23:46] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, Restore wdqs1025 after Bookworm reimage) xfer wikidata_main from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling source-only afterwards [00:26:41] RESOLVED: [2x] ConfdResourceFailed: confd resource _srv_config-master_pybal_eqiad_wdqs-internal-scholarly.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [00:28:41] (03PS2) 10Ryan Kemper: wdqs: make wdqs-all flag for wdqs-main hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1311585 (https://phabricator.wikimedia.org/T430880) [00:35:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:42:39] (03PS1) 10Urbanecm: linkrecommendation: Bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311586 [00:42:49] (03CR) 10Urbanecm: [C:03+2] linkrecommendation: Bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311586 (owner: 10Urbanecm) [00:45:01] (03Merged) 10jenkins-bot: linkrecommendation: Bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311586 (owner: 10Urbanecm) [00:52:49] !log urbanecm@deploy2003 helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply [00:53:30] !log urbanecm@deploy2003 helmfile [staging] START helmfile.d/services/linkrecommendation: apply [00:54:00] !log urbanecm@deploy2003 helmfile [codfw] START helmfile.d/services/linkrecommendation: apply [00:54:22] !log urbanecm@deploy2003 helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply [00:54:28] !log urbanecm@deploy2003 helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply [00:55:34] !log urbanecm@deploy2003 helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply [01:06:09] RECOVERY - PyBal IPVS diff check on lvs1020 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [01:07:57] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, Restore wdqs1027 after Bookworm reimage) xfer scholarly_articles from wdqs2026.codfw.wmnet -> wdqs1027.eqiad.wmnet, repooling both afterwards [01:08:00] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [01:08:09] RECOVERY - PyBal IPVS diff check on lvs1019 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [01:10:37] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [01:12:27] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [01:12:38] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1311587 [01:12:38] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1311587 (owner: 10TrainBranchBot) [01:12:58] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, Restore wdqs1025 after Bookworm reimage) xfer wikidata_main from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling source-only afterwards [01:13:02] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [01:20:41] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1311587 (owner: 10TrainBranchBot) [01:46:39] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12131165 (10Papaul) [02:00:22] !log mwpresync@deploy2003 Started scap build-images: Publishing wmf/next image [02:07:12] !log mwpresync@deploy2003 Finished scap build-images: Publishing wmf/next image (duration: 06m 49s) [02:10:43] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:15:43] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:37:02] !log ryankemper@cumin2003 conftool action : set/pooled=true; selector: dnsdisc=wdqs-internal-scholarly,name=eqiad [02:37:43] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, Restore categories on wdqs1025 after Bookworm reimage) xfer categories from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards [02:37:46] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [02:44:46] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, Restore categories on wdqs1025 after Bookworm reimage) xfer categories from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards [02:44:50] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [02:46:12] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, Restore categories on wdqs1025 after Bookworm reimage) xfer categories from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards [02:46:12] !log ryankemper@cumin2003 END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T430880, Restore categories on wdqs1025 after Bookworm reimage) xfer categories from wdqs2020.codfw.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards [02:53:48] FIRING: KubernetesCalicoDown: wikikube-worker1264.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1264.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [02:57:04] FIRING: HelmReleaseBadStatus: Helm release wdqs/scholarly-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [03:05:39] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [03:06:33] PROBLEM - MariaDB Replica Lag: s1 on clouddb1017 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 303.46 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [03:08:33] RECOVERY - MariaDB Replica Lag: s1 on clouddb1017 is OK: OK slave_sql_lag Replication lag: 0.25 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [04:35:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:19:42] (03CR) 10Trueg: [C:03+1] Add main/scholarly and internal/external specific configs for EventGate streams. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311547 (https://phabricator.wikimedia.org/T429407) (owner: 10Lerickson) [05:31:38] (03PS1) 10Ayounsi: Remove old stub OSPF definitions [homer/public] - 10https://gerrit.wikimedia.org/r/1311738 [05:41:20] (03PS1) 10Ayounsi: Configure OSPF for Arelion transports eqiad/esams eqiad/eqsin [homer/public] - 10https://gerrit.wikimedia.org/r/1311740 (https://phabricator.wikimedia.org/T428488) [05:54:09] 06SRE, 10SRE-Access-Requests: Requesting access to Superset for jniren-ctr - https://phabricator.wikimedia.org/T432273#12131355 (10jcrespo) [05:58:13] 06SRE, 10SRE-Access-Requests: Requesting access to Superset for jniren-ctr - https://phabricator.wikimedia.org/T432273#12131356 (10jcrespo) Thank you for confirming. Based on that, we should grant the **wmf group**, not nda, which is intended for volunteers with a separate NDA. @jniren-ctr Requesting the wmf g... [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260717T0600) [06:13:17] (03PS1) 10Kevin Bazira: ml-services: Add tts-section-generator deployment configs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311749 (https://phabricator.wikimedia.org/T432308) [06:15:58] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:33:30] (03PS1) 10Kevin Bazira: admin_ng: Add tts-section-generator namespace to LiftWing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311750 (https://phabricator.wikimedia.org/T432308) [06:43:45] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [06:48:32] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host netbox1003.eqiad.wmnet [06:52:41] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox1003.eqiad.wmnet [06:53:45] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [06:53:48] FIRING: KubernetesCalicoDown: wikikube-worker1264.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1264.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:57:05] FIRING: HelmReleaseBadStatus: Helm release wdqs/scholarly-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260717T0700) [07:04:10] !log elukey@cumin1003 START - Cookbook sre.puppet.disable-merges [07:04:11] !log elukey@cumin1003 END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) [07:04:16] !log btullis@cumin1003 END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster [07:05:04] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet [07:05:39] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:11:38] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet [07:11:58] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host puppetserver2002.codfw.wmnet [07:14:39] (03PS1) 10Trueg: WDQS: Upgrade wdqs-streaming-consumer to 1.2.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311752 [07:18:45] FIRING: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [07:18:56] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2002.codfw.wmnet [07:19:07] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet [07:19:49] 06SRE, 06Infrastructure-Foundations, 10netops: JunOS: Investigate BGP PIC Edge / Protection - https://phabricator.wikimedia.org/T432381#12131427 (10ayounsi) Another optimization to consider: https://www.juniper.net/documentation/us/en/software/junos/bgp/topics/topic-map/bgp-route-prioritization.html [07:20:25] FIRING: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:20:25] (03PS1) 10Kevin Bazira: tts-section-generator: Add deployment credentials and services proxy [puppet] - 10https://gerrit.wikimedia.org/r/1311753 (https://phabricator.wikimedia.org/T432308) [07:28:11] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet [07:28:21] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host puppetserver1001.eqiad.wmnet [07:28:45] FIRING: [3x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [07:37:23] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1001.eqiad.wmnet [07:37:53] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host puppetdb2003.codfw.wmnet [07:43:45] FIRING: [5x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [07:46:10] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetdb2003.codfw.wmnet [07:46:24] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host puppetdb1003.eqiad.wmnet [07:47:49] (03PS22) 10Federico Ceratto: mysql: update replication source [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) [07:50:25] FIRING: [2x] SystemdUnitFailed: ferm.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:52:25] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetdb1003.eqiad.wmnet [07:53:45] FIRING: [7x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [07:57:43] !log elukey@cumin2003 START - Cookbook sre.hosts.reboot-single for host cumin1003.eqiad.wmnet [08:01:16] (03CR) 10Federico Ceratto: "Based on feedback on IRC I deleted the pylint comments on top" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [08:03:34] !log elukey@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cumin1003.eqiad.wmnet [08:05:38] !log elukey@cumin1003 START - Cookbook sre.puppet.disable-merges [08:06:06] !log elukey@cumin1003 END (FAIL) - Cookbook sre.puppet.disable-merges (exit_code=99) [08:06:28] FIRING: KeyholderUnarmed: 2 unarmed Keyholder key(s) on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [08:09:05] !log cgoubert@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1335.eqiad.wmnet [08:09:09] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1335.eqiad.wmnet [08:09:44] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1335.eqiad.wmnet [08:09:49] (03PS3) 10Federico Ceratto: sre.mysql.pool: support esX sections [cookbooks] - 10https://gerrit.wikimedia.org/r/1311405 (https://phabricator.wikimedia.org/T430769) [08:10:27] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1335.eqiad.wmnet with OS trixie [08:10:57] !log cgoubert@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1335 [08:11:43] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [08:13:17] !log cgoubert@cumin2003 START - Cookbook sre.dns.netbox [08:13:45] FIRING: [7x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [08:14:27] !log elukey@cumin1003 START - Cookbook sre.puppet.disable-merges [08:14:28] !log elukey@cumin1003 END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0) [08:18:06] !log cgoubert@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1335 - cgoubert@cumin2003" [08:18:11] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1335 - cgoubert@cumin2003" [08:18:11] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [08:18:12] !log cgoubert@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1335.eqiad.wmnet 150.32.64.10.in-addr.arpa 0.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [08:18:15] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1335.eqiad.wmnet 150.32.64.10.in-addr.arpa 0.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [08:18:16] !log cgoubert@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1335 [08:18:45] FIRING: [7x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [08:18:45] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1335 [08:18:45] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1335 [08:20:25] RESOLVED: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:23:45] RESOLVED: [7x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [08:26:51] 06SRE, 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12131542 (10elukey) Hey folks! IIUC this task is to track both Bookworm's and Trixie's updates, but I don't s... [08:28:11] (03PS1) 10Kevin Bazira: service::catalog: Add tts-section-generator [puppet] - 10https://gerrit.wikimedia.org/r/1311794 (https://phabricator.wikimedia.org/T432308) [08:28:58] (03CR) 10Elukey: "ping! :)" [puppet] - 10https://gerrit.wikimedia.org/r/1310558 (https://phabricator.wikimedia.org/T432089) (owner: 10Elukey) [08:29:08] (03CR) 10Elukey: "ping :)" [puppet] - 10https://gerrit.wikimedia.org/r/1310986 (https://phabricator.wikimedia.org/T432089) (owner: 10Elukey) [08:30:47] !log cgoubert@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1335.eqiad.wmnet with reason: host reimage [08:30:57] FIRING: ProbeDown: Service text:80 has failed probes (http_text_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:33:36] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [08:34:27] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [08:34:28] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1335.eqiad.wmnet with reason: host reimage [08:35:39] (03PS1) 10Elukey: CHANGELOG: add changelogs for release v13.1.0 [software/spicerack] - 10https://gerrit.wikimedia.org/r/1311795 [08:35:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:35:57] RESOLVED: ProbeDown: Service text:80 has failed probes (http_text_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:36:03] (03PS2) 10Btullis: kubernetes: add the vanilla PostgreSQL 17 image to common_images [puppet] - 10https://gerrit.wikimedia.org/r/1311486 (https://phabricator.wikimedia.org/T432104) [08:36:23] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311486 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [08:36:28] RESOLVED: KeyholderUnarmed: 1 unarmed Keyholder key(s) on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [08:37:34] (03PS1) 10Btullis: dse-k8s: add the postgresql-superset-metrics deployment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311796 (https://phabricator.wikimedia.org/T432104) [08:39:55] (03CR) 10CI reject: [V:04-1] dse-k8s: add the postgresql-superset-metrics deployment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311796 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [08:42:00] (03CR) 10Elukey: [C:03+2] CHANGELOG: add changelogs for release v13.1.0 [software/spicerack] - 10https://gerrit.wikimedia.org/r/1311795 (owner: 10Elukey) [08:44:22] (03PS1) 10Elukey: Upstream release v13.1.0 [software/spicerack] (debian) - 10https://gerrit.wikimedia.org/r/1311798 [08:45:06] !log jiji@cumin2003 START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp1001.eqiad.wmnet [08:47:10] (03PS1) 10Kevin Bazira: conftool-data: Add tts-section-generator [puppet] - 10https://gerrit.wikimedia.org/r/1311799 (https://phabricator.wikimedia.org/T432308) [08:50:08] !log jiji@cumin2003 START - Cookbook sre.hosts.reboot-single for host testreduce1002.eqiad.wmnet [08:50:59] (03CR) 10Elukey: [C:03+2] Upstream release v13.1.0 [software/spicerack] (debian) - 10https://gerrit.wikimedia.org/r/1311798 (owner: 10Elukey) [08:51:13] !log jiji@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp1001.eqiad.wmnet [08:51:18] !log jiji@cumin2003 START - Cookbook sre.hosts.reboot-single for host wikikube-worker-exp2001.codfw.wmnet [08:51:33] (03PS1) 10Ayounsi: gNMI: override state/high-speed for sub-rated interfaces [puppet] - 10https://gerrit.wikimedia.org/r/1311802 (https://phabricator.wikimedia.org/T374614) [08:52:05] (03CR) 10CI reject: [V:04-1] gNMI: override state/high-speed for sub-rated interfaces [puppet] - 10https://gerrit.wikimedia.org/r/1311802 (https://phabricator.wikimedia.org/T374614) (owner: 10Ayounsi) [08:52:15] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1335.eqiad.wmnet with OS trixie [08:53:22] (03PS2) 10Ayounsi: gNMI: override state/high-speed for sub-rated interfaces [puppet] - 10https://gerrit.wikimedia.org/r/1311802 (https://phabricator.wikimedia.org/T374614) [08:53:34] (03CR) 10Fabfur: [C:03+1] varnish: Add SPDX license header to VCL files [puppet] - 10https://gerrit.wikimedia.org/r/1311540 (owner: 10BCornwall) [08:54:09] !log jiji@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host testreduce1002.eqiad.wmnet [08:55:06] (03CR) 10Fabfur: [C:03+1] "agree, now please let's insert `-*- mode: vcl-mode -*-` (joking)" [puppet] - 10https://gerrit.wikimedia.org/r/1311541 (owner: 10BCornwall) [08:55:20] !log jiji@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wikikube-worker-exp2001.codfw.wmnet [08:56:17] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Create alerting for saturation on sub-rated interfaces - https://phabricator.wikimedia.org/T374614#12131571 (10ayounsi) As said on IRC, dunno if I'm opening some kind of pandora box by using starlark/python, but that seems like a good appro... [08:59:18] (03CR) 10Hnowlan: [C:03+1] swift: Migrate storage of results to asyncio [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1310618 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup) [08:59:46] (03PS1) 10Fabfur: cache::haproxy: fix correlation id syntax [puppet] - 10https://gerrit.wikimedia.org/r/1311807 (https://phabricator.wikimedia.org/T426379) [09:01:35] (03CR) 10Btullis: [C:03+1] datahubsearch: remove DNS record [dns] - 10https://gerrit.wikimedia.org/r/1311538 (https://phabricator.wikimedia.org/T432148) (owner: 10Bking) [09:04:14] (03CR) 10Btullis: [C:03+1] "Looks good, with one whitespace nit." [puppet] - 10https://gerrit.wikimedia.org/r/1311533 (https://phabricator.wikimedia.org/T432148) (owner: 10Bking) [09:04:49] !log uploaded spicerack_13.1.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia [09:04:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:04:55] (03CR) 10Btullis: [C:03+1] opensearch: remove references to soon-to-be-decom'd service [alerts] - 10https://gerrit.wikimedia.org/r/1311542 (https://phabricator.wikimedia.org/T432148) (owner: 10Bking) [09:05:40] (03CR) 10Fabfur: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311807 (https://phabricator.wikimedia.org/T426379) (owner: 10Fabfur) [09:06:46] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1335.eqiad.wmnet [09:06:48] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1335.eqiad.wmnet [09:06:50] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1335.eqiad.wmnet [09:08:27] !log cgoubert@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1336.eqiad.wmnet [09:08:31] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1336.eqiad.wmnet [09:09:05] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1336.eqiad.wmnet [09:09:09] (03CR) 10DCausse: [C:03+1] cirrus: Stop building phrase suggester variant [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307211 (https://phabricator.wikimedia.org/T390858) (owner: 10Ebernhardson) [09:09:17] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1336.eqiad.wmnet with OS trixie [09:09:47] !log cgoubert@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1336 [09:09:56] !log elukey@cumin2002 START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet [09:10:17] !log cgoubert@cumin2003 START - Cookbook sre.dns.netbox [09:11:31] !log elukey@cumin2002 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet [09:11:43] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [09:14:37] !log cgoubert@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1336 - cgoubert@cumin2003" [09:14:41] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1336 - cgoubert@cumin2003" [09:14:42] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:14:42] !log cgoubert@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1336.eqiad.wmnet 152.32.64.10.in-addr.arpa 2.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [09:14:46] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1336.eqiad.wmnet 152.32.64.10.in-addr.arpa 2.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [09:14:46] !log cgoubert@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1336 [09:18:23] 06SRE, 10SRE-Access-Requests, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Grant Access to analytics-privatedata-users for zsinger - https://phabricator.wikimedia.org/T426458#12131604 (10BTullis) 05Open→03Resolved This change is now deployed and the user has confirmed that access to the Superset da... [09:18:59] (03PS3) 10Ayounsi: gNMI: override state/high-speed for sub-rated interfaces [puppet] - 10https://gerrit.wikimedia.org/r/1311802 (https://phabricator.wikimedia.org/T374614) [09:19:08] (03PS1) 10Elukey: sre.hosts.bmc-user-mgmt: add explicit remove for the root user [cookbooks] - 10https://gerrit.wikimedia.org/r/1311814 (https://phabricator.wikimedia.org/T426180) [09:19:43] !log elukey@cumin2002 START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet [09:20:06] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1336 [09:20:07] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1336 [09:21:08] !log elukey@cumin2002 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet [09:21:35] 10ops-codfw, 10ops-drmrs, 10ops-eqdfw, 10ops-eqiad, and 6 others: Fix cables with placeholders names and planned status - https://phabricator.wikimedia.org/T432317#12131611 (10ayounsi) And please don't forget to run Homer after changing a cable, as the cable ID is reflected in the interface description. [09:24:21] (03PS2) 10Elukey: sre.hosts.bmc-user-mgmt: add explicit remove for the root user [cookbooks] - 10https://gerrit.wikimedia.org/r/1311814 (https://phabricator.wikimedia.org/T426180) [09:24:44] (03PS2) 10Btullis: dse-k8s: add the postgresql-superset-metrics deployment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311796 (https://phabricator.wikimedia.org/T432104) [09:26:40] (03CR) 10CI reject: [V:04-1] dse-k8s: add the postgresql-superset-metrics deployment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311796 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [09:26:51] !log elukey@cumin2002 START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet [09:28:28] !log elukey@cumin2002 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet [09:29:08] (03CR) 10Fabfur: [C:03+2] cache::haproxy: fix correlation id syntax [puppet] - 10https://gerrit.wikimedia.org/r/1311807 (https://phabricator.wikimedia.org/T426379) (owner: 10Fabfur) [09:29:16] !log elukey@cumin2002 START - Cookbook sre.hosts.bmc-user-mgmt for host an-test-coord1002.eqiad.wmnet [09:29:22] !log elukey@cumin2002 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host an-test-coord1002.eqiad.wmnet [09:30:38] (03CR) 10Hnowlan: [C:03+2] "Change related to T432270" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311087 (owner: 10Hnowlan) [09:33:21] !log cgoubert@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1336.eqiad.wmnet with reason: host reimage [09:36:30] FIRING: Traffic bill over quota: Alert for device cr4-ulsfo.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [09:39:24] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1336.eqiad.wmnet with reason: host reimage [09:40:06] !log elukey@cumin2002 START - Cookbook sre.hosts.bmc-user-mgmt for host acmechief-test2001.codfw.wmnet,acmechief-test1001.eqiad.wmnet,an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet,db-test[2001-2002].codfw.wmnet,db-test[1001-1003].eqiad.wmn [09:40:06] et,dse-k8s-wdqs-test2001.codfw.wmnet,dse-k8s-wdqs-test1001.eqiad.wmnet,ganeti-test[2001-2003].codfw.wmnet,idm-test1001.wikimedia.org,idp-test[1005,2005].wikimedia.org,kafka-test[1006-1010].eqiad.wmnet,maps-test2001.codfw.wmnet,o11ytest2001.codfw.wmnet,o11ytest1001.eqiad.wmnet,parsoidtest1001.eqiad.wmnet,pybal-test2003.codfw.wmnet,sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet,testreduce1 [09:40:06] 002.eqiad.wmnet,zookeeper-test1002.eqiad.wmnet [09:40:21] need to fix this log sigh [09:44:11] (03CR) 10Ayounsi: [C:03+1] sre.hosts.bmc-user-mgmt: add explicit remove for the root user [cookbooks] - 10https://gerrit.wikimedia.org/r/1311814 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [09:44:31] (03CR) 10Elukey: [C:03+2] sre.hosts.bmc-user-mgmt: add explicit remove for the root user [cookbooks] - 10https://gerrit.wikimedia.org/r/1311814 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [09:45:09] !log elukey@cumin2002 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host acmechief-test2001.codfw.wmnet,acmechief-test1001.eqiad.wmnet,an-test-client1002.eqiad.wmnet,an-test-coord[1001-1002].eqiad.wmnet,an-test-druid1001.eqiad.wmnet,an-test-master[1001-1004].eqiad.wmnet,an-test-presto1001.eqiad.wmnet,an-test-ui1001.eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet,db-test[2001-2002].codfw.wmnet,db-test[1 [09:45:09] 001-1003].eqiad.wmnet,dse-k8s-wdqs-test2001.codfw.wmnet,dse-k8s-wdqs-test1001.eqiad.wmnet,ganeti-test[2001-2003].codfw.wmnet,idm-test1001.wikimedia.org,idp-test[1005,2005].wikimedia.org,kafka-test[1006-1010].eqiad.wmnet,maps-test2001.codfw.wmnet,o11ytest2001.codfw.wmnet,o11ytest1001.eqiad.wmnet,parsoidtest1001.eqiad.wmnet,pybal-test2003.codfw.wmnet,sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqia [09:45:09] d.wmnet,testreduce1002.eqiad.wmnet,zookeeper-test1002.eqiad.wmnet [09:47:18] (03PS1) 10Elukey: sre.hosts.bmc-user-mgmt: limit the log message [cookbooks] - 10https://gerrit.wikimedia.org/r/1311818 [09:47:45] !log elukey@cumin2002 START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts (check the cookbook's logs for more details.) [09:47:50] !log elukey@cumin2002 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts (check the cookbook's logs for more details.) [09:47:56] yeah better now [09:50:18] (03PS3) 10Btullis: kubernetes: add the vanilla PostgreSQL 17 image to common_images [puppet] - 10https://gerrit.wikimedia.org/r/1311486 (https://phabricator.wikimedia.org/T432104) [09:51:47] (03PS2) 10Trueg: WDQS: Upgrade wdqs-streaming-consumer to 1.2.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311752 (https://phabricator.wikimedia.org/T430782) [09:55:22] (03CR) 10Ayounsi: [C:03+1] sre.hosts.bmc-user-mgmt: limit the log message (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1311818 (owner: 10Elukey) [09:55:38] (03PS4) 10Btullis: kubernetes: add the vanilla PostgreSQL 17 image to common_images [puppet] - 10https://gerrit.wikimedia.org/r/1311486 (https://phabricator.wikimedia.org/T432104) [09:56:02] (03CR) 10Brouberol: [C:03+1] kubernetes: add the vanilla PostgreSQL 17 image to common_images [puppet] - 10https://gerrit.wikimedia.org/r/1311486 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [09:56:30] RESOLVED: Traffic bill over quota: Alert for device cr4-ulsfo.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [09:58:13] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1336.eqiad.wmnet with OS trixie [10:03:07] (03PS2) 10Elukey: sre.hosts.bmc-user-mgmt: fix SAL log and improve root removal [cookbooks] - 10https://gerrit.wikimedia.org/r/1311818 [10:03:40] !log elukey@cumin2002 START - Cookbook sre.hosts.bmc-user-mgmt for 13 hosts (check the cookbook's logs for more details.) [10:04:56] (03CR) 10Btullis: [C:03+2] kubernetes: add the vanilla PostgreSQL 17 image to common_images [puppet] - 10https://gerrit.wikimedia.org/r/1311486 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [10:05:16] !log elukey@cumin2002 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 13 hosts (check the cookbook's logs for more details.) [10:05:19] cgoubert@cumin2003 renumber-node (PID 2244617) is awaiting input [10:07:03] (03CR) 10Trueg: [C:03+2] Add main/scholarly and internal/external specific configs for EventGate streams. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311547 (https://phabricator.wikimedia.org/T429407) (owner: 10Lerickson) [10:07:20] (03CR) 10Trueg: [C:03+2] Update default Eventgate URL in the wdqs-proxy config. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311460 (https://phabricator.wikimedia.org/T432336) (owner: 10Lerickson) [10:07:53] (03CR) 10Trueg: [C:03+2] wdqs: version bump qlever [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310533 (https://phabricator.wikimedia.org/T431271) (owner: 10Gmodena) [10:09:14] (03Merged) 10jenkins-bot: Add main/scholarly and internal/external specific configs for EventGate streams. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311547 (https://phabricator.wikimedia.org/T429407) (owner: 10Lerickson) [10:09:39] !log cwilliams@cumin1003 START - Cookbook sre.mysql.multiinstance_reboot for db1176.eqiad.wmnet [10:09:54] (03Merged) 10jenkins-bot: Update default Eventgate URL in the wdqs-proxy config. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311460 (https://phabricator.wikimedia.org/T432336) (owner: 10Lerickson) [10:10:08] !log cwilliams@cumin1003 END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db1176.eqiad.wmnet [10:10:33] (03Merged) 10jenkins-bot: wdqs: version bump qlever [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310533 (https://phabricator.wikimedia.org/T431271) (owner: 10Gmodena) [10:10:35] (03PS3) 10Elukey: sre.hosts.bmc-user-mgmt: fix SAL log and improve root removal [cookbooks] - 10https://gerrit.wikimedia.org/r/1311818 [10:10:44] (03CR) 10Elukey: sre.hosts.bmc-user-mgmt: fix SAL log and improve root removal (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1311818 (owner: 10Elukey) [10:11:29] !log cwilliams@cumin1003 START - Cookbook sre.mysql.multiinstance_reboot for db1176.eqiad.wmnet [10:13:11] (03PS1) 10Hnowlan: mailman: migrate mailman_hours_until_empty_outbound_queue nrpe check [alerts] - 10https://gerrit.wikimedia.org/r/1311820 (https://phabricator.wikimedia.org/T370157) [10:15:08] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1336.eqiad.wmnet [10:15:10] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1336.eqiad.wmnet [10:15:12] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1336.eqiad.wmnet [10:15:22] (03PS3) 10Trueg: WDQS: Upgrade wdqs-streaming-consumer to 0.2.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311752 (https://phabricator.wikimedia.org/T430782) [10:15:36] RECOVERY - jenkins_service_running on contint1002 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [10:15:58] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:16:07] (03CR) 10CI reject: [V:04-1] mailman: migrate mailman_hours_until_empty_outbound_queue nrpe check [alerts] - 10https://gerrit.wikimedia.org/r/1311820 (https://phabricator.wikimedia.org/T370157) (owner: 10Hnowlan) [10:16:21] (03PS1) 10Hnowlan: lists: disable check in advance of removal [puppet] - 10https://gerrit.wikimedia.org/r/1311821 (https://phabricator.wikimedia.org/T370157) [10:16:42] !log cgoubert@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1337.eqiad.wmnet [10:16:46] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1337.eqiad.wmnet [10:16:59] !log cwilliams@cumin1003 END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db1176.eqiad.wmnet [10:17:20] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1337.eqiad.wmnet [10:17:50] (03CR) 10Ayounsi: [C:03+1] sre.hosts.bmc-user-mgmt: fix SAL log and improve root removal [cookbooks] - 10https://gerrit.wikimedia.org/r/1311818 (owner: 10Elukey) [10:17:56] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1337.eqiad.wmnet with OS trixie [10:18:27] !log cgoubert@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1337 [10:18:36] PROBLEM - jenkins_service_running on contint1002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [10:19:48] (03PS2) 10Hnowlan: mailman: migrate mailman_hours_until_empty_outbound_queue nrpe check [alerts] - 10https://gerrit.wikimedia.org/r/1311820 (https://phabricator.wikimedia.org/T370157) [10:21:22] (03CR) 10Elukey: [C:03+2] sre.hosts.bmc-user-mgmt: fix SAL log and improve root removal [cookbooks] - 10https://gerrit.wikimedia.org/r/1311818 (owner: 10Elukey) [10:21:30] cgoubert@cumin2003 renumber-node (PID 2259682) is awaiting input [10:21:47] !log cgoubert@cumin2003 START - Cookbook sre.dns.netbox [10:26:22] !log cgoubert@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1337 - cgoubert@cumin2003" [10:26:26] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1337 - cgoubert@cumin2003" [10:26:27] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [10:26:27] !log cgoubert@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1337.eqiad.wmnet 154.32.64.10.in-addr.arpa 4.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [10:26:31] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1337.eqiad.wmnet 154.32.64.10.in-addr.arpa 4.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [10:26:31] !log cgoubert@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1337 [10:26:36] RECOVERY - jenkins_service_running on contint1002 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [10:27:02] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1337 [10:27:02] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1337 [10:27:45] !log trueg@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [10:28:02] !log cwilliams@cumin1003 START - Cookbook sre.mysql.multiinstance_reboot for db-test[1001-1003].eqiad.wmnet [10:29:36] PROBLEM - jenkins_service_running on contint1002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [10:30:30] !log trueg@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [10:34:30] !log trueg@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [10:34:32] !log trueg@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [10:39:00] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[1001-1003].eqiad.wmnet [10:39:20] RESOLVED: HelmReleaseBadStatus: Helm release wdqs/scholarly-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:39:20] !log cgoubert@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1337.eqiad.wmnet with reason: host reimage [10:39:26] !log trueg@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [10:40:33] !log cwilliams@cumin1003 START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet [10:43:44] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1337.eqiad.wmnet with reason: host reimage [10:48:04] (03CR) 10Hnowlan: [C:03+1] shellbox-video: disable debug logging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1124752 (owner: 10Effie Mouzeli) [10:49:24] (03Abandoned) 10Hnowlan: (WIP) rest-gateway: add rest.php routes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1173345 (https://phabricator.wikimedia.org/T400132) (owner: 10Hnowlan) [10:50:04] !log trueg@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [10:51:10] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[2001-2002].codfw.wmnet [10:51:33] (03Abandoned) 10Hnowlan: grafana: replace grafana read-only with IDP-authentication [puppet] - 10https://gerrit.wikimedia.org/r/1248419 (https://phabricator.wikimedia.org/T418671) (owner: 10Hnowlan) [10:53:48] FIRING: KubernetesCalicoDown: wikikube-worker1264.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1264.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:55:30] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns7002 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is , dns.git is 508411e3df7d829a07600fac3eab6739321f5836) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260717T0700) [11:00:05] jelto, arnoldokoth, mutante, and arnaudb: It is that lovely time of the day again! You are hereby commanded to deploy GitLab version upgrades. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260717T1100). [11:02:38] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1337.eqiad.wmnet with OS trixie [11:05:30] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns7002 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is , dns.git is 508411e3df7d829a07600fac3eab6739321f5836) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:05:39] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:08:45] (03PS3) 10Tiziano Fogli: sloth: add rule to detect missing slo:sli_error:ratio_rate metric [alerts] - 10https://gerrit.wikimedia.org/r/1310604 (https://phabricator.wikimedia.org/T432140) [11:17:48] (03CR) 10Gmodena: [C:03+2] WDQS: Upgrade wdqs-streaming-consumer to 0.2.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311752 (https://phabricator.wikimedia.org/T430782) (owner: 10Trueg) [11:17:59] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1337.eqiad.wmnet [11:18:01] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1337.eqiad.wmnet [11:18:02] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1337.eqiad.wmnet [11:19:05] !log cgoubert@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1338.eqiad.wmnet [11:19:08] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1338.eqiad.wmnet [11:19:18] (03PS3) 10Gmodena: wdqs: bump init container memory and startup probe timeout [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311429 (https://phabricator.wikimedia.org/T429150) [11:19:43] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1338.eqiad.wmnet [11:20:24] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1338.eqiad.wmnet with OS trixie [11:20:28] (03Merged) 10jenkins-bot: WDQS: Upgrade wdqs-streaming-consumer to 0.2.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311752 (https://phabricator.wikimedia.org/T430782) (owner: 10Trueg) [11:20:30] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns7002 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is , dns.git is 508411e3df7d829a07600fac3eab6739321f5836) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [11:20:42] !log cgoubert@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1338 [11:20:51] (03CR) 10Gmodena: [V:03+2] wdqs: roll back wdqs-proxy to v0.3.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311438 (https://phabricator.wikimedia.org/T432336) (owner: 10Gmodena) [11:20:58] !log cgoubert@cumin2003 START - Cookbook sre.dns.netbox [11:21:13] (03CR) 10Clément Goubert: [C:03+2] modules/rsync: add pidfile to stunnel.conf [puppet] - 10https://gerrit.wikimedia.org/r/1310589 (https://phabricator.wikimedia.org/T432108) (owner: 10Kamila Součková) [11:21:15] (03PS3) 10Gmodena: wdqs: roll back wdqs-proxy to v0.3.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311438 (https://phabricator.wikimedia.org/T432336) [11:21:50] (03CR) 10Gmodena: [C:03+2] wdqs: bump init container memory and startup probe timeout [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311429 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [11:23:59] !log cwilliams@cumin1003 START - Cookbook sre.mysql.multiinstance_reboot for db2230.codfw.wmnet [11:24:03] (03Merged) 10jenkins-bot: wdqs: bump init container memory and startup probe timeout [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311429 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [11:25:32] !log cgoubert@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1338 - cgoubert@cumin2003" [11:25:36] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1338 - cgoubert@cumin2003" [11:25:36] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:25:36] !log cgoubert@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1338.eqiad.wmnet 155.32.64.10.in-addr.arpa 5.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [11:25:40] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1338.eqiad.wmnet 155.32.64.10.in-addr.arpa 5.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [11:25:41] !log cgoubert@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1338 [11:26:06] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1338 [11:26:06] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1338 [11:26:12] (03CR) 10Gmodena: [C:03+2] wdqs: roll back wdqs-proxy to v0.3.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311438 (https://phabricator.wikimedia.org/T432336) (owner: 10Gmodena) [11:28:40] (03Merged) 10jenkins-bot: wdqs: roll back wdqs-proxy to v0.3.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311438 (https://phabricator.wikimedia.org/T432336) (owner: 10Gmodena) [11:30:47] (03CR) 10Gmodena: [C:03+1] Add main/scholarly and internal/external specific configs for EventGate streams. (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311547 (https://phabricator.wikimedia.org/T429407) (owner: 10Lerickson) [11:31:52] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db2230.codfw.wmnet [11:37:20] (03PS1) 10Klausman: hiera/deployment_server: Add pseudo-secrtes for new LW service tts-section-generator [labs/private] - 10https://gerrit.wikimedia.org/r/1311831 (https://phabricator.wikimedia.org/T430536) [11:37:25] FIRING: SystemdUnitFailed: stunnel4.service on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:37:54] (03PS2) 10Klausman: hiera/deployment_server: Add pseudo-secrets for new LW service tts-section-generator [labs/private] - 10https://gerrit.wikimedia.org/r/1311831 (https://phabricator.wikimedia.org/T430536) [11:38:15] (03CR) 10Klausman: [V:03+2 C:03+2] hiera/deployment_server: Add pseudo-secrets for new LW service tts-section-generator [labs/private] - 10https://gerrit.wikimedia.org/r/1311831 (https://phabricator.wikimedia.org/T430536) (owner: 10Klausman) [11:38:34] !log cgoubert@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1338.eqiad.wmnet with reason: host reimage [11:39:58] (03CR) 10Klausman: [V:03+2 C:03+2] admin_ng: Add tts-section-generator namespace to LiftWing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311750 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [11:40:27] (03PS1) 10Urbanecm: [Growth] Deploy automated mentor list cleaner to all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311832 (https://phabricator.wikimedia.org/T431804) [11:42:26] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1338.eqiad.wmnet with reason: host reimage [11:42:50] (03PS1) 10Hnowlan: team-sre: make CertAlmostExpired team-aware [alerts] - 10https://gerrit.wikimedia.org/r/1311834 (https://phabricator.wikimedia.org/T414662) [11:43:50] (03CR) 10CWilliams: [C:03+1] "Testing showed the desired confirmations and expected failures appearing, including the replica check. The only issues that I hit were rel" [cookbooks] - 10https://gerrit.wikimedia.org/r/1290806 (https://phabricator.wikimedia.org/T420203) (owner: 10FNegri) [11:47:25] RESOLVED: SystemdUnitFailed: stunnel4.service on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:48:43] (03Merged) 10jenkins-bot: admin_ng: Add tts-section-generator namespace to LiftWing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311750 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [11:51:36] !log klausman@deploy2003 helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [11:53:03] !log klausman@deploy2003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [11:54:58] !log klausman@deploy2003 helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [11:56:13] !log klausman@deploy2003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [11:56:50] (03CR) 10CWilliams: mysql: update replication source (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [11:59:15] !log klausman@deploy2003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [12:01:26] !log klausman@deploy2003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [12:02:07] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1338.eqiad.wmnet with OS trixie [12:04:53] (03CR) 10Clément Goubert: [C:03+1] tts-section-generator: Add deployment credentials and services proxy [puppet] - 10https://gerrit.wikimedia.org/r/1311753 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [12:04:57] (03CR) 10Clément Goubert: [C:03+1] service::catalog: Add tts-section-generator [puppet] - 10https://gerrit.wikimedia.org/r/1311794 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [12:05:08] (03CR) 10Clément Goubert: [C:03+1] conftool-data: Add tts-section-generator [puppet] - 10https://gerrit.wikimedia.org/r/1311799 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [12:07:49] (03CR) 10Federico Ceratto: mysql: update replication source (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [12:11:33] (03CR) 10Klausman: [V:03+2 C:03+2] ml-services: Add tts-section-generator deployment configs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311749 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [12:13:32] (03Merged) 10jenkins-bot: ml-services: Add tts-section-generator deployment configs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311749 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [12:13:33] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [12:14:45] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [12:15:15] (03CR) 10CWilliams: "I have additionally suggested moving out functionality that is reusable, especially given that this is a new cookbook. The other is a repl" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [12:15:44] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [12:16:32] (03PS2) 10Hnowlan: kafka: migrate check_kafka_ssl to alertmanager [puppet] - 10https://gerrit.wikimedia.org/r/1307405 (https://phabricator.wikimedia.org/T407117) [12:16:57] (03CR) 10Hnowlan: "Good catch, thank you!" [puppet] - 10https://gerrit.wikimedia.org/r/1307405 (https://phabricator.wikimedia.org/T407117) (owner: 10Hnowlan) [12:17:02] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [12:17:43] (03PS1) 10DCausse: flink-k8s-operator: enable savepoint.dispose-on-delete [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311840 (https://phabricator.wikimedia.org/T432455) [12:18:58] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1338.eqiad.wmnet [12:19:00] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1338.eqiad.wmnet [12:19:02] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1338.eqiad.wmnet [12:20:16] (03CR) 10Trueg: [C:03+1] wdqs: bump init container memory and startup probe timeout (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311429 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [12:26:28] (03CR) 10Cathal Mooney: "Overall lgtm, one small comment in-line. I guess we should look at adjusting the wmf-plugin to set the [20Gbps] in the description based " [puppet] - 10https://gerrit.wikimedia.org/r/1311802 (https://phabricator.wikimedia.org/T374614) (owner: 10Ayounsi) [12:27:43] (03PS1) 10Trueg: WDQSv2: Several small adjustments [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311841 [12:30:39] (03PS2) 10Trueg: WDQSv2: Several small adjustments [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311841 [12:31:40] 06SRE, 10SRE-Access-Requests: Requesting access to Superset for jniren-ctr - https://phabricator.wikimedia.org/T432273#12132194 (10Jniren-ctr) Thanks @jcrespo - I have requested that permission for wmf group. Re other two points: Contract end date - Its currently 31st July 2026 but an extension to 30 Septembe... [12:31:47] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [12:31:57] !log trueg@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [12:32:06] !log trueg@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [12:32:07] (03CR) 10Cathal Mooney: gNMI: override state/high-speed for sub-rated interfaces (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311802 (https://phabricator.wikimedia.org/T374614) (owner: 10Ayounsi) [12:32:15] !log trueg@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [12:35:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:37:51] (03CR) 10FNegri: [C:03+2] sre.mysql.multiinstance_reboot: new cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1290806 (https://phabricator.wikimedia.org/T420203) (owner: 10FNegri) [12:39:23] (03CR) 10Cathal Mooney: gNMI: override state/high-speed for sub-rated interfaces (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311802 (https://phabricator.wikimedia.org/T374614) (owner: 10Ayounsi) [12:42:01] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on backup1019 - https://phabricator.wikimedia.org/T431367#12132216 (10jcrespo) All good: ` backup1019:~$ sudo perccli64 /c0 show Generating detailed summary of the adapter, it may take a while to complete. CLI Version = 007.1910.0000.0000 Oct 08, 2021 Operati... [12:43:30] (03Merged) 10jenkins-bot: sre.mysql.multiinstance_reboot: new cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1290806 (https://phabricator.wikimedia.org/T420203) (owner: 10FNegri) [12:43:33] (03PS4) 10Ayounsi: gNMI: override state/high-speed for sub-rated interfaces [puppet] - 10https://gerrit.wikimedia.org/r/1311802 (https://phabricator.wikimedia.org/T374614) [12:43:37] (03CR) 10Ayounsi: "yeah that would be ideal, but it can be manual until we get there." [puppet] - 10https://gerrit.wikimedia.org/r/1311802 (https://phabricator.wikimedia.org/T374614) (owner: 10Ayounsi) [12:48:54] (03PS1) 10Btullis: airflow: let DAGs manage run-scoped scratch volumes on TopoLVM [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311846 (https://phabricator.wikimedia.org/T428235) [12:49:17] !log jiji@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb1014.eqiad.wmnet [12:51:45] (03CR) 10CWilliams: mysql: update replication source (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [12:51:59] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [12:52:40] (03CR) 10CWilliams: mysql: update replication source (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [12:54:06] (03CR) 10Cathal Mooney: gNMI: override state/high-speed for sub-rated interfaces (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311802 (https://phabricator.wikimedia.org/T374614) (owner: 10Ayounsi) [12:54:50] (03PS3) 10Klausman: profile::k8s::deployment_server: add config for tts-section-generator [puppet] - 10https://gerrit.wikimedia.org/r/1311843 (https://phabricator.wikimedia.org/T432308) [12:54:58] (03CR) 10Klausman: [C:03+2] profile::k8s::deployment_server: add config for tts-section-generator [puppet] - 10https://gerrit.wikimedia.org/r/1311843 (https://phabricator.wikimedia.org/T432308) (owner: 10Klausman) [12:55:24] !log jiji@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1014.eqiad.wmnet [12:55:28] !log jiji@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb1016.eqiad.wmnet [12:57:07] 06SRE, 10SRE-Access-Requests: Requesting access to Superset for jniren-ctr - https://phabricator.wikimedia.org/T432273#12132246 (10jcrespo) **end date** - Whatever is true at the moment is ok, we will not remove access without asking first as it is common there are extensions, which is why we ask for a: **Cont... [12:58:10] 06SRE, 10SRE-Access-Requests: Requesting access to Superset for jniren-ctr - https://phabricator.wikimedia.org/T432273#12132247 (10Jniren-ctr) @jcrespo great. Then its 30th June and Nazneen Nawaz. Thank you. [12:59:52] (03CR) 10Elukey: conftool-data: Add tts-section-generator (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311799 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [13:00:51] !log fnegri@cumin1003 START - Cookbook sre.mysql.multiinstance_reboot for 15 hosts [13:01:02] !log jiji@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1016.eqiad.wmnet [13:01:06] !log jiji@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb2012.codfw.wmnet [13:01:23] !log fnegri@cumin1003 END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for 15 hosts [13:03:11] 06SRE, 10SRE-Access-Requests: Requesting access to Superset for jniren-ctr - https://phabricator.wikimedia.org/T432273#12132254 (10jcrespo) [13:03:16] (03CR) 10CDobbins: "Hey, @sbassett@wikimedia.org, sorry to bug you again, but could I get a +1? Gerrit removed the previous one after I got rid of the unneede" [puppet] - 10https://gerrit.wikimedia.org/r/1311069 (owner: 10CDobbins) [13:03:56] !log cwilliams@cumin1003 START - Cookbook sre.mysql.update-replication [13:04:04] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0) [13:04:33] (03PS1) 10EMcFarland: EventStreamConfig: remove stream used in past experiments [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311851 (https://phabricator.wikimedia.org/T428265) [13:05:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:06:21] !log jiji@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2012.codfw.wmnet [13:06:25] !log jiji@cumin2003 START - Cookbook sre.hosts.reboot-single for host rdb2014.codfw.wmnet [13:06:42] 06SRE, 10SRE-Access-Requests: Requesting access to Superset for jniren-ctr - https://phabricator.wikimedia.org/T432273#12132263 (10jcrespo) @Fabfur handing this to you, this is ready for the group and LDAP data.yaml patch as soon as the WMF group is approved by IF https://ldap.toolforge.org/user/jniren-ctr [13:11:31] !log jiji@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2014.codfw.wmnet [13:12:54] 10SRE-swift-storage, 10Thumbor: Gradually drop all thumbnails as a one-off clean up - https://phabricator.wikimedia.org/T379942#12132266 (10Ladsgroup) [13:13:22] 10SRE-swift-storage, 10Thumbor: Gradually drop all thumbnails as a one-off clean up - https://phabricator.wikimedia.org/T379942#12132271 (10Ladsgroup) [13:20:27] !log cwilliams@cumin1003 START - Cookbook sre.mysql.update-replication [13:20:58] (03CR) 10Blake: wmnet: Add CNAME records for mw-pretrain. (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1311074 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [13:21:36] (03CR) 10Blake: [C:03+2] wmnet: Add CNAME records for mw-pretrain. [dns] - 10https://gerrit.wikimedia.org/r/1311074 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [13:21:46] (03PS1) 10Fabfur: admin: record ldap access for jniren-ctr [puppet] - 10https://gerrit.wikimedia.org/r/1311852 (https://phabricator.wikimedia.org/T432273) [13:22:21] !log blake@dns1004 START - running authdns-update [13:23:34] cwilliams@cumin1003 update-replication (PID 45319) is awaiting input [13:24:19] !log blake@dns1004 END - running authdns-update [13:27:52] (03PS1) 10Andrew Bogott: codfw1dev cloudcontrol: Don't install barbican [puppet] - 10https://gerrit.wikimedia.org/r/1311853 (https://phabricator.wikimedia.org/T432349) [13:27:53] !log cwilliams@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) [13:27:54] (03PS1) 10Andrew Bogott: Openstack: remove files and references to barbican [puppet] - 10https://gerrit.wikimedia.org/r/1311854 (https://phabricator.wikimedia.org/T432349) [13:29:18] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311853 (https://phabricator.wikimedia.org/T432349) (owner: 10Andrew Bogott) [13:32:17] (03PS2) 10Andrew Bogott: codfw1dev cloudcontrol: Don't install barbican [puppet] - 10https://gerrit.wikimedia.org/r/1311853 (https://phabricator.wikimedia.org/T432349) [13:32:17] (03PS2) 10Andrew Bogott: Openstack: remove files and references to barbican [puppet] - 10https://gerrit.wikimedia.org/r/1311854 (https://phabricator.wikimedia.org/T432349) [13:36:04] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311853 (https://phabricator.wikimedia.org/T432349) (owner: 10Andrew Bogott) [13:36:07] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311854 (https://phabricator.wikimedia.org/T432349) (owner: 10Andrew Bogott) [13:37:33] (03PS1) 10FNegri: sre.mysql.multiinstance_reboot: fix replica check [cookbooks] - 10https://gerrit.wikimedia.org/r/1311857 (https://phabricator.wikimedia.org/T420203) [13:37:46] (03PS16) 10Federico Ceratto: cookbooks/sre/mysql/decommission: add cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) [13:39:17] !log fnegri@cumin1003 START - Cookbook sre.mysql.multiinstance_reboot for clouddb1013.eqiad.wmnet [13:40:58] (03CR) 10Andrew Bogott: "I plan to remove barbican packages and files by hand after merging this. Seems easier than having puppet do it." [puppet] - 10https://gerrit.wikimedia.org/r/1311853 (https://phabricator.wikimedia.org/T432349) (owner: 10Andrew Bogott) [13:45:53] !log fnegri@cumin1003 END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for clouddb1013.eqiad.wmnet [13:46:38] (03CR) 10Lerickson: Add main/scholarly and internal/external specific configs for EventGate streams. (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311547 (https://phabricator.wikimedia.org/T429407) (owner: 10Lerickson) [13:47:23] (03CR) 10FNegri: "Tested on clouddb1013 and it fixed the issue. @cwilliams@wikimedia.org can you please re-test it on one of your hosts to verify that it st" [cookbooks] - 10https://gerrit.wikimedia.org/r/1311857 (https://phabricator.wikimedia.org/T420203) (owner: 10FNegri) [13:48:21] (03PS2) 10FNegri: sre.mysql.multiinstance_reboot: fix replica check [cookbooks] - 10https://gerrit.wikimedia.org/r/1311857 (https://phabricator.wikimedia.org/T420203) [13:50:03] (03CR) 10FNegri: [C:03+1] codfw1dev cloudcontrol: Don't install barbican [puppet] - 10https://gerrit.wikimedia.org/r/1311853 (https://phabricator.wikimedia.org/T432349) (owner: 10Andrew Bogott) [13:51:53] (03PS1) 10Blake: api-gateway: remove DNS for api-gateway service. [dns] - 10https://gerrit.wikimedia.org/r/1311856 (https://phabricator.wikimedia.org/T432445) [13:52:00] (03PS1) 10Blake: api-gateway: Remove probes. [puppet] - 10https://gerrit.wikimedia.org/r/1311859 (https://phabricator.wikimedia.org/T432445) [13:52:05] (03PS1) 10Blake: api-gateway: Switch to service_setup. [puppet] - 10https://gerrit.wikimedia.org/r/1311860 (https://phabricator.wikimedia.org/T432445) [13:52:09] (03PS1) 10Blake: api-gateway: Remove service definition [puppet] - 10https://gerrit.wikimedia.org/r/1311861 (https://phabricator.wikimedia.org/T432445) [13:55:05] (03PS4) 10Blake: kubernetes: Add a debug deployment for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1311049 (https://phabricator.wikimedia.org/T427668) [13:55:38] (03CR) 10CI reject: [V:04-1] kubernetes: Add a debug deployment for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1311049 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [13:56:26] (03PS5) 10Blake: kubernetes: Add a debug deployment for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1311049 (https://phabricator.wikimedia.org/T427668) [13:59:25] (03CR) 10Clément Goubert: [C:03+1] conftool-data: Add tts-section-generator (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311799 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [14:01:03] (03CR) 10Ayounsi: [C:03+2] Configure OSPF for Arelion transports eqiad/esams eqiad/eqsin [homer/public] - 10https://gerrit.wikimedia.org/r/1311740 (https://phabricator.wikimedia.org/T428488) (owner: 10Ayounsi) [14:01:37] (03CR) 10Ayounsi: [C:03+2] "Self-merging (metric of 5000) to start collecting uptime stats." [homer/public] - 10https://gerrit.wikimedia.org/r/1311740 (https://phabricator.wikimedia.org/T428488) (owner: 10Ayounsi) [14:02:25] (03Merged) 10jenkins-bot: Configure OSPF for Arelion transports eqiad/esams eqiad/eqsin [homer/public] - 10https://gerrit.wikimedia.org/r/1311740 (https://phabricator.wikimedia.org/T428488) (owner: 10Ayounsi) [14:02:55] !log cgoubert@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1339.eqiad.wmnet [14:02:59] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1339.eqiad.wmnet [14:03:32] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1339.eqiad.wmnet [14:06:17] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1339.eqiad.wmnet with OS trixie [14:06:46] !log cgoubert@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1339 [14:09:49] cgoubert@cumin2003 renumber-node (PID 2308257) is awaiting input [14:09:55] !log cgoubert@cumin2003 START - Cookbook sre.dns.netbox [14:10:37] (03CR) 10Btullis: [C:03+2] kubernetes: add deploy tokens for the postgresql-superset-metrics namespace [puppet] - 10https://gerrit.wikimedia.org/r/1311480 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [14:12:02] (03PS1) 10CDobbins: dns: set `sudo_user` as `authdns` [puppet] - 10https://gerrit.wikimedia.org/r/1311862 [14:14:24] !log cgoubert@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1339 - cgoubert@cumin2003" [14:14:29] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1339 - cgoubert@cumin2003" [14:14:29] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:14:29] !log cgoubert@cumin2003 START - Cookbook sre.dns.wipe-cache wikikube-worker1339.eqiad.wmnet 156.32.64.10.in-addr.arpa 6.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [14:14:33] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1339.eqiad.wmnet 156.32.64.10.in-addr.arpa 6.5.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [14:14:34] !log cgoubert@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1339 [14:15:24] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1339 [14:15:24] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1339 [14:15:58] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:19:56] (03CR) 10Scott French: [C:03+1] kubernetes: Add a debug deployment for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1311049 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [14:27:19] (03PS3) 10CDobbins: dns: set `sudo_user` as `authdns` [puppet] - 10https://gerrit.wikimedia.org/r/1311862 [14:27:54] !log cgoubert@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1339.eqiad.wmnet with reason: host reimage [14:27:59] (03CR) 10Ssingh: "Yeah this should work. Please run PCC for !dns7002 as well." [puppet] - 10https://gerrit.wikimedia.org/r/1311862 (owner: 10CDobbins) [14:32:31] FIRING: Traffic bill over quota: Alert for device cr2-eqsin.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [14:33:42] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1339.eqiad.wmnet with reason: host reimage [14:42:53] (03CR) 10CDobbins: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9028/co" [puppet] - 10https://gerrit.wikimedia.org/r/1311862 (owner: 10CDobbins) [14:44:57] (03CR) 10Ssingh: "Looks good, one minor nit about the commit message. I will +1 on Monday. I see no concerns merging it today but yeah, why bother on a Frid" [puppet] - 10https://gerrit.wikimedia.org/r/1311862 (owner: 10CDobbins) [14:46:57] (03PS4) 10CDobbins: P:dns::auth::update: set `sudo_user` as `authdns` [puppet] - 10https://gerrit.wikimedia.org/r/1311862 [14:49:41] (03CR) 10Ssingh: "Looks good. will +1 on Monday!" [puppet] - 10https://gerrit.wikimedia.org/r/1311862 (owner: 10CDobbins) [14:49:48] !log kamila@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1264.eqiad.wmnet with OS trixie [14:49:52] !log kamila@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1264.eqiad.wmnet with OS trixie [14:50:29] !log kamila@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1264.eqiad.wmnet with OS trixie [14:51:48] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1339.eqiad.wmnet with OS trixie [14:52:31] RESOLVED: Traffic bill over quota: Alert for device cr2-eqsin.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [14:53:48] (03PS3) 10Trueg: WDQSv2: Several small adjustments [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311841 [14:53:48] FIRING: KubernetesCalicoDown: wikikube-worker1264.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1264.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:54:23] (03CR) 10SBassett: [C:03+1] varnish: update CSP report-only header [puppet] - 10https://gerrit.wikimedia.org/r/1311069 (owner: 10CDobbins) [15:05:39] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:07:19] (03CR) 10Brouberol: airflow: let DAGs manage run-scoped scratch volumes on TopoLVM (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311846 (https://phabricator.wikimedia.org/T428235) (owner: 10Btullis) [15:12:26] (03PS4) 10Trueg: WDQSv2: Several small adjustments [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311841 [15:13:56] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1339.eqiad.wmnet [15:13:58] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1339.eqiad.wmnet [15:14:01] !log cgoubert@cumin2003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1339.eqiad.wmnet [15:35:51] !log jhathaway@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-out1001.wikimedia.org with reason: T431659 [15:39:03] (03CR) 10Dzahn: "setting author to Arnaudb - he did the original patch that this re-reverts" [dns] - 10https://gerrit.wikimedia.org/r/1310135 (owner: 10Dzahn) [15:41:21] !log kamila@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1264.eqiad.wmnet with OS trixie [15:41:46] !log kamila@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1264.eqiad.wmnet with OS trixie [15:56:22] (03PS4) 10Tiziano Fogli: sloth: add rule to detect missing slo:sli_error:ratio_rate metric [alerts] - 10https://gerrit.wikimedia.org/r/1310604 (https://phabricator.wikimedia.org/T432140) [15:57:44] (03CR) 10Bking: [C:03+1] wdqs: make wdqs-all flag for wdqs-main hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1311585 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [15:59:06] (03PS5) 10Tiziano Fogli: sloth: add rule to detect missing slo:sli_error:ratio_rate metric [alerts] - 10https://gerrit.wikimedia.org/r/1310604 (https://phabricator.wikimedia.org/T432140) [16:00:18] !log jhathaway@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-out2001.wikimedia.org with reason: T431659 [16:01:07] !log kamila@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1264.eqiad.wmnet with reason: host reimage [16:03:29] (03PS1) 10TheDJ: generatePngAndMidi.sh: Remove SCORE_SAFE [extensions/Score] (wmf/1.47.0-wmf.11) - 10https://gerrit.wikimedia.org/r/1311872 (https://phabricator.wikimedia.org/T432418) [16:05:05] !log kamila@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1264.eqiad.wmnet with reason: host reimage [16:06:33] (03PS6) 10Tiziano Fogli: sloth: add rule to detect missing slo:sli_error:ratio_rate metric [alerts] - 10https://gerrit.wikimedia.org/r/1310604 (https://phabricator.wikimedia.org/T432140) [16:07:28] !log jhathaway@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-in1001.wikimedia.org with reason: T431659 [16:10:09] !log jhathaway@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-in2001.wikimedia.org with reason: T431659 [16:10:43] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:15:43] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:26:16] !log kamila@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1264.eqiad.wmnet with OS trixie [16:28:23] (03CR) 10Reedy: [C:03+2] generatePngAndMidi.sh: Remove SCORE_SAFE [extensions/Score] (wmf/1.47.0-wmf.11) - 10https://gerrit.wikimedia.org/r/1311872 (https://phabricator.wikimedia.org/T432418) (owner: 10TheDJ) [16:29:43] (03Merged) 10jenkins-bot: generatePngAndMidi.sh: Remove SCORE_SAFE [extensions/Score] (wmf/1.47.0-wmf.11) - 10https://gerrit.wikimedia.org/r/1311872 (https://phabricator.wikimedia.org/T432418) (owner: 10TheDJ) [16:31:35] !log reedy@deploy2003 Started scap sync-world: Backport for [[gerrit:1311872|generatePngAndMidi.sh: Remove SCORE_SAFE (T432418)]] [16:31:39] T432418: Score extension broken - everything "Impossible de compiler" - https://phabricator.wikimedia.org/T432418 [16:33:29] !log reedy@deploy2003 reedy, hartman: Backport for [[gerrit:1311872|generatePngAndMidi.sh: Remove SCORE_SAFE (T432418)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:34:11] !log reedy@deploy2003 reedy, hartman: Continuing with deployment [16:40:28] (03PS1) 10Btullis: kubernetes: enable TopoLVM on the SSD-backed dse-k8s workers [puppet] - 10https://gerrit.wikimedia.org/r/1311877 (https://phabricator.wikimedia.org/T432109) [16:40:30] (03PS1) 10Btullis: kubernetes: enable TopoLVM on the HDD-backed dse-k8s workers [puppet] - 10https://gerrit.wikimedia.org/r/1311878 (https://phabricator.wikimedia.org/T432109) [16:42:04] !log reedy@deploy2003 Finished scap sync-world: Backport for [[gerrit:1311872|generatePngAndMidi.sh: Remove SCORE_SAFE (T432418)]] (duration: 10m 29s) [16:42:08] T432418: Score extension broken - everything "Impossible de compiler" - https://phabricator.wikimedia.org/T432418 [16:46:50] (03PS1) 10Bking: stat hosts: Set up S3 config for search platform team. [puppet] - 10https://gerrit.wikimedia.org/r/1311880 (https://phabricator.wikimedia.org/T432249) [16:47:07] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311880 (https://phabricator.wikimedia.org/T432249) (owner: 10Bking) [16:51:49] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311877 (https://phabricator.wikimedia.org/T432109) (owner: 10Btullis) [16:52:08] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311878 (https://phabricator.wikimedia.org/T432109) (owner: 10Btullis) [16:55:18] (03CR) 10Gmodena: WDQSv2: Several small adjustments (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311841 (owner: 10Trueg) [16:58:34] (03PS2) 10Bking: stat hosts: Set up S3 config for search platform team. [puppet] - 10https://gerrit.wikimedia.org/r/1311880 (https://phabricator.wikimedia.org/T432249) [17:01:03] !log kamila@cumin2003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1264.eqiad.wmnet [17:01:07] !log kamila@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1264.eqiad.wmnet [17:01:09] !log kamila@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1264.eqiad.wmnet [17:01:21] !log kamila@cumin2003 START - Cookbook sre.hosts.reimage for host wikikube-worker1264.eqiad.wmnet with OS trixie [17:01:43] !log kamila@cumin2003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1264 [17:01:43] !log kamila@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1264 [17:02:00] (03CR) 10JHathaway: [C:03+2] Puppet 8: Replace legacy facts in profile pontoon [puppet] - 10https://gerrit.wikimedia.org/r/1311566 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [17:04:31] (03PS2) 10JHathaway: Puppet 8: Replace legacy facts in role alerting_host [puppet] - 10https://gerrit.wikimedia.org/r/1311568 (https://phabricator.wikimedia.org/T372666) [17:04:36] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311568 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [17:08:17] (03CR) 10Btullis: stat hosts: Set up S3 config for search platform team. (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1311880 (https://phabricator.wikimedia.org/T432249) (owner: 10Bking) [17:08:36] (03CR) 10Btullis: stat hosts: Set up S3 config for search platform team. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311880 (https://phabricator.wikimedia.org/T432249) (owner: 10Bking) [17:20:09] 06SRE, 10SRE-Access-Requests: Requesting access to ml-lab-users for dcausse - https://phabricator.wikimedia.org/T432347#12133072 (10Dzahn) [17:20:57] (03PS1) 10Andrew Bogott: wmcs-backup.py: report max total potential size of a vm backup run [puppet] - 10https://gerrit.wikimedia.org/r/1311884 (https://phabricator.wikimedia.org/T430018) [17:21:36] (03CR) 10Bking: stat hosts: Set up S3 config for search platform team. (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1311880 (https://phabricator.wikimedia.org/T432249) (owner: 10Bking) [17:21:49] !log kamila@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1264.eqiad.wmnet with reason: host reimage [17:21:53] 06SRE, 10SRE-Access-Requests: Requesting access to ml-lab-users for dcausse - https://phabricator.wikimedia.org/T432347#12133075 (10Dzahn) Thanks @calbon ! Since David already has shell access and signed the L3 over 10 years ago..all the boxes are checked except one. We just need the approval from @pfischer... [17:23:10] (03CR) 10CI reject: [V:04-1] wmcs-backup.py: report max total potential size of a vm backup run [puppet] - 10https://gerrit.wikimedia.org/r/1311884 (https://phabricator.wikimedia.org/T430018) (owner: 10Andrew Bogott) [17:23:15] (03CR) 10BCornwall: [C:03+1] Remove mirrors.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/1311584 (https://phabricator.wikimedia.org/T432422) (owner: 10Urbanecm) [17:25:27] (03PS3) 10Btullis: dse-k8s: add the postgresql-superset-metrics deployment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311796 (https://phabricator.wikimedia.org/T432104) [17:25:45] (03PS1) 10Dzahn: admin: add dcausse to ml-lab-users [puppet] - 10https://gerrit.wikimedia.org/r/1311886 (https://phabricator.wikimedia.org/T432347) [17:26:10] (03PS3) 10Bking: stat hosts: Set up S3 config for search platform team. [puppet] - 10https://gerrit.wikimedia.org/r/1311880 (https://phabricator.wikimedia.org/T432249) [17:27:45] 06SRE, 10SRE-Access-Requests: Requesting access to ml-lab-users for ebernhardson - https://phabricator.wikimedia.org/T432345#12133084 (10Dzahn) [17:28:23] !log kamila@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1264.eqiad.wmnet with reason: host reimage [17:28:59] (03PS1) 10Dzahn: admin: add ebernhardson to ml-lab-users [puppet] - 10https://gerrit.wikimedia.org/r/1311890 (https://phabricator.wikimedia.org/T432345) [17:30:16] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to ml-lab-users for ebernhardson - https://phabricator.wikimedia.org/T432345#12133101 (10Dzahn) Same as T432347#12133072 Existing shell user and all boxes are checked except we need the approval from @pfischer Uploaded patch that can be m... [17:30:35] (03CR) 10Btullis: stat hosts: Set up S3 config for search platform team. (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1311880 (https://phabricator.wikimedia.org/T432249) (owner: 10Bking) [17:30:40] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to ml-lab-users for ebernhardson - https://phabricator.wikimedia.org/T432345#12133107 (10Dzahn) a:03pfischer [17:30:46] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to ml-lab-users for dcausse - https://phabricator.wikimedia.org/T432347#12133108 (10Dzahn) a:03pfischer [17:31:12] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to ml-lab-users for dcausse - https://phabricator.wikimedia.org/T432347#12133110 (10Dzahn) 05Open→03In progress [17:31:20] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to ml-lab-users for ebernhardson - https://phabricator.wikimedia.org/T432345#12133111 (10Dzahn) 05Open→03In progress [17:33:21] (03CR) 10Btullis: airflow: let DAGs manage run-scoped scratch volumes on TopoLVM (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311846 (https://phabricator.wikimedia.org/T428235) (owner: 10Btullis) [17:40:47] (03PS2) 10Urbanecm: Remove mirrors.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/1311584 (https://phabricator.wikimedia.org/T432422) [17:41:22] (03CR) 10Dzahn: [C:03+1] "a bit confused because I already see "Host mirrors.wikimedia.org not found: 3(NXDOMAIN)" but +1 nevertheless" [dns] - 10https://gerrit.wikimedia.org/r/1311584 (https://phabricator.wikimedia.org/T432422) (owner: 10Urbanecm) [17:41:36] mutante: it is a CNAME to a removed record [17:42:04] urbanecm: ah! ok, another +1 :) [17:42:20] confused me too the other day :D [17:42:28] (03CR) 10Dzahn: [C:03+2] "CNAME to a deleted record" [dns] - 10https://gerrit.wikimedia.org/r/1311584 (https://phabricator.wikimedia.org/T432422) (owner: 10Urbanecm) [17:42:41] !log dzahn@dns1006 START - running authdns-update [17:43:45] eh, let me do this the right way [17:44:25] DNS update feels much slower than usual [17:44:32] !log dzahn@dns1006 END - running authdns-update [17:44:40] oh, it just jumped from 1 to 14.. ok [17:44:55] !log dzahn@dns1006 START - running authdns-update [17:46:54] !log dzahn@dns1006 END - running authdns-update [17:47:31] 06SRE, 10DNS, 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: Remove mirrors.wikimedia.org from public DNS - https://phabricator.wikimedia.org/T432422#12133132 (10Dzahn) 05Open→03Resolved a:03Dzahn merged and deployed - since it was just a CNAME to an already deleted record [17:49:57] !log kamila@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1264.eqiad.wmnet with OS trixie [17:54:05] kamila@cumin2003 renumber-node (PID 2347382) is awaiting input [17:55:53] mutante: yeah, around two mins is the average time [17:56:16] we run some cleanup during the first Monday of the month that helps prune things a bit [17:56:23] perhaps we can do it more frequently [17:57:04] sukhe: thanks. gotcha. it wasn't a problem. just felt like different from what I am used to [17:57:23] yeah, but your observation is not incorrect [17:57:25] > command => "/usr/bin/git -C ${workingdir} maintenance run", [17:57:32] this is what we run on the first Monday of a month [17:57:38] aha! [17:57:52] there is more deeper work involved in speeding things up but yeah [18:06:29] (03PS4) 10Bking: stat hosts: Set up S3 config for search platform team. [puppet] - 10https://gerrit.wikimedia.org/r/1311880 (https://phabricator.wikimedia.org/T432249) [18:06:44] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311880 (https://phabricator.wikimedia.org/T432249) (owner: 10Bking) [18:09:11] (03CR) 10Bking: stat hosts: Set up S3 config for search platform team. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311880 (https://phabricator.wikimedia.org/T432249) (owner: 10Bking) [18:11:38] (03PS5) 10Bking: stat hosts: Set up S3 config for search platform team. [puppet] - 10https://gerrit.wikimedia.org/r/1311880 (https://phabricator.wikimedia.org/T432249) [18:16:24] !log kamila@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1264.eqiad.wmnet [18:16:27] !log kamila@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1264.eqiad.wmnet [18:16:30] !log kamila@cumin2003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1264.eqiad.wmnet [18:25:30] (03CR) 10Bking: [C:03+2] opensearch: remove references to soon-to-be-decom'd service [alerts] - 10https://gerrit.wikimedia.org/r/1311542 (https://phabricator.wikimedia.org/T432148) (owner: 10Bking) [18:27:57] (03CR) 10Bking: [C:03+2] datahubsearch: remove DNS record [dns] - 10https://gerrit.wikimedia.org/r/1311538 (https://phabricator.wikimedia.org/T432148) (owner: 10Bking) [18:28:59] !log bking@dns1004 START - running authdns-update [18:30:47] (03CR) 10JHathaway: "@cwhite@wikimedia.org these module specific patches are ready for review, thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1311549 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [18:30:57] !log bking@dns1004 END - running authdns-update [18:50:45] (03PS3) 10Bking: datahubsearch: remove references to Puppet plans [puppet] - 10https://gerrit.wikimedia.org/r/1311533 (https://phabricator.wikimedia.org/T432148) [18:50:45] (03PS4) 10Bking: datahubsearch: move to 'insetup' role [puppet] - 10https://gerrit.wikimedia.org/r/1311530 (https://phabricator.wikimedia.org/T432148) [18:50:45] (03PS1) 10Bking: datahubsearch: move service from production to lvs_setup [puppet] - 10https://gerrit.wikimedia.org/r/1311899 (https://phabricator.wikimedia.org/T432148) [19:05:39] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:15:53] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs2017.codfw.wmnet with OS bookworm [19:16:14] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs2017 [19:16:14] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2017 [19:16:14] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs1026.eqiad.wmnet with OS bookworm [19:16:31] (03PS2) 10Andrew Bogott: wmcs-backup.py: report max total potential size of a vm backup run [puppet] - 10https://gerrit.wikimedia.org/r/1311884 (https://phabricator.wikimedia.org/T430018) [19:16:31] (03PS1) 10Andrew Bogott: cloud-vps VM backups: move many more projects to cloudbackup1003 [puppet] - 10https://gerrit.wikimedia.org/r/1311905 (https://phabricator.wikimedia.org/T430018) [19:19:20] (03CR) 10CI reject: [V:04-1] wmcs-backup.py: report max total potential size of a vm backup run [puppet] - 10https://gerrit.wikimedia.org/r/1311884 (https://phabricator.wikimedia.org/T430018) (owner: 10Andrew Bogott) [19:23:26] (03CR) 10Andrew Bogott: [C:03+2] cloud-vps VM backups: move many more projects to cloudbackup1003 [puppet] - 10https://gerrit.wikimedia.org/r/1311905 (https://phabricator.wikimedia.org/T430018) (owner: 10Andrew Bogott) [19:26:08] 06SRE, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 06Traffic-Icebox: CentralNotice: Allow Varnish caching of Special:BannerLoader HTML - https://phabricator.wikimedia.org/T149873#12133353 (10Krinkle) [19:30:32] (03PS3) 10Andrew Bogott: wmcs-backup.py: report max total potential size of a vm backup run [puppet] - 10https://gerrit.wikimedia.org/r/1311884 (https://phabricator.wikimedia.org/T430018) [19:33:41] !log bking@deploy2003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [19:33:45] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [19:33:46] !log bking@deploy2003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 12s) [19:34:32] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1026.eqiad.wmnet with reason: host reimage [19:34:47] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2017.codfw.wmnet with reason: host reimage [19:37:36] !log bking@deploy2003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [19:37:48] !log bking@deploy2003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 16s) [19:39:34] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1026.eqiad.wmnet with reason: host reimage [19:39:53] !log bking@deploy2003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [19:39:57] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [19:40:02] !log bking@deploy2003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 15s) [19:40:19] !log bking@deploy2003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [19:43:00] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2017.codfw.wmnet with reason: host reimage [19:50:18] !log bking@deploy2003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 10m 03s) [19:50:22] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [19:50:39] !log bking@deploy2003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [19:50:43] !log bking@deploy2003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 08s) [19:55:15] !log bking@deploy2003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [19:55:21] !log bking@deploy2003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 09s) [19:55:24] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [19:55:29] !log bking@deploy2003 Started deploy [wdqs/wdqs@e8fb00c]: T430880 [19:55:37] !log bking@deploy2003 Finished deploy [wdqs/wdqs@e8fb00c]: T430880 (duration: 00m 12s) [20:06:05] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1026.eqiad.wmnet with OS bookworm [20:08:06] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1022.eqiad.wmnet -> wdqs1026.eqiad.wmnet, repooling source-only afterwards [20:08:10] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [20:08:35] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2017.codfw.wmnet with OS bookworm [20:10:49] (03CR) 10Andrew Bogott: [C:03+2] codfw1dev cloudcontrol: Don't install barbican [puppet] - 10https://gerrit.wikimedia.org/r/1311853 (https://phabricator.wikimedia.org/T432349) (owner: 10Andrew Bogott) [20:11:40] (03CR) 10Andrew Bogott: [C:03+2] Openstack: remove files and references to barbican [puppet] - 10https://gerrit.wikimedia.org/r/1311854 (https://phabricator.wikimedia.org/T432349) (owner: 10Andrew Bogott) [20:11:56] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs2026.codfw.wmnet -> wdqs2017.codfw.wmnet, repooling source-only afterwards [20:15:43] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:57:16] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs2026.codfw.wmnet -> wdqs2017.codfw.wmnet, repooling source-only afterwards [20:57:20] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [20:59:08] (03PS1) 10Bking: WIP: apt: Introduce vector.dev repository [puppet] - 10https://gerrit.wikimedia.org/r/1311939 (https://phabricator.wikimedia.org/T324335) [21:05:03] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs1022.eqiad.wmnet -> wdqs1026.eqiad.wmnet, repooling source-only afterwards [21:05:07] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [21:12:12] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs2026.codfw.wmnet with OS bookworm [21:12:34] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs2026 [21:12:34] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2026 [21:13:18] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs2018.codfw.wmnet with OS bookworm [21:13:47] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs2018 [21:14:48] !log bking@cumin2003 START - Cookbook sre.dns.netbox [21:20:42] bking@cumin2003 reimage (PID 2402292) is awaiting input [21:23:22] PROBLEM - statsv Varnishkafka log producer on cp3068 is CRITICAL: PROCS CRITICAL: 2 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [21:24:22] RECOVERY - statsv Varnishkafka log producer on cp3068 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [21:26:26] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2018 - bking@cumin2003" [21:26:31] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs2018 - bking@cumin2003" [21:26:31] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [21:26:31] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache wdqs2018.codfw.wmnet 155.32.192.10.in-addr.arpa 5.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [21:26:35] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs2018.codfw.wmnet 155.32.192.10.in-addr.arpa 5.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [21:26:36] !log bking@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wdqs2018 [21:26:50] !log bking@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs2018 [21:26:51] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs2018 [21:32:47] 06SRE, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 06Traffic-Icebox: CentralNotice: Allow using Varnish cache of Special:BannerLoader HTML for logged-in users - https://phabricator.wikimedia.org/T149873#12133632 (10Krinkle) [21:32:59] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2026.codfw.wmnet with reason: host reimage [21:38:04] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2026.codfw.wmnet with reason: host reimage [21:45:38] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2018.codfw.wmnet with reason: host reimage [21:49:07] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2018.codfw.wmnet with reason: host reimage [22:02:46] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2026.codfw.wmnet with OS bookworm [22:11:19] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2018.codfw.wmnet with OS bookworm [23:05:39] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:08:03] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs2017.codfw.wmnet -> wdqs2026.codfw.wmnet, repooling source-only afterwards [23:08:07] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880 [23:09:34] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430880, restore data on newly-reimaged host) xfer wdqs-all from wdqs2022.codfw.wmnet -> wdqs2018.codfw.wmnet, repooling source-only afterwards [23:42:24] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1311984 [23:42:25] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1311984 (owner: 10TrainBranchBot) [23:50:21] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1311984 (owner: 10TrainBranchBot) [23:53:18] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T430880, restore data on newly-reimaged host) xfer scholarly_articles from wdqs2017.codfw.wmnet -> wdqs2026.codfw.wmnet, repooling source-only afterwards [23:53:22] T430880: Migrate WDQS hosts to Bookworm or later - https://phabricator.wikimedia.org/T430880