[00:03:00] FIRING: [2x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:08:59] RESOLVED: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [00:12:59] FIRING: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [00:17:59] RESOLVED: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [00:40:59] FIRING: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [00:41:06] jouncebot: nowandnext [00:41:06] No deployments scheduled for the next 5 hour(s) and 18 minute(s) [00:41:06] In 5 hour(s) and 18 minute(s): MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260731T0600) [00:43:05] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316832 (owner: 10Jforrester) [00:45:59] RESOLVED: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [00:46:23] (03Merged) 10jenkins-bot: Drop old support for non-temporary accounts wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1316832 (owner: 10Jforrester) [00:46:48] !log dreamyjazz@deploy1003 Started scap sync-world: Backport for [[gerrit:1316832|Drop old support for non-temporary accounts wikis]] [00:47:07] (03PS2) 10Aaron Schulz: Further simplify api-gateway chart by assuming rest-gateway mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319122 (https://phabricator.wikimedia.org/T428625) [00:48:35] !log dreamyjazz@deploy1003 dreamyjazz, jforrester: Backport for [[gerrit:1316832|Drop old support for non-temporary accounts wikis]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [00:52:30] Still testing :D [00:53:42] !log dreamyjazz@deploy1003 dreamyjazz, jforrester: Continuing with deployment [00:53:52] Temporary account editing appears unaffected, continuing... [00:57:53] !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for [[gerrit:1316832|Drop old support for non-temporary accounts wikis]] (duration: 11m 04s) [00:58:01] I should be done [01:12:27] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12173259 (10Papaul) [01:12:32] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1319559 [01:12:32] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1319559 (owner: 10TrainBranchBot) [01:12:58] (03PS1) 10Aaron Schulz: Rename api-gateway to common-application-gateway [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319560 (https://phabricator.wikimedia.org/T428625) [01:22:50] 06SRE, 10SRE-Access-Requests, 10SRE-tools, 10Cumin, and 3 others: add dcops group to run sre.hosts.downtime cookbook - https://phabricator.wikimedia.org/T433409#12173276 (10wiki_willy) Thanks for looping me in. Before I approve though, just wanted to provide space for any comments or concerns that anyone f... [01:25:24] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1319559 (owner: 10TrainBranchBot) [01:35:59] FIRING: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [01:47:59] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [02:00:39] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:07:33] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 54s) [02:08:51] (03PS3) 10Aaron Schulz: Further simplify api-gateway chart by assuming rest-gateway mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319122 (https://phabricator.wikimedia.org/T428625) [02:08:51] (03PS1) 10Aaron Schulz: Remove API Gateway specific ratelimiting from api-gateway [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319563 (https://phabricator.wikimedia.org/T428625) [02:10:59] RESOLVED: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [02:12:59] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [02:14:07] (03PS2) 10Aaron Schulz: Rename api-gateway to common-application-gateway [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319560 (https://phabricator.wikimedia.org/T428625) [02:15:16] (03PS2) 10Aaron Schulz: Remove API Gateway specific ratelimiting from api-gateway [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319563 (https://phabricator.wikimedia.org/T428625) [02:16:12] (03PS8) 10Aaron Schulz: api-gateway: remove some API Gateway specific template files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319114 (https://phabricator.wikimedia.org/T428625) [02:17:31] (03PS4) 10Aaron Schulz: api-gateway: further simplify chart by assuming REST Gateway mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319122 (https://phabricator.wikimedia.org/T428625) [02:17:53] (03PS3) 10Aaron Schulz: api-gateway:: remove API Gateway specific ratelimiting logic [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319563 (https://phabricator.wikimedia.org/T428625) [02:17:59] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:19:54] (03CR) 10Scott French: [C:03+1] "Thanks, Blake!" [software/httpbb] - 10https://gerrit.wikimedia.org/r/1318682 (https://phabricator.wikimedia.org/T428972) (owner: 10Blake) [02:22:59] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:23:59] FIRING: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [02:26:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:28:59] RESOLVED: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [02:36:59] FIRING: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [02:41:59] RESOLVED: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [02:44:59] FIRING: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [03:04:59] RESOLVED: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [03:06:39] (03CR) 10AKhatun: [C:03+1] eventstreams: expose page_html_feature_counts_change.v1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319117 (https://phabricator.wikimedia.org/T433507) (owner: 10Ottomata) [03:08:10] (03CR) 10AKhatun: [C:03+1] "Do we need any changes in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/wmf-config/ext-Ev" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319116 (https://phabricator.wikimedia.org/T433507) (owner: 10Ottomata) [03:09:59] FIRING: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [03:36:43] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12173316 (10Papaul) [03:45:07] (03PS1) 10Samwilson: Use ImageMagick's -background=none flag for Alpha WebP images [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1319567 (https://phabricator.wikimedia.org/T283646) [03:52:59] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2033 is not sending any prefix to lsw1-b7-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [04:20:15] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12173344 (10Papaul) [04:23:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.17% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:28:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.17% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:33:51] FIRING: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-603-eqsin:ethernet-1/55 (Core: cr2-eqsin:et-0/0/2) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-603-eqsin:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [04:43:51] RESOLVED: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-603-eqsin:ethernet-1/55 (Core: cr2-eqsin:et-0/0/2) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-603-eqsin:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [05:47:59] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260731T0600) [06:13:00] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [06:17:59] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:26:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:27:52] (03CR) 10Slyngshede: [C:03+1] "I think we can just merge it." [puppet] - 10https://gerrit.wikimedia.org/r/1318214 (owner: 10Ammarpad) [06:30:53] (03PS1) 10Filippo Giunchedi: installserver: move cloudvirt1048 to efi [puppet] - 10https://gerrit.wikimedia.org/r/1319749 (https://phabricator.wikimedia.org/T431682) [06:33:36] (03CR) 10Filippo Giunchedi: [C:03+2] "Simple enough and need this to unblock reimage, self-merging" [puppet] - 10https://gerrit.wikimedia.org/r/1319749 (https://phabricator.wikimedia.org/T431682) (owner: 10Filippo Giunchedi) [06:34:24] !log filippo@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1048.eqiad.wmnet with OS trixie [06:34:49] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12173440 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt1048.eqiad.wmnet with OS trixie executed with e... [06:35:15] (03PS1) 10Kevin Bazira: ml-services: update tts-section-generator image to add batch generation script [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319750 (https://phabricator.wikimedia.org/T433594) [06:37:41] 06SRE, 10SRE-Access-Requests, 10SRE-tools, 10Cumin, and 3 others: add dcops group to run sre.hosts.downtime cookbook - https://phabricator.wikimedia.org/T433409#12173444 (10elukey) @Dzahn technically we are adding a sudo rule so the I/F team should +1 it, but I already anticipate it is a +1. I'll try to ma... [06:37:43] (03CR) 10Kevin Bazira: [C:03+2] ml-services: update tts-section-generator image to add batch generation script [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319750 (https://phabricator.wikimedia.org/T433594) (owner: 10Kevin Bazira) [06:40:02] (03Merged) 10jenkins-bot: ml-services: update tts-section-generator image to add batch generation script [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319750 (https://phabricator.wikimedia.org/T433594) (owner: 10Kevin Bazira) [06:44:17] !log filippo@cumin1003 START - Cookbook sre.hosts.reimage for host cloudvirt1048.eqiad.wmnet with OS trixie [06:44:40] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12173452 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1003 for host cloudvirt1048.eqiad.wmnet with OS trixie [06:46:04] (03CR) 10Filippo Giunchedi: [C:03+1] "Nice!" [puppet] - 10https://gerrit.wikimedia.org/r/1307368 (https://phabricator.wikimedia.org/T431017) (owner: 10Dpogorzelski) [06:48:17] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [06:48:18] !log elukey@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [06:55:45] (03PS1) 10Elukey: sre.hosts.provision: default to wmfroot when --no-users is supplied [cookbooks] - 10https://gerrit.wikimedia.org/r/1319753 (https://phabricator.wikimedia.org/T426180) [06:56:52] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [06:57:01] !log elukey@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260731T0700) [07:09:47] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "cloudvirt1048 - filippo@cumin1003" [07:09:52] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "cloudvirt1048 - filippo@cumin1003" [07:09:59] FIRING: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [07:11:31] !log filippo@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "cloudvirt1048 - filippo@cumin1003" [07:11:46] !log filippo@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "cloudvirt1048 - filippo@cumin1003" [07:13:14] !log filippo@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1048.eqiad.wmnet with reason: host reimage [07:15:06] (03CR) 10Slyngshede: [C:03+1] "I think that makes a lot of sense and we can always revert." [puppet] - 10https://gerrit.wikimedia.org/r/1296495 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [07:18:03] (03CR) 10Slyngshede: [C:03+1] "Host is in fact gone, according to Horizon." [puppet] - 10https://gerrit.wikimedia.org/r/1314050 (owner: 10Andrew Bogott) [07:19:15] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1048.eqiad.wmnet with reason: host reimage [07:31:22] 06SRE, 10SRE-swift-storage, 06Infrastructure-Foundations: Unable to reimage or reprovision ms-be2082 due to redfish connection errors - https://phabricator.wikimedia.org/T433635#12173480 (10elukey) @MatthewVernon I tried to reset the BMC via Redfish but it didn't help, and everything boils down to the /redfi... [07:32:06] (03CR) 10Ayounsi: [C:03+1] sre.hosts.provision: default to wmfroot when --no-users is supplied [cookbooks] - 10https://gerrit.wikimedia.org/r/1319753 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [07:32:18] (03CR) 10Elukey: [C:03+2] sre.hosts.provision: default to wmfroot when --no-users is supplied [cookbooks] - 10https://gerrit.wikimedia.org/r/1319753 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [07:35:26] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps new: Rerack mc-gp2006 and mc2046 - https://phabricator.wikimedia.org/T432716#12173498 (10jijiki) >>! In T432716#12172126, @Jhancock.wm wrote: > @jijiki i can get this tomorrow some time. I will ping when i can start it. Grand, thanks! [07:45:23] 06SRE, 10SRE-swift-storage, 06Infrastructure-Foundations: Unable to reimage or reprovision ms-be2082 due to redfish connection errors - https://phabricator.wikimedia.org/T433635#12173504 (10MatthewVernon) @elukey thanks for taking a look! I had a meander through https://wikitech.wikimedia.org/wiki/Management... [07:52:59] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2033 is not sending any prefix to lsw1-b7-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [08:00:44] !log filippo@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1048 [08:01:07] !log filippo@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1048 [08:03:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [08:03:49] (03PS1) 10Ayounsi: Add Python 3.14 support [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 [08:06:03] !log filippo@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1048 [08:07:50] !log filippo@cumin1003 END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudvirt1048 [08:07:53] !log filippo@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1048 [08:07:59] !log filippo@cumin1003 END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host cloudvirt1048 [08:08:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [08:11:04] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [08:11:12] !log elukey@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [08:13:07] (03PS9) 10Effie Mouzeli: api-gateway: remove some API Gateway specific template files #1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319114 (https://phabricator.wikimedia.org/T428625) (owner: 10Aaron Schulz) [08:13:30] (03PS5) 10Effie Mouzeli: api-gateway: further simplify chart by assuming REST Gateway mode #2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319122 (https://phabricator.wikimedia.org/T428625) (owner: 10Aaron Schulz) [08:14:00] (03PS4) 10Effie Mouzeli: api-gateway:: remove API Gateway specific ratelimiting logic #3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319563 (https://phabricator.wikimedia.org/T428625) (owner: 10Aaron Schulz) [08:14:31] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12173560 (10fgiunchedi) I am working on getting cloudvirt1048 up an running in C8 after the move. The current situation is the following: * host is 'active' in ne... [08:17:43] PROBLEM - Host ms-be2082 is DOWN: PING CRITICAL - Packet loss = 100% [08:18:39] RECOVERY - Host ms-be2082 is UP: PING OK - Packet loss = 0%, RTA = 31.55 ms [08:24:10] (03CR) 10CI reject: [V:04-1] Add Python 3.14 support [software/homer] - 10https://gerrit.wikimedia.org/r/1319796 (owner: 10Ayounsi) [08:25:06] (03PS1) 10Ayounsi: Revert "Don't fetch the cable label" [software/homer] - 10https://gerrit.wikimedia.org/r/1319798 [08:30:14] 06SRE, 06ServiceOps new, 10VisualEditor, 10VisualEditor Suggestion Mode, and 3 others: Deploy Headless VE in k8s for technical pilot - https://phabricator.wikimedia.org/T431497#12173584 (10GGoncalves-WMF) Sounds good to me, and we should assign to a point of contact in Editing to drive this. [08:35:41] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [08:35:52] (03CR) 10Ozge: [C:03+1] "I think we can deploy them together in the same patch." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318662 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [08:36:13] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [08:37:00] !log brouberol@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [08:37:01] (03CR) 10Gkyziridis: [C:03+2] ml-services: Deploy latest version of revise-tone-task-generator on ml-serve-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318662 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [08:37:55] !log brouberol@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [08:38:00] (03CR) 10Gkyziridis: [C:03+2] "I prefer to do it in separated patches. I am merging this and I will open a follow up patch." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318662 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [08:39:29] (03Merged) 10jenkins-bot: ml-services: Deploy latest version of revise-tone-task-generator on ml-serve-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318662 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [08:42:02] !log gkyziridis@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . [08:45:06] (03PS1) 10Gkyziridis: ml-services: Deploy latest version of revise-tone-task-generator on ml-eqiad-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319800 (https://phabricator.wikimedia.org/T429675) [08:45:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [08:46:58] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [08:47:07] (03CR) 10Ozge: [C:03+1] ml-services: Deploy latest version of revise-tone-task-generator on ml-eqiad-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319800 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [08:47:46] (03CR) 10Gkyziridis: [C:03+2] ml-services: Deploy latest version of revise-tone-task-generator on ml-eqiad-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319800 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [08:48:05] PROBLEM - Host ms-be2082 is DOWN: PING CRITICAL - Packet loss = 100% [08:50:01] (03Merged) 10jenkins-bot: ml-services: Deploy latest version of revise-tone-task-generator on ml-eqiad-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319800 (https://phabricator.wikimedia.org/T429675) (owner: 10Gkyziridis) [08:51:37] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' . [08:51:39] RECOVERY - Host ms-be2082 is UP: PING OK - Packet loss = 0%, RTA = 31.52 ms [08:53:17] PROBLEM - Host ms-be2082 is DOWN: PING CRITICAL - Packet loss = 100% [08:57:09] !log elukey@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [08:57:13] (03CR) 10Elukey: [C:03+2] admin: allow dcops to run the downtime cookbook [puppet] - 10https://gerrit.wikimedia.org/r/1319125 (https://phabricator.wikimedia.org/T433409) (owner: 10Elukey) [08:57:19] RECOVERY - Host ms-be2082 is UP: PING OK - Packet loss = 0%, RTA = 31.58 ms [08:58:24] 06SRE, 10SRE-Access-Requests, 10SRE-tools, 10Cumin, and 3 others: add dcops group to run sre.hosts.downtime cookbook - https://phabricator.wikimedia.org/T433409#12173638 (10elukey) 05Open→03Resolved Done! [09:05:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:05:46] 06SRE, 10SRE-swift-storage, 06Infrastructure-Foundations: Unable to reimage or reprovision ms-be2082 due to redfish connection errors - https://phabricator.wikimedia.org/T433635#12173690 (10elukey) So the Bios endpoint issue seemed to be just waiting for a host reboot: IIUC on Supermicro X12 the BMC is just... [09:12:57] 06SRE, 06Traffic: Prometheus Ferm MSS export does not work for IPv6 - https://phabricator.wikimedia.org/T433672 (10cmooney) 03NEW p:05Triage→03Low [09:16:46] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 10 hosts [09:18:14] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 10 hosts [09:28:22] (03PS1) 10Elukey: sre.hosts.bmc-user-mgmt: test ipmi as success measure [cookbooks] - 10https://gerrit.wikimedia.org/r/1319805 (https://phabricator.wikimedia.org/T426180) [09:30:40] 06SRE, 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: Support LVS backend servers using nftables - https://phabricator.wikimedia.org/T433601#12173746 (10cmooney) >>! In T433601#12171476, @ssingh wrote: > The primary change that I think we need to make is in `modules/profile/manifests/lvs/real... [09:36:27] (03PS2) 10Elukey: sre.hosts.bmc-user-mgmt: test ipmi as success measure [cookbooks] - 10https://gerrit.wikimedia.org/r/1319805 (https://phabricator.wikimedia.org/T426180) [09:48:00] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:51:18] (03PS1) 10Elukey: sre.hosts.provision: do not fail if ipmi's check_connection fails [cookbooks] - 10https://gerrit.wikimedia.org/r/1319806 (https://phabricator.wikimedia.org/T426180) [09:52:02] !log elukey@cumin1003 START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [09:53:51] PROBLEM - Host ms-be2082 is DOWN: PING CRITICAL - Packet loss = 100% [09:54:59] RESOLVED: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from RU) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [09:56:26] 06SRE, 06Infrastructure-Foundations, 10netops: Consider removing BFD on datacentre IBGP peerings - https://phabricator.wikimedia.org/T433675 (10cmooney) 03NEW p:05Triage→03Low [09:56:39] RECOVERY - Host ms-be2082 is UP: PING OK - Packet loss = 0%, RTA = 31.61 ms [09:59:03] PROBLEM - Host ms-be2082 is DOWN: PING CRITICAL - Packet loss = 100% [10:01:47] RECOVERY - Host ms-be2082 is UP: PING OK - Packet loss = 0%, RTA = 31.58 ms [10:02:11] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [10:08:02] (03CR) 10Cathal Mooney: [C:03+1] sre.hosts.bmc-user-mgmt: test ipmi as success measure [cookbooks] - 10https://gerrit.wikimedia.org/r/1319805 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [10:13:00] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [10:18:00] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:24:45] (03CR) 10Elukey: [C:03+2] sre.hosts.bmc-user-mgmt: test ipmi as success measure [cookbooks] - 10https://gerrit.wikimedia.org/r/1319805 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [10:26:38] (03PS1) 10Ozge: ml-services: Set TORCH_BLAS_PREFER_HIPBLASLT=0 for jina-embeddings-v5-text-nano [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319808 (https://phabricator.wikimedia.org/T432717) [10:26:38] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be2082.codfw.wmnet with OS trixie [10:26:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:26:57] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12173869 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be2082.codfw.wmnet with OS trixie [10:27:07] (03CR) 10Ozge: [V:03+2 C:03+2] ml-services: Set TORCH_BLAS_PREFER_HIPBLASLT=0 for jina-embeddings-v5-text-nano [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319808 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:27:12] (03PS1) 10Jcrespo: mariadb: Reenable notifications for db1265 & db1285 [puppet] - 10https://gerrit.wikimedia.org/r/1319809 (https://phabricator.wikimedia.org/T407942) [10:27:30] (03PS2) 10Jcrespo: mariadb: Reenable notifications for db1265 & db1285 [puppet] - 10https://gerrit.wikimedia.org/r/1319809 (https://phabricator.wikimedia.org/T407942) [10:27:34] (03CR) 10CI reject: [V:04-1] mariadb: Reenable notifications for db1265 & db1285 [puppet] - 10https://gerrit.wikimedia.org/r/1319809 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [10:27:43] (03CR) 10Dpogorzelski: [C:03+1] ml-services: Set TORCH_BLAS_PREFER_HIPBLASLT=0 for jina-embeddings-v5-text-nano [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319808 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:28:28] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:40:49] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2082.codfw.wmnet with reason: host reimage [10:40:58] (03PS1) 10Ozge: ml-services: Set ATTN_IMPLEMENTATION=sdpa for jina-embeddings [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319810 (https://phabricator.wikimedia.org/T432717) [10:41:20] (03CR) 10Ozge: [V:03+2 C:03+2] ml-services: Set ATTN_IMPLEMENTATION=sdpa for jina-embeddings [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319810 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:42:15] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:44:17] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2082.codfw.wmnet with reason: host reimage [10:46:22] (03CR) 10Ayounsi: [C:03+2] Revert "Don't fetch the cable label" [software/homer] - 10https://gerrit.wikimedia.org/r/1319798 (owner: 10Ayounsi) [11:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260731T0700) [11:00:05] jelto, arnoldokoth, mutante, and arnaudb: GitLab version upgrades (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260731T1100). Please do the needful. [11:01:18] (03PS1) 10Slyngshede: Release version 0.1.18 [software/bitu] - 10https://gerrit.wikimedia.org/r/1319812 [11:02:40] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2082.codfw.wmnet with OS trixie [11:02:51] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12173926 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be2082.codfw.wmnet with OS trixie completed: - ms-be2082 (**PASS*... [11:02:53] (03Merged) 10jenkins-bot: Revert "Don't fetch the cable label" [software/homer] - 10https://gerrit.wikimedia.org/r/1319798 (owner: 10Ayounsi) [11:03:18] (03PS1) 10Ayounsi: Add support for Nokia SR-Linux v26 [homer/public] - 10https://gerrit.wikimedia.org/r/1319813 (https://phabricator.wikimedia.org/T433105) [11:07:57] !log aokoth@cumin1003 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T433606 [11:08:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:13:44] RESOLVED: RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:18:17] 10ops-eqiad, 06SRE, 10Ceph, 06cloud-services-team, and 3 others: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267#12173976 (10VRiley-WMF) 05Open→03In progress [11:19:44] (03PS1) 10Ayounsi: sre.network.tls: add SR-Linux v26 support [cookbooks] - 10https://gerrit.wikimedia.org/r/1319814 (https://phabricator.wikimedia.org/T433105) [11:22:08] (03PS1) 10Slyngshede: wmnet: update CNAME records for DB masters to codfw [dns] - 10https://gerrit.wikimedia.org/r/1319815 (https://phabricator.wikimedia.org/T433363) [11:22:15] (03CR) 10Cathal Mooney: [C:03+1] "Nice one! LGTM provided it applies ok on lswtest-d8-eqiad and no diff on the others." [homer/public] - 10https://gerrit.wikimedia.org/r/1319813 (https://phabricator.wikimedia.org/T433105) (owner: 10Ayounsi) [11:24:11] aokoth@cumin1003 aokoth: The backup on gitlab1004 is complete, ready to proceed with upgrade. [11:24:37] (03PS1) 10Slyngshede: geo-maps: update map default to list codfw first [dns] - 10https://gerrit.wikimedia.org/r/1319816 (https://phabricator.wikimedia.org/T433363) [11:26:42] (03PS3) 10Hnowlan: mailman: migrate mailman_hours_until_empty_outbound_queue nrpe check [alerts] - 10https://gerrit.wikimedia.org/r/1311820 (https://phabricator.wikimedia.org/T370157) [11:27:40] !log aokoth@cumin1003 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T433606 [11:28:04] !log aokoth@cumin1003 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T433606 [11:30:25] (03CR) 10Cathal Mooney: [C:03+1] "Nice work! Function to get the version is great :)" [cookbooks] - 10https://gerrit.wikimedia.org/r/1319814 (https://phabricator.wikimedia.org/T433105) (owner: 10Ayounsi) [11:33:32] (03CR) 10Ayounsi: [C:03+2] Add support for Nokia SR-Linux v26 [homer/public] - 10https://gerrit.wikimedia.org/r/1319813 (https://phabricator.wikimedia.org/T433105) (owner: 10Ayounsi) [11:35:33] (03Merged) 10jenkins-bot: Add support for Nokia SR-Linux v26 [homer/public] - 10https://gerrit.wikimedia.org/r/1319813 (https://phabricator.wikimedia.org/T433105) (owner: 10Ayounsi) [11:36:24] (03CR) 10Hnowlan: [C:03+1] "I think we're in business!" [puppet] - 10https://gerrit.wikimedia.org/r/1306792 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [11:37:41] !log cmooney@cumin1003 START - Cookbook sre.network.peering with action 'configure' for AS: 6515 [11:37:42] (03CR) 10Hnowlan: [C:03+1] opensearch: add security-plugin specific configuration to opensearch.yml (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1318792 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [11:40:49] (03PS1) 10Slyngshede: debug.json: order codfw (primary) DC backends first [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319818 [11:42:01] !log cmooney@cumin1003 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 6515 [11:44:20] aokoth@cumin1003 aokoth: The backup on gitlab1004 is complete, ready to proceed with upgrade. [11:50:15] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 2353 bytes in 0.016 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [11:51:15] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 28690 bytes in 0.097 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [11:52:59] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2033 is not sending any prefix to lsw1-b7-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [11:53:38] !log aokoth@cumin1003 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T433606 [11:56:25] FIRING: [2x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:59:28] (03PS1) 10Slyngshede: wmnet: update deployment CNAME record to deploy2003 [dns] - 10https://gerrit.wikimedia.org/r/1319825 (https://phabricator.wikimedia.org/T433363) [12:03:05] (03PS1) 10Slyngshede: hieradata: update deployment_server to deploy2002 [puppet] - 10https://gerrit.wikimedia.org/r/1319826 (https://phabricator.wikimedia.org/T433363) [12:03:30] (03PS2) 10Slyngshede: hieradata: update deployment_server to deploy2003 [puppet] - 10https://gerrit.wikimedia.org/r/1319826 (https://phabricator.wikimedia.org/T433363) [12:03:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:06:47] 10ops-eqiad, 06SRE, 10Ceph, 06cloud-services-team, and 3 others: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267#12174095 (10VRiley-WMF) Updated firmware on iDRAC and BIOS. No change of yet. Grabbing TSR and will be opening up a ticket with Dell [12:07:55] (03PS1) 10Hnowlan: team-sre: Add data-engineering tag [alerts] - 10https://gerrit.wikimedia.org/r/1319827 (https://phabricator.wikimedia.org/T432376) [12:07:57] (03PS1) 10Hnowlan: team-sre: add o11y team tag [alerts] - 10https://gerrit.wikimedia.org/r/1319828 (https://phabricator.wikimedia.org/T432376) [12:08:00] (03PS1) 10Hnowlan: team-sre: use traffic team tag [alerts] - 10https://gerrit.wikimedia.org/r/1319829 (https://phabricator.wikimedia.org/T432376) [12:08:02] (03PS1) 10Hnowlan: team-sre: add data-persistence [alerts] - 10https://gerrit.wikimedia.org/r/1319830 (https://phabricator.wikimedia.org/T432376) [12:08:05] (03PS1) 10Hnowlan: team-sre: add dcops tag [alerts] - 10https://gerrit.wikimedia.org/r/1319831 (https://phabricator.wikimedia.org/T432376) [12:08:07] (03PS1) 10Hnowlan: team-sre: add infrastructure-foundations tag [alerts] - 10https://gerrit.wikimedia.org/r/1319832 (https://phabricator.wikimedia.org/T432376) [12:08:18] oops [12:08:22] (03CR) 10Cathal Mooney: [C:03+1] "lgtm!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1319806 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [12:10:44] (03CR) 10CI reject: [V:04-1] team-sre: add o11y team tag [alerts] - 10https://gerrit.wikimedia.org/r/1319828 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [12:10:58] (03CR) 10CI reject: [V:04-1] team-sre: add dcops tag [alerts] - 10https://gerrit.wikimedia.org/r/1319831 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [12:10:58] (03CR) 10CI reject: [V:04-1] team-sre: add data-persistence [alerts] - 10https://gerrit.wikimedia.org/r/1319830 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [12:11:04] (03CR) 10CI reject: [V:04-1] team-sre: add infrastructure-foundations tag [alerts] - 10https://gerrit.wikimedia.org/r/1319832 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [12:11:07] (03CR) 10CI reject: [V:04-1] team-sre: Add data-engineering tag [alerts] - 10https://gerrit.wikimedia.org/r/1319827 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [12:11:24] (03CR) 10CI reject: [V:04-1] team-sre: use traffic team tag [alerts] - 10https://gerrit.wikimedia.org/r/1319829 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [12:16:52] 06SRE, 06Infrastructure-Foundations, 10netops: JunOS: Investigate BGP PIC Edge / Protection - https://phabricator.wikimedia.org/T432381#12174142 (10cmooney) >>! In T432381#12131427, @ayounsi wrote: > Another optimization to consider: https://www.juniper.net/documentation/us/en/software/junos/bgp/topics/topic... [12:18:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:18:53] (03CR) 10Majavah: [C:04-1] Add script to get constructive edits for all wikis (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1272633 (https://phabricator.wikimedia.org/T428490) (owner: 10Clare Ming) [12:23:25] (03CR) 10Elukey: [C:03+2] sre.hosts.provision: do not fail if ipmi's check_connection fails [cookbooks] - 10https://gerrit.wikimedia.org/r/1319806 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [12:23:44] RESOLVED: RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:25:53] 06SRE, 06Infrastructure-Foundations, 10WMDE Analytics: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12174180 (10SLyngshede-WMF) Who needs to be listed as the approvers for the group? [12:27:04] (03PS1) 10Slyngshede: P:idm add airflow-wmde-ops as a managed group [puppet] - 10https://gerrit.wikimedia.org/r/1319835 (https://phabricator.wikimedia.org/T431077) [12:27:43] (03CR) 10Slyngshede: "Pending list of approvers" [puppet] - 10https://gerrit.wikimedia.org/r/1319835 (https://phabricator.wikimedia.org/T431077) (owner: 10Slyngshede) [12:28:17] 06SRE, 06Infrastructure-Foundations, 10WMDE Analytics, 13Patch-For-Review: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12174204 (10SLyngshede-WMF) 05Open→03In progress [12:30:00] (03PS2) 10Hnowlan: team-sre: Add data-engineering tag [alerts] - 10https://gerrit.wikimedia.org/r/1319827 (https://phabricator.wikimedia.org/T432376) [12:33:05] (03Abandoned) 10Slyngshede: P:idm remove approver [puppet] - 10https://gerrit.wikimedia.org/r/1236668 (owner: 10Slyngshede) [12:33:36] (03Abandoned) 10Slyngshede: P:idm enable bitu-account-manager permission request. [puppet] - 10https://gerrit.wikimedia.org/r/1093275 (owner: 10Slyngshede) [12:35:18] (03CR) 10CWilliams: [C:03+1] "LGTM for after the weekend" [puppet] - 10https://gerrit.wikimedia.org/r/1319809 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [12:35:35] (03PS1) 10Hashar: ci: allow inbound ssh from controller on prod agents [puppet] - 10https://gerrit.wikimedia.org/r/1319839 (https://phabricator.wikimedia.org/T418521) [12:38:31] (03PS2) 10Ayounsi: sre.network.tls: add SR-Linux v26 support [cookbooks] - 10https://gerrit.wikimedia.org/r/1319814 (https://phabricator.wikimedia.org/T433105) [12:38:58] (03PS1) 10Kevin Bazira: ml-services: update tts isvc to reject over-long text instead of silently dropping words [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319840 (https://phabricator.wikimedia.org/T433594) [12:40:12] (03PS1) 10Ayounsi: junos_set_interface_config: remove cable ID from description [cookbooks] - 10https://gerrit.wikimedia.org/r/1319841 (https://phabricator.wikimedia.org/T432689) [12:42:23] (03PS2) 10Ayounsi: junos_set_interface_config: remove cable ID from description [cookbooks] - 10https://gerrit.wikimedia.org/r/1319841 (https://phabricator.wikimedia.org/T432689) [12:43:25] (03CR) 10Kevin Bazira: [C:03+2] ml-services: update tts isvc to reject over-long text instead of silently dropping words [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319840 (https://phabricator.wikimedia.org/T433594) (owner: 10Kevin Bazira) [12:43:38] (03CR) 10Hashar: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319839 (https://phabricator.wikimedia.org/T418521) (owner: 10Hashar) [12:43:47] (03CR) 10Cathal Mooney: [C:03+1] junos_set_interface_config: remove cable ID from description [cookbooks] - 10https://gerrit.wikimedia.org/r/1319841 (https://phabricator.wikimedia.org/T432689) (owner: 10Ayounsi) [12:45:58] (03Merged) 10jenkins-bot: ml-services: update tts isvc to reject over-long text instead of silently dropping words [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319840 (https://phabricator.wikimedia.org/T433594) (owner: 10Kevin Bazira) [12:47:51] (03PS1) 10Clément Goubert: role::docker_registry: Ownership transfer [puppet] - 10https://gerrit.wikimedia.org/r/1319842 [12:49:04] !log ayounsi@cumin1003 START - Cookbook sre.network.tls for network device lswtest-d8-eqiad [12:49:04] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lswtest-d8-eqiad [12:49:54] (03CR) 10Ayounsi: [C:03+2] junos_set_interface_config: remove cable ID from description [cookbooks] - 10https://gerrit.wikimedia.org/r/1319841 (https://phabricator.wikimedia.org/T432689) (owner: 10Ayounsi) [12:50:35] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [12:53:12] (03Merged) 10jenkins-bot: junos_set_interface_config: remove cable ID from description [cookbooks] - 10https://gerrit.wikimedia.org/r/1319841 (https://phabricator.wikimedia.org/T432689) (owner: 10Ayounsi) [12:56:27] (03CR) 10Clément Goubert: [C:04-1] "I can't review the content of the file, but as far as the patch itself goes, there is a `.well-known` directory in the docroot and this fi" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315128 (https://phabricator.wikimedia.org/T432412) (owner: 10Tsevener) [13:01:16] (03CR) 10Clément Goubert: "However that will make the file available for every `wikipedia` site." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315128 (https://phabricator.wikimedia.org/T432412) (owner: 10Tsevener) [13:04:06] !log filippo@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1048 [13:04:21] !log filippo@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1048 [13:05:21] !log filippo@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1048 [13:05:36] !log filippo@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1048 [13:07:10] (03CR) 10Elukey: [C:03+1] "Somehow I don't feel that happy, but LGTM :D" [puppet] - 10https://gerrit.wikimedia.org/r/1319842 (owner: 10Clément Goubert) [13:07:35] (03CR) 10Elukey: [C:03+1] "Adding Lukasz for final confirmation" [puppet] - 10https://gerrit.wikimedia.org/r/1319842 (owner: 10Clément Goubert) [13:07:59] (03PS2) 10Hashar: ci: allow inbound ssh from controller on prod agents [puppet] - 10https://gerrit.wikimedia.org/r/1319839 (https://phabricator.wikimedia.org/T418521) [13:10:10] (03CR) 10Hashar: "Somehow contint2003 got forgotten. The PCC output is at https://puppet-compiler.wmflabs.org/output/1319839/7424/" [puppet] - 10https://gerrit.wikimedia.org/r/1319839 (https://phabricator.wikimedia.org/T418521) (owner: 10Hashar) [13:11:49] !log cwilliams@cumin1003 START - Cookbook sre.mysql.depool depool db2195: Testing [13:12:23] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2195: Testing [13:15:51] (03CR) 10LSobanski: [C:03+1] role::docker_registry: Ownership transfer [puppet] - 10https://gerrit.wikimedia.org/r/1319842 (owner: 10Clément Goubert) [13:18:51] !log cwilliams@cumin1003 START - Cookbook sre.mysql.depool depool db2195: Testing [13:19:03] !log cwilliams@cumin1003 END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2195: Testing [13:19:14] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12174444 (10elukey) To keep archives happy - in order to reimage the host, you'd need to use: test-cookbook --python-path=/home/your-home-dir/spicerack --no... [13:22:26] !log cwilliams@cumin1003 START - Cookbook sre.mysql.depool depool db2195: Testing [13:22:38] !log cwilliams@cumin1003 END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool db2195: Testing [13:23:56] 10SRE-tools, 10Spicerack: Clarify when spicerack/cumin is going to retry - https://phabricator.wikimedia.org/T433698 (10fgiunchedi) 03NEW [13:23:59] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Clarify when spicerack/cumin is going to retry - https://phabricator.wikimedia.org/T433698#12174470 (10fgiunchedi) p:05Triage→03Low [13:27:22] (03PS1) 10DCausse: opensearch-semantic-search: disable query-insights [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319845 (https://phabricator.wikimedia.org/T433697) [13:30:23] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1048.eqiad.wmnet with OS trixie [13:30:39] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12174510 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt1048.eqiad.wmnet with OS trixie completed: - cl... [13:31:22] (03PS1) 10AikoChou: ml-services: scale revertrisk-wikidata to 3 replicas [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319847 (https://phabricator.wikimedia.org/T420883) [13:32:43] (03PS2) 10AikoChou: ml-services: scale revertrisk-wikidata to 3 replicas [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319847 (https://phabricator.wikimedia.org/T420883) [13:38:00] (03CR) 10Gkyziridis: [C:03+1] "Thnx for working on that one!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319847 (https://phabricator.wikimedia.org/T420883) (owner: 10AikoChou) [13:38:13] (03CR) 10Clément Goubert: [C:03+2] role::docker_registry: Ownership transfer [puppet] - 10https://gerrit.wikimedia.org/r/1319842 (owner: 10Clément Goubert) [13:40:51] 06SRE, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 5 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12174539 (10brouberol) [13:47:59] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [13:49:12] (03PS1) 10Brouberol: admin: allow analytics-admin members to impersonate any analytics user [puppet] - 10https://gerrit.wikimedia.org/r/1319853 (https://phabricator.wikimedia.org/T421952) [13:57:07] (03PS1) 10Mhorsey: enable CampaignEvents worklists [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319854 (https://phabricator.wikimedia.org/T429507) [13:57:11] (03CR) 10DLynch: Add script to get constructive edits for all wikis (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1272633 (https://phabricator.wikimedia.org/T428490) (owner: 10Clare Ming) [13:59:34] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, August 03 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319854 (https://phabricator.wikimedia.org/T429507) (owner: 10Mhorsey) [14:03:09] (03CR) 10AikoChou: [C:03+2] ml-services: scale revertrisk-wikidata to 3 replicas [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319847 (https://phabricator.wikimedia.org/T420883) (owner: 10AikoChou) [14:04:27] !log filippo@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1048 [14:04:40] !log filippo@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1048 [14:05:36] (03Merged) 10jenkins-bot: ml-services: scale revertrisk-wikidata to 3 replicas [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319847 (https://phabricator.wikimedia.org/T420883) (owner: 10AikoChou) [14:07:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [14:10:32] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12174654 (10fgiunchedi) Ok baby steps: reimage wasn't completing because puppet wasn't able to complete a successful run. Specifically `nova-compute.service` can't... [14:12:22] (03CR) 10Ebernhardson: [C:03+1] opensearch-semantic-search: disable query-insights [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319845 (https://phabricator.wikimedia.org/T433697) (owner: 10DCausse) [14:12:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [14:13:00] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [14:13:23] 10ops-eqiad, 06SRE, 10Ceph, 06cloud-services-team, and 3 others: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267#12174656 (10VRiley-WMF) opened ticket 229798904 with dell [14:15:07] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12174658 (10fgiunchedi) pinging hosts on `cloud-private` though still doesn't work: ` root@cloudvirt1048:~# ping -c2 -4 rabbitmq01.eqiad1.wikimediacloud.org PING... [14:15:47] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12174664 (10MatthewVernon) OK, I've had a poke at this host (which uses `storcli` rather than `perccli`, which is a bit confusing, but I guess expected?). I... [14:16:22] (03CR) 10Ottomata: "No we shouldn't. For better or worse, EventStreamConfig doesn't know about Kafka details. A stream might be on several clusters, dependin" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319116 (https://phabricator.wikimedia.org/T433507) (owner: 10Ottomata) [14:16:52] !log aikochou@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [14:17:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [14:18:00] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:18:09] (03PS1) 10Atsuko: airflow: undo dev instance version pinning [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319858 (https://phabricator.wikimedia.org/T433388) [14:19:59] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2195.codfw.wmnet with reason: Testing [14:21:25] PROBLEM - Host mc2046 is DOWN: PING CRITICAL - Packet loss = 100% [14:21:38] !log cwilliams@cumin1003 START - Cookbook sre.mysql.depool depool db2195: Testing [14:21:50] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be209[7,8] - https://phabricator.wikimedia.org/T424892#12174703 (10MatthewVernon) [should this task be less public if we're talking vendors and figures?] Thanks for the update - I think that email had slipped... [14:21:56] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2195: Testing [14:22:30] !log aikochou@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [14:23:27] (03CR) 10JHathaway: [C:03+1] Release version 0.1.18 [software/bitu] - 10https://gerrit.wikimedia.org/r/1319812 (owner: 10Slyngshede) [14:32:49] (03PS4) 10Atsuko: airflow: support for v3.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318719 (https://phabricator.wikimedia.org/T432993) [14:36:08] (03PS3) 10CWilliams: sre.mysql.depool: regression of T427381 [cookbooks] - 10https://gerrit.wikimedia.org/r/1319509 (https://phabricator.wikimedia.org/T433564) [14:37:48] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12174803 (10taavi) The switch port was missing the required tagged VLANs (cloud-instances and cloud-private), I fixed that with https://netbox.wikimedia.org/extra... [14:38:21] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: Rerack mc-gp2006 and mc2046 - https://phabricator.wikimedia.org/T432716#12174815 (10Jhancock.wm) @jijiki it's moved. but unless you want to reimage it, you'll need to redo whatever magic cathal did. [14:38:31] (03PS1) 10Elukey: icinga: allow more retries in wait_for_optimal [software/spicerack] - 10https://gerrit.wikimedia.org/r/1319862 (https://phabricator.wikimedia.org/T433266) [14:39:28] (03PS2) 10Elukey: icinga: allow more retries in wait_for_optimal [software/spicerack] - 10https://gerrit.wikimedia.org/r/1319862 (https://phabricator.wikimedia.org/T433266) [14:41:09] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12174829 (10fgiunchedi) Thank you @taavi, appreciate it! I don't understand what this line in the reimage cookbook output did or didn't do: ` [info] Set cloudsw1... [14:41:50] (03CR) 10CWilliams: [C:03+1] "LGTM, thanks for the quick fix!" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1319862 (https://phabricator.wikimedia.org/T433266) (owner: 10Elukey) [14:43:58] (03PS2) 10Hnowlan: team-sre: add o11y team tag [alerts] - 10https://gerrit.wikimedia.org/r/1319828 (https://phabricator.wikimedia.org/T432376) [14:47:59] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2195: Testing [14:48:23] (03CR) 10Krinkle: "We don't generally vary the interpertion of a given docroot between virtual hosts. For consistency and to ease debugging, I suggest approa" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315128 (https://phabricator.wikimedia.org/T432412) (owner: 10Tsevener) [14:48:30] !log pt1979@cumin2002 START - Cookbook sre.dns.netbox [14:49:03] (03CR) 10Brouberol: "Looks great, but this also requires a chart version bumo" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318719 (https://phabricator.wikimedia.org/T432993) (owner: 10Atsuko) [14:49:04] (03CR) 10Krinkle: "(I agree as a temporary test this is fine. I figured it's worth sharing the context.)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315128 (https://phabricator.wikimedia.org/T432412) (owner: 10Tsevener) [14:49:20] (03CR) 10Brouberol: "bump" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318719 (https://phabricator.wikimedia.org/T432993) (owner: 10Atsuko) [14:49:28] (03CR) 10Brouberol: [C:03+1] airflow: undo dev instance version pinning [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319858 (https://phabricator.wikimedia.org/T433388) (owner: 10Atsuko) [14:50:08] (03PS1) 10Bking: cirrus: enable ship_server_logs for CODFW [puppet] - 10https://gerrit.wikimedia.org/r/1319863 (https://phabricator.wikimedia.org/T324335) [14:51:03] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319863 (https://phabricator.wikimedia.org/T324335) (owner: 10Bking) [14:51:35] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12174859 (10taavi) Good question. Looking at the Netbox changelog that change was first made in [[ https://netbox.wikimedia.org/extras/changelog/290184/ | 290184 ]... [14:54:39] pt1979@cumin2002 netbox (PID 2864327) is awaiting input [14:58:26] 10ops-eqiad, 06SRE, 06DC-Ops: Check list of PXE miss-configs for eqiad - https://phabricator.wikimedia.org/T401441#12174875 (10Jhancock.wm) [15:01:25] (03CR) 10Elukey: [C:03+2] icinga: allow more retries in wait_for_optimal [software/spicerack] - 10https://gerrit.wikimedia.org/r/1319862 (https://phabricator.wikimedia.org/T433266) (owner: 10Elukey) [15:01:53] (03PS5) 10Atsuko: airflow: support for v3.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318719 (https://phabricator.wikimedia.org/T432993) [15:02:35] !log pt1979@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mr1 ge-0/0/3 ipv4 - pt1979@cumin2002" [15:02:40] !log pt1979@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add mr1 ge-0/0/3 ipv4 - pt1979@cumin2002" [15:02:40] !log pt1979@cumin2002 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:03:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [15:04:19] (03PS1) 10Bking: cirrus: enable ship_server_logs for EQIAD [puppet] - 10https://gerrit.wikimedia.org/r/1319866 (https://phabricator.wikimedia.org/T324335) [15:05:05] (03CR) 10CI reject: [V:04-1] cirrus: enable ship_server_logs for EQIAD [puppet] - 10https://gerrit.wikimedia.org/r/1319866 (https://phabricator.wikimedia.org/T324335) (owner: 10Bking) [15:05:53] (03PS2) 10Bking: cirrus: enable ship_server_logs for EQIAD [puppet] - 10https://gerrit.wikimedia.org/r/1319866 (https://phabricator.wikimedia.org/T324335) [15:07:42] (03CR) 10Clément Goubert: "Agreed with Timo. I do wonder whether we will have to do more things like this, making it potentially worth doing now, or if we table that" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315128 (https://phabricator.wikimedia.org/T432412) (owner: 10Tsevener) [15:08:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [15:09:58] (03CR) 10Brouberol: [C:03+1] airflow: support for v3.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318719 (https://phabricator.wikimedia.org/T432993) (owner: 10Atsuko) [15:10:15] (03CR) 10Atsuko: [C:03+2] airflow: support for v3.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318719 (https://phabricator.wikimedia.org/T432993) (owner: 10Atsuko) [15:10:24] (03CR) 10Atsuko: [C:03+2] airflow: undo dev instance version pinning [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319858 (https://phabricator.wikimedia.org/T433388) (owner: 10Atsuko) [15:12:14] (03PS1) 10Krinkle: logging: Remove mention of 'fatal' channel that no longer exists [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319868 (https://phabricator.wikimedia.org/T247113) [15:12:49] (03Merged) 10jenkins-bot: airflow: undo dev instance version pinning [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319858 (https://phabricator.wikimedia.org/T433388) (owner: 10Atsuko) [15:13:47] (03Merged) 10jenkins-bot: airflow: support for v3.3.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318719 (https://phabricator.wikimedia.org/T432993) (owner: 10Atsuko) [15:16:45] (03PS2) 10Hnowlan: team-sre: use traffic team tag [alerts] - 10https://gerrit.wikimedia.org/r/1319829 (https://phabricator.wikimedia.org/T432376) [15:16:52] (03PS2) 10Hnowlan: team-sre: add data-persistence [alerts] - 10https://gerrit.wikimedia.org/r/1319830 (https://phabricator.wikimedia.org/T432376) [15:18:40] (03CR) 10Hnowlan: [C:03+1] thumbor: Restrict access to thumb file types [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319481 (https://phabricator.wikimedia.org/T430528) (owner: 10Ladsgroup) [15:19:34] (03PS2) 10Hnowlan: team-sre: add dcops tag [alerts] - 10https://gerrit.wikimedia.org/r/1319831 (https://phabricator.wikimedia.org/T432376) [15:19:40] (03PS2) 10Hnowlan: team-sre: add infrastructure-foundations tag [alerts] - 10https://gerrit.wikimedia.org/r/1319832 (https://phabricator.wikimedia.org/T432376) [15:32:39] 06SRE, 10SRE-Access-Requests, 10SRE-tools, 10Cumin, and 2 others: add dcops group to run sre.hosts.downtime cookbook - https://phabricator.wikimedia.org/T433409#12174984 (10Dzahn) @elukey ACK, after making that comment I realized there is a difference between "add a member to the group"-approver and "c... [15:33:27] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2195: Testing [15:34:44] (03CR) 10Dzahn: [C:03+2] admin: Add new SSH public key for Ammarpad [puppet] - 10https://gerrit.wikimedia.org/r/1318214 (owner: 10Ammarpad) [15:35:22] claime: wanna join "multiple"? [15:36:21] (03CR) 10Dzahn: [C:03+2] puppet-diffs: remove hiera data for pcc-db1001.yaml [puppet] - 10https://gerrit.wikimedia.org/r/1314050 (owner: 10Andrew Bogott) [15:37:20] I see it's just about role owner.. just doing it [15:38:16] mutante: Oh so sorry [15:38:20] I got distracted [15:38:24] :( [15:39:02] (03CR) 10Dzahn: "ACK! thanks :)" [puppet] - 10https://gerrit.wikimedia.org/r/1319839 (https://phabricator.wikimedia.org/T418521) (owner: 10Hashar) [15:39:04] (03CR) 10Dzahn: [C:03+2] ci: allow inbound ssh from controller on prod agents [puppet] - 10https://gerrit.wikimedia.org/r/1319839 (https://phabricator.wikimedia.org/T418521) (owner: 10Hashar) [15:39:15] claime: no worries, already done now [15:39:23] once I saw what the change was [15:42:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:52:00] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319866 (https://phabricator.wikimedia.org/T324335) (owner: 10Bking) [15:53:00] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2033 is not sending any prefix to lsw1-b7-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [15:56:32] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12175045 (10RobH) @MatthewVernon: What disk shoudl disk 18 swap with? We don't have spares of this exact hard disk, as this config with SM is new and nothin... [15:56:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:57:55] (03PS2) 10Bking: cirrussearch CODFW: Allow, but disable the security plugin. [puppet] - 10https://gerrit.wikimedia.org/r/1319524 (https://phabricator.wikimedia.org/T350516) [15:58:04] (03CR) 10Bking: [C:03+2] cirrussearch CODFW: Allow, but disable the security plugin. [puppet] - 10https://gerrit.wikimedia.org/r/1319524 (https://phabricator.wikimedia.org/T350516) (owner: 10Bking) [16:08:00] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:08:06] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12175058 (10Jhancock.wm) @MatthewVernon i got an 8GB from a decommissioned backup server. would that work? [16:09:46] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12175066 (10CDiggs-WMF) @Dzahn So I now have the wmf ldap access, but I'm not sure exactly where to go to get into Matomo now. Any help is appreciated! And if not I may hav... [16:11:03] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12175075 (10fgiunchedi) In the meantime I have restore nova-compute and brought it up: ` | 79d5cff2-1f5b-423d-bd9a-6119597f8aaa | nova-compute | cloudvirt1048... [16:17:28] 06SRE, 10SRE-Access-Requests, 10LDAP-Access-Requests: Grant Access to wmf for Chandler Diggs - https://phabricator.wikimedia.org/T433302#12175087 (10Dzahn) Hi @CDiggs-WMF here are some docs about it: https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Matomo So since this used to be called Piwiki b... [16:18:00] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:19:37] (03CR) 10Krinkle: [C:03+1] wmf-config/logging.php: Adjustments for WMF_MAINTENANCE_OFFLINE (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315968 (https://phabricator.wikimedia.org/T433104) (owner: 10Ahmon Dancy) [16:26:35] (03PS2) 10Bking: cirrussearch EQIAD: Allow, but disable the security plugin. [puppet] - 10https://gerrit.wikimedia.org/r/1319525 (https://phabricator.wikimedia.org/T350516) [16:28:05] (03CR) 10Bking: [C:03+2] cirrussearch EQIAD: Allow, but disable the security plugin. [puppet] - 10https://gerrit.wikimedia.org/r/1319525 (https://phabricator.wikimedia.org/T350516) (owner: 10Bking) [16:28:36] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12175118 (10RobH) The disk ordered in this host is HDD-3A08T-1FECR(x24)HDD, 3.5",SAS,8TB,7.2K,Fast format,Enterprise,CMR,STD. If the 8GB disk from a dcom ho... [16:40:57] (03CR) 10Clément Goubert: [C:03+1] "Seen, will deploy on Monday" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1312695 (owner: 10Aaron Schulz) [16:44:12] (03CR) 10Bking: [C:03+1] "Sorry the the long delay! Feel free to ping me in IRC (inflatador) or Slack if I'm not responding to a review request." [alerts] - 10https://gerrit.wikimedia.org/r/1311834 (https://phabricator.wikimedia.org/T414662) (owner: 10Hnowlan) [16:47:06] (03CR) 10Bking: [C:03+2] cirrus: enable ship_server_logs for CODFW [puppet] - 10https://gerrit.wikimedia.org/r/1319863 (https://phabricator.wikimedia.org/T324335) (owner: 10Bking) [16:52:27] (03CR) 10RLazarus: [C:03+1] httpbb: Add a --request-header argument. [software/httpbb] - 10https://gerrit.wikimedia.org/r/1318682 (https://phabricator.wikimedia.org/T428972) (owner: 10Blake) [16:53:01] 10ops-codfw, 06DC-Ops, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: move wikikube-worker servers in same rack - https://phabricator.wikimedia.org/T431585#12175222 (10Clement_Goubert) Hey @Jhancock.wm you can move them whenever you want, just depool them first and repool them after, or we can coordinate to... [17:13:27] (03CR) 10Clément Goubert: [C:03+1] "This is just a test fixtures, what matters is that it is in https://gerrit.wikimedia.org/g/operations/deployment-charts/+/6fdd2a05f755619b" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310654 (owner: 10Aaron Schulz) [17:23:10] (03PS1) 10Jdlrobson: Enable page images on recipe namespace [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315895 [17:23:14] (03PS2) 10Jdlrobson: Enable page images on recipe namespace [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315895 [17:23:55] (03PS1) 10Jdlrobson: Disable wgMFCustomSiteModules on English Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319804 (https://phabricator.wikimedia.org/T375538) [17:40:28] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [17:40:59] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [17:41:00] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [17:41:29] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [17:42:19] (03PS6) 10Clare Ming: Add script to get constructive edits for all wikis [puppet] - 10https://gerrit.wikimedia.org/r/1272633 (https://phabricator.wikimedia.org/T428490) [17:44:49] (03CR) 10Clare Ming: Add script to get constructive edits for all wikis (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1272633 (https://phabricator.wikimedia.org/T428490) (owner: 10Clare Ming) [17:47:46] (03CR) 10Clare Ming: Add script to get constructive edits for all wikis (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1272633 (https://phabricator.wikimedia.org/T428490) (owner: 10Clare Ming) [17:48:00] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:59:42] (03CR) 10Dzahn: "Dan, let's decide whether we want both zuul main and zuul executor to be on physical hardware or the executors stay VMs." [puppet] - 10https://gerrit.wikimedia.org/r/1319547 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [18:03:51] (03PS1) 10Ahmon Dancy: services/shellbox: Add traindev environment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319890 (https://phabricator.wikimedia.org/T412941) [18:08:40] (03PS2) 10Ahmon Dancy: services/shellbox: Add traindev environment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319890 (https://phabricator.wikimedia.org/T412941) [18:13:00] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [18:46:31] (03PS1) 10Bking: cirrus: Add a defined type for master-eligibles [puppet] - 10https://gerrit.wikimedia.org/r/1319891 (https://phabricator.wikimedia.org/T433726) [18:47:02] (03PS2) 10Bking: cirrus: Add a defined type for master-eligibles [puppet] - 10https://gerrit.wikimedia.org/r/1319891 (https://phabricator.wikimedia.org/T433726) [18:47:22] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319891 (https://phabricator.wikimedia.org/T433726) (owner: 10Bking) [18:55:11] 10ops-eqsin, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12175477 (10Papaul) [18:55:35] (03PS3) 10Bking: cirrus: Add a defined type for master-eligibles [puppet] - 10https://gerrit.wikimedia.org/r/1319891 (https://phabricator.wikimedia.org/T433726) [19:03:13] (03CR) 10Bking: [C:03+2] cirrus: enable ship_server_logs for EQIAD [puppet] - 10https://gerrit.wikimedia.org/r/1319866 (https://phabricator.wikimedia.org/T324335) (owner: 10Bking) [19:42:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:43:29] (03CR) 10Tsevener: "Thanks for the review! I agree, I would rather set up something a little more long-term here than revert temporary code once our testing i" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1315128 (https://phabricator.wikimedia.org/T432412) (owner: 10Tsevener) [19:53:00] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2033 is not sending any prefix to lsw1-b7-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [19:56:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:18:56] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:26:25] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:27:55] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:28:02] (03PS1) 10Slyngshede: Revert "admin: Add new SSH public key for Ammarpad" [puppet] - 10https://gerrit.wikimedia.org/r/1319897 [21:29:29] (03CR) 10Slyngshede: [C:03+2] Revert "admin: Add new SSH public key for Ammarpad" [puppet] - 10https://gerrit.wikimedia.org/r/1319897 (owner: 10Slyngshede) [21:33:33] (03CR) 10Slyngshede: [C:03+1] "Sorry @ammarpad@yahoo.com, nothing you did, but I forgot to follow prober procedure. I'll get back to you with how we can verify your key." [puppet] - 10https://gerrit.wikimedia.org/r/1318214 (owner: 10Ammarpad) [21:37:30] (03CR) 10Dzahn: [C:03+1] "sorry, same for me. we'll get you verified asap." [puppet] - 10https://gerrit.wikimedia.org/r/1319897 (owner: 10Slyngshede) [21:48:00] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [21:48:18] (03CR) 10Dzahn: [C:03+2] "@Ammarpad can you please create a new key and for verification purposes put the public key in your home directory on the deployment server" [puppet] - 10https://gerrit.wikimedia.org/r/1318214 (owner: 10Ammarpad) [21:52:15] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [puppet] - 10https://gerrit.wikimedia.org/r/1318792 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [21:54:16] (03CR) 10Andrea Denisse: [C:03+1] "LGTM!!" [puppet] - 10https://gerrit.wikimedia.org/r/1306792 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [22:13:00] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [22:26:07] PROBLEM - Blazegraph Port for wdqs-blazegraph on wdqs2022 is CRITICAL: connect to address 127.0.0.1 and port 9999: Connection refused https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [22:27:03] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs2022 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [23:42:22] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1319901 [23:42:23] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1319901 (owner: 10TrainBranchBot) [23:42:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:51:31] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1319901 (owner: 10TrainBranchBot) [23:53:00] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2033 is not sending any prefix to lsw1-b7-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes