[00:22:45] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [00:32:45] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [01:11:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:12:34] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1309316 [01:12:34] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1309316 (owner: 10TrainBranchBot) [01:20:25] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1309316 (owner: 10TrainBranchBot) [01:35:40] (03PS3) 10Scott French: P:etcd::v3: Add support for explicitly enabling the v2 API [puppet] - 10https://gerrit.wikimedia.org/r/1309315 (https://phabricator.wikimedia.org/T428495) [01:35:40] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1309315 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [01:50:31] (03PS6) 10Scott French: zookeeper: Support installation from component/zookeeper34 [puppet] - 10https://gerrit.wikimedia.org/r/1309257 (https://phabricator.wikimedia.org/T428495) [01:50:31] (03PS4) 10Scott French: hieradata: Use component/zookeeper34 on conf* hosts (codfw) [puppet] - 10https://gerrit.wikimedia.org/r/1309260 (https://phabricator.wikimedia.org/T428495) [01:50:54] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1309257 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [01:51:10] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1309260 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [01:51:29] PROBLEM - Check if dnsdist.service has been restarted after /etc/dnsdist/dnsdist.conf was changed on doh5004 is CRITICAL: CRITICAL: Service dnsdist.service has not been restarted after /etc/dnsdist/dnsdist.conf was changed (gt 4h). https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring%23Service_Restart_Check [02:00:26] !log mwpresync@deploy2003 Started scap build-images: Publishing wmf/next image [02:07:08] !log mwpresync@deploy2003 Finished scap build-images: Publishing wmf/next image (duration: 06m 41s) [02:37:33] PROBLEM - Wikidough DoH Check -IPv4- on doh5004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring%23Wikidough_Basic_Check [02:38:23] RECOVERY - Wikidough DoH Check -IPv4- on doh5004 is OK: HTTP OK: HTTP/1.1 200 OK - 595 bytes in 1.020 second response time https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring%23Wikidough_Basic_Check [02:52:45] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [02:57:43] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [03:02:45] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [04:32:45] FIRING: WidespreadPuppetFailure: Puppet has failed in esams - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [04:36:45] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [04:40:03] (03PS1) 10Ryan Kemper: rolling-operation: add per-batch preflight checks [cookbooks] - 10https://gerrit.wikimedia.org/r/1309320 (https://phabricator.wikimedia.org/T426862) [04:42:45] FIRING: [2x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [04:45:20] (03PS2) 10Ryan Kemper: rolling-operation: add per-batch preflight checks [cookbooks] - 10https://gerrit.wikimedia.org/r/1309320 (https://phabricator.wikimedia.org/T426862) [04:46:45] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [04:47:45] FIRING: [3x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [04:52:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [05:07:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [05:11:28] (03CR) 10Giuseppe Lavagetto: [C:03+1] P:etcd::v3: Add support for explicitly enabling the v2 API [puppet] - 10https://gerrit.wikimedia.org/r/1309315 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [05:11:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:12:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [05:27:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [05:52:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [05:57:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260710T0600) [06:02:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [06:32:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [06:45:11] !log jynus@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 10 hosts with reason: reboot & upgrade [06:57:43] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260710T0700) [07:02:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [07:02:29] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 80%, RTA = 8354.96 ms [07:07:15] RESOLVED: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [07:07:31] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 222.60 ms [07:23:03] !log jynus@cumin1003 START - Cookbook sre.hosts.remove-downtime for 10 hosts [07:23:09] !log jynus@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 10 hosts [07:23:25] (03PS1) 10Mszwarc: Revert "Temporarily change plwiki tagline for 1.7M articles" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309552 [07:23:38] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 13 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309552 (owner: 10Mszwarc) [07:33:27] (03PS1) 10Bartosz Wójtowicz: ml-services: move qwen36-27b to llm namespace. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309554 (https://phabricator.wikimedia.org/T425680) [07:41:13] (03CR) 10Anzx: "please add phabricator ticket ID in commit message, we usually delete files fews days after revert so that broken logos doesn't appear on " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309552 (owner: 10Mszwarc) [07:42:47] (03PS2) 10Mszwarc: Revert "Temporarily change plwiki tagline for 1.7M articles" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309552 (https://phabricator.wikimedia.org/T430512) [07:43:10] (03CR) 10Mszwarc: "Oops, I missed it" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309552 (https://phabricator.wikimedia.org/T430512) (owner: 10Mszwarc) [07:44:06] !log dcausse@deploy2003 helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [07:44:17] !log dcausse@deploy2003 helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [07:47:15] !log dcausse@deploy2003 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [07:47:26] !log dcausse@deploy2003 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [07:47:47] (03PS1) 10Mszwarc: plwiki: Switch back to normal tagline [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309559 (https://phabricator.wikimedia.org/T430512) [07:48:44] (03CR) 10CI reject: [V:04-1] plwiki: Switch back to normal tagline [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309559 (https://phabricator.wikimedia.org/T430512) (owner: 10Mszwarc) [07:48:59] (03PS8) 10Filippo Giunchedi: dumps: add nfs_client for dumps-nfs.w.o [puppet] - 10https://gerrit.wikimedia.org/r/1308126 (https://phabricator.wikimedia.org/T411248) [07:48:59] (03PS8) 10Filippo Giunchedi: statistics: optional support for dumps-nfs.w.o [puppet] - 10https://gerrit.wikimedia.org/r/1308127 (https://phabricator.wikimedia.org/T411248) [07:48:59] (03PS8) 10Filippo Giunchedi: wmcs: add dumps-nfs.w.o support [puppet] - 10https://gerrit.wikimedia.org/r/1308128 (https://phabricator.wikimedia.org/T411248) [07:49:00] (03PS3) 10Filippo Giunchedi: labstore: don't check after umount [puppet] - 10https://gerrit.wikimedia.org/r/1309200 (https://phabricator.wikimedia.org/T411248) [07:49:06] (03CR) 10Filippo Giunchedi: "Thank you for the review !" [puppet] - 10https://gerrit.wikimedia.org/r/1308126 (https://phabricator.wikimedia.org/T411248) (owner: 10Filippo Giunchedi) [07:49:24] !log jynus@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 11 hosts with reason: reboot & upgrade [07:51:17] (03CR) 10CI reject: [V:04-1] dumps: add nfs_client for dumps-nfs.w.o [puppet] - 10https://gerrit.wikimedia.org/r/1308126 (https://phabricator.wikimedia.org/T411248) (owner: 10Filippo Giunchedi) [07:52:12] !log dcausse@deploy2003 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [07:52:12] (03PS2) 10Mszwarc: plwiki: Switch back to normal tagline [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309559 (https://phabricator.wikimedia.org/T430512) [07:52:12] (03PS3) 10Mszwarc: Revert "Temporarily change plwiki tagline for 1.7M articles" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309552 (https://phabricator.wikimedia.org/T430512) [07:52:27] !log dcausse@deploy2003 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [07:53:24] (03CR) 10Mszwarc: "I have split the patch into changing the tagline and removing the files from the server, then." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309552 (https://phabricator.wikimedia.org/T430512) (owner: 10Mszwarc) [07:55:50] (03CR) 10Anzx: [C:03+1] plwiki: Switch back to normal tagline [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309559 (https://phabricator.wikimedia.org/T430512) (owner: 10Mszwarc) [07:56:05] (03CR) 10Anzx: [C:03+1] Revert "Temporarily change plwiki tagline for 1.7M articles" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309552 (https://phabricator.wikimedia.org/T430512) (owner: 10Mszwarc) [07:56:35] (03PS9) 10Filippo Giunchedi: statistics: optional support for dumps-nfs.w.o [puppet] - 10https://gerrit.wikimedia.org/r/1308127 (https://phabricator.wikimedia.org/T411248) [07:56:35] (03PS9) 10Filippo Giunchedi: wmcs: add dumps-nfs.w.o support [puppet] - 10https://gerrit.wikimedia.org/r/1308128 (https://phabricator.wikimedia.org/T411248) [07:56:36] (03PS4) 10Filippo Giunchedi: labstore: don't check after umount [puppet] - 10https://gerrit.wikimedia.org/r/1309200 (https://phabricator.wikimedia.org/T411248) [07:56:37] (03CR) 10Filippo Giunchedi: statistics: optional support for dumps-nfs.w.o (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1308127 (https://phabricator.wikimedia.org/T411248) (owner: 10Filippo Giunchedi) [07:58:45] FIRING: CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnstable [08:00:29] !log jynus@cumin1003 START - Cookbook sre.hosts.remove-downtime for db[2183-2184].codfw.wmnet [08:00:30] !log jynus@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db[2183-2184].codfw.wmnet [08:03:45] FIRING: [2x] CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnstable [08:06:32] (03PS9) 10Filippo Giunchedi: dumps: add nfs_client for dumps-nfs.w.o [puppet] - 10https://gerrit.wikimedia.org/r/1308126 (https://phabricator.wikimedia.org/T411248) [08:06:32] (03PS10) 10Filippo Giunchedi: statistics: optional support for dumps-nfs.w.o [puppet] - 10https://gerrit.wikimedia.org/r/1308127 (https://phabricator.wikimedia.org/T411248) [08:06:32] (03PS10) 10Filippo Giunchedi: wmcs: add dumps-nfs.w.o support [puppet] - 10https://gerrit.wikimedia.org/r/1308128 (https://phabricator.wikimedia.org/T411248) [08:06:33] (03PS5) 10Filippo Giunchedi: labstore: don't check after umount [puppet] - 10https://gerrit.wikimedia.org/r/1309200 (https://phabricator.wikimedia.org/T411248) [08:06:40] (03CR) 10Kevin Bazira: [C:03+1] ml-services: move qwen36-27b to llm namespace. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309554 (https://phabricator.wikimedia.org/T425680) (owner: 10Bartosz Wójtowicz) [08:09:16] 06SRE, 10SRE-swift-storage, 07Essential-Work, 13Patch-For-Review: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12107919 (10MatthewVernon) [08:16:32] FIRING: SLOBudgetBurn: Search update lag is below 95% target in codfw - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [08:17:37] (03CR) 10Majavah: [C:03+1] dumps: add nfs_client for dumps-nfs.w.o [puppet] - 10https://gerrit.wikimedia.org/r/1308126 (https://phabricator.wikimedia.org/T411248) (owner: 10Filippo Giunchedi) [08:17:43] (03CR) 10Majavah: [C:03+1] statistics: optional support for dumps-nfs.w.o [puppet] - 10https://gerrit.wikimedia.org/r/1308127 (https://phabricator.wikimedia.org/T411248) (owner: 10Filippo Giunchedi) [08:17:56] (03CR) 10Majavah: [C:03+1] wmcs: add dumps-nfs.w.o support [puppet] - 10https://gerrit.wikimedia.org/r/1308128 (https://phabricator.wikimedia.org/T411248) (owner: 10Filippo Giunchedi) [08:18:02] (03CR) 10Majavah: [C:03+1] labstore: don't check after umount [puppet] - 10https://gerrit.wikimedia.org/r/1309200 (https://phabricator.wikimedia.org/T411248) (owner: 10Filippo Giunchedi) [08:18:10] !log dcausse@deploy2003 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [08:18:14] !log dcausse@deploy2003 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [08:21:04] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: move qwen36-27b to llm namespace. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309554 (https://phabricator.wikimedia.org/T425680) (owner: 10Bartosz Wójtowicz) [08:21:28] 06SRE-OnFire, 10corto: Cortobot help command should not spam the main channel - https://phabricator.wikimedia.org/T421858#12107963 (10MatthewVernon) I was coming here to ask for something similar - even if it'll reply to cortobot: help in-channel, it should have an option to respond to help over privmsg if add... [08:23:27] (03Merged) 10jenkins-bot: ml-services: move qwen36-27b to llm namespace. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309554 (https://phabricator.wikimedia.org/T425680) (owner: 10Bartosz Wójtowicz) [08:24:01] !log dcausse@deploy2003 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [08:24:04] !log dcausse@deploy2003 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [08:25:48] FIRING: PuppetFailure: Puppet has failed on maps1012:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [08:28:45] FIRING: [2x] CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnstable [08:29:12] !log dcausse@deploy2003 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [08:29:14] !log dcausse@deploy2003 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [08:30:17] !log Deploying Refinery at 1abf22ea for changes 1308121/T427068 1306491/T430020 and 1308190 [08:30:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:30:55] !log javiermonton@deploy2003 Started deploy [analytics/refinery@1abf22e] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@1abf22ea] [08:32:58] !log javiermonton@deploy2003 Finished deploy [analytics/refinery@1abf22e] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@1abf22ea] (duration: 02m 03s) [08:33:19] !log javiermonton@deploy2003 Started deploy [analytics/refinery@1abf22e]: Regular analytics weekly train [analytics/refinery@1abf22ea] [08:33:27] (03PS1) 10Gmodena: wdqs: pool backend on index load [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309597 (https://phabricator.wikimedia.org/T431253) [08:34:35] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [08:35:11] 06SRE-OnFire, 10corto: Corto command to just show notes URL - https://phabricator.wikimedia.org/T431793 (10MatthewVernon) 03NEW [08:36:27] 06SRE-OnFire, 10corto, 10Incident Tooling: Corto command to just show notes URL - https://phabricator.wikimedia.org/T431793#12108083 (10MatthewVernon) [08:36:32] FIRING: [10x] SLOBudgetBurn: Search update lag is below 95% target in codfw - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [08:38:19] !log bwojtowicz@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [08:38:36] !log javiermonton@deploy2003 Finished deploy [analytics/refinery@1abf22e]: Regular analytics weekly train [analytics/refinery@1abf22ea] (duration: 05m 17s) [08:38:45] RESOLVED: [2x] CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnstable [08:38:49] !log javiermonton@deploy2003 Started deploy [analytics/refinery@1abf22e] (thin): Regular analytics weekly train THIN [analytics/refinery@1abf22ea] [08:40:40] (03CR) 10Giuseppe Lavagetto: [C:03+1] "lgtm, thanks for taking care of this." [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309303 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [08:41:00] !log javiermonton@deploy2003 Finished deploy [analytics/refinery@1abf22e] (thin): Regular analytics weekly train THIN [analytics/refinery@1abf22ea] (duration: 02m 11s) [08:44:31] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [08:45:48] RESOLVED: PuppetFailure: Puppet has failed on maps1012:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [08:46:32] FIRING: [10x] SLOBudgetBurn: Search update lag is below 95% target in codfw - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [08:49:33] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 223.00 ms [08:56:31] (03PS1) 10Trueg: wdqs: Update docker image versions + config values for debugging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309606 (https://phabricator.wikimedia.org/T431253) [09:01:33] (03CR) 10Gmodena: [C:03+2] wdqs: Update docker image versions + config values for debugging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309606 (https://phabricator.wikimedia.org/T431253) (owner: 10Trueg) [09:02:50] (03CR) 10Brouberol: [C:03+2] Define public/discovery DNS records for WDQS/qlever [dns] - 10https://gerrit.wikimedia.org/r/1309183 (https://phabricator.wikimedia.org/T429150) (owner: 10Brouberol) [09:03:07] !log brouberol@dns1004 START - running authdns-update [09:03:34] (03CR) 10Filippo Giunchedi: [C:03+2] dumps: add nfs_client for dumps-nfs.w.o [puppet] - 10https://gerrit.wikimedia.org/r/1308126 (https://phabricator.wikimedia.org/T411248) (owner: 10Filippo Giunchedi) [09:04:06] (03Merged) 10jenkins-bot: wdqs: Update docker image versions + config values for debugging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309606 (https://phabricator.wikimedia.org/T431253) (owner: 10Trueg) [09:04:54] !log brouberol@dns1004 END - running authdns-update [09:06:01] !log blake@cumin1003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1053.eqiad.wmnet [09:06:04] !log blake@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet [09:06:39] !log blake@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet [09:06:53] !log blake@cumin1003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker1053.eqiad.wmnet [09:08:08] !log blake@cumin1003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1053.eqiad.wmnet [09:08:13] !log blake@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet [09:08:13] !log blake@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet [09:08:23] !log blake@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-worker1053.eqiad.wmnet with OS trixie [09:08:37] (03PS1) 10Gmodena: wdqs: pool backend on index load [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309597 (https://phabricator.wikimedia.org/T431253) [09:11:02] !log blake@cumin1003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1053 [09:11:13] !log blake@cumin1003 START - Cookbook sre.dns.netbox [09:11:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:11:58] (03PS2) 10Gmodena: wdqs: pool backend on index load [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309597 (https://phabricator.wikimedia.org/T431253) [09:15:28] !log blake@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1053 - blake@cumin1003" [09:15:32] !log blake@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1053 - blake@cumin1003" [09:15:32] !log blake@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:15:32] !log blake@cumin1003 START - Cookbook sre.dns.wipe-cache wikikube-worker1053.eqiad.wmnet 48.32.64.10.in-addr.arpa 8.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [09:15:36] !log blake@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1053.eqiad.wmnet 48.32.64.10.in-addr.arpa 8.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [09:15:37] !log blake@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1053 [09:16:00] !log blake@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1053 [09:16:00] !log blake@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1053 [09:16:32] RESOLVED: [9x] SLOBudgetBurn: Search update lag is below 95% target in codfw - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [09:16:43] (03PS1) 10Kevin Bazira: tts: ml-services: deploy latest TTS isvc model-server [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309616 (https://phabricator.wikimedia.org/T430536) [09:18:42] (03CR) 10JavierMonton: streams: webrequest - pageview - trending (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1308656 (https://phabricator.wikimedia.org/T430675) (owner: 10JavierMonton) [09:19:23] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:21:53] (03CR) 10Bartosz Wójtowicz: [C:03+1] "Thank you!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309616 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [09:24:23] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:26:41] (03CR) 10Kevin Bazira: [C:03+2] tts: ml-services: deploy latest TTS isvc model-server [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309616 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [09:27:05] (03CR) 10Trueg: [C:03+1] "makes sense" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309597 (https://phabricator.wikimedia.org/T431253) (owner: 10Gmodena) [09:28:30] 06SRE-OnFire, 10corto, 10Incident Tooling: Corto should by default write to the open incident - https://phabricator.wikimedia.org/T431801 (10MatthewVernon) 03NEW [09:29:07] (03Merged) 10jenkins-bot: tts: ml-services: deploy latest TTS isvc model-server [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309616 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [09:29:48] (03CR) 10Gmodena: [C:03+2] wdqs: pool backend on index load [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309597 (https://phabricator.wikimedia.org/T431253) (owner: 10Gmodena) [09:32:08] (03Merged) 10jenkins-bot: wdqs: pool backend on index load [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309597 (https://phabricator.wikimedia.org/T431253) (owner: 10Gmodena) [09:36:27] !log blake@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1053.eqiad.wmnet with reason: host reimage [09:36:34] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [09:37:02] !log apply new IBGP outbound policy on lsw1-e2-codfw T423430 [09:37:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:37:06] T423430: Don't announce OSPF routes in unicast BGP on Nokia SR-Linux - https://phabricator.wikimedia.org/T423430 [09:41:51] !log blake@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1053.eqiad.wmnet with reason: host reimage [09:42:16] (03PS1) 10Gmodena: wdqs: install external releases in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309625 (https://phabricator.wikimedia.org/T429150) [09:46:37] 06SRE, 06Infrastructure-Foundations, 10netops: Don't announce OSPF routes in unicast BGP on Nokia SR-Linux - https://phabricator.wikimedia.org/T423430#12108326 (10cmooney) I was able to test this in containerlab, which then gave me the confidence to try in prod and it's simple enough. Resulting policy is si... [09:49:12] (03CR) 10Trueg: [C:03+1] wdqs: install external releases in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309625 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [09:50:39] (03CR) 10Gmodena: [C:03+2] wdqs: install external releases in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309625 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [09:51:49] !log cgoubert@cumin1003 START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on A:wikikube-worker-codfw [09:52:00] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2001-2002,2005-2006,2011-2012].codfw.wmnet [09:53:09] (03Merged) 10jenkins-bot: wdqs: install external releases in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309625 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [09:55:18] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [09:55:22] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2001-2002,2005-2006,2011-2012].codfw.wmnet [09:55:51] !log cgoubert@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-worker2001.codfw.wmnet with OS bookworm [10:00:33] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [10:02:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [10:06:55] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [10:07:49] FIRING: HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:07:51] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [10:08:21] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [10:08:59] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [10:11:13] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [10:11:47] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [10:12:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [10:12:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:15:09] !log cgoubert@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2001.codfw.wmnet with reason: host reimage [10:15:29] PROBLEM - Host ms-backup1004 is DOWN: PING CRITICAL - Packet loss = 100% [10:16:59] RECOVERY - Host ms-backup1004 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [10:17:30] that was a reboot, I must have missed it for downtime [10:17:51] (03CR) 10Klausman: [C:03+1] kubernetes: run lvmd as a systemd service on selected dse-k8s workers [puppet] - 10https://gerrit.wikimedia.org/r/1309121 (https://phabricator.wikimedia.org/T429325) (owner: 10Btullis) [10:18:06] yep, I downtimed ms-backup1*.codfw.wmnet by mistake, ignore [10:18:20] (03CR) 10Jforrester: "Huh, yeah, gerrit clearly hasn't indexed it so searches for "Ia5e6f0a6" just come back to here as the singleton result. "Ia5e6f0a65" OTOH " [deployment-charts] - 10https://gerrit.wikimedia.org/r/1308788 (https://phabricator.wikimedia.org/T427864) (owner: 10RLazarus) [10:24:03] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [10:24:07] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [10:25:09] !log cgoubert@cumin1003 END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker2001.codfw.wmnet with reason: host reimage [10:27:14] !log cgoubert@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2001.codfw.wmnet with OS bookworm [10:27:28] !log cgoubert@cumin1003 END (FAIL) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=1) rolling reimage on A:wikikube-worker-codfw [10:27:33] FIRING: KubernetesCalicoDown: wikikube-worker2001.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2001.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:30:11] !log aokoth@cumin1003 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T431656 [10:30:20] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2002,2005-2006,2011-2012].codfw.wmnet [10:30:24] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2002,2005-2006,2011-2012].codfw.wmnet [10:30:43] !log cgoubert@cumin1003 START - Cookbook sre.hosts.remove-downtime for wikikube-worker[2002,2005-2006,2011-2012].codfw.wmnet [10:30:46] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker[2002,2005-2006,2011-2012].codfw.wmnet [10:35:44] !log adjust IBGP outbound policy on lsw1-e2-codfw T423430 towards ssw1-e1-codfw [10:35:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:35:47] T423430: Don't announce OSPF routes in unicast BGP on Nokia SR-Linux - https://phabricator.wikimedia.org/T423430 [10:38:16] !log cgoubert@cumin1003 START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker2001*} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [10:38:20] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet [10:38:22] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet [10:38:38] !log cgoubert@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-worker2001.codfw.wmnet with OS trixie [10:45:25] blake@cumin1003 renumber-node (PID 3370018) is awaiting input [10:46:06] aokoth@cumin1003 aokoth: The backup on gitlab1004 is complete, ready to proceed with upgrade. [10:48:23] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on backup1019 - https://phabricator.wikimedia.org/T431367#12108453 (10jcrespo) FYI I am about to restart this host as part of a kernel upgrade, will be back online in a few minutes. [10:57:14] !log aokoth@cumin1003 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T431656 [10:57:36] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on backup1019 - https://phabricator.wikimedia.org/T431367#12108493 (10jcrespo) Host is back online and in service. [10:57:43] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [10:58:32] (03PS1) 10Brouberol: dse-k8s/wdqs: add the public DNS records to the namespace certificate SANs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309642 (https://phabricator.wikimedia.org/T429150) [10:58:39] ACKNOWLEDGEMENT - Dell PowerEdge or Supermicro Broadcom RAID Controller on backup1019 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 0 OK : virtual_disk: 1 Dgrd : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T431811 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [10:58:48] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on backup1019 - https://phabricator.wikimedia.org/T431811 (10ops-monitoring-bot) 03NEW [10:58:49] !log cgoubert@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2001.codfw.wmnet with reason: host reimage [10:59:21] (03PS1) 10Cathal Mooney: Nokia SR-Linux: add export policy for unicast IBGP clusters [homer/public] - 10https://gerrit.wikimedia.org/r/1309643 (https://phabricator.wikimedia.org/T423430) [11:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260710T0700) [11:00:05] jelto, arnoldokoth, mutante, and arnaudb: #bothumor My software never has bugs. It just develops random features. Rise for GitLab version upgrades. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260710T1100). [11:01:33] (03PS2) 10Brouberol: dse-k8s/wdqs: add the public DNS records to the namespace certificate SANs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309642 (https://phabricator.wikimedia.org/T429150) [11:01:54] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [11:02:27] !log blake@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1053.eqiad.wmnet with OS trixie [11:02:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:03:17] (03PS2) 10Cathal Mooney: Nokia SR-Linux: add export policy for unicast IBGP clusters [homer/public] - 10https://gerrit.wikimedia.org/r/1309643 (https://phabricator.wikimedia.org/T423430) [11:03:24] (03PS1) 10Gmodena: wdqs: bump init image and set qlever memory per-release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309644 (https://phabricator.wikimedia.org/T429150) [11:03:54] (03PS3) 10Cathal Mooney: Nokia SR-Linux: add export policy for unicast IBGP clusters [homer/public] - 10https://gerrit.wikimedia.org/r/1309643 (https://phabricator.wikimedia.org/T423430) [11:05:44] (03PS4) 10Cathal Mooney: Nokia SR-Linux: add export policy for unicast IBGP clusters [homer/public] - 10https://gerrit.wikimedia.org/r/1309643 (https://phabricator.wikimedia.org/T423430) [11:06:35] (03CR) 10Aklapper: [V:03+2 C:03+2] "Thanks! Applies cleanly and `/settings/panel/language/` still loads :)" [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1309241 (owner: 10Pppery) [11:07:34] (03PS1) 10Gmodena: wdqs: install external releases in codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309646 (https://phabricator.wikimedia.org/T429150) [11:08:50] !log cgoubert@cumin1003 END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker2001.codfw.wmnet with reason: host reimage [11:09:29] (03CR) 10Aklapper: [V:03+2 C:03+2] Remove the commented out bit about AM/PM [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1309247 (https://phabricator.wikimedia.org/T428957) (owner: 10Pppery) [11:09:40] (03CR) 10Trueg: [C:03+1] wdqs: bump init image and set qlever memory per-release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309644 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [11:09:40] (03CR) 10Aklapper: [V:03+2 C:03+2] Fix broken link in scripts/ and support/ docs [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1309245 (https://phabricator.wikimedia.org/T429078) (owner: 10Pppery) [11:09:55] (03CR) 10Trueg: [C:03+1] wdqs: install external releases in codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309646 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [11:10:42] (03CR) 10Gmodena: [C:03+2] wdqs: install external releases in codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309646 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [11:10:55] (03CR) 10Trueg: [C:03+1] "ty" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309642 (https://phabricator.wikimedia.org/T429150) (owner: 10Brouberol) [11:11:11] (03CR) 10Gmodena: [C:03+2] wdqs: bump init image and set qlever memory per-release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309644 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [11:11:49] (03CR) 10Neriah: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307836 (https://phabricator.wikimedia.org/T431334) (owner: 10Danielyepezgarces) [11:12:49] (03CR) 10CI reject: [V:04-1] extwiki: Rename wgSitename to Güiquipedia and remove obsolete namespace alias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307836 (https://phabricator.wikimedia.org/T431334) (owner: 10Danielyepezgarces) [11:12:51] (03CR) 10Gmodena: [C:04-1] "The route for scholarly should be query-scholarly-next.wikidata.org" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309642 (https://phabricator.wikimedia.org/T429150) (owner: 10Brouberol) [11:13:00] (03Merged) 10jenkins-bot: wdqs: install external releases in codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309646 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [11:13:41] (03Merged) 10jenkins-bot: wdqs: bump init image and set qlever memory per-release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309644 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [11:13:58] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [11:14:38] !log jynus@cumin1003 START - Cookbook sre.hosts.remove-downtime for 11 hosts [11:14:45] !log jynus@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 11 hosts [11:14:53] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [11:15:13] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [11:16:12] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [11:16:44] (03CR) 10Neriah: "ERROR:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307836 (https://phabricator.wikimedia.org/T431334) (owner: 10Danielyepezgarces) [11:17:12] (03CR) 10Neriah: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309302 (https://phabricator.wikimedia.org/T431765) (owner: 10Danielyepezgarces) [11:17:28] RESOLVED: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [11:17:49] RESOLVED: [2x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:17:53] blake@cumin1003 renumber-node (PID 3370018) is awaiting input [11:19:03] (03CR) 10Aklapper: [V:03+2 C:03+2] "Thanks! (Yay, I don't see any whitespace warnings anymore when applying this one.)" [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1309254 (owner: 10Pppery) [11:19:23] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:20:01] !log cgoubert@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2001.codfw.wmnet with OS trixie [11:21:31] !log jynus@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 18 hosts with reason: reboot & upgrade [11:21:55] (03PS1) 10CWilliams: admin: Added dotfiles for cwilliams [puppet] - 10https://gerrit.wikimedia.org/r/1309650 [11:22:33] RESOLVED: KubernetesCalicoDown: wikikube-worker2001.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2001.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:23:01] cgoubert@cumin1003 roll-reimage-nodes (PID 3378855) is awaiting input [11:24:23] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:27:49] FIRING: HelmReleaseBadStatus: Helm release wdqs/scholarly-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:29:05] (03CR) 10Aklapper: [V:03+2 C:03+2] Add a bunch more locale files [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1309266 (https://phabricator.wikimedia.org/T412651) (owner: 10Pppery) [11:30:07] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [11:30:13] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [11:31:04] !log cgoubert@cumin1003 END (FAIL) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=1) rolling reimage on P{wikikube-worker2001*} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [11:31:25] !log cgoubert@cumin1003 START - Cookbook sre.hosts.remove-downtime for wikikube-worker2001.codfw.wmnet [11:31:26] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker2001.codfw.wmnet [11:32:23] !log blake@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1053.eqiad.wmnet [11:32:24] !log blake@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1053.eqiad.wmnet [11:32:25] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2001.codfw.wmnet [11:32:26] !log blake@cumin1003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1053.eqiad.wmnet [11:32:26] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2001.codfw.wmnet [11:44:05] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [11:47:36] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [11:47:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs/scholarly-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:48:21] !log gmodena@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [11:49:08] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [11:51:16] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [11:58:47] (03PS1) 10Kevin Bazira: ml-services: deploy latest TTS isvc that serializes predict() since pipeline is not thread-safe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309660 (https://phabricator.wikimedia.org/T430536) [12:00:51] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [12:00:53] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [12:01:25] (03CR) 10Kevin Bazira: [C:03+2] ml-services: deploy latest TTS isvc that serializes predict() since pipeline is not thread-safe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309660 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [12:02:48] FIRING: PuppetFailure: Puppet has failed on ping1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [12:03:39] (03Merged) 10jenkins-bot: ml-services: deploy latest TTS isvc that serializes predict() since pipeline is not thread-safe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309660 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [12:07:00] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [12:07:48] RESOLVED: PuppetFailure: Puppet has failed on ping1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [12:08:15] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [12:09:15] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [12:09:23] FIRING: JobUnavailable: Reduced availability for job mysql-dbstore in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:09:37] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [12:09:48] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [12:10:42] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [12:14:12] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [12:14:23] RESOLVED: JobUnavailable: Reduced availability for job mysql-dbstore in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:24:15] (03CR) 10Brouberol: dse-k8s/wdqs: add the public DNS records to the namespace certificate SANs (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309642 (https://phabricator.wikimedia.org/T429150) (owner: 10Brouberol) [12:24:30] (03PS3) 10Brouberol: dse-k8s/wdqs: add the public DNS records to the namespace certificate SANs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309642 (https://phabricator.wikimedia.org/T429150) [12:26:49] (03PS1) 10Gmodena: wdqs: allow codfw backends egress to eqiad RGW [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309670 (https://phabricator.wikimedia.org/T429150) [12:34:08] (03CR) 10Brouberol: [C:03+1] wdqs: allow codfw backends egress to eqiad RGW [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309670 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [12:34:27] (03CR) 10Gmodena: [C:03+2] wdqs: allow codfw backends egress to eqiad RGW [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309670 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [12:34:27] (03CR) 10Brouberol: [C:03+2] dse-k8s/wdqs: add the public DNS records to the namespace certificate SANs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309642 (https://phabricator.wikimedia.org/T429150) (owner: 10Brouberol) [12:36:42] (03Merged) 10jenkins-bot: wdqs: allow codfw backends egress to eqiad RGW [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309670 (https://phabricator.wikimedia.org/T429150) (owner: 10Gmodena) [12:37:36] (03CR) 10Cathal Mooney: [C:03+1] "lgtm!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1309030 (https://phabricator.wikimedia.org/T327300) (owner: 10Ayounsi) [12:37:56] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [12:38:29] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [12:39:02] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [12:39:16] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [12:42:41] (03CR) 10Gmodena: [C:03+1] dse-k8s/wdqs: add the public DNS records to the namespace certificate SANs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309642 (https://phabricator.wikimedia.org/T429150) (owner: 10Brouberol) [12:44:20] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [12:47:41] !log bking@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs1002.eqiad.wmnet with OS trixie [12:48:39] (03CR) 10Filippo Giunchedi: "Ready for merge" [puppet] - 10https://gerrit.wikimedia.org/r/1308626 (https://phabricator.wikimedia.org/T328502) (owner: 10Filippo Giunchedi) [12:48:50] (03PS2) 10Filippo Giunchedi: openstack: remove openstack::monitor::neutron::l3_agent_conntrack [puppet] - 10https://gerrit.wikimedia.org/r/1308626 (https://phabricator.wikimedia.org/T328502) [12:56:24] !log blake@cumin1003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1054.eqiad.wmnet [12:56:28] !log blake@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1054.eqiad.wmnet [12:56:49] FIRING: HelmReleaseBadStatus: Helm release wdqs/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [12:57:00] !log blake@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1054.eqiad.wmnet [12:57:13] !log blake@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-worker1054.eqiad.wmnet with OS trixie [12:59:51] !log blake@cumin1003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1054 [13:00:02] !log blake@cumin1003 START - Cookbook sre.dns.netbox [13:01:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:01:57] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudweb2002-dev.wikimedia.org [13:04:13] !log blake@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1054 - blake@cumin1003" [13:04:18] !log blake@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1054 - blake@cumin1003" [13:04:18] !log blake@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:04:18] !log blake@cumin1003 START - Cookbook sre.dns.wipe-cache wikikube-worker1054.eqiad.wmnet 49.32.64.10.in-addr.arpa 9.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [13:04:22] !log blake@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1054.eqiad.wmnet 49.32.64.10.in-addr.arpa 9.4.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [13:04:22] !log blake@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1054 [13:04:46] !log blake@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1054 [13:04:46] !log blake@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1054 [13:05:17] !log jgiannelos@deploy2003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [13:05:44] !log jgiannelos@deploy2003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [13:08:36] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudweb2002-dev.wikimedia.org [13:08:40] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudservices2004-dev.codfw.wmnet [13:11:56] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices2004-dev.codfw.wmnet [13:12:00] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudservices2005-dev.codfw.wmnet [13:12:39] FIRING: CoreBGPDown: ... [13:12:39] Core BGP session down between cloudsw1-b1-codfw and cloudservices2004-dev (2a02:ec80:a100:205::8) - group cloud_host6 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cloudsw1-b1-codfw:9804&var-bgp_group=cloud_host6&var-bgp_neighbor=cloudservices2004-dev - ... [13:12:39] https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:13:08] that's me ^ [13:14:43] PROBLEM - BFD status on cloudsw1-b1-codfw.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [13:15:43] RECOVERY - BFD status on cloudsw1-b1-codfw.mgmt is OK: UP: 10 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [13:15:49] PROBLEM - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1191 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 2 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [13:15:52] ACKNOWLEDGEMENT - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1191 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 2 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T431828 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [13:16:10] FIRING: [2x] BFDdown: BFD session down between cloudsw1-b1-codfw and 172.20.5.9 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cloudsw1-b1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:17:39] RESOLVED: [3x] CoreBGPDown: Core BGP session down between cloudsw1-b1-codfw and cloudservices2005-dev (172.20.5.9) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:18:06] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudservices2005-dev.codfw.wmnet [13:18:10] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudnet2005-dev.codfw.wmnet [13:21:10] RESOLVED: [2x] BFDdown: BFD session down between cloudsw1-b1-codfw and 172.20.5.9 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cloudsw1-b1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:23:46] !log blake@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1054.eqiad.wmnet with reason: host reimage [13:26:04] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2005-dev.codfw.wmnet [13:26:08] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudnet2006-dev.codfw.wmnet [13:26:53] FIRING: PuppetFailure: Puppet has failed on ldap-maint1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:33:41] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2006-dev.codfw.wmnet [13:33:45] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudidp2001-dev.codfw.wmnet [13:33:47] !log blake@cumin1003 END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker1054.eqiad.wmnet with reason: host reimage [13:34:35] (03PS1) 10Dreamrimmer: Enable the abuse filter block action on Hindi Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309675 (https://phabricator.wikimedia.org/T431830) [13:35:33] FIRING: KubernetesCalicoDown: wikikube-worker1054.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1054.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:37:45] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudidp2001-dev.codfw.wmnet [13:37:50] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudlb2002-dev.codfw.wmnet [13:40:43] PROBLEM - BFD status on cloudsw1-b1-codfw.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [13:41:28] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [13:41:33] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [13:41:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cloudsw1-b1-codfw and cloudlb2002-dev (172.20.5.3) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:41:43] RECOVERY - BFD status on cloudsw1-b1-codfw.mgmt is OK: UP: 10 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [13:42:10] FIRING: [2x] BFDdown: BFD session down between cloudsw1-b1-codfw and 172.20.5.3 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cloudsw1-b1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:42:55] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [13:44:29] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2002-dev.codfw.wmnet [13:44:33] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudlb2003-dev.codfw.wmnet [13:45:33] RESOLVED: KubernetesCalicoDown: wikikube-worker1054.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1054.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:46:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cloudsw1-b1-codfw and cloudlb2002-dev (172.20.5.3) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:46:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [13:47:10] RESOLVED: [2x] BFDdown: BFD session down between cloudsw1-b1-codfw and 172.20.5.3 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cloudsw1-b1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:47:43] PROBLEM - BFD status on cloudsw1-b1-codfw.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [13:48:09] FIRING: [4x] CoreBGPDown: Core BGP session down between cloudsw1-b1-codfw and cloudlb2002-dev (172.20.5.3) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:48:37] !log gmodena@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [13:48:40] FIRING: [4x] BFDdown: BFD session down between cloudsw1-b1-codfw and 172.20.5.4 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cloudsw1-b1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:48:43] RECOVERY - BFD status on cloudsw1-b1-codfw.mgmt is OK: UP: 10 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [13:51:54] RESOLVED: [4x] CoreBGPDown: Core BGP session down between cloudsw1-b1-codfw and cloudlb2002-dev (172.20.5.3) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:53:37] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2003-dev.codfw.wmnet [13:53:40] RESOLVED: [4x] BFDdown: BFD session down between cloudsw1-b1-codfw and 172.20.5.4 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cloudsw1-b1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:53:41] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudlb2004-dev.codfw.wmnet [13:56:43] PROBLEM - Kafka broker TLS certificate validity on kafka-test1009 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/Kafka/Administration%23Renew_TLS_certificate [13:56:43] PROBLEM - BFD status on cloudsw1-b1-codfw.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [13:57:03] PROBLEM - Kafka Broker Server on kafka-test1009 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [13:57:43] RECOVERY - BFD status on cloudsw1-b1-codfw.mgmt is OK: UP: 10 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [13:58:03] PROBLEM - Kafka Broker Server on kafka-test1007 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [13:58:13] PROBLEM - Kafka broker TLS certificate validity on kafka-test1007 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/Kafka/Administration%23Renew_TLS_certificate [14:00:49] FIRING: HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [14:01:49] RESOLVED: PuppetFailure: Puppet has failed on ldap-maint1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:02:29] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2004-dev.codfw.wmnet [14:02:33] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudgw2003-dev.codfw.wmnet [14:05:03] FIRING: KafkaUnderReplicatedPartitions: Under replicated partitions for Kafka cluster test-eqiad in eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration - https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-kafka_cluster=test-eqiad - https://alerts.wikimedia.org/?q=alertname%3DKafkaUnderReplicatedPartitions [14:08:58] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2003-dev.codfw.wmnet [14:09:02] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudgw2004-dev.codfw.wmnet [14:10:08] !log blake@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1054.eqiad.wmnet with OS trixie [14:12:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:13:07] (03PS1) 10Brouberol: trafficserver: redirect query-next.w.o/sparql to qlever in dse-k8s [puppet] - 10https://gerrit.wikimedia.org/r/1309679 (https://phabricator.wikimedia.org/T429150) [14:13:10] (03PS1) 10Brouberol: cache/text: enable caching for query-next.w.o [puppet] - 10https://gerrit.wikimedia.org/r/1309680 (https://phabricator.wikimedia.org/T429150) [14:13:58] (03CR) 10Brouberol: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1309679 (https://phabricator.wikimedia.org/T429150) (owner: 10Brouberol) [14:14:01] (03CR) 10Brouberol: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1309680 (https://phabricator.wikimedia.org/T429150) (owner: 10Brouberol) [14:14:10] (03PS7) 10Btullis: kubernetes: run lvmd as a systemd service on selected dse-k8s workers [puppet] - 10https://gerrit.wikimedia.org/r/1309121 (https://phabricator.wikimedia.org/T429325) [14:15:12] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2004-dev.codfw.wmnet [14:15:15] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcontrol2010-dev.codfw.wmnet [14:17:44] (03CR) 10Filippo Giunchedi: "LGTM overall, two main comments:" [puppet] - 10https://gerrit.wikimedia.org/r/1305183 (https://phabricator.wikimedia.org/T429630) (owner: 10MVernon) [14:20:31] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [14:21:27] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2010-dev.codfw.wmnet [14:21:31] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcontrol2006-dev.codfw.wmnet [14:22:20] (03CR) 10Gmodena: [C:03+1] trafficserver: redirect query-next.w.o/sparql to qlever in dse-k8s [puppet] - 10https://gerrit.wikimedia.org/r/1309679 (https://phabricator.wikimedia.org/T429150) (owner: 10Brouberol) [14:22:54] (03CR) 10Gmodena: [C:03+1] cache/text: enable caching for query-next.w.o [puppet] - 10https://gerrit.wikimedia.org/r/1309680 (https://phabricator.wikimedia.org/T429150) (owner: 10Brouberol) [14:29:32] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2006-dev.codfw.wmnet [14:29:35] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcontrol2005-dev.codfw.wmnet [14:30:41] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING WARNING - Packet loss = 50%, RTA = 1469.91 ms [14:32:32] FIRING: Traffic bill over quota: Alert for device cr2-eqsin.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [14:33:01] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1309121 (https://phabricator.wikimedia.org/T429325) (owner: 10Btullis) [14:37:03] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 75%, RTA = 7573.78 ms [14:37:44] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcontrol2005-dev.codfw.wmnet [14:37:48] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcephmon2005-dev.codfw.wmnet [14:39:29] (03CR) 10Btullis: "Could you please add a fixture for this to the airflow chart, with a dummy secret?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309204 (https://phabricator.wikimedia.org/T427457) (owner: 10Aqu) [14:39:32] blake@cumin1003 renumber-node (PID 3397387) is awaiting input [14:40:59] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2005-dev.codfw.wmnet [14:41:03] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcephmon2006-dev.codfw.wmnet [14:42:07] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING WARNING - Packet loss = 0%, RTA = 761.34 ms [14:47:09] (03CR) 10Btullis: [C:03+1] "Yes, this looks good to me, too. Here is the same command on the current standby node." [puppet] - 10https://gerrit.wikimedia.org/r/1304766 (https://phabricator.wikimedia.org/T407138) (owner: 10Hnowlan) [14:47:14] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2006-dev.codfw.wmnet [14:47:18] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcephmon2007-dev.codfw.wmnet [14:48:23] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 50%, RTA = 2832.64 ms [14:48:31] blake@cumin1003 renumber-node (PID 3397387) is awaiting input [14:50:00] 06SRE, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06tools-infrastructure-team: Upgrade cloudsw1-e4-eqiad - https://phabricator.wikimedia.org/T429013#12109624 (10fgiunchedi) >>! In T429013#12109258, @cmooney wrote: > Thanks @fgiunchedi. > > This isn't so urgent we want to cause stress for you g... [14:50:41] (03PS1) 10Dreamy Jazz: ProductionServices: Drop NodeJS iPoid URL [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309684 (https://phabricator.wikimedia.org/T416623) [14:51:01] !log blake@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1054.eqiad.wmnet [14:51:01] !log blake@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1054.eqiad.wmnet [14:51:04] !log blake@cumin1003 END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1054.eqiad.wmnet [14:52:32] RESOLVED: Traffic bill over quota: Alert for device cr2-eqsin.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [14:53:16] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephmon2007-dev.codfw.wmnet [14:53:20] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcephosd2004-dev.codfw.wmnet [14:54:56] 06SRE, 06Infrastructure-Foundations, 10netops: Edge BGP: Change order we apply export policies - https://phabricator.wikimedia.org/T431849 (10cmooney) 03NEW p:05Triage→03Low [14:59:32] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2004-dev.codfw.wmnet [14:59:36] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcephosd2005-dev.codfw.wmnet [15:02:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:03:49] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 428.05 ms [15:06:48] (03PS1) 10Atsuko: cirrussearch: alert on tripped circuit breakers [alerts] - 10https://gerrit.wikimedia.org/r/1309685 (https://phabricator.wikimedia.org/T431827) [15:07:18] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2005-dev.codfw.wmnet [15:07:22] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcephosd2006-dev.codfw.wmnet [15:14:48] !log filippo@cumin1003 END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cloudcephosd2006-dev.codfw.wmnet [15:14:51] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcephosd2007-dev.codfw.wmnet [15:14:52] !log filippo@cumin1003 END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudcephosd2007-dev.codfw.wmnet [15:17:20] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:18:25] FIRING: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:19:53] !log filippo@cumin1003 START - Cookbook sre.hosts.reboot-single for host cloudcephosd2007-dev.codfw.wmnet [15:21:37] (03CR) 10DCausse: [C:03+1] cirrussearch: alert on tripped circuit breakers [alerts] - 10https://gerrit.wikimedia.org/r/1309685 (https://phabricator.wikimedia.org/T431827) (owner: 10Atsuko) [15:26:02] !log filippo@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2007-dev.codfw.wmnet [15:32:20] (03CR) 10Atsuko: [C:03+2] cirrussearch: alert on tripped circuit breakers [alerts] - 10https://gerrit.wikimedia.org/r/1309685 (https://phabricator.wikimedia.org/T431827) (owner: 10Atsuko) [15:35:05] (03Merged) 10jenkins-bot: cirrussearch: alert on tripped circuit breakers [alerts] - 10https://gerrit.wikimedia.org/r/1309685 (https://phabricator.wikimedia.org/T431827) (owner: 10Atsuko) [15:42:21] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-esams:et-0/0/0 (Transport: Arelion (IC-398709) {#20260602}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-esams:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:42:28] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 77%, RTA = 3708.67 ms [15:44:54] PROBLEM - Kafka Broker Server on kafka-test1009 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [15:45:48] FIRING: PuppetFailure: Puppet has failed on maps1014:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:49:28] (03PS1) 10Scott French: trafficserver::backend: Restore X-W-D for /w/rest.php [puppet] - 10https://gerrit.wikimedia.org/r/1309689 (https://phabricator.wikimedia.org/T428909) [15:50:14] (03CR) 10CI reject: [V:04-1] trafficserver::backend: Restore X-W-D for /w/rest.php [puppet] - 10https://gerrit.wikimedia.org/r/1309689 (https://phabricator.wikimedia.org/T428909) (owner: 10Scott French) [15:50:48] RESOLVED: PuppetFailure: Puppet has failed on maps1014:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:50:52] (03PS2) 10Scott French: trafficserver::backend: Restore X-W-D for /w/rest.php [puppet] - 10https://gerrit.wikimedia.org/r/1309689 (https://phabricator.wikimedia.org/T428909) [15:52:39] FIRING: CirrusSearchBreakerTripped: OpenSearch breaker parent has tripped on production-search-codfw 31.525423728813557 times in past hour - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://w.wiki/SEVN - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBreakerTripped [15:52:42] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 228.95 ms [15:53:12] (03CR) 10Scott French: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1309689 (https://phabricator.wikimedia.org/T428909) (owner: 10Scott French) [15:57:39] FIRING: [2x] CirrusSearchBreakerTripped: OpenSearch breaker parent has tripped on production-search-codfw 31.525423728813557 times in past hour - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://w.wiki/SEVN - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBreakerTripped [16:00:02] PROBLEM - Check if Pybal has been restarted after pybal.conf was changed on lvs2012 is CRITICAL: CRITICAL: Service pybal.service has not been restarted after /etc/pybal/pybal.conf was changed (gt 1h). https://wikitech.wikimedia.org/wiki/PyBal%23Pybal_service_has_not_been_restarted [16:00:07] (03CR) 10Scott French: "FYI, I'm just creating this now, so we have it ready if we go this route." [puppet] - 10https://gerrit.wikimedia.org/r/1309689 (https://phabricator.wikimedia.org/T428909) (owner: 10Scott French) [16:00:27] (03PS2) 10Btullis: admin_ng: configure topolvm storage classes and node placement [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309261 (https://phabricator.wikimedia.org/T428235) [16:00:28] (03PS13) 10Btullis: admin_ng: enable the topolvm CSI driver on dse-k8s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1305978 (https://phabricator.wikimedia.org/T429331) [16:02:36] !log releases1003/releases2003 (releases.wikimedia.org backends) - rebooting for maintenance [16:02:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:02:39] FIRING: [2x] CirrusSearchBreakerTripped: OpenSearch breaker parent has tripped on production-search-codfw 31.525423728813557 times in past hour - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://w.wiki/SEVN - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBreakerTripped [16:02:52] jouncebot: now [16:02:52] For the next 14 hour(s) and 57 minute(s): No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260710T0700) [16:03:42] (03CR) 10RLazarus: "Even weirder, then, because it not only works for me now, it worked for me as soon as I posted it -- I remember clicking the links to chec" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1308788 (https://phabricator.wikimedia.org/T427864) (owner: 10RLazarus) [16:04:14] (03CR) 10Bking: cirrussearch: alert on tripped circuit breakers (032 comments) [alerts] - 10https://gerrit.wikimedia.org/r/1309685 (https://phabricator.wikimedia.org/T431827) (owner: 10Atsuko) [16:07:28] (03CR) 10Atsuko: [C:03+2] "bking: thanks, will fix in I901564582a57a02a00a6f31dd28b2cb148c481a1" [alerts] - 10https://gerrit.wikimedia.org/r/1309685 (https://phabricator.wikimedia.org/T431827) (owner: 10Atsuko) [16:07:39] RESOLVED: CirrusSearchBreakerTripped: OpenSearch breaker parent has tripped on production-search-codfw 31.525423728813557 times in past hour - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://w.wiki/SEVN - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBreakerTripped [16:07:47] (03PS2) 10Dzahn: gerrit: add support for multiple ssh host keys [puppet] - 10https://gerrit.wikimedia.org/r/1309271 (https://phabricator.wikimedia.org/T240266) [16:08:01] (03CR) 10Dzahn: "thank you! did the 2 suggested fixes" [puppet] - 10https://gerrit.wikimedia.org/r/1309271 (https://phabricator.wikimedia.org/T240266) (owner: 10Dzahn) [16:08:12] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:09:02] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 50%, RTA = 3223.33 ms [16:13:36] !log doc1004/doc2003 (doc.wikimedia.org backends) - rebooting [16:13:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:14:04] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 240.79 ms [16:14:22] (03CR) 10Jasmine: [C:03+2] sre.k8s.renumber-node: Increase puppet timeout and attempts [cookbooks] - 10https://gerrit.wikimedia.org/r/1307872 (https://phabricator.wikimedia.org/T430226) (owner: 10Jasmine) [16:17:15] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [16:17:32] !log etherpad1004/etherpad2002 - (etherpad.wikimedia.org) - rebooting [16:17:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:18:12] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:18:25] RESOLVED: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:20:22] (03Merged) 10jenkins-bot: sre.k8s.renumber-node: Increase puppet timeout and attempts [cookbooks] - 10https://gerrit.wikimedia.org/r/1307872 (https://phabricator.wikimedia.org/T430226) (owner: 10Jasmine) [16:23:12] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:25:11] !log gitlab-runners (production) rebooting cluster one by one [16:25:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:25:22] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 50%, RTA = 2615.03 ms [16:27:00] !log dzahn@cumin2002 START - Cookbook sre.hosts.reboot-single for host gitlab-runner2002.codfw.wmnet [16:27:01] PROBLEM - SSH on gitlab-runner1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [16:27:15] RESOLVED: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [16:27:50] RECOVERY - SSH on gitlab-runner1002 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u10 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [16:28:12] RESOLVED: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:28:27] (03CR) 10RLazarus: [C:03+1] zookeeper: Support installation from component/zookeeper34 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1309257 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [16:28:55] (03CR) 10RLazarus: [C:03+1] hieradata: Use component/zookeeper34 on conf* hosts (codfw) [puppet] - 10https://gerrit.wikimedia.org/r/1309260 (https://phabricator.wikimedia.org/T428495) (owner: 10Scott French) [16:33:23] !log dzahn@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2002.codfw.wmnet [16:33:24] 06SRE, 10SRE-swift-storage, 06Commons, 10media-backups, 10MediaWiki-File-management: Uncompressed TIFFs on commons - https://phabricator.wikimedia.org/T427949#12109972 (10Ladsgroup) Update on the compression script. I have switched it to start from compressing largest files first but it's failing basical... [16:33:36] (03PS2) 10CDobbins: varnish: update CSP report-only header [puppet] - 10https://gerrit.wikimedia.org/r/1308772 [16:35:16] !log dzahn@cumin2002 START - Cookbook sre.hosts.reboot-single for host gitlab-runner2003.codfw.wmnet [16:36:19] (03CR) 10CDobbins: varnish: update CSP report-only header (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1308772 (owner: 10CDobbins) [16:40:38] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 238.26 ms [16:41:24] !log dzahn@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2003.codfw.wmnet [16:41:27] (03PS1) 10Kevin Bazira: ml-services: deploy latest TTS isvc model-server [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309695 (https://phabricator.wikimedia.org/T430536) [16:42:22] !log dzahn@cumin2002 START - Cookbook sre.hosts.reboot-single for host gitlab-runner2004.codfw.wmnet [16:45:00] (03CR) 10Kevin Bazira: [C:03+2] ml-services: deploy latest TTS isvc model-server [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309695 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [16:47:21] (03Merged) 10jenkins-bot: ml-services: deploy latest TTS isvc model-server [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309695 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [16:48:31] !log dzahn@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner2004.codfw.wmnet [16:48:49] !log kevinbazira@deploy2003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [16:49:50] 06SRE, 06cloud-services-team, 06collaboration-services, 10Infrastructure Security, and 6 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12110010 (10colewhite) [16:51:03] !log dzahn@cumin2002 START - Cookbook sre.hosts.reboot-single for host gitlab-runner1003.eqiad.wmnet [16:51:45] FIRING: WidespreadPuppetFailure: Puppet has failed in eqiad - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [16:54:20] (03PS5) 10CDobbins: varnish: put testwiki back into enforce mode [puppet] - 10https://gerrit.wikimedia.org/r/1309263 [16:56:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [16:57:30] (03CR) 10Ottomata: streams: webrequest - pageview - trending (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1308656 (https://phabricator.wikimedia.org/T430675) (owner: 10JavierMonton) [16:57:35] !log dzahn@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner1003.eqiad.wmnet [16:58:05] (03CR) 10Ottomata: [C:03+1] streams: webrequest - pageview - trending [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1308656 (https://phabricator.wikimedia.org/T430675) (owner: 10JavierMonton) [16:59:32] (03CR) 10Ottomata: [C:03+1] streams: webrequest - pageview - trending (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1308656 (https://phabricator.wikimedia.org/T430675) (owner: 10JavierMonton) [17:03:00] !log dzahn@cumin2002 START - Cookbook sre.hosts.reboot-single for host gitlab-runner1004.eqiad.wmnet [17:05:24] (03PS1) 10JHathaway: pupetserver1002: depool [dns] - 10https://gerrit.wikimedia.org/r/1309699 (https://phabricator.wikimedia.org/T423282) [17:06:38] !log depooling puppetserver1002, cause of errors is still unknown [17:06:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:06:52] (03CR) 10JHathaway: [C:03+2] pupetserver1002: depool [dns] - 10https://gerrit.wikimedia.org/r/1309699 (https://phabricator.wikimedia.org/T423282) (owner: 10JHathaway) [17:07:03] !log jhathaway@dns1004 START - running authdns-update [17:08:49] !log jhathaway@dns1004 END - running authdns-update [17:09:36] !log dzahn@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab-runner1004.eqiad.wmnet [17:11:03] (03PS6) 10CDobbins: varnish: put testwiki back into enforce mode [puppet] - 10https://gerrit.wikimedia.org/r/1309263 [17:11:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [17:12:12] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [17:13:48] (03PS1) 10Btullis: Add a component/topolvm to bookworm-wikimedia in reprepro [puppet] - 10https://gerrit.wikimedia.org/r/1309701 (https://phabricator.wikimedia.org/T428895) [17:15:10] FIRING: BFDdown: BFD session down between cr1-codfw and 10.192.16.35 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:15:26] !log [doc1004:~] $ sudo systemctl start rsync-doc-host-data-sync (T431856) [17:15:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:15:29] T431856: SystemdUnitFailed - rsync on doc1004 - https://phabricator.wikimedia.org/T431856 [17:15:51] FIRING: [2x] ProbeDown: Service centrallog2002:6514 has failed probes (tcp_rsyslog_receiver_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#centrallog2002:6514 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:16:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [17:16:52] (03PS7) 10CDobbins: varnish: put testwiki back into enforce mode [puppet] - 10https://gerrit.wikimedia.org/r/1309263 [17:17:14] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 224.13 ms [17:20:10] RESOLVED: [2x] BFDdown: BFD session down between cr1-codfw and 10.192.16.35 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:20:23] (03CR) 10CDobbins: varnish: put testwiki back into enforce mode (035 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1309263 (owner: 10CDobbins) [17:20:50] RESOLVED: [2x] ProbeDown: Service centrallog2002:6514 has failed probes (tcp_rsyslog_receiver_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#centrallog2002:6514 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:21:45] RESOLVED: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [17:25:25] FIRING: [4x] BFDdown: BFD session down between cr1-codfw and 10.192.16.35 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:30:25] RESOLVED: [4x] BFDdown: BFD session down between cr1-codfw and 10.192.16.35 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:37:03] FIRING: [2x] ProbeDown: Service titan2002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:42:03] RESOLVED: [4x] ProbeDown: Service centrallog1002:6514 has failed probes (tcp_rsyslog_receiver_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:44:03] FIRING: [2x] ProbeDown: Service titan2002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:48:12] FIRING: [3x] JobUnavailable: Reduced availability for job thanos-query-frontend in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:48:36] 10SRE-swift-storage, 06MediaWiki-Media-Platform-Team, 10MediaWiki-Uploading, 07Wikimedia-production-error: MediaWiki\Upload\Exception\UploadChunkFileException: Error storing file in '{chunkPath}': backend-fail-internal; local-swift-codfw - https://phabricator.wikimedia.org/T430986#12110247 (10Umherirrender) [17:49:03] FIRING: [4x] ProbeDown: Service titan2001:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:53:12] RESOLVED: [4x] JobUnavailable: Reduced availability for job thanos-query-frontend in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:54:03] RESOLVED: [4x] ProbeDown: Service titan2001:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:02:04] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - thanos-web_443: Servers titan1001.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [18:02:19] FIRING: HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [18:02:21] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5) {#122601}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [18:03:04] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:06:52] (03PS1) 10Cathal Mooney: Adjust order BGP_outfilter is applied to routes announced [homer/public] - 10https://gerrit.wikimedia.org/r/1309707 (https://phabricator.wikimedia.org/T431849) [18:07:33] FIRING: KafkaUnderReplicatedPartitions: Under replicated partitions for Kafka cluster test-eqiad in eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration - https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-kafka_cluster=test-eqiad - https://alerts.wikimedia.org/?q=alertname%3DKafkaUnderReplicatedPartitions [18:09:49] (03PS2) 10Cathal Mooney: Adjust order BGP_outfilter is applied to routes announced [homer/public] - 10https://gerrit.wikimedia.org/r/1309707 (https://phabricator.wikimedia.org/T431849) [18:15:59] !log swfrench@cumin2002 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-high-traffic2-codfw [18:16:22] RECOVERY - Check if Pybal has been restarted after pybal.conf was changed on lvs2012 is OK: OK: pybal.service was restarted after /etc/pybal/pybal.conf was changed. https://wikitech.wikimedia.org/wiki/PyBal%23Pybal_service_has_not_been_restarted [18:16:24] RECOVERY - PyBal connections to etcd on lvs2012 is OK: OK: 6 connections established with conf2004.codfw.wmnet:4001 (min=6) https://wikitech.wikimedia.org/wiki/PyBal [18:16:33] \i/ [18:16:35] !log swfrench@cumin2002 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-high-traffic2-codfw [18:16:42] yay! [18:20:34] PROBLEM - Host lists2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:21:18] !log lists2001, phab2003 - rebooting (not the active hosts) [18:21:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:22:01] RECOVERY - Host lists2001 is UP: PING OK - Packet loss = 0%, RTA = 30.47 ms [18:22:44] PROBLEM - Host phab2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:23:05] !log vrts2002 rebooting (not the active host) [18:23:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:23:38] RECOVERY - Host phab2003 is UP: PING OK - Packet loss = 0%, RTA = 31.72 ms [18:24:13] FIRING: [2x] ProbeDown: Service phab2003:25 has failed probes (tcp_phabricator_smtp_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#phab2003:25 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:24:34] PROBLEM - Host vrts2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:25:45] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [18:26:02] RECOVERY - Host vrts2002 is UP: PING OK - Packet loss = 0%, RTA = 32.09 ms [18:26:27] swfrench-wmf: Not related, I hope? [18:26:50] brett: these are presumably mutante's reboots? [18:27:06] oh, derp, my eyeballs helpfully ignored those lines [18:27:18] well, the Cirrus thing is not related to me [18:27:30] brett: better safe than sorry! thanks for noticing :) [18:27:36] mutante: ack, thanks [18:28:01] PROBLEM - freshclam running on vrts2002 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 110 (clamav), command name freshclam https://wikitech.wikimedia.org/wiki/VRT_System%23ClamAV [18:28:59] not sure what the deal is with CirrusStreamingUpdaterUnknownErrors - that was happening intermittently yesterday as well, IIRC [18:29:01] RECOVERY - freshclam running on vrts2002 is OK: PROCS OK: 1 process with UID = 110 (clamav), command name freshclam https://wikitech.wikimedia.org/wiki/VRT_System%23ClamAV [18:29:13] RESOLVED: [4x] ProbeDown: Service phab2003:25 has failed probes (tcp_phabricator_smtp_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:30:38] I am stopping now and all that was related to me has recovered [18:35:45] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [18:42:38] 06SRE, 10LDAP-Access-Requests: Grant Access to for - https://phabricator.wikimedia.org/T431871 (10Europepetrol) 03NEW [18:48:39] 06SRE, 10LDAP-Access-Requests: Grant Access to for  - https://phabricator.wikimedia.org/T431871#12110408 (10Dzahn) 05Open→03Invalid [18:49:34] 06SRE, 10LDAP-Access-Requests: Grant Access to for  - https://phabricator.wikimedia.org/T431871#12110425 (10Dzahn) This ticket seemed invalid; there was no actual user name or group requested in it. If this was a real request please add those details and reopen it. [18:49:39] FIRING: CirrusSearchBreakerTripped: OpenSearch breaker parent has tripped on production-search-eqiad 100 times in past 3 hours - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://w.wiki/SEVN - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBreakerTripped [18:59:03] swfrench-wmf we just updated OpenSearch major versions and there are some hiccups with the streaming updater. Search platform is aware of the issue, I suppressed that alert until Monday [18:59:54] inflatador: ah, thank you! [19:01:17] np, I also suppressed those circuit breaker alerts ^^ [19:05:24] (03PS1) 10Scott French: linked-artifacts: Use /healthz handler in blackbox probes [puppet] - 10https://gerrit.wikimedia.org/r/1309712 (https://phabricator.wikimedia.org/T431108) [19:06:28] (03PS1) 10JHathaway: Revert "pupetserver1002: depool" [dns] - 10https://gerrit.wikimedia.org/r/1309713 (https://phabricator.wikimedia.org/T423282) [19:07:51] (03PS2) 10JHathaway: Revert "pupetserver1002: depool" [dns] - 10https://gerrit.wikimedia.org/r/1309713 (https://phabricator.wikimedia.org/T423282) [19:08:04] (03CR) 10Scott French: "Thanks for the review!" [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309303 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [19:08:07] (03PS3) 10JHathaway: Revert "pupetserver1002: depool" [dns] - 10https://gerrit.wikimedia.org/r/1309713 (https://phabricator.wikimedia.org/T423282) [19:08:10] (03CR) 10Scott French: [C:03+2] Consolidate and modernize test dependencies and entrypoint [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309303 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [19:09:48] (03Merged) 10jenkins-bot: Consolidate and modernize test dependencies and entrypoint [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309303 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [19:10:30] (03CR) 10JHathaway: [C:03+2] Revert "pupetserver1002: depool" [dns] - 10https://gerrit.wikimedia.org/r/1309713 (https://phabricator.wikimedia.org/T423282) (owner: 10JHathaway) [19:10:40] !log jhathaway@dns1004 START - running authdns-update [19:12:30] !log jhathaway@dns1004 END - running authdns-update [19:14:47] (03CR) 10Bking: admin_ng: configure topolvm storage classes and node placement (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309261 (https://phabricator.wikimedia.org/T428235) (owner: 10Btullis) [19:17:23] (03CR) 10RLazarus: [C:03+1] linked-artifacts: Use /healthz handler in blackbox probes [puppet] - 10https://gerrit.wikimedia.org/r/1309712 (https://phabricator.wikimedia.org/T431108) (owner: 10Scott French) [19:19:36] (03CR) 10Bking: [C:03+1] Add a component/topolvm to bookworm-wikimedia in reprepro [puppet] - 10https://gerrit.wikimedia.org/r/1309701 (https://phabricator.wikimedia.org/T428895) (owner: 10Btullis) [19:21:00] (03CR) 10Bking: [C:03+1] "I realized the previous comment as bikesheddish, just wanted to make it clear I don't think the names should hold up this CR." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309261 (https://phabricator.wikimedia.org/T428235) (owner: 10Btullis) [19:21:05] (03CR) 10Bking: [C:03+1] admin_ng: configure topolvm storage classes and node placement (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309261 (https://phabricator.wikimedia.org/T428235) (owner: 10Btullis) [19:53:48] (03PS1) 10Scott French: Bump etcdmirror package version: 0.0.12 [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309718 (https://phabricator.wikimedia.org/T418915) [19:53:50] (03PS1) 10Scott French: Release etcd-mirror 0.0.12 [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309719 (https://phabricator.wikimedia.org/T418915) [20:04:16] (03CR) 10Scott French: "Thanks in advance for the review!" [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309718 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [20:04:18] (03CR) 10Scott French: "Note: the non-voting test should pass once I merge and tag the previous patch in the servies." [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309719 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [20:09:46] (03CR) 10RLazarus: [C:03+1] Bump etcdmirror package version: 0.0.12 [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309718 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [20:10:09] (03CR) 10RLazarus: [C:03+1] "Agreed! LGTM once that's done." [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309719 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [20:11:37] (03CR) 10Scott French: [C:03+2] Bump etcdmirror package version: 0.0.12 [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309718 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [20:12:46] (03Merged) 10jenkins-bot: Bump etcdmirror package version: 0.0.12 [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309718 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [20:16:12] (03CR) 10Scott French: "recheck" [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309719 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [20:24:59] (03PS1) 10Esanders: Set wgMFFallbackEditor to 'visual' on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309727 (https://phabricator.wikimedia.org/T431858) [20:25:40] (03CR) 10Scott French: "Well, it _would_ have passed, if (1) webproxy were still reachable from the CI container and (2) the presumably very old cowbuilder image " [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309719 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [20:30:52] (03CR) 10Scott French: [C:03+2] Release etcd-mirror 0.0.12 [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309719 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [20:32:07] (03Merged) 10jenkins-bot: Release etcd-mirror 0.0.12 [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1309719 (https://phabricator.wikimedia.org/T418915) (owner: 10Scott French) [20:33:58] FIRING: [2x] ProbeDown: Service prometheus2008:443 has failed probes (http_prometheus_codfw_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#prometheus2008:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:37:28] 06SRE, 06Infrastructure-Foundations, 10Puppet-Infrastructure: Timeouts on puppetserver1002 past reboot - https://phabricator.wikimedia.org/T423282#12110761 (10jhathaway) 05Resolved→03Open This issue arose again at 2026-07-10 03:00 UTC, https://grafana.wikimedia.org/goto/bfrpyuspzo3r4a?orgId=default I pu... [20:37:48] (03Abandoned) 10Esanders: MobileFrontend: remove override for default mobile editor [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1119197 (https://phabricator.wikimedia.org/T361134) (owner: 10DLynch) [20:38:06] (03CR) 10EggRoll97: [C:03+1] Enable the abuse filter block action on Hindi Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309675 (https://phabricator.wikimedia.org/T431830) (owner: 10Dreamrimmer) [20:38:41] (03CR) 10DLynch: "🥲" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1119197 (https://phabricator.wikimedia.org/T361134) (owner: 10DLynch) [20:38:58] RESOLVED: [2x] ProbeDown: Service prometheus2008:443 has failed probes (http_prometheus_codfw_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#prometheus2008:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:00:04] (03CR) 10Ebernhardson: [C:03+1] cirrus: change the restart strategy to exponential-delay [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309210 (https://phabricator.wikimedia.org/T428863) (owner: 10DCausse) [21:03:58] FIRING: [2x] ProbeDown: Service prometheus2006:443 has failed probes (http_prometheus_codfw_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#prometheus2006:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:13:58] RESOLVED: [2x] ProbeDown: Service prometheus2006:443 has failed probes (http_prometheus_codfw_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#prometheus2006:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:18:01] FIRING: [2x] ProbeDown: Service prometheus2005:443 has failed probes (http_prometheus_codfw_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#prometheus2005:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:27:58] RESOLVED: [2x] ProbeDown: Service prometheus2005:443 has failed probes (http_prometheus_codfw_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#prometheus2005:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:47:21] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5) {#122601}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [21:48:33] FIRING: [4x] ProbeDown: Service prometheus1007:443 has failed probes (http_prometheus_eqiad_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:53:33] RESOLVED: [4x] ProbeDown: Service prometheus1007:443 has failed probes (http_prometheus_eqiad_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:57:03] FIRING: [2x] ProbeDown: Service prometheus1006:443 has failed probes (http_prometheus_eqiad_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#prometheus1006:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:02:03] RESOLVED: [4x] ProbeDown: Service prometheus1006:443 has failed probes (http_prometheus_eqiad_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:02:33] RESOLVED: KafkaUnderReplicatedPartitions: Under replicated partitions for Kafka cluster test-eqiad in eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration - https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-kafka_cluster=test-eqiad - https://alerts.wikimedia.org/?q=alertname%3DKafkaUnderReplicatedPartitions [22:04:52] FIRING: HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [22:07:03] FIRING: KafkaUnderReplicatedPartitions: Under replicated partitions for Kafka cluster test-eqiad in eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration - https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-kafka_cluster=test-eqiad - https://alerts.wikimedia.org/?q=alertname%3DKafkaUnderReplicatedPartitions [22:10:42] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:37:54] RECOVERY - Kafka Broker Server on kafka-test1009 is OK: PROCS OK: 1 process with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [22:40:54] PROBLEM - Kafka Broker Server on kafka-test1009 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [22:57:36] PROBLEM - SSH on doh5004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [22:57:36] PROBLEM - Wikidough DoH Check -IPv4- on doh5004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring%23Wikidough_Basic_Check [22:57:36] PROBLEM - Wikidough DoT Check -IPv6- on doh5004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring%23Wikidough_Basic_Check [22:57:36] PROBLEM - Wikidough DoH Check -IPv6- on doh5004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring%23Wikidough_Basic_Check [22:57:38] PROBLEM - Wikidough DoT Check -IPv4- on doh5004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring%23Wikidough_Basic_Check [23:00:42] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:01:26] RECOVERY - Wikidough DoT Check -IPv6- on doh5004 is OK: TCP OK - 0.462 second response time on 2001:df2:e500:3:103:102:166:99 port 853 https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring%23Wikidough_Basic_Check [23:01:28] RECOVERY - Wikidough DoH Check -IPv6- on doh5004 is OK: HTTP OK: HTTP/1.1 200 OK - 595 bytes in 0.972 second response time https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring%23Wikidough_Basic_Check [23:01:28] RECOVERY - SSH on doh5004 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u9 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [23:01:28] RECOVERY - Wikidough DoT Check -IPv4- on doh5004 is OK: TCP OK - 0.535 second response time on 103.102.166.99 port 853 https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring%23Wikidough_Basic_Check [23:01:30] RECOVERY - Wikidough DoH Check -IPv4- on doh5004 is OK: HTTP OK: HTTP/1.1 200 OK - 595 bytes in 2.548 second response time https://wikitech.wikimedia.org/wiki/Wikidough/Monitoring%23Wikidough_Basic_Check [23:05:42] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:08:54] RECOVERY - Kafka Broker Server on kafka-test1007 is OK: PROCS OK: 1 process with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [23:10:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:11:54] PROBLEM - Kafka Broker Server on kafka-test1007 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [23:25:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:25:42] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:30:42] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:32:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:40:42] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:42:58] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1309752 [23:42:58] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1309752 (owner: 10TrainBranchBot) [23:47:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:50:51] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1309752 (owner: 10TrainBranchBot)