[00:01:24] !log pt1979@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [00:01:47] !log pt1979@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" [00:01:49] !log pt1979@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl2006.codfw.wmnet with OS trixie [00:02:01] 06SRE, 06ServiceOps new, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12027390 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host wikikube-ctrl2006.codf... [00:13:23] RESOLVED: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [00:14:23] FIRING: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [00:30:35] (03PS2) 10RLazarus: tox: Bump flake8 to 7.3.0 [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1302989 [00:30:35] (03PS2) 10RLazarus: tox: Test up to Python 3.14 [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1302990 [00:30:35] (03PS2) 10RLazarus: Release 4.0.5 [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1302991 [00:39:23] RESOLVED: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [00:40:23] FIRING: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [00:53:54] (03PS1) 10Jasmine: Add Kubernetes POD IP reverse range delegations for wikikube-ctrl1005 [dns] - 10https://gerrit.wikimedia.org/r/1302996 (https://phabricator.wikimedia.org/T418920) [01:00:23] RESOLVED: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [01:12:15] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1302997 [01:12:15] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1302997 (owner: 10TrainBranchBot) [01:17:41] (03CR) 10ArielGlenn: rest-gateway: emit 401 if rate limit is 0 (033 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1298031 (https://phabricator.wikimedia.org/T428184) (owner: 10Daniel Kinzler) [01:19:23] FIRING: SLOBudgetBurn: Standalone event system success rate is below 99.9% target - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [01:20:36] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1302997 (owner: 10TrainBranchBot) [01:30:06] (03PS1) 10DDesouza: Add English Wikipedia Mobile App Survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1302998 (https://phabricator.wikimedia.org/T428876) [01:31:48] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, June 17 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1302998 (https://phabricator.wikimedia.org/T428876) (owner: 10DDesouza) [01:47:35] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:52:35] FIRING: [2x] SystemdUnitFailed: database-backups-snapshots.service on cumin2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed