[06:41:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on doh7003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:47:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cp1110:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:21:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on doh7003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:32:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cp1110:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:42:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy3002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:47:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy3002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:57:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cp7007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:09:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cp7016:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:17:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cp7007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:19:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cp7016:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:30:12] 06Traffic, 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27): Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12108249 (10BWojtowicz-WMF) The model is now deployed in the `llm` namespace and is exposed via REST Gateway with LiftWingLLM rate limits. Additionally, I've a... [09:32:17] 06Traffic, 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27): Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12108253 (10BWojtowicz-WMF) 05Open→03Resolved [09:46:37] 10netops, 06Infrastructure-Foundations, 06SRE: Don't announce OSPF routes in unicast BGP on Nokia SR-Linux - https://phabricator.wikimedia.org/T423430#12108326 (10cmooney) I was able to test this in containerlab, which then gave me the confidence to try in prod and it's simple enough. Resulting policy is si... [10:08:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cp7006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:23:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cp7006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [11:22:48] FIRING: PuppetFailure: Puppet has failed on durum3005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:42:48] RESOLVED: PuppetFailure: Puppet has failed on durum3005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:46:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cp7015:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [12:49:48] FIRING: PuppetFailure: Puppet has failed on cp6005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:09:48] RESOLVED: PuppetFailure: Puppet has failed on cp6005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:26:48] FIRING: PuppetFailure: Puppet has failed on cp7010:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:31:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cp7015:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:09:48] FIRING: PuppetFailure: Puppet has failed on cp6009:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:14:48] RESOLVED: PuppetFailure: Puppet has failed on cp6009:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:16:48] RESOLVED: PuppetFailure: Puppet has failed on cp7010:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:17:54] Do we have a policy on ATS mapping deploys on a friday? [14:25:11] (I'll abstain by default) [14:48:17] I don't know about anything specific to ATS, but IME deployments generally aren't done on Fridays [14:50:01] 10netops, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE, 06tools-infrastructure-team: Upgrade cloudsw1-e4-eqiad - https://phabricator.wikimedia.org/T429013#12109624 (10fgiunchedi) >>! In T429013#12109258, @cmooney wrote: > Thanks @fgiunchedi. > > This isn't so urgent we want to cause stress for you g... [14:52:48] FIRING: PuppetFailure: Puppet has failed on cp1101:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:54:56] 10netops, 06Infrastructure-Foundations, 06SRE: Edge BGP: Change order we apply export policies - https://phabricator.wikimedia.org/T431849 (10cmooney) 03NEW p:05Triage→03Low [15:04:48] FIRING: PuppetFailure: Puppet has failed on cp1110:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:08:23] 06Traffic, 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27): Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12109745 (10achou) [15:13:05] FIRING: PuppetFailure: Puppet has failed on ncredir3005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:13:09] RESOLVED: PuppetFailure: Puppet has failed on cp1101:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:29:48] RESOLVED: PuppetFailure: Puppet has failed on cp1110:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:29:49] RESOLVED: PuppetFailure: Puppet has failed on cp1110:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:37:53] whoa [15:42:34] * cjd91 looks at puppetboard [15:42:40] that's...a lot of red [15:44:15] seems -foundations has been notified already [15:46:48] RESOLVED: PuppetFailure: Puppet has failed on ncredir3005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:37:59] hello traffic friends o/ it looks like something weird happened on Wednesday and PyBal on lvs2012 get restarted after puppet-agent applied [0] (so "Check if Pybal has been restarted after pybal.conf was changed" and "PyBal connections to etcd" are firing). [16:37:59] (note that it really is connected to etcd - just not the node the icinga check expects) [16:37:59] [0] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308653 [16:38:33] I can restart it now if that's something you're comfortable with happening on a Friday, but I want to defer to you all on that. [16:44:08] lol ... that should read "lvs2012 *didn't* get restarted" [16:49:50] 06Traffic, 06cloud-services-team, 06collaboration-services, 10Infrastructure Security, and 6 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12110010 (10colewhite) [17:14:50] ^ cc: brett or cjd91 re: lvs2012 weirdness above [18:01:26] swfrench-wmf: I say go ahead and do that. It's not ideal but it's better to not leave it inconsistent over the weekend. thanks for bringing it up [18:04:09] brett: sounds good. I'll do that shortly w/ sre.loadbalancer.restart-pybal