[02:46:25] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:11:05] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:11:04] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [06:46:40] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:57:51] elukey: what's the process for rolling out spicerack changes these days? would like to get https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1167299 released [07:19:10] ryankemper: I'll take care of the release! ETA probably early next week :) [07:25:19] we also need to drop elasticsearch from setup.py, don't we? [08:46:25] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:17:34] 10homer, 10SRE-tools, 06Infrastructure-Foundations: Homer: add parallelization support - https://phabricator.wikimedia.org/T250415#11263494 (10Volans) [note for future self] If we can wait for Python 3.14 to be around in our systems then we should evaluate also the new [[ https://docs.python.org/3/library/co... [10:51:38] 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 06Product Safety and Integrity, and 2 others: Provide a sender email address alias for use in Special:Emailuser (aka 2-way email relay) - https://phabricator.wikimedia.org/T150421#11263939 (10OKryva-WMF) [12:52:48] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cumin1003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [12:57:24] moritzm: https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1195208 [12:57:48] RESOLVED: PuppetConstantChange: Puppet performing a change on every puppet run on cumin1003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [13:08:52] +1d [13:10:47] and I think we also need https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1195211 [13:24:50] indeed, also +1d [14:13:48] FIRING: PuppetFailure: Puppet has failed on cumin1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:53:48] RESOLVED: PuppetFailure: Puppet has failed on cumin1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:16:35] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Decom Hurricane Electric Transit/Peering circuit eqiad - https://phabricator.wikimedia.org/T407008 (10cmooney) 03NEW p:05Triage→03Medium [18:15:08] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Decom Hurricane Electric Transit/Peering circuit eqiad - https://phabricator.wikimedia.org/T407008#11265323 (10RobH) [18:16:11] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Decom Hurricane Electric Transit/Peering circuit eqiad - https://phabricator.wikimedia.org/T407008#11265326 (10RobH) I've pulled my old records off the google sheet and updated this task and netbox with the patch panel landing info and... [18:19:20] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Decom Hurricane Electric Transit/Peering circuit eqiad - https://phabricator.wikimedia.org/T407008#11265342 (10RobH) @VRiley-WMF, I don't want to accidentally disconnect the wrong thing, and as my old (and possibly outdated) google sh... [18:22:00] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Decom Hurricane Electric Transit/Peering circuit eqiad - https://phabricator.wikimedia.org/T407008#11265352 (10VRiley-WMF) Sure, I will look into that [18:43:47] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Decom Hurricane Electric Transit/Peering circuit eqiad - https://phabricator.wikimedia.org/T407008#11265424 (10VRiley-WMF) I have confirmed that patch ID 3909 lands on PP:0000:103234:P1/2 and then onto cr1-eqiad:xe-3/1/5. Also, it seem... [20:50:27] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Decom Hurricane Electric Transit/Peering circuit eqiad - https://phabricator.wikimedia.org/T407008#11265826 (10RobH) [20:51:14] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Decom Hurricane Electric Transit/Peering circuit eqiad - https://phabricator.wikimedia.org/T407008#11265827 (10RobH) Deinstall order 1-252958908448 submitted, feel free to unplug the patch cable 3909 from both ends. Once they confirm... [20:51:54] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Decom Hurricane Electric Transit/Peering circuit eqiad - https://phabricator.wikimedia.org/T407008#11265831 (10RobH) [20:54:26] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Decom Hurricane Electric Transit/Peering circuit eqiad - https://phabricator.wikimedia.org/T407008#11265835 (10RobH) @cmooney: Shouldn't the circuit be 'decommissioning' status in netbox at this point? https://netbox.wikimedia.org/cir...