[03:11:05] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:11:04] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [06:53:05] 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11253359 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host idp-test2005.wikimedia.org with OS trixie [06:55:16] 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11253364 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host idp-test2005.wikimedia.org with OS trixie executed with errors: - idp-test2005 (**... [07:22:57] 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11253401 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host idp-test2005.wikimedia.org with OS trixie [08:03:50] 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11253529 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host idp-test2005.wikimedia.org with OS trixie executed with errors: - idp-test2005 (**... [09:36:56] 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11253832 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host idp-test2005.wikimedia.org with OS trixie [09:51:56] 10netops, 06Infrastructure-Foundations, 06SRE: cr1-esams: MPC7E 3D 40XGE line card in slot 0 failure [Oct 2025] - https://phabricator.wikimedia.org/T406705 (10cmooney) 03NEW p:05Triage→03High [10:14:46] 10netops, 06Infrastructure-Foundations, 06SRE: cr1-esams: MPC7E 3D 40XGE line card in slot 0 failure [Oct 2025] - https://phabricator.wikimedia.org/T406705#11253997 (10cmooney) JTAC Case 2025-1008-891506 raised. [10:17:46] 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11254009 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host idp-test2005.wikimedia.org with OS trixie executed with errors: - idp-test2005 (**... [10:37:33] 10netops, 06Infrastructure-Foundations, 06SRE: cr1-esams: MPC7E 3D 40XGE line card in slot 0 failure [Oct 2025] - https://phabricator.wikimedia.org/T406705#11254135 (10cmooney) Typical lackluster from Juniper. After finally looking at the logs they requested we re-seat the card, so I will work to create a r... [10:53:27] 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11254232 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host idp-test2005.wikimedia.org with OS bookworm [11:34:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr2-eqiad: fan failure on left tray [Oct 2025] - https://phabricator.wikimedia.org/T406554#11254376 (10Jclark-ctr) Replaced fan modular with spare from storage room. Pending tac ticket with juniper [11:34:06] 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11254377 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host idp-test2005.wikimedia.org with OS bookworm executed with errors: - idp-test2005 (... [11:35:26] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr2-eqiad: fan failure on left tray [Oct 2025] - https://phabricator.wikimedia.org/T406554#11254391 (10Jclark-ctr) Verified fan speeds compared with cr1 ` jclark@re0.cr1-eqiad> show chassis fan Item Status... [11:52:08] 10netops, 06Infrastructure-Foundations, 06SRE: cr1-esams: MPC7E 3D 40XGE line card in slot 0 failure [Oct 2025] - https://phabricator.wikimedia.org/T406705#11254678 (10cmooney) Remote hands request CS3302125 has been raised to the Digital Realty staff on site in AMS9 Science Park. [12:05:30] 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11254751 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by slyngshede@cumin1003 for hosts: `idp-test2005.wikimedia.org` - idp-test2005.wikimedia.org (**WARN**) - //Host not... [12:10:30] 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11254772 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host idp-test2005.wikimedia.org with OS trixie [12:32:49] FYI, I'll be reimaging sretest1003 for some tests [12:49:19] 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11254960 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host idp-test2005.wikimedia.org with OS trixie completed: - idp-t... [13:31:41] 10netops, 06Infrastructure-Foundations, 06SRE: cr1-esams: MPC7E 3D 40XGE line card in slot 0 failure [Oct 2025] - https://phabricator.wikimedia.org/T406705#11255104 (10cmooney) Looks like the card re-seat did the trick: ` cmooney@re0.cr1-esams> show chassis fpc 0 detail Slot 0 information: State... [14:04:51] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: Move lvs1020 link from ssw1-f1-eqiad to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T404959#11255300 (10VRiley-WMF) Okay, was looking at this issue a bit. There are currently two fiber cables involved with this process. Afte... [14:20:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr2-eqiad: fan failure on left tray [Oct 2025] - https://phabricator.wikimedia.org/T406554#11255372 (10cmooney) 05Open→03Resolved Thanks @Jclark-ctr. As you say it seems the one that has gone in is the same model as came out.... [14:23:48] 10netops, 06Infrastructure-Foundations, 06SRE: cr1-esams: MPC7E 3D 40XGE line card in slot 0 failure [Oct 2025] - https://phabricator.wikimedia.org/T406705#11255395 (10cmooney) 05Open→03Resolved esams has been re-pooled and traffic levels have returned to normal for the site. closing this task now,... [17:45:45] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11256390 (10RobH) [17:46:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11256391 (10RobH)