[03:11:05] <jinxer-wm>	 FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts
[04:11:05] <jinxer-wm>	 RESOLVED: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts
[06:24:22] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 06SRE, 10Wikimedia-Mailing-lists: Replace Exim on lists.wikimedia.org with Postfix - https://phabricator.wikimedia.org/T378021#11248577 (10ABran-WMF)
[07:09:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 82.6% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[07:14:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 84.69% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[07:26:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 86.06% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[07:31:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 83.82% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[09:07:42] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr2-eqiad: fan failure on left tray [Oct 2025] - https://phabricator.wikimedia.org/T406554 (10cmooney) 03NEW p:05Triage→03High
[09:08:19] <wikibugs>	 07Puppet, 06Data-Engineering, 06Data-Engineering-Icebox, 10observability: Upgrade prometheus-jmx-exporter on all services using it - https://phabricator.wikimedia.org/T192948#11249022 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff We have 0.15.0 running fleet-wide, resolving this t...
[09:14:37] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: Remove lvs1018 L2 link to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T405499#11249042 (10cmooney) @BCornwall I'm hoping to make progress on this one, can you review the gerrit patch when you have a moment?  In terms of how...
[10:34:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 84.23% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[10:49:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 82.63% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[11:23:27] <wikibugs>	 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11249553 (10ops-monitoring-bot) Host idp-test2005.wikimedia.org rebooted by slyngshede@cumin1003 with reason: memory upgrade
[11:28:14] <wikibugs>	 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11249582 (10ops-monitoring-bot) VM idp-test2005.wikimedia.org rebooted by slyngshede@cumin1003 with reason: memory upgrade
[11:33:48] <wikibugs>	 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11249631 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host idp-test2005.wikimedia.org with OS trixie
[11:40:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 81.02% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[11:45:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 81.07% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[12:15:39] <wikibugs>	 10CAS-SSO, 06Infrastructure-Foundations: Upgrade Apereo CAS to version 7.2 - https://phabricator.wikimedia.org/T406455#11249763 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host idp-test2005.wikimedia.org with OS trixie executed with errors: - idp-test2005 (**...
[13:47:35] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: lvs1020: reimage to move primary IP from private1-d-eqiad to private1-d7-eqiad vlan - https://phabricator.wikimedia.org/T405630#11250100 (10cmooney) I'm actually not sure if this is going to be a possibility.  Unfortunately the Nokia SR-Linux platfo...
[13:48:19] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1019: reimage to move primary IP from private1-c-eqiad to private1-c7-eqiad vlan - https://phabricator.wikimedia.org/T405632#11250105 (10cmooney) See T405630#11250099, I'm not sure this will be possible.
[13:57:03] <inflatador>	 Hey IF! I'm getting the error `spicerack.redfish.RedfishError: POST https://10.193.1.237/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset returned HTTP 409` when I try to update the bios of wdqs2017. Does that means there's already a job in the queue or something?
[14:24:52] <jhathaway>	 inflatador: not sure, let me check the web gui
[14:25:48] <jhathaway>	 it has some critical errors, "Unable to power on the server because of an unidentified cable or a device is disconnected. This may be a device seating issue, connection issues in the device cable on SL 0, or other disconnected SL cables nearby."
[14:27:04] <jhathaway>	 inflatador: there is also a firmware update in the queue, but I assume it can't complete because of the hardware issue
[14:27:43] <inflatador>	 jhathaway Ah, thanks for taking a look. I'll send a ticket to DC Ops for that guy unless you think there's anything else we can do
[14:28:16] <jhathaway>	 inflatador: yeah I agree, dcops should take a look
[14:53:22] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr2-eqiad: fan failure on left tray [Oct 2025] - https://phabricator.wikimedia.org/T406554#11250521 (10VRiley-WMF) Yes, it seems like there is an issue with the fan, it is showing the warning lights for the fan. Is it okay to proceed w...
[15:01:41] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11250579 (10Papaul)
[15:02:03] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11250581 (10Papaul)
[15:19:47] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11250715 (10cmooney)
[15:47:12] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11250847 (10RobH)
[15:47:17] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602#11250848 (10RobH)
[15:47:47] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11250849 (10RobH)
[15:47:49] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602#11250850 (10RobH)
[17:27:07] <jinxer-wm^>	 FIRING: MaxConntrack: Max conntrack at 82.04% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[17:30:55] <jinxer-wm^>	 RESOLVED: MaxConntrack: Max conntrack at 80.88% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[19:39:25] <jinxer-wm^>	 FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag
[20:09:25] <jinxer-wm^>	 RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag
[20:50:03] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: Move lvs1020 link from ssw1-f1-eqiad to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T404959#11252319 (10wiki_willy) a:05cmooney→03VRiley-WMF
[20:50:35] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr2-eqiad: fan failure on left tray [Oct 2025] - https://phabricator.wikimedia.org/T406554#11252323 (10wiki_willy) a:05cmooney→03VRiley-WMF
[20:57:19] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr2-eqiad: fan failure on left tray [Oct 2025] - https://phabricator.wikimedia.org/T406554#11252329 (10cmooney) >>! In T406554#11250521, @VRiley-WMF wrote: > Yes, it seems like there is an issue with the fan, it is showing the warning...
[21:33:32] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr2-eqiad: fan failure on left tray [Oct 2025] - https://phabricator.wikimedia.org/T406554#11252463 (10VRiley-WMF) Hey @cmooney I just checked the filter, and it looked clean. I also reseated the fans as well, however it still is showi...