[02:38:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12152189 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=b95b784a-5c97-432d-a8f4-184a296fddc1) set by pt1979@cumin1003 for 1 day, 0... [02:49:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12152195 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8c1d9ab0-7c1c-4f48-9f20-9323d7443de0) set by pt1979@cumin1003 for 1 day, 0... [03:26:46] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12152217 (10Papaul) We are having some mgmt issues in eqsin so i had to setup some downtime. We need to check the connections ge-0/0/1 on mr1-eqsin is... [05:52:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12152292 (10ayounsi) p:05Medium→03High [06:28:18] 06Traffic, 10Wikidata, 06Wikidata-Omega (The Board): Service at query-scholarly-next.wikidata.org not available in esams - https://phabricator.wikimedia.org/T432932#12152312 (10ArthurTaylor) 05Open→03Resolved a:03ArthurTaylor Hi @ssingh, The issue seems to be resolved now. There were a few more co... [06:28:58] sukhe: The issue is resolved now - thanks for taking a look [06:29:15] I have no idea if it was some further configuration changes that resolved it, or a cache expiring [07:11:42] sukhe: (low priority) re your earlier `so we did depool before the reimaging and then repooled after the reimaging?` yes exactly that, seems like the pybal seeing old IP issue can manifest without a depool failure being necessary to cause it [09:46:11] sukhe: now that https://gerrit.wikimedia.org/r/c/operations/puppet/+/1313085 is deployed I've sent https://gerrit.wikimedia.org/r/c/operations/alerts/+/1315778 as an exemple on how to use that new metric for service owners [10:12:41] 06Traffic, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 2 others: codfw: rack B8 maintenance - Thursday July 23rd 14:00 UTC - https://phabricator.wikimedia.org/T430929#12152732 (10cmooney) 05Open→03Resolved [10:14:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12152756 (10cmooney) [13:49:31] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12153283 (10Papaul) I put in a remote hands order 1-262642171482 we are working on this [14:28:24] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12153394 (10cmooney) @papaul resolved the issue by shutting down ge-0/0/2 (connected to msw2) on mr1-eqsin. In brief the mgmt connectivity is working... [14:36:32] Hello traffic friends. I've got a patch here that I'd like you to review ahead of a Monday deploy, if possible. Thanks. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1315900 [15:10:21] btullis: +1'd [15:37:14] 10netops, 06Traffic, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097 (10RobH) 03NEW p:05Triage→03High [15:38:57] 10netops, 06Traffic, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12153909 (10RobH) [15:39:08] 10netops, 06Traffic, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12153910 (10RobH) a:03ssingh Sukhbir, Please review the task description and provide feedback on when we could depool eqsin for the proposed w... [16:09:50] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux: Update Homer automation to support v26 - https://phabricator.wikimedia.org/T433105 (10cmooney) 03NEW p:05Triage→03Medium [16:11:54] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux: Update Homer automation to support v26 - https://phabricator.wikimedia.org/T433105#12154065 (10cmooney) [16:14:28] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux: Update Homer automation to support v26 - https://phabricator.wikimedia.org/T433105#12154069 (10cmooney) Actually I double-checked the ACL one. If you display the config `as json` it is identical in v25 and v26. Seems they just changed how it is... [16:57:46] 10netops, 06Traffic, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12154233 (10BCornwall) @ssingh I am willing to be available on my Monday evening/Tuesday Singapore time.