[00:19:25] FIRING: SystemdUnitFailed: rsyslog-imfile-remedy.service on aux-k8s-worker2006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:38:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12152189 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=b95b784a-5c97-432d-a8f4-184a296fddc1) set by pt1979@cumin1003 for 1 day, 0... [02:49:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12152195 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8c1d9ab0-7c1c-4f48-9f20-9323d7443de0) set by pt1979@cumin1003 for 1 day, 0... [03:19:25] RESOLVED: SystemdUnitFailed: rsyslog-imfile-remedy.service on aux-k8s-worker2006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:26:46] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12152217 (10Papaul) We are having some mgmt issues in eqsin so i had to setup some downtime. We need to check the connections ge-0/0/1 on mr1-eqsin is... [05:52:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12152292 (10ayounsi) p:05Medium→03High [08:37:47] Dear I/F, cont what I wrote yesterday [08:38:22] we have two hosts which at "Failed to get Netbox script results, try manually" [08:42:06] I had a read of https://wikitech.wikimedia.org/wiki/Netbox#Custom%20Scripts, but sadly I am still unsure what to do :) [09:05:42] effie: where do you get that error? [09:07:00] xhttps://phabricator.wikimedia.org/T421711#12150803 [09:09:38] so the job (script) completed properly - https://netbox.wikimedia.org/core/jobs/371306/ [09:10:20] so all the cookbook has properly run I think [09:10:25] as I htink that's the last ste [09:10:26] p [09:14:44] yeah I know what happened, we did a lot of IP change work, and because of https://phabricator.wikimedia.org/T361549 it triggered the Capirca script many times, which blocked the queue [09:15:18] The ImportPuppetDB script was scheduled at 2026-07-23 16:26:20 but only started at 2026-07-23 19:01:44 to be completed at 2026-07-23 19:01:46 [10:14:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12152756 (10cmooney) [10:19:07] XioNoX: thinking out loud - can we adjust the event rule to only trigger if the IP that was changed was on an interface of a server? [10:19:24] I didn't realise but I caused that adding all the IPs for the new switches in eqiad a/b [10:43:22] topranks: I tried but I don't think it's possible. If it causes too much issues I'll just put it on a timer [11:27:16] XioNoX: no major deal thanks for looking [11:28:01] it's rare we add so many IPs so quickly anyway we can perhaps disable it if we need to do it again [11:28:27] we understand the potential issue now at least, hadn't occurred to me prior [11:33:14] discovering the new feature toi [11:33:16] too [11:50:54] topranks: quick thought, we should be able to use the reimage cookbook with --move-vlan when moving servers across racks, just need one "hack": change the host's vlan to one of the legacy vlans before running the re-image. [11:51:05] not tested but I think that would work [11:52:12] yes quite likely [11:52:18] you thinking in terms of the public-vlan ones? [11:52:39] we could improve the move-vlan `def _is_move_required(self)` to check the host's IP's vlan and check if it match the current vlan (or ideal vlan) [11:53:19] topranks: I thought about that when looking at moves like https://phabricator.wikimedia.org/T432716 [11:53:41] ok yep [11:53:46] could be worth testing that flow with those hosts [11:53:51] could save quite some time [11:54:23] yeah [11:54:58] I think it totally makes sense to have _is_move_required() check the server's rack location, and if the rack location is invalid for the vlan then to to the same vlan-move logic [11:55:07] s/to/do [11:55:21] yeah I started to think and work on that yesterday, but it's more complex than it looks like [11:55:41] hmm ok [11:55:49] especially for example those hosts have been configured with the proper vlan [11:55:54] but it's the IP that is wrong [11:56:04] so ideally we need to check multiple elements [11:56:05] how is that possible? [11:56:27] topranks: dcops changed it manually [11:56:57] but only the vlan/interface/cable, but not the IP [11:57:32] so for what I saw there are lots of edge cases [11:57:36] surely that's something the update to the cookbook is aiming to ensure is not needed? [11:58:02] my thinking would be to update it to support the "basic" case.... rack does not match the configured switch port vlan [11:58:15] but the move-vlan cookbook doesn't take care of cabling/interface, etc only vlan/IP [11:58:19] anything beyond that caused by manual edits will need to be fixed by manual edits [11:58:25] we're in the world of manual edits at that point [11:58:37] we need to have a look at the big picture and full flow [11:58:47] ah ok yeah I hadn't considered that [11:58:50] yeah needs more work [11:59:13] and update that terrible process: https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Move_existing_server_between_rows/racks,_changing_IPs [11:59:19] automate some of it, etc.. [12:00:02] anyway, I just wanted to share my possible workaround/hack for those 2 hosts and maybe other similar usecases just in case [12:00:17] if it can save some time [12:02:03] I'm out for today! have a good weekend [12:02:50] if anyone fancy some friday reviews I have https://gerrit.wikimedia.org/r/c/operations/alerts/+/1315778 pending :) [12:51:43] where does that leave the SRE re-IPing stuff ? :) [13:17:34] effie: the above only relates to moving servers, and having to re-ip to the appropriate vlan/subnet for the new location [13:18:01] if you are doing such a thing I can help and we can whandangle it [13:18:45] regular reimages from row-wide to per-rack vlans work via the cookbook, we're just discussing how to expand it to support servers moving about [13:19:48] topranks: I had a meeting earlier and then lunch. We were only re-iping servers [13:20:45] part of https://phabricator.wikimedia.org/T421711. I just want to know what to do to continue with the rest of the list, as well as if I can repool 1071 and 1072 :) [13:21:36] effie: yep for those the cookbook should do the trick, if it doesn't feel free to ping me [13:26:49] ta [13:49:31] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12153283 (10Papaul) I put in a remote hands order 1-262642171482 we are working on this [14:28:24] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12153394 (10cmooney) @papaul resolved the issue by shutting down ge-0/0/2 (connected to msw2) on mr1-eqsin. In brief the mgmt connectivity is working... [15:37:14] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097 (10RobH) 03NEW p:05Triage→03High [15:38:57] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12153909 (10RobH) [15:39:08] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12153910 (10RobH) a:03ssingh Sukhbir, Please review the task description and provide feedback on when we could depool eqsin for the proposed w... [16:09:50] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux: Update Homer automation to support v26 - https://phabricator.wikimedia.org/T433105 (10cmooney) 03NEW p:05Triage→03Medium [16:11:54] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux: Update Homer automation to support v26 - https://phabricator.wikimedia.org/T433105#12154065 (10cmooney) [16:14:28] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux: Update Homer automation to support v26 - https://phabricator.wikimedia.org/T433105#12154069 (10cmooney) Actually I double-checked the ACL one. If you display the config `as json` it is identical in v25 and v26. Seems they just changed how it is... [16:57:46] 10netops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE, 06Traffic: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12154233 (10BCornwall) @ssingh I am willing to be available on my Monday evening/Tuesday Singapore time.