[09:25:17] 10netops, 06Infrastructure-Foundations: Setup cr3-esams - https://phabricator.wikimedia.org/T442691#12521384 (10ayounsi) [10:16:37] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12521571 (10cmooney) [10:16:40] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12521570 (10cmooney) [10:17:11] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12521573 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=351d61a3-389f-44b4-9fe8-5fea6f1228f9) set by cmooney@cumin1004 for 1:00:00 on 1 host(s) and the... [14:10:52] 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade to CAS 8.0.X - https://phabricator.wikimedia.org/T439509#12522318 (10BTullis) Hi @SLyngshede-WMF - How would you feel about waiting for the CAS 8.1.0 version to be released, before continuing with the upgrade? There is a [[https://github.c... [14:30:48] 10netops, 06Infrastructure-Foundations, 07Incident Severity 2, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12522432 (10MLechvien-WMF) [14:38:47] 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade to CAS 8.0.X - https://phabricator.wikimedia.org/T439509#12522483 (10MoritzMuehlenhoff) >>! In T439509#12522318, @BTullis wrote: > Hi @SLyngshede-WMF - How would you feel about waiting for the CAS 8.1.0 version to be released, before contin... [14:43:26] 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade to CAS 8.0.X - https://phabricator.wikimedia.org/T439509#12522528 (10BTullis) >>! In T439509#12522483, @MoritzMuehlenhoff wrote: > Only two CAS release branches are security-supported at a time, so we need to move forward with 8.0 regardles... [15:18:00] 10SRE-tools, 06Infrastructure-Foundations, 13Patch-For-Review: Debmonitor: backend-changeable settings are stored in the browser's session storage - https://phabricator.wikimedia.org/T240457#12522764 (10Lizimedia) @Volans we should either merge this speculatively or close it, I think [15:45:45] 07Puppet, 06Release-Engineering-Team, 06SRE, 10Continuous-Integration-Config: Upgrade operations/puppet CI from Bullseye to Bookworm - https://phabricator.wikimedia.org/T443798 (10hashar) 03NEW [15:45:53] 07Puppet, 06Release-Engineering-Team, 06SRE, 10Continuous-Integration-Config: Upgrade operations/puppet CI from Bullseye to Bookworm - https://phabricator.wikimedia.org/T443798#12522973 (10hashar) [17:32:00] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523829 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=37a3a0c5-db75-4a5f-b3af-203b15319a02) set by cmooney@cumin1004 for 3 days, 0:00:00 on 1 host(s)... [17:32:33] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523837 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a759d84c-b17a-40ad-9d3b-e8171fe3782b) set by cmooney@cumin1004 for 3 days, 0:00:00 on 1 host(s)... [17:33:17] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523840 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=83c9e035-90af-41e4-bc00-e2734e08a539) set by cmooney@cumin1004 for 5:00:00 on 2 host(s) and the... [17:35:25] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523870 (10cmooney) FWIW I did some more testing today, starting with no network-instances, irb sub-interface etc configured, and built the config up bit by bit. See P973... [18:00:05] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523955 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=866cbc1d-696d-4894-9d5d-2aa4aa1c50da) set by cmooney@cumin1004 for 3 days, 0:00:00 on 2 host(s)... [18:05:24] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:10:24] FIRING: [2x] SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:26:50] hi I/F! we got https://gerrit.wikimedia.org/r/1361577 in the puppet window, but I think it needs a real review from someone with subject-matter understanding, can you take a look please? [20:31:48] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12524464 (10cmooney) In an effort to try and work out what version things still worked in I downgraded lsw1-b3-eqiad to 26.7.1, same issue. I then went to 26.3.3 - same th... [20:38:41] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12524495 (10cmooney) p:05High→03Medium [20:39:27] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12524496 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8099810e-6408-4be2-82b4-5240b5d6c531) set by cmooney@c... [20:59:32] 10SRE-tools, 06Infrastructure-Foundations, 13Patch-For-Review: Debmonitor: backend-changeable settings are stored in the browser's session storage - https://phabricator.wikimedia.org/T240457#12524621 (10Volans) Reviewed the code, LGTM. [21:20:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12524731 (10ops-monitoring-bot) Host lvs1019.eqiad.wmnet rebooted by cmooney@cumin1004 with reason: reconfigure vlan ints [22:10:24] FIRING: [2x] SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:44:17] 10SRE-tools, 10Ganeti, 06Infrastructure-Foundations, 06SRE: Cookbook to failover the Ganeti master - https://phabricator.wikimedia.org/T283320#12525135 (10Lizimedia) a:03Lizimedia [23:46:22] 07Puppet, 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Cannot rollout Trixie hosts in deployment-prep without rm -rf /var/lib/puppet/ssl - https://phabricator.wikimedia.org/T443373#12525336 (10Southparkfan) >As far as I know, agents on Bookworm hosts do not...