[01:07:17] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO:Switch refresh diagram - https://phabricator.wikimedia.org/T408511#11625878 (10Papaul) [01:32:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO:Switch refresh diagram - https://phabricator.wikimedia.org/T408511#11625923 (10Papaul) @ayounsi I need you input here. et-0/0/1 on cr3/4-ulsfo are connected to asw2-22/23 the goal was to wait until phase 2 to move et-0/0/1 to the... [08:58:24] 10SRE-tools, 06Infrastructure-Foundations: Outdated cookbooks cleanup - https://phabricator.wikimedia.org/T379259#11626491 (10Volans) 05Open→03Resolved Resolving this old task, there was some cleanup done at the time, in case it's deemed necessary to do a new pass a new task should be created. [10:21:24] 10CAS-SSO, 06Infrastructure-Foundations, 06tools-infrastructure-team: Unable to log in to labtesthorizon, cloudidp-dev - https://phabricator.wikimedia.org/T417755 (10taavi) 03NEW [10:22:29] 10CAS-SSO, 06Infrastructure-Foundations, 06tools-infrastructure-team: Unable to log in to labtesthorizon, cloudidp-dev - https://phabricator.wikimedia.org/T417755#11626903 (10taavi) Seeing errors like: ` Feb 18 10:20:53 cloudidp2001-dev tomcat10[535471]: 2026-02-18 10:20:53,011 WARN [org.apereo.cas.util.func... [10:25:43] 10netops, 06Infrastructure-Foundations: ulsfo: upgrade routers (2026) - https://phabricator.wikimedia.org/T416562#11626918 (10ayounsi) a:03Papaul Assigning this to Papaul, to be bundled at some point with the switch replacement (not urgent, but a good opportunity). [10:25:45] 10netops, 06Infrastructure-Foundations: eqsin: upgrade routers (2026) - https://phabricator.wikimedia.org/T416563#11626920 (10ayounsi) a:03Papaul Assigning this to Papaul, to be bundled at some point with the switch replacement (not urgent, but a good opportunity). [10:33:11] 10CAS-SSO, 06Infrastructure-Foundations, 06tools-infrastructure-team: Unable to log in to labtesthorizon, cloudidp-dev - https://phabricator.wikimedia.org/T417755#11626962 (10taavi) 05Open→03Resolved Fixed with d5611fc9f in private-puppet. [10:40:05] 24h without monitoring noise (the only alert was real, due to the magru work) [10:45:21] 10CAS-SSO, 06Infrastructure-Foundations, 06tools-infrastructure-team: Unable to log in to labtesthorizon, cloudidp-dev - https://phabricator.wikimedia.org/T417755#11627006 (10taavi) 05Resolved→03Open Eh, no, now it's complaining about: ` Feb 18 10:44:51 cloudidp2001-dev tomcat10[617265]: 2026-02-18 10:44... [11:32:26] 10CAS-SSO, 06Infrastructure-Foundations, 06tools-infrastructure-team: Unable to log in to labtesthorizon, cloudidp-dev - https://phabricator.wikimedia.org/T417755#11627197 (10taavi) Turning CAS logging up at least reveals which key it is complaining about: ` Feb 18 11:27:32 cloudidp2001-dev tomcat10[621906]:... [11:36:30] 10CAS-SSO, 06Infrastructure-Foundations, 06tools-infrastructure-team: Unable to log in to labtesthorizon, cloudidp-dev - https://phabricator.wikimedia.org/T417755#11627227 (10taavi) 05Open→03Resolved Replacing that key with an another one of the exact same length made CAS let me log in again to Horizon. [15:26:50] 07Puppet, 06cloud-services-team, 10Cloud-VPS: Prune old reports from /var/lib/puppetserver/server_data/facts on Cloud VPS puppet servers - https://phabricator.wikimedia.org/T417795 (10taavi) 03NEW p:05Triage→03High [15:28:16] 07Puppet, 06cloud-services-team, 10Cloud-VPS: Prune old reports from /var/lib/puppetserver/server_data/facts on Cloud VPS puppet servers - https://phabricator.wikimedia.org/T417795#11628178 (10taavi) [16:00:39] 07Puppet, 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Prune old reports from /var/lib/puppetserver/server_data/facts on Cloud VPS puppet servers - https://phabricator.wikimedia.org/T417795#11628332 (10taavi) 05Open→03Resolved [17:00:46] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: FY2526 Q3 ulsfo: switch refresh - https://phabricator.wikimedia.org/T408510#11628604 (10RobH) [17:17:46] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO:Switch refresh diagram - https://phabricator.wikimedia.org/T408511#11628686 (10Papaul) Created ticket Case Order #01144222 for initial racking and wiring of the new Nokia switches. [19:37:18] XioNoX: are you available for https://phabricator.wikimedia.org/T417817 ? (sorry we didn't get to it while Cathal was around, I was sick) [19:38:43] Raine: I can have a look tomorrow [19:38:53] awesome, thank you <3 [23:55:25] FIRING: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed