[00:01:00] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 22.4R3 - https://phabricator.wikimedia.org/T364092#10190074 (10Papaul) [00:02:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: cr3-ulsfo incident 22 Sep 2024 - https://phabricator.wikimedia.org/T375345#10190077 (10Papaul) Junos upgrade complete for the system Icinga checks back green. All good on the router, site can be pool back Thanks [06:55:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: cr3-ulsfo incident 22 Sep 2024 - https://phabricator.wikimedia.org/T375345#10190475 (10ayounsi) 05Open→03Resolved Thanks, all is good now ! [07:15:44] 10netops, 06Infrastructure-Foundations: Juniper: regularly run `request system configuration rescue save` - https://phabricator.wikimedia.org/T376005#10190489 (10ayounsi) ` cr3-ulsfo> request vmhost snapshot ? Possible completions: <[Enter]> Execute this command config Sychronise C... [07:36:16] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10190518 (10ayounsi) [11:05:35] anyone to +1 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1076987 ? [13:27:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: cr3-ulsfo incident 22 Sep 2024 - https://phabricator.wikimedia.org/T375345#10191716 (10Papaul) 05Resolved→03Open a:05ayounsi→03Papaul I have to update netbox with the inventory and new serial number [13:47:04] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10191790 (10cmooney) I looked into how we might generate the required NS records in the appropriate zones to delegate ranges used by k8s to the appropriat... [14:14:42] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:servers migration task - https://phabricator.wikimedia.org/T375151#10191898 (10Jhancock.wm) [14:29:25] FIRING: SystemdUnitFailed: user@0.service on cp2030:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:30:31] why though... [14:30:57] restarted [14:34:25] RESOLVED: SystemdUnitFailed: user@0.service on cp2030:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:50:48] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#10192040 (10Papaul) [14:51:02] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#10192044 (10Papaul) [15:25:28] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#10192253 (10Papaul) [15:26:59] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE: cp307[12] thermal issues - https://phabricator.wikimedia.org/T374986#10192267 (10ssingh) Hi @RobH: Is this confirmed for tomorrow Oct 2? [15:28:19] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE: cp307[12] thermal issues - https://phabricator.wikimedia.org/T374986#10192278 (10RobH) Yes, they'll be showing up onsite around 09:00 CET / 00:00 Pacific. We'll want to fully depool and power down these two hosts in advance of their arrival. I figured I would just... [15:34:16] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE: cp307[12] thermal issues - https://phabricator.wikimedia.org/T374986#10192348 (10ssingh) Thanks @RobH, that works for us. @Vgutierrez will depool the two hosts in advance of the event and downtime. [15:46:52] 06Traffic, 06collaboration-services, 06SRE, 13Patch-For-Review, 10Release-Engineering-Team (Radar): implement anti-abuse features for GitLab (Move GitLab behind the CDN) - https://phabricator.wikimedia.org/T366882#10192476 (10Jelto) [15:52:44] 06Traffic, 06collaboration-services, 06SRE, 13Patch-For-Review, 10Release-Engineering-Team (Radar): implement anti-abuse features for GitLab (Move GitLab behind the CDN) - https://phabricator.wikimedia.org/T366882#10192543 (10Jelto) 05Open→03Resolved Throttling is active for around one month on a... [15:58:17] 10netops, 06Infrastructure-Foundations: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#10192596 (10Papaul) [17:00:42] 06Traffic, 06Data-Engineering-Icebox, 06SRE, 10WMF-General-or-Unknown, and 2 others: Requests for /static get an invalid WMF-Last-Access cookie for wikipedia.org on non-Wikipedia requests - https://phabricator.wikimedia.org/T261803#10192971 (10matmarex) This is still a problem today, and it makes for a dis... [18:13:27] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: cr3-ulsfo incident 22 Sep 2024 - https://phabricator.wikimedia.org/T375345#10193306 (10Papaul) 05Open→03Resolved Add both power supplies in Netbox under inventory [18:18:21] 06Traffic, 06DC-Ops, 10ops-esams, 06SRE: cp307[12] thermal issues - https://phabricator.wikimedia.org/T374986#10193313 (10RobH) > Your appointment has been scheduled between Wed, Oct 2, 2024 8:00 AM and Wed, Oct 2, 2024 12:00 PM. Please check back here for updates. > Your technician is scheduled to arriv... [18:22:56] 10netops, 06Infrastructure-Foundations: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#10193328 (10Papaul) [18:23:50] 10netops, 06Infrastructure-Foundations: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#10193330 (10Papaul) 05Open→03Resolved This is complete. @ayounsi thanks for the patch [18:35:11] 06Traffic, 06Data-Engineering-Icebox, 06SRE, 10WMF-General-or-Unknown, and 2 others: Requests for /static get an invalid WMF-Last-Access cookie for wikipedia.org on non-Wikipedia requests - https://phabricator.wikimedia.org/T261803#10193353 (10Tgr) Yeah, the wider issue here is that setting the cookie on c... [23:06:48] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:rack/install/configuration new firewalls - https://phabricator.wikimedia.org/T374176#10194218 (10Papaul) [23:14:14] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:servers migration task - https://phabricator.wikimedia.org/T375151#10194252 (10Papaul) [23:50:29] 06Traffic, 10Infrastructure Security, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog, 07Security: Integrate In-App Internet censorship circumvention by domain fronting - https://phabricator.wikimedia.org/T327286#10194390 (10ZauberViolino) >Recent days, WMF somehow changed their GeoDNS so th...