[00:03:37] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Management routers to 23.4R2-S2 - https://phabricator.wikimedia.org/T369504#10236283 (10Papaul) [00:06:03] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Management routers to 23.4R2-S2 - https://phabricator.wikimedia.org/T369504#10236284 (10Papaul) 05Open→03Resolved This is complete [03:47:04] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr1-eqiad: disk failure - https://phabricator.wikimedia.org/T372781#10236448 (10Papaul) @VRiley-WMF thank you for following up on this. It looks like the router is back running on re0 and disks are all there. We can close. @ayounsi any... [04:20:51] 10netops, 06Infrastructure-Foundations: mr1-eqsin performance issue - https://phabricator.wikimedia.org/T362522#10236460 (10Papaul) I checked the router again today after the Junos upgrade and reboot no core-dump file so far. ` show system core-dumps no-forwarding /var/crash/*core*: No such file or directory... [04:25:43] 10netops, 06Infrastructure-Foundations: cr2-codfw - Host 0 ECC single bit parity error - https://phabricator.wikimedia.org/T371868#10236461 (10Papaul) since August until now no errors so far ` cr2-codfw> show system alarms No alarms currently active [06:37:10] 10netops, 06Infrastructure-Foundations: mr1-eqsin performance issue - https://phabricator.wikimedia.org/T362522#10236570 (10ayounsi) We will need to monitor it a bit more, at they seem to happen once a month or about. [06:38:37] 10netops, 06Infrastructure-Foundations: cr2-codfw - Host 0 ECC single bit parity error - https://phabricator.wikimedia.org/T371868#10236571 (10ayounsi) 05Open→03Resolved a:03ayounsi Perfect, thanks ! [06:40:12] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr1-eqiad: disk failure - https://phabricator.wikimedia.org/T372781#10236574 (10ayounsi) ` re1.cr1-eqiad> show system alarms 1 alarms currently active Alarm time Class Description 2024-07-18 16:11:37 UTC Minor Backup... [06:42:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr1-eqiad: disk failure - https://phabricator.wikimedia.org/T372781#10236576 (10ayounsi) [06:42:50] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10236577 (10ayounsi) [07:47:05] 10netops, 06Infrastructure-Foundations, 06SRE: Re-IP codfw private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T354869#10236702 (10ayounsi) [09:34:39] 10netops, 06Infrastructure-Foundations, 06SRE: Re-IP codfw private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T354869#10236937 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host phab2002.codfw.wmnet with OS bullseye executed... [09:54:54] 10netops, 06Infrastructure-Foundations, 06SRE: Re-IP codfw private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T354869#10236988 (10cmooney) [13:45:55] 14Varnish, 10Maps, 06SRE, 06Traffic-Icebox: Tilerator should purge Varnish cache - https://phabricator.wikimedia.org/T109776#10237879 (10akosiaris) 05Open→03Invalid Tilerator exists no more in the WMF environment. I 'll close this av `invalid`, feel free to reopen. [14:38:53] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10238146 (10cmooney) [14:44:53] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10238176 (10cmooney) [14:51:21] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10238216 (10cmooney) [15:25:14] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10238331 (10cmooney) [15:26:42] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10238348 (10cmooney) [15:32:43] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10238391 (10cmooney) [15:33:51] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10238395 (10cmooney) [15:45:46] 06Traffic, 06Infrastructure-Foundations, 06SRE: NetworkProbeLimit cookie should set samesite attribute - https://phabricator.wikimedia.org/T342624#10238465 (10Krinkle) This change introduces the following error, repeated in the console for me when logged-in. Note that, unlike the original message in the task... [16:17:12] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10238641 (10RobH) [18:59:49] hello traffic friends, any objections to / conflicts with a DNS update (adding some new A and PTR records in the svc zone [0]) in the not-too-distant future? [18:59:49] [0] https://gerrit.wikimedia.org/r/c/operations/dns/+/1080778 [19:26:04] I'll take that as a no :)