[02:14:34] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12126803 (10Papaul) [02:16:27] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12126805 (10Papaul) [03:11:38] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12126823 (10Papaul) [03:25:28] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12126832 (10Papaul) [06:47:13] 10netops, 06Infrastructure-Foundations, 06tools-infrastructure-team, 06Traffic: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12126934 (10fgiunchedi) [06:47:16] 10netops, 10Cloud-VPS, 06Data-Platform-SRE, 10Data-Services, and 5 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12126935 (10fgiunchedi) [10:41:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: Netbox report: alert on cables in 'planned' state for more than two months - https://phabricator.wikimedia.org/T432329 (10cmooney) 03NEW p:05Triage→03Low [11:51:23] 10netops, 06Infrastructure-Foundations, 06SRE: Investigate using BGP addpath for unicast IBGP spine/leaf pods - https://phabricator.wikimedia.org/T402640#12127963 (10cmooney) >>! In T402640#11121128, @ayounsi wrote: > If I understand correctly we currently get some "per rack" load balancing, where `E3` might... [11:54:50] 10netops, 06Infrastructure-Foundations, 06SRE: Investigate using BGP addpath for unicast IBGP spine/leaf pods - https://phabricator.wikimedia.org/T402640#12127974 (10ayounsi) Yep, I refreshed my memory since then and I aadd path is a must have for proper balancing! +1 [12:07:43] 10netops, 06Infrastructure-Foundations: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12128005 (10ayounsi) I had a closer look to this, hopefully all the bad bugs and security vulnerabilities has been ironed out with time. The idea is to implement it behind a feature flag to... [12:07:47] 10netops, 06Infrastructure-Foundations: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12128006 (10ayounsi) a:03ayounsi [12:22:36] XioNoX: topranks: for the rack maint tasks, would it ok for you folks to include the due date in the task title? it would help a lot when skimming through the task lists [12:22:58] effie: that's a very good idea [12:23:05] down there for dancing :) [12:23:26] I'll update the other ones we have planned later to do and do it in future (unless Arzhel objects) [12:23:27] what do you think of using the "Due Date" field ? [12:23:42] https://phabricator.wikimedia.org/maniphest/task/edit/430928/ at the very bottom [12:23:59] we also sometimes make the task before it's scheduled - so I guess no data in the title would indicate it's not been scheduled [12:24:14] but yeah no objection [12:25:08] XioNoX: could be, though I suspect it will not be visible in the workboard, eg https://usercontent.irccloud-cdn.com/file/Py5DYgUe/image.png [12:25:38] thank you both! [12:25:54] yeah I just set the "due date" for that example task [12:27:11] effie: indeed looks like it doesn't show up [12:27:43] 10netops, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12128073 (10cmooney) [12:27:48] it would be nice to be able to customize the workboard, and for example hide all the other task's tags [12:28:01] yeah just tested there and doesn't seem visible on a work board [12:28:13] +1 from me to add the date to the title when the maintenance is scheduled [12:28:22] +1 [12:28:27] cool [12:42:21] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12128124 (10cmooney) After cr1-eqiad booted yesterday it took a disturbingly long time before routes were announced to some transits, so overall I think we should give t... [13:41:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Netbox report: alert on cables in 'planned' state for more than two months - https://phabricator.wikimedia.org/T432329#12128355 (10ayounsi) Above patch tested in Netbox-next: {F94004219} [16:02:51] 10Mail, 06cloud-services-team, 06collaboration-services, 06Infrastructure-Foundations, 10VPS-project-Phabricator: @wikimedia.org email addresses don't seem to be receiving emails sent by the test Phabricator instance - https://phabricator.wikimedia.org/T422559#12129019 (10Dzahn) Just ran into this again... [16:08:51] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10VPS-project-Phabricator: @wikimedia.org email addresses don't seem to be receiving emails sent by the test Phabricator instance - https://phabricator.wikimedia.org/T422559#12129092 (10taavi) [16:12:52] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10VPS-project-Phabricator: @wikimedia.org email addresses don't seem to be receiving emails sent by the test Phabricator instance - https://phabricator.wikimedia.org/T422559#12129123 (10taavi) >>! In T422559#11945766, @jhathaway wrote: > 1. mx... [16:31:28] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: cr1-drmrs: No power on PEM 0 - https://phabricator.wikimedia.org/T432370 (10cmooney) 03NEW p:05Triage→03High [16:33:19] 10netops, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12129249 (10cmooney) [16:34:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12129252 (10cmooney) [16:43:49] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: cr1-drmrs: No power on PEM 0 - https://phabricator.wikimedia.org/T432370#12129297 (10RobH) 05Open→03In progress [16:55:27] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: cr1-drmrs: No power on PEM 0 - https://phabricator.wikimedia.org/T432370#12129329 (10RobH) Filed urgent work request CS4924823, this includes out of hours rates. Listed myself, Papaul, Cathal, and Arzhel for ticket updates. > Support, > > We... [17:33:05] 10netops, 06Infrastructure-Foundations, 06SRE: Don't announce OSPF routes in unicast BGP on Nokia SR-Linux - https://phabricator.wikimedia.org/T423430#12129544 (10cmooney) 05Open→03Resolved Re-resolving after updating the alert for filtered prefixes to also monitor peers in the 'ibgp' group we use on... [17:34:17] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: cr1-drmrs: No power on PEM 0 - https://phabricator.wikimedia.org/T432370#12129548 (10Papaul) ` Jul 16 14:52:36 cr1-drmrs chassisd[18580]: CHASSISD_SNMP_TRAP6: SNMP trap generated: Power Supply failed (jnxContentsContainerIndex 2, jnxContentsL1Ind... [17:37:02] 10netops, 06Infrastructure-Foundations, 06SRE: cr1-drmrs: unexpected reboot Jul 16 2026 - https://phabricator.wikimedia.org/T432380 (10cmooney) 03NEW p:05Triage→03High [17:37:15] 10netops, 06Infrastructure-Foundations, 06SRE: cr1-drmrs: unexpected reboot Jul 16 2026 - https://phabricator.wikimedia.org/T432380#12129571 (10cmooney) [17:37:17] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: cr1-drmrs: No power on PEM 0 - https://phabricator.wikimedia.org/T432370#12129570 (10cmooney) [17:37:22] 10netops, 06Infrastructure-Foundations, 06SRE: cr1-drmrs: unexpected reboot Jul 16 2026 - https://phabricator.wikimedia.org/T432380#12129572 (10cmooney) 05Open→03Resolved [17:39:06] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: cr1-drmrs: No power on PEM 0 - https://phabricator.wikimedia.org/T432370#12129577 (10cmooney) 05In progress→03Resolved Remote hands were able to restore power: ` Jul 16 17:12:41 cr1-drmrs alarmd[23564]: Alarm cleared: PS SFXPC id=53687098... [17:51:02] topranks no rush on this one, but here's the WCQS CR as promised https://gerrit.wikimedia.org/r/c/operations/puppet/+/1311502 [17:54:24] 10netops, 06Infrastructure-Foundations, 06SRE: JunOS: Investigate BGP PIC Edge / Protection - https://phabricator.wikimedia.org/T432381 (10cmooney) 03NEW p:05Triage→03Low [18:17:23] inflatador: thank you!