[02:14:34] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12126803 (10Papaul) [02:16:27] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12126805 (10Papaul) [03:11:38] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12126823 (10Papaul) [03:25:28] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12126832 (10Papaul) [06:47:13] 10netops, 06Traffic, 06Infrastructure-Foundations, 06tools-infrastructure-team: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12126934 (10fgiunchedi) [06:47:16] 10netops, 06Traffic, 10Cloud-VPS, 06Data-Platform-SRE, and 5 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12126935 (10fgiunchedi) [10:41:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: Netbox report: alert on cables in 'planned' state for more than two months - https://phabricator.wikimedia.org/T432329 (10cmooney) 03NEW p:05Triage→03Low [11:51:23] 10netops, 06Infrastructure-Foundations, 06SRE: Investigate using BGP addpath for unicast IBGP spine/leaf pods - https://phabricator.wikimedia.org/T402640#12127963 (10cmooney) >>! In T402640#11121128, @ayounsi wrote: > If I understand correctly we currently get some "per rack" load balancing, where `E3` might... [11:54:51] 10netops, 06Infrastructure-Foundations, 06SRE: Investigate using BGP addpath for unicast IBGP spine/leaf pods - https://phabricator.wikimedia.org/T402640#12127974 (10ayounsi) Yep, I refreshed my memory since then and I aadd path is a must have for proper balancing! +1 [12:06:21] FIRING: DnsboxServiceMismatch: Service ntp-b state mismatch on dns5004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqsin&var-instance=dns5004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [12:07:43] 10netops, 06Infrastructure-Foundations: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12128005 (10ayounsi) I had a closer look to this, hopefully all the bad bugs and security vulnerabilities has been ironed out with time. The idea is to implement it behind a feature flag to... [12:07:47] 10netops, 06Infrastructure-Foundations: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12128006 (10ayounsi) a:03ayounsi [12:11:20] RESOLVED: DnsboxServiceMismatch: Service ntp-b state mismatch on dns5004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqsin&var-instance=dns5004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [12:27:44] 10netops, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12128073 (10cmooney) [12:42:21] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12128124 (10cmooney) After cr1-eqiad booted yesterday it took a disturbingly long time before routes were announced to some transits, so overall I think we should give t... [13:40:20] FIRING: DnsboxServiceMismatch: Service ntp-b state mismatch on dns7002:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=magru&var-instance=dns7002:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:40:48] hmm [13:41:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Netbox report: alert on cables in 'planned' state for more than two months - https://phabricator.wikimedia.org/T432329#12128355 (10ayounsi) Above patch tested in Netbox-next: {F94004219} [13:45:20] RESOLVED: DnsboxServiceMismatch: Service ntp-b state mismatch on dns7002:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=magru&var-instance=dns7002:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:45:32] nice [13:58:09] 06Traffic, 06MediaWiki-API-Platform-Team, 10MediaWiki-File-management, 10MediaWiki-General, and 5 others: Allow MediaWiki to make authenticated requests to other instances of MediaWiki - https://phabricator.wikimedia.org/T424768#12128391 (10Tgr) I think owner-only tokens will get you through edge throttlin... [13:59:24] 06Traffic, 06MediaWiki-API-Platform-Team, 10MediaWiki-File-management, 10MediaWiki-General, and 5 others: Allow MediaWiki to make authenticated requests to other instances of MediaWiki - https://phabricator.wikimedia.org/T424768#12128400 (10Tgr) >>! In T424768#12128391, @Tgr wrote: > Not sure if any of the... [14:21:16] dear traffic, heads up, I am merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1310103 [14:29:14] effie: ok thanks for letting us know [16:31:28] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: cr1-drmrs: No power on PEM 0 - https://phabricator.wikimedia.org/T432370 (10cmooney) 03NEW p:05Triage→03High [16:33:20] 10netops, 06cloud-services-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12129249 (10cmooney) [16:33:28] 06Traffic, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 2 others: codfw: rack B8 maintenance - Thursday July 23rd 14:00 UTC - https://phabricator.wikimedia.org/T430929#12129250 (10cmooney) [16:34:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12129252 (10cmooney) [16:42:04] folks it seems a bunch of BFD sessions to dns/doh/durum hosts in eqiad also failed yesterday after their reboots [16:42:24] I've reset with "sudo birdc restart bfd1" using cumin and all ok again [16:43:49] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: cr1-drmrs: No power on PEM 0 - https://phabricator.wikimedia.org/T432370#12129297 (10RobH) 05Open→03In progress [16:55:27] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: cr1-drmrs: No power on PEM 0 - https://phabricator.wikimedia.org/T432370#12129329 (10RobH) Filed urgent work request CS4924823, this includes out of hours rates. Listed myself, Papaul, Cathal, and Arzhel for ticket updates. > Support, > > We... [16:57:06] 06Traffic, 10bot-traffic-requests: Link preview from iMessage on macOS blocked - https://phabricator.wikimedia.org/T432373 (10Krinkle) 03NEW [17:00:18] oh no [17:00:22] thanks topranks [17:00:33] out of curiosity, where do you get these alerts? [17:00:44] we should track them in Traffic as well [17:07:24] I noticed them in alertmanager [17:10:28] This is the one I think: [17:10:30] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/refs/heads/master/team-netops/bfd.yaml [17:11:00] but it's generic, the alert just lists the IP. not sure we have a way right now to find the specific peer and route to the right team [17:15:38] 06Traffic, 10bot-traffic-requests: Link preview from iMessage on macOS blocked - https://phabricator.wikimedia.org/T432373#12129456 (10Krinkle) [17:19:50] 06Traffic, 10bot-traffic-requests: Link preview from iMessage on macOS blocked - https://phabricator.wikimedia.org/T432373#12129469 (10Krinkle) It seems Google Messages may be blocked for the same reason. Unlike the macOS one, I have not emperically confirmed this with my own messages as I don't have an Androi... [17:21:52] topranks: ok, thanks [17:22:07] and yeah, routing to Traffic is easy, it goes under team-traffic [17:22:19] and that way alerts to us. but specific peer, not sure [17:33:05] 10netops, 06Infrastructure-Foundations, 06SRE: Don't announce OSPF routes in unicast BGP on Nokia SR-Linux - https://phabricator.wikimedia.org/T423430#12129544 (10cmooney) 05Open→03Resolved Re-resolving after updating the alert for filtered prefixes to also monitor peers in the 'ibgp' group we use on... [17:34:18] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: cr1-drmrs: No power on PEM 0 - https://phabricator.wikimedia.org/T432370#12129548 (10Papaul) ` Jul 16 14:52:36 cr1-drmrs chassisd[18580]: CHASSISD_SNMP_TRAP6: SNMP trap generated: Power Supply failed (jnxContentsContainerIndex 2, jnxContentsL1Ind... [17:37:02] 10netops, 06Infrastructure-Foundations, 06SRE: cr1-drmrs: unexpected reboot Jul 16 2026 - https://phabricator.wikimedia.org/T432380 (10cmooney) 03NEW p:05Triage→03High [17:37:15] 10netops, 06Infrastructure-Foundations, 06SRE: cr1-drmrs: unexpected reboot Jul 16 2026 - https://phabricator.wikimedia.org/T432380#12129571 (10cmooney) [17:37:17] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: cr1-drmrs: No power on PEM 0 - https://phabricator.wikimedia.org/T432370#12129570 (10cmooney) [17:37:22] 10netops, 06Infrastructure-Foundations, 06SRE: cr1-drmrs: unexpected reboot Jul 16 2026 - https://phabricator.wikimedia.org/T432380#12129572 (10cmooney) 05Open→03Resolved [17:39:06] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: cr1-drmrs: No power on PEM 0 - https://phabricator.wikimedia.org/T432370#12129577 (10cmooney) 05In progress→03Resolved Remote hands were able to restore power: ` Jul 16 17:12:41 cr1-drmrs alarmd[23564]: Alarm cleared: PS SFXPC id=53687098... [17:54:24] 10netops, 06Infrastructure-Foundations, 06SRE: JunOS: Investigate BGP PIC Edge / Protection - https://phabricator.wikimedia.org/T432381 (10cmooney) 03NEW p:05Triage→03Low [19:00:15] 06Traffic, 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12129847 (10bking) [21:02:45] brett thanks for the review, I didn't even have to ask ;) [21:08:29] I'm doing my part!