[01:46:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12131165 (10Papaul) [07:19:49] 10netops, 06Infrastructure-Foundations, 06SRE: JunOS: Investigate BGP PIC Edge / Protection - https://phabricator.wikimedia.org/T432381#12131427 (10ayounsi) Another optimization to consider: https://www.juniper.net/documentation/us/en/software/junos/bgp/topics/topic-map/bgp-route-prioritization.html [08:26:51] 06Traffic, 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12131542 (10elukey) Hey folks! IIUC this task is to track both Bookworm's and Trixie's updates, but I don... [08:33:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on lvs4009:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:43:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on lvs4009:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:56:18] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Create alerting for saturation on sub-rated interfaces - https://phabricator.wikimedia.org/T374614#12131571 (10ayounsi) As said on IRC, dunno if I'm opening some kind of pandora box by using starlark/python, but that seems like a good appro... [12:08:27] 10netops, 06Infrastructure-Foundations, 10netbox: Automatically run Capirca Netbox script regularly - https://phabricator.wikimedia.org/T361549#12132133 (10ayounsi) a:03ayounsi On Netbox-next: I created an "event-rule" - https://netbox-next.wikimedia.org/extras/event-rules/1/ {F94211857} Configured to ru... [12:35:17] 10netops, 06Infrastructure-Foundations, 10netbox: Automatically run Capirca Netbox script regularly - https://phabricator.wikimedia.org/T361549#12132210 (10cmooney) That's a great approach very nice! Haven't looked at the event-rule config but overall the idea is good with me +1 [13:27:32] 06Traffic, 06MediaWiki-API-Platform-Team, 06ServiceOps new, 10ServiceOps-SharedInfra, 07Epic: Remove api-gateway service traffic - https://phabricator.wikimedia.org/T432445#12132329 (10jijiki) 05Open→03In progress p:05Triage→03Medium [13:29:11] 06Traffic, 06MediaWiki-API-Platform-Team, 06ServiceOps new, 10ServiceOps-SharedInfra, 07Epic: Remove api-gateway service traffic - https://phabricator.wikimedia.org/T432445#12132343 (10jijiki) Dear #traffic: we will be prepping the patches, please let us know who could pair up with us and deploy them to... [13:29:45] 06Traffic, 06MediaWiki-API-Platform-Team, 06ServiceOps new, 10ServiceOps-SharedInfra, 07Epic: Remove api-gateway service traffic - https://phabricator.wikimedia.org/T432445#12132346 (10jijiki) [13:31:16] 06Traffic, 06MediaWiki-API-Platform-Team, 06ServiceOps new, 10ServiceOps-SharedInfra, 07Epic: Remove api-gateway service traffic - https://phabricator.wikimedia.org/T432445#12132359 (10jijiki) [13:31:52] 06Traffic, 06MediaWiki-API-Platform-Team, 06ServiceOps new, 10ServiceOps-SharedInfra, 07Epic: Remove api-gateway service traffic - https://phabricator.wikimedia.org/T432445#12132364 (10jijiki) [13:39:37] 06Traffic, 06MediaWiki-API-Platform-Team, 06ServiceOps new, 10ServiceOps-SharedInfra, and 2 others: Remove api-gateway service traffic - https://phabricator.wikimedia.org/T432445#12132393 (10jijiki) [13:53:43] 06Traffic, 06MediaWiki-API-Platform-Team, 06ServiceOps new, 10ServiceOps-SharedInfra, and 2 others: Remove api-gateway service traffic - https://phabricator.wikimedia.org/T432445#12132450 (10Blake) Patches have been uploaded and are ready for review. Thanks @jijiki for the overview today! [14:37:43] FIRING: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3072&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [14:37:48] oh no [14:42:43] FIRING: [2x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [14:47:43] RESOLVED: [2x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [17:47:31] 06Traffic, 10DNS, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Remove mirrors.wikimedia.org from public DNS - https://phabricator.wikimedia.org/T432422#12133132 (10Dzahn) 05Open→03Resolved a:03Dzahn merged and deployed - since it was just a CNAME to an already deleted record [20:30:39] Girl Scouts are hiring for VP of Agentic AI Innovation. Apply now at https://linkedin.com/jobs/view/4415733532 and build the agentic, data-driven future of American scouting.