[09:53:33] 06Traffic, 06SRE, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Have CDN edge set the `X-Request-Id` header for incoming external requests - https://phabricator.wikimedia.org/T221976#11267952 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez [09:54:11] not bad... that task was from 2019 :) [09:55:38] cured and aged perfectly [10:45:51] 10netops, 06Infrastructure-Foundations, 06SRE: drmrs: cr1-drmrs <-> asw1-b13-drmrs link down [Oct 2025] - https://phabricator.wikimedia.org/T407107 (10cmooney) 03NEW p:05Triage→03High [10:50:43] FIRING: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3070&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [10:55:43] FIRING: [14x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [11:00:43] FIRING: [14x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [11:05:43] RESOLVED: [14x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [11:29:52] 10netops, 06Infrastructure-Foundations, 06SRE: drmrs: cr1-drmrs <-> asw1-b13-drmrs link down [Oct 2025] - https://phabricator.wikimedia.org/T407107#11268170 (10cmooney) Remote hands request id CS3321949 [13:11:14] 10netops, 06Infrastructure-Foundations, 06SRE: drmrs: cr1-drmrs <-> asw1-b13-drmrs link down [Oct 2025] - https://phabricator.wikimedia.org/T407107#11268439 (10cmooney) They checked the fibres and reseated but no change. ` We have reseated the fiber and SFPs on both sides. However, QSFP port 2 on the cr1-drm... [14:10:59] o/ I'd like to do one last rest.php disable-puppet dance on A:cp soon, would that be okay? about to bring rest.php enwiki to rest-gateway at 100% [14:11:20] sounds good to me [14:11:29] thanks! [14:23:45] done [14:39:59] 10netops, 06Infrastructure-Foundations, 06SRE: drmrs: cr1-drmrs <-> asw1-b13-drmrs link down [Oct 2025] - https://phabricator.wikimedia.org/T407107#11268789 (10cmooney) The QSFP replacement in cr1-drmrs did the trick: ` Oct 13 13:49:58 asw1-b13-drmrs mib2d[9179]: SNMP_TRAP_LINK_UP: ifIndex 574, ifAdminStatu... [14:41:17] 10netops, 06Infrastructure-Foundations, 06SRE: drmrs: cr1-drmrs <-> asw1-b13-drmrs link down [Oct 2025] - https://phabricator.wikimedia.org/T407107#11268792 (10cmooney) p:05High→03Low [15:48:44] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140 (10taavi) 03NEW [15:49:14] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11269090 (10taavi) /cc @Andrew @cmooney [16:31:10] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11269245 (10cmooney) Thanks for the task @taavi ! Option 1 is nice and neat, but as you say there is a lot of work to do in terms... [17:40:43] FIRING: [4x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [17:45:43] FIRING: [10x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [17:55:43] RESOLVED: [10x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [19:32:28] 10Domains, 10DNS, 06SRE: Request to create the 25.wikipedia.org domain + 301 redirect to the org site - https://phabricator.wikimedia.org/T407156 (10SCampos-WMF) 03NEW [20:05:47] 10Domains, 10DNS, 06SRE: Request to create the 25.wikipedia.org domain + 301 redirect to the org site - https://phabricator.wikimedia.org/T407156#11269579 (10SCampos-WMF)