[00:22:18] 06Traffic, 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Fix Hive event.development_network_probe table - https://phabricator.wikimedia.org/T400360#11257642 (10Ahoelzl) [07:47:31] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: cr2-eqiad: fan failure on left tray [Oct 2025] - https://phabricator.wikimedia.org/T406554#11258314 (10Jclark-ctr) a:05VRiley-WMF→03Jclark-ctr [08:30:14] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Move lvs1020 link from ssw1-f1-eqiad to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T404959#11258496 (10cmooney) >>! In T404959#11255300, @VRiley-WMF wrote: > Okay, was looking at this issue a bit. There are currently two fibe... [13:54:38] I'd like to start the group2-minus-enwiki rollout for rest-gateway rest.php once the backport window is done. any objections to me doing the usual disable-puppet dance briefly? [13:55:36] hnowlan: vg and fabrizio were rolling out a utf-8 change [13:55:44] fabfur: ^ all done on that I think? [13:56:24] no problem for me, puppet has been enabled again on all cp hosts some times ago [14:06:23] great, thanks! [15:42:10] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE: Create boot environment of Bullseye with a 6.1 kernel - https://phabricator.wikimedia.org/T405102#11260770 (10MoritzMuehlenhoff) I built a bullseye d-i environment with the Linux 6.1 from Debian LTS (https://tracker.debian.org/pkg/linux-6.1) and after some needed fi... [15:54:02] doing one last A:cp puppet dance fyi [16:00:43] undoing it now [16:01:55] Thursday @ 6pm.. that's tight :D [16:02:17] it's only enwiki, how big could it be [16:02:29] :D [16:02:33] I'm staying around to babysit, not moving it past 10% [16:03:12] sukhe: https://phabricator.wikimedia.org/T405102#11260770 :( [16:03:35] interesting-but-unsurprising: rest.php for everything but enwiki barely cracks 200 rps per dc https://grafana.wikimedia.org/goto/3pFKxI6Hg?orgId=1 [16:04:08] elukey: yeah thanks, discussing with moritzm as well [16:04:12] will pick it up in a bit [16:28:56] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11261019 (10Jhancock.wm) @elukey license uploaded for cp2056. should be good to try that one again. [16:43:43] 06Traffic, 10Beta-Cluster-Infrastructure: Copy the Traffic team on alerts for deployment-cache* hosts - https://phabricator.wikimedia.org/T406650#11261114 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [17:30:51] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Move lvs1020 link from ssw1-f1-eqiad to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T404959#11261370 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=366dc32d-e9d7-437c-98a7-7a9cd7979655) set by cmooney@cumi... [17:48:18] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad: 2 VM request for hCaptcha - https://phabricator.wikimedia.org/T406166#11261445 (10ssingh) 05Open→03Resolved a:03ssingh Rolled out. [18:04:43] FIRING: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3067&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [18:09:43] FIRING: [13x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [18:19:43] FIRING: [13x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [18:24:43] RESOLVED: [13x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [18:35:07] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Move lvs1020 link from ssw1-f1-eqiad to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T404959#11261706 (10cmooney) 05Open→03Resolved Link has been moved, port is up on ssw1-e1-eqiad and MACs learnt on all vlans: ` cmoone... [18:36:55] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Traffic host migrations - https://phabricator.wikimedia.org/T405623#11261714 (10RobH) So I'm going to outline a few assumptions here and steal this back. If any of the following assumptions are incorrect, please let me know. We now have a move ahead... [18:36:57] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Traffic host migrations - https://phabricator.wikimedia.org/T405623#11261715 (10RobH) a:05BCornwall→03RobH [18:40:52] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Traffic host migrations - https://phabricator.wikimedia.org/T405623#11261725 (10RobH) [21:44:43] FIRING: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3073&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [21:54:43] RESOLVED: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3073&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages