[00:02:03] 06Traffic, 10DNS, 13Patch-For-Review: Setting up Wikimedia Trust and Safety Help Center with Zendesk product: Seeking Guidance on host mapping - https://phabricator.wikimedia.org/T400952#11227536 (10BCornwall) @JAbrams: I have the code ready to go for both trustandsafety.wikimedia.support and committee.wikim... [08:02:02] 1 [08:02:12] 222 [08:21:59] 06Traffic, 06Data-Engineering: Request for a new request dataset for caching research - https://phabricator.wikimedia.org/T401331#11228010 (10yazhuoz) @GGoncalves-WMF That's great! Thanks for all the efforts. Please let me know if there is anything I can help with :) [14:15:41] 06Traffic, 06Data-Platform-SRE: Validate pybal config in CI - https://phabricator.wikimedia.org/T394789#11229562 (10bking) Thanks @ssingh , that's completely understandable. If PyBal is going away, it's probably not worth the effort to fix it. But I'm wondering if Liberica has the same issue? In other words,... [14:22:59] 06Traffic, 06Data-Platform-SRE: Validate pybal config in CI - https://phabricator.wikimedia.org/T394789#11229602 (10ssingh) >>! In T394789#11229562, @bking wrote: > Thanks @ssingh , that's completely understandable. If PyBal is going away, it's probably not worth the effort to fix it. > > But I'm wondering i... [14:30:14] 06Traffic: Move HTTP/1.0 requests rejections at HAProxy level - https://phabricator.wikimedia.org/T365456#11229630 (10Fabfur) Problem is that PyBal (twisted) [[ https://github.com/twisted/twisted/blob/twisted-18.9.0/src/twisted/web/http.py#L499 | defaults to a HTTP1.0 ]] client so healthchecks in eqiad|codfw wil... [14:30:49] 06Traffic: Move HTTP/1.0 requests rejections at HAProxy level - https://phabricator.wikimedia.org/T365456#11229634 (10Fabfur) 05In progress→03Stalled p:05Triage→03Low [14:49:51] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Move lvs1020 link from ssw1-f1-eqiad to ssw1-e1-eqiad - https://phabricator.wikimedia.org/T404959#11229706 (10VRiley-WMF) Hey @cmooney is there a good time to schedual this move? [15:35:35] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad: new structured cabling required for fr-tech expansion and row a/b switch refresh - https://phabricator.wikimedia.org/T402432#11229937 (10RobH) @wiki_willy, I'm not 100% sure on how to process this and I wanted to check with you... [16:05:45] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11230052 (10Jhancock.wm) cp2056 has had the card replaced and i've assigned the mgmt ip to it. if you needed a clean slate for any reason it's ready. [17:06:59] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Rollout Phase 3] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11230394 (10BCornwall) [17:16:09] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Rollout Phase 3] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11230427 (10Krinkle) [17:19:27] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11230434 (10Papaul) [17:23:01] 06Traffic, 06MediaWiki-Platform-Team (Radar): Report progress on rollout of unified mobile routing (Hadoop query) - https://phabricator.wikimedia.org/T405429#11230451 (10Krinkle) [17:23:19] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw:frack:rack/install/configuration new switches in rack F5 - https://phabricator.wikimedia.org/T405618#11230453 (10Papaul) [17:26:45] 06Traffic, 06MediaWiki-Platform-Team (Radar): [Pilot Rollout] Implement unified mobile routing and enable on wikitech.wikimedia.org - https://phabricator.wikimedia.org/T401595#11230464 (10Krinkle) [17:37:15] 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Rollout Phase 3] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11230520 (10BCornwall) [18:53:43] FIRING: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2033&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [19:03:43] RESOLVED: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2033&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [19:08:07] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad: new structured cabling required for fr-tech expansion and row a/b switch refresh - https://phabricator.wikimedia.org/T402432#11230861 (10wiki_willy) Hi @RobH - thanks for checking on this. Can you start off by sending me an ema... [20:05:35] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad: new structured cabling required for fr-tech expansion and row a/b switch refresh - https://phabricator.wikimedia.org/T402432#11231001 (10RobH) a:03wiki_willy @wiki_willy, I stand corrected, I figured out how to get this order... [21:16:14] 06Traffic, 06DC-Ops, 10ops-magru: planned power redundancy depreciation 2025-09-20 @ 18:00 GMT to 2025-09-21 @ 21:00 GMT - https://phabricator.wikimedia.org/T402818#11231299 (10RobH) 05Open→03Resolved a:03RobH