[07:22:52] 06Traffic, 06SRE, 13Patch-For-Review: WE5.2.13 Dumps UA enforcement - https://phabricator.wikimedia.org/T427836#12063820 (10SLyngshede-WMF) 05In progress→03Resolved User-Agent check as been deployed. [08:51:53] 10netops, 06Infrastructure-Foundations, 06SRE: Blackbox probe for TLS cert expriy failing on multiple eqiad SR-Linux nodes - https://phabricator.wikimedia.org/T429242#12064314 (10ayounsi) 05Resolved→03Open Alerts are back for eqiad C/D (+spines) - https://alerts.wikimedia.org/?q=scope%3Dnetwork&q=alertna... [10:04:20] 06Traffic, 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12064615 (10Clement_Goubert) [10:07:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12064630 (10cmooney) p:05Medium→03High [10:19:07] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations: bird bfd session with 172.20.1.1 down - Bad packet from 172.20.1.1 - unknown session id - https://phabricator.wikimedia.org/T427202#12064697 (10cmooney) 05Open→03Resolved a:03cmooney Gonna close this one for now, if it be... [10:47:54] 06Traffic: Test HAProxy 3.2 with AWS-LC libraries - https://phabricator.wikimedia.org/T419825#12064864 (10Ladsgroup) I'm curios too. If you don't mind sharing with me as well 👀 [12:13:09] 06Traffic, 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12065080 (10isarantopoulos) Thanks for all the work here! Regarding the OOM errors: since we’re using 2x24 GB partitions, shouldn’t the... [12:55:06] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux: applying analytics-in acl to irb sub-interface blocks ARP - https://phabricator.wikimedia.org/T429499#12065297 (10cmooney) FWIW this behaviour is not evident on SR-Linux v25.10.1 (tested on lswtest-d8-eqiad). It seems to be yet another 24.x SR-Linux b... [13:06:22] 06Traffic, 06Content-Transform-Team, 10Maps, 06SRE, 07affects-Kiwix-and-openZIM: Wikipedia wikis have broken maps URLs in infobox: "Bad GeoJSON - unknown \"type\" property \"ExternalData\"" - https://phabricator.wikimedia.org/T424046#12065358 (10Jgiannelos) I debugged this example: > Article URL: https:... [13:33:17] 06Traffic, 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12065527 (10gkyziridis) ==== Update ==== Qwen3.6-27B-FP8 KServe Deployment — Blocking Issue The Problem I parsed is that even the logs o... [13:41:25] 10netops, 06Traffic, 06Data-Persistence, 06Data-Platform-SRE, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12065635 (10Papaul) [13:48:36] 10netops, 06Traffic, 06Data-Persistence, 06Data-Platform-SRE, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12065694 (10Papaul) [13:51:39] 10netops, 06Traffic, 06Data-Persistence, 06Data-Platform-SRE, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12065734 (10Papaul) [13:54:06] 10netops, 06Infrastructure-Foundations, 06SRE: Blackbox probe for TLS cert expriy failing on multiple eqiad SR-Linux nodes - https://phabricator.wikimedia.org/T429242#12065756 (10cmooney) >>! In T429242#12064314, @ayounsi wrote: > Alerts are back for eqiad C/D (+spines) - https://alerts.wikimedia.org/?q=scop... [15:47:59] 06Traffic, 06SRE, 13Patch-For-Review: WE5.2.13 Dumps UA enforcement - https://phabricator.wikimedia.org/T427836#12066761 (10HCoplin-WMF) Thank you!! Really appreciate it :) [16:00:50] 06Traffic, 06MediaWiki-Platform-Team (Kanban Board): Decide on how to serve ESI fragments - https://phabricator.wikimedia.org/T430330#12066872 (10BCornwall) [16:03:22] 06Traffic, 06MediaWiki-Platform-Team (Kanban Board): Decide on how to serve ESI fragments - https://phabricator.wikimedia.org/T430330#12066884 (10BCornwall) There was an additional patch proposed: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1305243 - Do we want this new messaging? [16:05:43] 06Traffic, 06SRE, 13Patch-For-Review: WE5.2.13 Dumps UA enforcement - https://phabricator.wikimedia.org/T427836#12066894 (10BCornwall) There was an additional patch proposed: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1305243 - Do we want this new messaging? [16:32:01] 06Traffic, 06SRE, 13Patch-For-Review: WE5.2.13 Dumps UA enforcement - https://phabricator.wikimedia.org/T427836#12067027 (10HCoplin-WMF) oh -- yes please! @BTullis said he would merge it sometime today, but whoever gets to it first works for me. [16:43:49] 06Traffic, 06SRE, 13Patch-For-Review: WE5.2.13 Dumps UA enforcement - https://phabricator.wikimedia.org/T427836#12067095 (10BCornwall) Not sure why pcc is no-op-ing on both clouddumps but I merged anyway since it's just an HTML change. [16:44:24] brett: btw, ^ that is unfortunately just How It Works if puppet is managing a whole directory recursively [16:44:38] fantastic [16:44:41] I love it [17:01:26] 06Traffic, 06SRE, 13Patch-For-Review: WE5.2.13 Dumps UA enforcement - https://phabricator.wikimedia.org/T427836#12067176 (10HCoplin-WMF) Great! Thank you, again. I see the changes on the site now. Really appreciate the support on this :) [19:16:53] Hi, I'm sure your plate is not overflowing with work, so would you like some dessert? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306395 [19:20:43] brett can take this as entree even [19:20:48] he loves VCL [19:23:45] the same way Manuel loves s4 database? [19:26:03] I need to be afk for two hours now, afterwards we can deploy it if they are around [19:26:08] (and have capacity) [19:47:01] looking! [22:00:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo: ULSFO: OOB IPV6 down - https://phabricator.wikimedia.org/T430599 (10Papaul) 03NEW [22:00:14] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo: ULSFO: OOB IPV6 down - https://phabricator.wikimedia.org/T430599#12068267 (10Papaul) p:05Triage→03Medium [22:51:27] brett: Hi, I can't find a documentation on gradual roll out of a vcl change. Do you have it handy or do you want to deploy the change yourself? I'd be grateful either way [23:04:55] Amir1: hm, not sure that we have such a thing. Closest would be https://wikitech.wikimedia.org/wiki/CDN#Site_deployment_ordering but that doesn't say the how [23:05:53] it's basically, disable puppet fleetwide, merge, run-puppet-agent on a random host, and make sure it's fine. Then do a rolling run-puppet-agent I guess [23:05:58] is that okay if I do it? [23:05:58] yep [23:06:06] be my guest :) [23:06:10] cool cool [23:06:36] (and then re-enable puppet on every host ofc) [23:08:09] yup [23:40:34] 10netops, 06Traffic, 06Data-Persistence, 06Data-Platform-SRE, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12068484 (10BCornwall) What is the duration of the maintenance? [23:50:43] 06Traffic, 07Sustainability (Incident Followup): Ensure the pre-repooling checklist includes to restart liberica services whenever realserver IPs has changed - https://phabricator.wikimedia.org/T426299#12068502 (10BCornwall) 05In progress→03Resolved Given that I've reorg'ed/cleaned up the general liber...