[03:40:41] 06Traffic, 06Commons, 06MediaWiki-Media-Platform-Team: SVG thumbnail downloads as WebP instead of PNG in Chrome browsers - https://phabricator.wikimedia.org/T443394#12514141 (10Krinkle) 05Invalid→03Open It may be worth skipping the WebP optimization on `?download` requests (ref T442313). [06:34:35] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12514245 (10ayounsi) 05Resolved→03Open Looks like `ssw1-a1-codfw` needs the fix too... [08:38:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-esams, 06SRE: esams router and switch maint - https://phabricator.wikimedia.org/T440349#12514462 (10ayounsi) [09:07:49] Morning, MPEG-DASH is enabled in testwiki. You can play with it in https://test.wikipedia.org/w/index.php?title=User:Ladsgroup/sandbox&oldid=782822 you see chunks like https://upload.wikimedia.org/wikipedia/test/transcoded/7/72/BBH_gravitational_lensing_of_gw150914_-_local.webm/chunk-2-00007.3x1080p.vp9.webm and [09:07:49] https://upload.wikimedia.org/wikipedia/test/transcoded/7/72/BBH_gravitational_lensing_of_gw150914_-_local.webm/chunk-0-00002.3x1080p.vp9.webm note that 108p doesn't mean it's 1080p, it's 0th in the ladder (360p) [09:08:47] The files here are 2.4MB. Chunk size is 4s. That should make most (if not all) videos cacheable on the edge. [09:19:35] \o/ [09:22:02] Amir1: player seems quite aggressive fetching chunks [09:22:58] it's standard library :D [09:23:17] if fetched 6 chunks in 0.8 seconds [09:23:32] wondering if this could trigger some rate limits [09:23:51] cause those are still cache misses for requestctl/varnish [09:58:24] vgutierrez: if you got time I was hoping to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1361867/2 [09:58:51] I spoke to Sukhe about it yesterday he thought it best to run by you too before doing any work [09:58:56] sure [09:59:09] it's annoying doing it this close to being finished, but it's the easier way forward to unblock our network migration [09:59:37] the task lists the steps I'd like to do, merge that, test lvs1020 looks ok, fail over to it, then repeat with the similar patch for lvs1019 [10:00:35] that sounds good to me [10:01:30] I can take care of it I think. When it comes to failing over from lvs1019 -> lvs1020 what is the best approach? [10:01:49] in the past I think we've done a systemctl stop pybal.service, not sure if that's still the best way? [10:01:58] after stopping puppet and downtiming the host [10:06:49] vgutierrez: What do you think of setting a different rate limit for chunks? [10:08:59] it should be easy to detect those given they have the /chunk- in the uri path [10:12:47] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12514701 (10ops-monitoring-bot) Host lvs1020.eqiad.wmnet rebooted by cmooney@cumin1004 with reason: apply vlan changes on lvs1020 [13:02:05] 06Traffic: trafficserver: remove support for running multiple instances - https://phabricator.wikimedia.org/T443596 (10dkertesz) 03NEW [13:34:16] 06Traffic, 06MediaWiki-Core-Platform-Team (Kanban): Decide on how to serve ESI fragments - https://phabricator.wikimedia.org/T430330#12515593 (10pmiazga) 05Open→03Resolved [13:38:12] 06Traffic, 06MediaWiki-Core-Platform-Team (Kanban): Decide on how to serve ESI fragments - https://phabricator.wikimedia.org/T430330#12515633 (10pmiazga) The decision is tu use the `rest.php` and introduce an internal REST module called `fragment`. On the edge, we will do URL rewriting; MediaWiki should re... [14:03:46] topranks: yes that works, thanks [14:04:02] disable puppet, stop pybal and downtime [14:06:56] hi, I'm a WMF employee working on a new project that is a microsite promoting the Wikipedia mobile apps. as part of the microsite, we would like to show a preview of the Nearby tab in the app which shows articles on a map (the site won't actually geolocate the user--we're using sample points). We'd like to use Wikimedia Maps for the map tiles. [14:07:52] it looks like we need to get our dev site (https://mobile-app-microsite-dev.toolforge.org/) on an allowlist, and would also like to set it up so that when we go to prod, that is allowed as well. could someone share what the process for this is? [14:08:17] the dev site doesn't have the map yet, FYI--this issue was spotted in code review [14:12:20] cklimas: toolforge should already in our allow-list, per the VCL config [14:12:35] # Then WMCS domains [14:12:36] |wmflabs\.org|toolforge\.org|wmcloud\.org [14:12:47] and subdomains are already included [14:13:07] ah, cool. sorry, I assumed because when we tested off of localhost, it failed. [14:13:24] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/varnish/templates/upload-frontend.inc.vcl.erb#86 [14:13:35] this then gets templated to the the line above, so it should cover it [14:13:41] no worries, if it still does not work, please reach out to us [14:14:28] we plan for the prod site to live on a subdomain of wikipedia.org. I'm hoping that <%= @wikimedia_domains_regex %> in the config would cover that? [14:14:52] yes that should cover it. for external domains not covered, the process involves filing a task [https://wikitech.wikimedia.org/wiki/Maps/External_usage] and then getting approval [14:15:02] (just in case you ever need an external one :) [14:15:10] awesome, thank you! [14:15:28] but if for whatever reason it does not work, please file a phab task and tag "Traffic" [14:16:08] will do [14:42:26] 10netops, 06Infrastructure-Foundations: eqsin: upgrade routers (2026) - https://phabricator.wikimedia.org/T416563#12516074 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=74ccf2e5-5748-4b05-8ce2-227a2efb7716) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:02:23] 06Traffic, 13Patch-For-Review: trafficserver: remove support for running multiple instances - https://phabricator.wikimedia.org/T443596#12516288 (10dkertesz) just to be sure we're not leaving breadcrumbs behind, I ran `sudo cumin 'A:cp' 'ls -ld /srv/trafficserver'` and got all negative [16:42:52] 06Traffic, 10Lift-Wing, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 06Machine-Learning-Team (Q2 FY2026-27): Migrate inference and inference-staging to IPIP - https://phabricator.wikimedia.org/T441352#12517369 (10Gehel) [16:44:31] 06Traffic, 10Lift-Wing, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work, 06Machine-Learning-Team (Q2 FY2026-27): Migrate inference and inference-staging to IPIP - https://phabricator.wikimedia.org/T441352#12517395 (10Gehel) p:05Triage→03Medium [16:58:10] 06Traffic: Handle exception raised when OCSP response is an error - https://phabricator.wikimedia.org/T435104#12517606 (10BCornwall) 05Open→03Invalid Closing as duplicate of T441025 [16:58:24] 06Traffic: Remove OCSP support from acme-chief - https://phabricator.wikimedia.org/T441025#12517611 (10BCornwall) a:05CDobbins→03BCornwall [17:13:22] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12517784 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=df199593-4830-4db0-8b30-ba2e71011704) set by cmooney@cum... [18:13:48] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (non-WMF native apps treated as bot-traffic) - https://phabricator.wikimedia.org/T413570#12518446 (10ssingh) @Nylki: Is there anything pending from your end on this ticket? If not, I will go ahead and boldly resolve it, since I think the validat... [19:45:39] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687 (10cmooney) 03NEW p:05Triage→03High [19:46:15] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535#12518898 (10cmooney) [19:46:16] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12518897 (10cmooney) [20:38:50] 06Traffic, 10FY26-27 ST6.4 - Logged-in performance, 06MediaWiki-Core-Platform-Team: Implement Varnish URL Rewriting for Internal ESI Fragment Endpoints - https://phabricator.wikimedia.org/T443703 (10pmiazga) 03NEW [20:40:34] 06Traffic, 10FY26-27 ST6.4 - Logged-in performance: ESI: Enforce public blocking of both /esi and internal fragment rotes - https://phabricator.wikimedia.org/T443704 (10pmiazga) 03NEW [20:44:37] 06Traffic, 10FY26-27 ST6.4 - Logged-in performance, 06MediaWiki-Core-Platform-Team: Implement Varnish URL Rewriting for Internal ESI Fragment Endpoints - https://phabricator.wikimedia.org/T443703#12519234 (10pmiazga) @Catrope, before we jump into this topic, I wanted to check that the rewriting rules ( ADR-6... [21:28:33] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (non-WMF native apps treated as bot-traffic) - https://phabricator.wikimedia.org/T413570#12519569 (10Nylki) @ssingh please do so, thank you. [21:35:31] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (non-WMF native apps treated as bot-traffic) - https://phabricator.wikimedia.org/T413570#12519587 (10ssingh) 05Open→03Resolved a:03ssingh