[12:43:14] 10Traffic, 10Operations, 10Technical-blog-posts: 3rd part of blog post series: the evolution of Wikimedia's Content Delivery Network - https://phabricator.wikimedia.org/T270074 (10ema) >>! In T270074#6699924, @srodlund wrote: > @ema I published this Thanks! > Will you look it over and let me know if you s... [14:28:29] 10Traffic, 10MediaWiki-Docker, 10Operations, 10serviceops, and 2 others: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10ema) 05Open→03Resolved >>! In T270270#6699809, @ema wrote: > I think we can revert the no-s... [14:32:47] jayme: phew, this was fun ^ :) [14:33:30] * jayme reading [14:33:51] the best thing is that it all started with: [14:33:55] > you maybe got a minute to talk about T270270 ? [14:33:55] T270270: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 [14:34:16] a varnish minute [14:34:44] Well, tbh it probably all started with someone at docker not reading HTTP RFC :) [14:37:46] ema: thanks for the write up and for the fix ofc. I like how you described what you did to figure out (as I was lost there) [14:50:04] I think the docker client is innocent at the end of the day, they just expect to get Content-Length with a HEAD request [14:51:45] this has been a fun one to follow, and also a fun one to be happy about not needing to troubleshoot it personally :D [14:54:02] one bit of fun you missed while I was trying to reproduce with kostajh: for some reason his requests were occasionally going out with IPv4 and other times with IPv6, so he wasn't always c-hashed to the same node [14:54:17] fantastic [14:54:42] as always I remain amazed that anything ever works at all ;) [14:54:48] seriously [14:54:51] but isn't content-lenght a non-mandatory field after all? [14:55:16] s/field/header/ [14:57:15] a pragmatic interpretation might be that ideally the HEAD should return either CL or CE:chunked [14:57:25] but I'm not sure that either is strictly always required [14:57:52] the gospels say that payload header fields MAY be omitted [14:58:07] https://tools.ietf.org/html/rfc7231#section-4.3.2 [15:00:02] ha! :) [15:02:01] In case that got missed: We where not the only ones bitten by that change in docker: https://github.com/goharbor/harbor/issues/13740 [15:04:39] side project for the holidays: markov chain coming up with README.md files for CNCF repos [15:05:48] open source trusted cloud native registry project role foundation policy slack api [15:05:59] mixing and matching these we're gonna go far ^ [15:07:03] :D + scalable distributed [15:07:30] :) [15:07:38] 10Traffic, 10MediaWiki-Docker, 10Operations, 10serviceops, and 2 others: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10hashar) Thank you @ema for the full explanation (and for the fix of course)! [15:07:45] I'm gonna write an AI that predicts from the README.md whether the dockerfile is exploitable or compromised. [15:08:11] [the hidden internal implementation will just run a visual spinner for a while as if it's "thinking" and then "return true"] [15:08:15] ema: thank you so much to have taken the time to write a detailed resolution report for the docker pull issue! [15:09:09] hashar: yw, I was taking notes due to early symptoms of senility anyways :) [15:09:30] bblack: make sure to add a link to donate.wikimedia.org to let folks opt-in for premium scanner with granting faster reports (well that might be illegal though) [15:09:57] ema: I really love when people give a nice explanation at the end of a task. Definitely better than "ok fixed." :] [15:10:59] bblack: https://lobste.rs/s/dipypa/51_4m_docker_images_have_critical is fun [15:13:39] only? I would have though more :D [15:15:17] volans: *critical* is the keyword [15:15:30] Non-vulnerable images: 20% :D [15:15:34] lol [15:31:01] it always comes down to point of view and definitiosn though [15:31:28] if you think the best purpose of the internet is to increase the supply of cryptocurrency, many of these may be features rather than problems :) [15:32:13] [and if you think the best purpose of the internet is converting people into mindless non-contributory consumers, MediaWiki might be an exploit software that needs squashing!] [16:53:11] 10HTTPS, 10Traffic, 10Diff-blog, 10Operations: Send HSTS header on diff.wikimedia.org - https://phabricator.wikimedia.org/T270034 (10RLazarus) a:03RLazarus Emailed Comms about it, will route this appropriately when I hear back. [16:55:58] 10Traffic, 10Operations, 10Technical-blog-posts: 3rd part of blog post series: the evolution of Wikimedia's Content Delivery Network - https://phabricator.wikimedia.org/T270074 (10srodlund) @ema these should all be fixed now. :-) I'll send out an announcement today. [17:11:14] 10HTTPS, 10Traffic, 10Diff-blog, 10Operations: Send HSTS header on diff.wikimedia.org - https://phabricator.wikimedia.org/T270034 (10RLazarus) a:05RLazarus→03Varnent Thanks @Varnent for offering to look at this, as our primary contact with VIP. It turns out two other VIP-hosted domains, techblog.wikime... [17:11:44] 10HTTPS, 10Traffic, 10Diff-blog, 10Operations: Send HSTS header on all VIP-hosted domains - https://phabricator.wikimedia.org/T270034 (10RLazarus) p:05Triage→03Medium [19:42:14] It looks like one of my old test sites has vanished due to some refactors; I'm hoping that https://gerrit.wikimedia.org/r/c/operations/puppet/+/650587 is all I need to get it back. Would appreciate a review if anyone is around. [19:42:28] And if the answer is 'this is much more complicated' then I might just skip it for now [19:47:51] pinging cdanis since he was in a helpful mood earlier ^ [19:49:38] oh dang, earlier may have been months ago due to bouncer confusion [20:07:30] 10Traffic, 10Operations: Image fails to load with CORS violation - https://phabricator.wikimedia.org/T270209 (10RLazarus) I can't repro the CORS issue exactly, but I am getting a 503 from Varnish for `https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/Circuit_de_la_Sarthe_track_map.svg/2880px-Circuit_de... [20:08:31] 10Traffic, 10Operations: Image fails to load with CORS violation - https://phabricator.wikimedia.org/T270209 (10RLazarus) >>! In T270209#6702440, @RLazarus wrote: > only on the 2880px- URL -- the smaller ones work fine. Meant to add -- that's also the reason for this: >>! In T270209#6697248, @RoySmith wrote... [20:20:59] 10Traffic, 10Operations: Image fails to load with CORS violation - https://phabricator.wikimedia.org/T270209 (10RoySmith) Yeah, I can repro that here. On the command line: curl -v https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/Circuit_de_la_Sarthe_track_map.svg/2560px-Circuit_de_la_Sarthe_track_ma... [21:13:39] 10Traffic, 10Operations: Set CORS headers on error pages? - https://phabricator.wikimedia.org/T270526 (10RLazarus) p:05Triage→03Medium [21:13:56] 10Traffic, 10Operations: Set CORS headers on error pages? - https://phabricator.wikimedia.org/T270526 (10RLazarus) [21:53:31] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10Thumbor: Thumbnail rendering of complex SVG file leads to Error 500 or Error 429 instead of Error 408 - https://phabricator.wikimedia.org/T226318 (10AntiCompositeNumber)