[01:31:32] 10Traffic, 10DC-Ops, 10Operations: poll power data for redeployment of esams/knams - https://phabricator.wikimedia.org/T225720 (10RobH) Ok, as it is now peak hours (according to @bblack) for eqiad, I'm re-pulling all the power data now. Please note that I'll update the task description AFTER this post (and... [04:57:50] 10Traffic, 10Operations, 10Performance-Team, 10Performance: Sometimes pages load slowly for European users (due to some factor outside of Wikimedia cluster) - https://phabricator.wikimedia.org/T226048 (10ema) >>! In T226048#5270181, @CDanis wrote: > My guess is that the beginning of this problem correlates... [05:35:15] 10Traffic, 10Operations, 10Performance-Team, 10Performance: Sometimes pages load slowly for European users (due to some factor outside of Wikimedia cluster) - https://phabricator.wikimedia.org/T226048 (10Wurgl) @ema It was not a single timeout. A chunk of data (the start of the page) was rendered by the br... [06:32:35] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Matrix, and 2 others: Configure wikimedia.org to enable *:wikimedia.org Matrix user IDs - https://phabricator.wikimedia.org/T223835 (10Tgr) https://wikimedia.org/.well-known/matrix/server works corrently. https://wikimedia.org/.well-known/matrix/client... [08:16:01] ema: let me know when the varnish patch is actually deployed [08:17:53] gilles: between 08:09 and 08:39 (but we can speed that up by forcing a puppet run if you prefer) [08:18:02] no, that's fine [08:18:15] ack [08:18:20] I just want to verify that it works as expected once it's out there [08:18:59] if you tell me the cache frontend you get chashed to, I can run puppet there [08:19:47] cp3049 [08:21:16] gilles: done, you can test now [08:22:32] ema: it works 👍 [08:22:43] yay! [08:22:47] 10Traffic, 10Operations, 10Performance-Team, 10media-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Gilles) [08:22:51] 10Traffic, 10Operations, 10Performance-Team, 10Patch-For-Review: Normalize thumbnail request URLs in Varnish to avoid cachebusting - https://phabricator.wikimedia.org/T216339 (10Gilles) 05Open→03Resolved [08:22:59] 10Traffic, 10Operations, 10Performance-Team, 10media-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Gilles) 05Stalled→03Open [08:23:57] 10Traffic, 10Operations, 10Performance-Team, 10media-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Gilles) @aaron your concern has been addressed now, the Varnish-level thumbnail URL normalization is live. We can now proceed... [08:25:37] 10Traffic, 10Operations, 10Performance-Team, 10media-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Gilles) @jijiki @fgiunchedi Have Swift proxies been restarted since https://gerrit.wikimedia.org/r/#/c/mediawiki/vagrant/+/489... [08:27:39] 10Traffic, 10Operations, 10Performance-Team, 10media-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10jijiki) @Gilles can it wait till next week when we will all be back, unless it is urgent, in which case we will figure it out [08:30:16] 10Traffic, 10Operations, 10Performance-Team, 10media-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Gilles) It can wait. Basically I want to figure out where we're at in regards to that patch, what's actually deployed and runn... [08:31:01] 10Traffic, 10Operations, 10Performance-Team, 10media-storage, and 2 others: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10jijiki) [08:33:47] gilles: https://gerrit.wikimedia.org/r/#/c/mediawiki/vagrant/+/489021/ does not seem to be applied to the puppet repo at all [08:34:20] ema: oh, my bad, I thought it was a production patch at a glance [08:34:24] I shall make one, then [08:35:09] 10Traffic, 10Operations, 10Performance-Team, 10media-storage, and 2 others: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Gilles) Nevermind, this was the Vagrant patch... I'm going to make the production one now [08:52:34] 10Traffic, 10Operations, 10Proton, 10Reading-Infrastructure-Team-Backlog, and 3 others: Document and possibly fine-tune how Proton interacts with Varnish - https://phabricator.wikimedia.org/T213371 (10Joe) [09:45:38] 10Traffic, 10Operations, 10Performance-Team, 10media-storage, and 2 others: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Gilles) [09:45:42] 10Traffic, 10Operations, 10Performance-Team, 10Patch-For-Review: Normalize thumbnail request URLs in Varnish to avoid cachebusting - https://phabricator.wikimedia.org/T216339 (10Gilles) 05Resolved→03Open [10:41:35] 10Traffic, 10Operations, 10Performance-Team, 10Performance: Sometimes pages load slowly for European users (due to some factor outside of Wikimedia cluster) - https://phabricator.wikimedia.org/T226048 (10Jheald) ^^ I saw exactly what Wurgl reports, browsing long pages on en-wiki (eg the Fram case) in Londo... [12:37:57] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team (Kanban): Set up a subdomain for Phame to enable caching - https://phabricator.wikimedia.org/T226044 (10mmodell) This seems like a good idea. The upstream documentation has a warning that there are some issues with an external blog / dedic... [13:58:18] 10Traffic, 10Operations, 10Performance-Team, 10media-storage, and 2 others: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Gilles) [13:58:22] 10Traffic, 10Operations, 10Performance-Team, 10Patch-For-Review: Normalize thumbnail request URLs in Varnish to avoid cachebusting - https://phabricator.wikimedia.org/T216339 (10Gilles) 05Open→03Resolved This has caused a spike of thumbor thumbnailing requests, by virtue of making some objects hotter t... [14:21:39] 10Traffic, 10Operations, 10Patch-For-Review: cp3041 - Varnish frontend child restarted icinga alert - https://phabricator.wikimedia.org/T224694 (10ema) 05Open→03Resolved a:03ema All cache nodes are currently running Varnish 5.1.3-1wm10, which fixes this. [14:29:48] 10Traffic, 10Operations, 10Performance-Team, 10Performance: Study performance impact of disabling TCP selective acknowledgments - https://phabricator.wikimedia.org/T225998 (10ema) All cache nodes are now running Linux 4.9.168-1+deb9u3. Next week we can thus re-enable SACKs on part of the cache fleet and fu... [16:19:00] 10Traffic, 10Gerrit, 10Operations: When downloading from git using HTTPS: HTTP 500 / GnuTLS recv error (-110) - https://phabricator.wikimedia.org/T225347 (10sbassett) Hello @Ciencia_Al_Poder - Apologies for the delay on a response to this issue. Due to an ongoing security incident [0], certain IP ranges co... [16:56:04] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Image thumbnail (cache?) broken on Wikimedia Commons, e.g. Information.svg, when viewing non-default resolution (e.g. 241px) - https://phabricator.wikimedia.org/T226271 (10Krinkle) [17:13:31] 10Traffic, 10Gerrit, 10Operations: When downloading from git using HTTPS: HTTP 500 / GnuTLS recv error (-110) - https://phabricator.wikimedia.org/T225347 (10Ciencia_Al_Poder) Ok, thanks for the update. I only need a read-only access to the repos, so I'd have to live with the github mirror... [19:27:12] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Image thumbnail (cache?) broken on Wikimedia Commons, e.g. Information.svg, when viewing non-default resolution (e.g. 241px) - https://phabricator.wikimedia.org/T226271 (10JJMC89) I deleted it and created protected the page as a... [19:38:35] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Image thumbnail (cache?) broken on Wikimedia Commons, e.g. Information.svg, when viewing non-default resolution (e.g. 241px) - https://phabricator.wikimedia.org/T226271 (10JJMC89) I reuploaded the file as [[ https://en.wikipedia... [19:49:19] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Image thumbnail (cache?) broken on Wikimedia Commons, e.g. Information.svg, when viewing non-default resolution (e.g. 241px) - https://phabricator.wikimedia.org/T226271 (10Xaosflux) Possibly related to T30299 ? I'd like to see... [19:56:31] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Image thumbnail (cache?) broken on English Wikipedia, e.g. Information.svg, when viewing non-default resolution (e.g. 241px) - https://phabricator.wikimedia.org/T226271 (10JJMC89) [19:56:42] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Image thumbnail (cache?) broken on English Wikipedia, e.g. Information.svg, when viewing non-default resolution (e.g. 241px) - https://phabricator.wikimedia.org/T226271 (10Krinkle) >>! In T226271#5274599, @Xaosflux wrote: > Poss... [20:21:48] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Image thumbnail (cache?) broken on English Wikipedia, e.g. Information.svg, when viewing non-default resolution (e.g. 241px) - https://phabricator.wikimedia.org/T226271 (10JJMC89) //(In case any of this is helpful to those inves... [20:35:15] 10Traffic, 10Gerrit, 10Operations: When downloading from git using HTTPS: HTTP 500 / GnuTLS recv error (-110) - https://phabricator.wikimedia.org/T225347 (10sbassett) 05Open→03Resolved