[07:06:42] <wikibugs>	 10Traffic, 10Commons, 10MediaWiki-File-management, 10Operations, 10SRE-swift-storage: Cached thumbnails and originals are sometimes not being purged correctly/quickly - https://phabricator.wikimedia.org/T256313 (10ema) 05Open→03Resolved a:03ema There's been no new report of stale images since the l...
[08:57:01] <wikibugs>	 10netops, 10Operations, 10observability, 10Patch-For-Review: Migrate role::netmon to Buster - https://phabricator.wikimedia.org/T247967 (10fgiunchedi)
[10:06:44] <wikibugs>	 10netops, 10Operations, 10observability, 10Patch-For-Review, 10User-fgiunchedi: Migrate role::netmon to Buster - https://phabricator.wikimedia.org/T247967 (10fgiunchedi)
[10:44:26] <wikibugs>	 10Wikimedia-Apache-configuration, 10Operations, 10Developer Productivity, 10Patch-For-Review, 10Performance-Team (Radar): VirtualHost for mod_status breaks debugging Apache/MediaWiki from localhost (on jobrunners) - https://phabricator.wikimedia.org/T190111 (10Joe) My take on this is that we should rathe...
[11:29:08] <wikibugs>	 10Wikimedia-Apache-configuration, 10Operations: Encoding discrepancy in https://wikipedia.org redirects - https://phabricator.wikimedia.org/T257608 (10Joe) I'm not sure the problem is what you think it is. wikipedia.org isn't even supposed to respond to such requests IIRC. I'll dig deeper.
[12:02:34] <wikibugs>	 10Traffic, 10Operations, 10serviceops, 10affects-Kiwix-and-openZIM: ETAG response headers not always with double-quotes - https://phabricator.wikimedia.org/T256217 (10ema) As it turns out, this is due to a deliberate bug in Swift:  https://bugs.launchpad.net/swift/+bug/1099087  > When swift was originally...
[12:07:34] <wikibugs>	 10Traffic, 10Operations, 10SRE-swift-storage, 10affects-Kiwix-and-openZIM: Swift sends ETAG without double-quotes - https://phabricator.wikimedia.org/T256217 (10ema)
[12:15:42] <wikibugs>	 10Traffic, 10Operations, 10SRE-swift-storage, 10affects-Kiwix-and-openZIM: Swift sends ETAG without double-quotes - https://phabricator.wikimedia.org/T256217 (10fgiunchedi) >>! In T256217#6322433, @ema wrote: > As it turns out, this is due to a deliberate bug in Swift:  https://bugs.launchpad.net/swift/+bu...
[12:34:05] <wikibugs>	 10Traffic, 10Operations, 10SRE-swift-storage, 10affects-Kiwix-and-openZIM: Swift sends ETAG without double-quotes - https://phabricator.wikimedia.org/T256217 (10ema) >>! In T256217#6322461, @fgiunchedi wrote: >>>! In T256217#6322433, @ema wrote: >> Openstack/swift 2.24.0 fixes the issue (or removes the fea...
[12:55:03] <cdanis>	 Traffic friends, do you think it would make sense to add me to your Monday syncup?  think about it, no worries either way :)
[12:56:27] <ema>	 definitely +1 from my side
[13:08:12] <vgutierrez>	 cdanis: bring 🍻
[13:08:25] <cdanis>	 11am local isn't too early I suppose
[13:15:18] <ema>	 cdanis: I wanted to reply https://isitbeeroclock.com/ but that's a 502 right now
[13:15:39] <ema>	 so apparently it isn't *always* beer o'clock, after all
[13:17:06] <cdanis>	 wow, that's very sad!
[13:17:10] <cdanis>	 now how will we know :(
[13:34:38] <vgutierrez>	 http://isbeeroclock.ynaas.com/
[13:34:39] <vgutierrez>	 :D
[13:34:55] <vgutierrez>	 yes or no as a service FTW
[13:35:36] <cdanis>	 the way that site is implemented still astounds me
[13:36:21] <vgutierrez>	 but it works better than isitbeeroclock.com apparently
[13:37:30] <Reedy>	 lol
[13:40:36] <wikibugs>	 10Traffic, 10Operations, 10SRE-swift-storage, 10affects-Kiwix-and-openZIM: Swift sends ETAG without double-quotes - https://phabricator.wikimedia.org/T256217 (10Kelson) @ema not really this is case which had to be handled in MWoffliner. This is all.
[13:43:38] <wikibugs>	 10Traffic, 10Operations, 10SRE-swift-storage, 10affects-Kiwix-and-openZIM: Swift sends ETAG without double-quotes - https://phabricator.wikimedia.org/T256217 (10ema) p:05Medium→03Low
[13:44:23] <wikibugs>	 10Traffic, 10Operations, 10SRE-swift-storage, 10affects-Kiwix-and-openZIM: Swift sends ETAG without double-quotes - https://phabricator.wikimedia.org/T256217 (10ema) @Kelson: alright, this will be solved with the upgrade to Bullseye then. Thank you!
[14:19:48] <wikibugs>	 10Wikimedia-Apache-configuration, 10Operations: Encoding discrepancy in https://wikipedia.org redirects - https://phabricator.wikimedia.org/T257608 (10Joe) The problem presents even if we start with `www.wikipedia.org`:  ` $ curl -I -L "https://www.wikipedia.org/wiki/Special:Redirect/file/(61-365)_Can_you_imag...
[14:27:51] <wikibugs>	 10Wikimedia-Apache-configuration, 10Operations, 10serviceops: Encoding discrepancy in https://wikipedia.org redirects - https://phabricator.wikimedia.org/T257608 (10Joe) p:05Triage→03Low
[14:39:07] <ejegg>	 Hi traffic folks - we started serving the donation thank you page at a wikipedia subdomain (donate.wikipedia.org) without testing on mobile first unfortunately
[14:40:06] <ejegg>	 and it looks like all donate.wikipedia.org/wiki/Thank_You requests from mobile get redirected to donate.M.wikipedia.org/wiki/Thank_You
[14:40:44] <ejegg>	 while the donate.wikimedia.org/wiki/Thank_You page does not redirect
[14:41:19] <ejegg>	 both are being served by the same apache site configuration and donatewiki LocalSettings
[14:41:43] <ejegg>	 so I imagine that redirect is handled by some *.wikipedia.org redirect
[14:42:02] <ejegg>	 can we make an exception for donate.wikipedia.org?
[14:42:29] <ejegg>	 Or would it be better to make that same site also listen on donate.m.wikipedia.org?
[14:44:00] <bblack>	 we could pursue either (especially if it were an actual mobile-friendly variant in the latter case)
[14:44:57] <ejegg>	 bblack since we never used a variant URL for the pages served from donate.wikiMedia.org, I think it's OK to just stop redirecting
[14:45:39] <ejegg>	 I'm pretty sure pcoombe has used done all the work in CSS media queries to make the pages mobile friendly
[14:45:45] <bblack>	 ok
[14:46:27] <bblack>	 ejegg: I think https://gerrit.wikimedia.org/r/c/operations/puppet/+/615229/ will fix it
[14:46:41] <bblack>	 but I have a meeting start shortly, and maybe someone else familiar with VCL should double-check
[14:46:51] <cdanis>	 bblack: I can take this on
[14:46:55] <ejegg>	 ahh, it's in the varnish cache
[14:46:55] <cdanis>	 I'll write a VTC too
[14:47:01] <ejegg>	 thanks so much bblack and cdanis!
[14:47:03] <bblack>	 cdanis: awesome, thanks
[14:47:05] <cdanis>	 ejegg: yeah, a lot of request rewriting/routing logic is in there
[14:47:59] <ejegg>	 ok, i see, and that line is a list of the exceptions to the .m. redirect
[14:48:19] <ejegg>	 well, the ^((?!etc part at the start
[14:49:27] <ejegg>	 TIL we had a 9/11 wiki
[15:19:58] <wikibugs>	 10Traffic, 10Discovery, 10Operations, 10Wikidata, and 3 others: Wikidata maxlag repeatedly over 5s since Jan 20, 2020 (primarily caused by the query service) - https://phabricator.wikimedia.org/T243701 (10ArthurPSmith) Something seems to be going on very recently that's a different pattern - did something...
[15:37:35] <wikibugs>	 10netops, 10Operations, 10ops-eqsin: cr3-eqsin disk 1 failure - https://phabricator.wikimedia.org/T257154 (10RobH)
[15:39:14] <wikibugs>	 10HTTPS, 10Traffic, 10Operations, 10Pywikibot: Configure HTTPS for pywikibot.org - https://phabricator.wikimedia.org/T257537 (10ema) p:05Triage→03Medium
[16:19:03] <cdanis>	 btw, I believe text/21-sec-warning.vtc fails at HEAD
[16:19:09] <cdanis>	 for now I will ignore that
[16:19:24] <bblack>	 vgutierrez: ^
[16:20:27] <cdanis>	 (I finally got running VTC tests working on my machine again, after spending an embarrassing amount of time realizing that I needed to `vagrant destroy` because I had an ancient stretch image that didn't have `python3-jenkinsapi` available, after first figuring out that the error message about the package not being installed was coming from inside the vagrant machine, and not mine where it clearly
[16:20:29] <cdanis>	 was installed...)
[16:26:26] <cdanis>	 hmm maybe that isn't true
[16:26:53] <vgutierrez>	 hmm ema tested it this EU morning on cp3050 and everything was happy
[16:27:18] <cdanis>	 yeah I just tested it w/ cp3052 and using an unrelated patchset and no modifications in my checkout
[16:27:30] <cdanis>	 and it failed
[16:27:45] <cdanis>	 but now it succeeds on 615229
[16:35:03] <cdanis>	 could I trouble someone for a review of https://gerrit.wikimedia.org/r/c/operations/puppet/+/615229 ?
[16:48:47] <cdanis>	 ema: vgutierrez: either of you still around for the day? if not no worries
[16:52:46] <bblack>	 I can take a peek 
[16:53:48] <bblack>	 looks legit visually
[16:53:57] <bblack>	 what was different when VTC failed?
[16:54:09] <bblack>	 cdanis: ^
[17:01:19] <cdanis>	 nothing
[17:01:22] <cdanis>	 and it was a different VTC
[17:07:29] <cdanis>	 ejegg: okay, that patch is merged, so the redirects should stop happening after the next half hour :)
[17:27:19] <ejegg>	 thanks cdanis !
[17:27:29] <ejegg>	 and thanks again bblack 
[17:27:30] <cdanis>	 np!
[19:05:27] <wikibugs>	 10Traffic, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Operations, and 2 others: TY pages in a subdomain of wikipedia and set hide banner cookie - https://phabricator.wikimedia.org/T251780 (10DStrine) 05Open→03Resolved
[20:07:07] <wikibugs>	 10Traffic, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Operations, and 2 others: TY pages in a subdomain of wikipedia and set hide banner cookie - https://phabricator.wikimedia.org/T251780 (10Krinkle) Serving the same wiki through multiple hostnames is wholly unsupported in MediaWiki and...