[01:27:53] 10Wikimedia-Apache-configuration: Titles containing a dot give HTTP 403 when accessed using action=raw and the short URL form - https://phabricator.wikimedia.org/T126183#2007056 (10TTO) 3NEW [01:30:58] 10Traffic, 10Wikimedia-Apache-configuration, 6operations, 5Patch-For-Review: wikiversity.org and wikinews.org redirects to /503.html - https://phabricator.wikimedia.org/T109226#2007064 (10TTO) Is this fixed now? The links in the task description seem to work correctly. [01:36:45] 10Wikimedia-Apache-configuration: Titles containing a dot give HTTP 403 when accessed using action=raw and the short URL form - https://phabricator.wikimedia.org/T126183#2007065 (10TTO) ...unless something's changed since 1.26.2: https://translatewiki.net/wiki/Alt._BGN_Series?action=raw I don't have a local MW... [01:46:57] 10Wikimedia-Apache-configuration: Titles containing a dot give HTTP 403 when accessed using action=raw and the short URL form - https://phabricator.wikimedia.org/T126183#2007067 (10Krenair) This comes from WebRequest::checkUrlExtension [02:00:15] 10Traffic, 10Wikimedia-Apache-configuration, 6operations, 5Patch-For-Review: wikiversity.org and wikinews.org redirects to /503.html - https://phabricator.wikimedia.org/T109226#2007088 (10Krenair) Sounds like the bug only appears when HHVM is broken and a request to wikiversity.org returns a 503 and gets c... [02:16:56] 10Wikimedia-Apache-configuration: Titles containing a dot give HTTP 403 when accessed using action=raw and the short URL form - https://phabricator.wikimedia.org/T126183#2007109 (10Krenair) ShoutWiki does not have this issue because they run nginx, which is not listed in IEUrlExtension::haveUndecodedRequestUri [02:46:34] 10Wikimedia-Apache-configuration: Titles containing a dot give HTTP 403 when accessed using action=raw and the short URL form - https://phabricator.wikimedia.org/T126183#2007110 (10TTO) It looks as though the IEUrlExtension code is there to prevent XSS attacks in IE6 and earlier. Is it worth keeping this securit... [04:25:11] 10Traffic, 10Wikimedia-Apache-configuration, 6operations, 5Patch-For-Review: wikiversity.org and wikinews.org redirects to /503.html - https://phabricator.wikimedia.org/T109226#2007137 (10BBlack) @Krenair - basically, the problem is that our Apache configuration has some implementation bugs with how 301 re... [04:35:58] 10Wikimedia-Apache-configuration: Titles containing a dot give HTTP 403 when accessed using action=raw and the short URL form - https://phabricator.wikimedia.org/T126183#2007155 (10BBlack) IE6 is effectively shut out of WMF wikis anyways, because we require HTTPS, and we don't support SSLv3 (or lower), which is... [04:36:37] 10Wikimedia-Apache-configuration: Titles containing a dot give HTTP 403 when accessed using action=raw and the short URL form - https://phabricator.wikimedia.org/T126183#2007156 (10BBlack) (although I guess there are ways around that which wouldn't likely be popular - e.g. using IE6 behind some trusted proxy doi... [08:02:09] 7HTTPS, 10Wikimedia-Blog: Switch blog to HTTPS-only - https://phabricator.wikimedia.org/T105905#2007291 (10Tbayer) >>! In T105905#1979351, @Varnent wrote: > Is there a status update on this? Just curious as it came up on IRC. :) Yes, so I followed up with Automattic in November, and they eventually offered a... [10:32:54] 10Traffic, 6operations: Upgrade to Varnish 4: things to remember - https://phabricator.wikimedia.org/T126206#2007434 (10ema) 3NEW a:3ema [10:42:03] 10Traffic, 6operations: Upgrade to Varnish 4: things to remember - https://phabricator.wikimedia.org/T126206#2007468 (10ori) [ ] Make sure that [[ https://github.com/wikimedia/operations-puppet/blob/24cc170e/modules/varnish/files/varnishlog.py | `modules/varnish/files/varnishlog.py` ]] (and the metric loggers... [11:27:04] 10Traffic, 10Wikimedia-Apache-configuration, 6operations, 5Patch-For-Review: Sometimes apache error 503s redirect to /503.html and get cached - https://phabricator.wikimedia.org/T109226#2007520 (10Krenair) [11:27:23] 10Traffic, 10Wikimedia-Apache-configuration, 6operations, 5Patch-For-Review: Sometimes apache error 503s redirect to /503.html and this redirect gets cached - https://phabricator.wikimedia.org/T109226#1543344 (10Krenair) [12:06:12] <_joe_> I'm going to package the new pybal version, and install it on the backups in ulsfo and codfw now [15:27:28] 10Traffic, 6Zero, 6operations, 5Patch-For-Review: Merge mobile cache into text cache - https://phabricator.wikimedia.org/T109286#2007923 (10BBlack) [15:27:41] 10Traffic, 6Zero, 6operations: Use Text IP for Mobile hostnames to gain SPDY/H2 coalesce between the two - https://phabricator.wikimedia.org/T124482#2007927 (10BBlack) [15:27:44] 10Traffic, 6Discovery, 10Maps, 6operations, 5Patch-For-Review: Set up proper edge Varnish caching for maps cluster - https://phabricator.wikimedia.org/T109162#2007928 (10BBlack) [15:27:47] 10Traffic, 6Zero, 6operations, 5Patch-For-Review: Merge mobile cache into text cache - https://phabricator.wikimedia.org/T109286#2007924 (10BBlack) 5Open>3Resolved [15:38:27] 10Traffic, 6operations, 10ops-eqiad: eqiad cache cluster re-arrangements - https://phabricator.wikimedia.org/T125486#2007975 (10BBlack) [15:57:32] 10Traffic, 10Deployment-Systems, 6Performance-Team, 6operations, 5Patch-For-Review: Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime - https://phabricator.wikimedia.org/T99096#2008045 (10Krinkle) [16:01:25] <_joe_> bblack: I'm back to installing pybal [16:06:35] _joe_: ok I'm still paused on all related things right now [16:53:01] bblack: so the decom of mobile is done now? [16:54:00] ema: yup [16:54:06] nice! :) [17:21:54] 10Traffic, 6operations: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#2008345 (10BBlack) [17:31:01] <_joe_> we're never gonna get rid of pinkunicorn, are we? [17:31:20] actually it's probably not appropriate for this, but we can set up something else in its place [17:31:41] just because pinkunicorn's frontends talk to the rest of the varnish3 text cluster... [17:31:55] either that or make config changes to isolate it [18:13:13] 10Traffic, 6Zero, 6operations, 3Mobile-Content-Service, and 3 others: Enable X-CS headers for non-mobile domains - https://phabricator.wikimedia.org/T126053#2008599 (10jhobs) [18:25:31] 10Traffic, 6operations: Upgrade to Varnish 4: things to remember - https://phabricator.wikimedia.org/T126206#2008698 (10Johsthao) [18:25:51] 10Wikimedia-Apache-configuration: Titles containing a dot give HTTP 403 when accessed using action=raw and the short URL form - https://phabricator.wikimedia.org/T126183#2008707 (10Johsthao) [18:33:28] 10Traffic, 6operations: Upgrade to Varnish 4: things to remember - https://phabricator.wikimedia.org/T126206#2008815 (10matmarex) 5duplicate>3Open [18:33:48] 10Wikimedia-Apache-configuration: Titles containing a dot give HTTP 403 when accessed using action=raw and the short URL form - https://phabricator.wikimedia.org/T126183#2008829 (10matmarex) 5duplicate>3Open [19:12:49] 10Traffic, 6operations: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#2009050 (10ori) Since {e10801bfe8} rolled out on Friday, we can use the presence or absence of the Backend-Timing header to tell apart what percentage of traffic is served from cache objects that are older... [19:24:40] 10Traffic, 6operations: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#2009122 (10BBlack) My guess would be you're already seeing 75%-ish or more. [19:42:22] 10Traffic, 6operations, 7Performance: Estimate effective cache time for text - https://phabricator.wikimedia.org/T126063#2009228 (10ori) [19:42:25] 10Traffic, 6operations: Decrease max object TTL in varnishes - https://phabricator.wikimedia.org/T124954#2009229 (10ori) [19:52:26] 10Traffic, 6Performance-Team, 6operations, 5Patch-For-Review: Disable SPDY on cache_text for a week - https://phabricator.wikimedia.org/T125979#2009316 (10ori) p:5Triage>3Normal [20:27:18] 10Traffic, 6Zero, 6operations, 3Mobile-Content-Service, and 3 others: Send X-Carrier + X-Carrier-Meta headers on all responses - https://phabricator.wikimedia.org/T126053#2009467 (10BBlack) [20:28:33] 10Traffic, 6Zero, 6operations, 3Mobile-Content-Service, and 3 others: Send X-Carrier + X-Carrier-Meta headers on all responses - https://phabricator.wikimedia.org/T126053#2003432 (10BBlack) Updated task title, in a recent meeting we decided to avoid the legacy X-CS[2] header names (which are full of legacy... [22:07:59] 10Traffic, 6operations, 10ops-esams: cp30[34]x hw/firmware/BMC issues - https://phabricator.wikimedia.org/T126062#2009923 (10BBlack) [22:31:44] got a RAID error cronspam from cp4010 [22:31:48] too tired to check it out, sorry [22:36:27] heh [22:36:41] I rarely look at cronspam, sorry [22:49:41] it's fixing itself slowly now, will be a couple of hours [22:52:44] hmm I remembered the tuning knobs, now it's going quick :)