[00:42:51] 10Traffic, 10Commons, 10Multimedia, 10Operations, and 3 others: Disable serving unpatrolled new files to Wikipedia Zero users - https://phabricator.wikimedia.org/T167400#3787682 (10Tgr) So to summarize / spec out the code parts a bit: == Core functionality: * Make `File::getContentHeaders()` look up patro... [08:33:57] 10Traffic, 10CheckUser, 10Operations: Log source port for anonymous users and expose it for sysops/checkusers - https://phabricator.wikimedia.org/T181368#3787859 (10ema) p:05Triage>03Normal [08:34:20] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3787860 (10ema) p:05Triage>03Normal [08:45:40] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3787868 (10Samat) I re-uploaded the .har file. I will give access to it as soon as I will know who should I share with :) [08:46:23] 10Traffic, 10CheckUser, 10Operations: Log source port for anonymous users and expose it for sysops/checkusers - https://phabricator.wikimedia.org/T181368#3787475 (10ema) >>! In T181368#3787516, @Krenair wrote: > I would expect $_SERVER['REMOTE_PORT'] to be useless inside WMF infrastructure Yeah I'... [08:56:09] 10Traffic, 10CheckUser, 10Operations: Log source port for anonymous users and expose it for sysops/checkusers - https://phabricator.wikimedia.org/T181368#3787475 (10Legoktm) >>! In T181368#3787869, @ema wrote: >>>! In T181368#3787516, @Krenair wrote: >> I would expect $_SERVER['REMOTE_PORT'] to be... [09:29:08] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3786217 (10Gilles) @Samat can you email the HAR file to performance-team@wikimedia.org ? [10:21:12] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3788079 (10Samat) @Gilles I emailed the file to this address. Thank you in advance! [10:24:01] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3788102 (10Gilles) @Samat sorry I gave you the wrong address, the correct one is: performance-team@lists.wikimedia.org [10:24:19] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3788103 (10Samat) Hmm: I received from Google the following message: "Message not delivered: 550 Address performance-team@wikimedia.org does not exist" [10:24:42] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3788106 (10Gilles) See my reply above [12:13:26] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3788370 (10Gilles) It is indeed startling. Does this happen to you on all browsers? You mention that you can trigger this easily, it would be interesting if... [14:23:48] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3788748 (10Samat) @Gilles 160 secs was an extreme example, but in the 10-60 second range it happens daily. I have several screenshots, but earlier I didn't k... [14:32:59] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3788779 (10Peter) @Samat Just want to check, FF 57 is mentioned in the description but you got this before on older versions too (just want to make sure it i... [14:38:35] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3786217 (10ema) We've recently started logging requests taking longer than 60 seconds (from varnish's point of view) and sending the logs to logstash. [[http... [15:06:44] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3788870 (10Samat) >>! In T181315#3788779, @Peter wrote: > @Samat Just want to check, FF 57 is mentioned in the description but you got this before on older v... [15:30:40] 10Traffic, 10Operations, 10Phabricator, 10Zero: Missing IP addresses for Maroc Telecom - https://phabricator.wikimedia.org/T174342#3558443 (10Mholloway) >>! In T174342#3787625, @Tgr wrote: >we have the IP configuration on Zerowiki that's in theory maintained by Maroc employees The IP ranges on ZeroWiki for... [17:16:48] 10netops, 10Analytics-Kanban, 10Operations, 10monitoring, 10User-Elukey: Pull netflow data in realtime from Kafka via Tranquillity/Spark - https://phabricator.wikimedia.org/T181036#3789390 (10Milimetric) [18:22:42] 10Traffic, 10CheckUser, 10Operations: Log source port for anonymous users and expose it for sysops/checkusers - https://phabricator.wikimedia.org/T181368#3789662 (10Krenair) Per @legoktm this has nothing to do with #Analytics as far as I am aware. >>! In T181368#3787869, @ema wrote: >> I would not... [20:17:00] 10Traffic, 10Operations, 10Phabricator, 10Zero: Missing IP addresses for Maroc Telecom - https://phabricator.wikimedia.org/T174342#3790158 (10Keegan) >>! In T174342#3787625, @Tgr wrote: > So what does it take for this task to be resolved? Is someone actually looking into it or is it just being pushed aroun... [20:31:12] 10Traffic, 10Operations, 10Phabricator, 10Zero: Missing IP addresses for Maroc Telecom - https://phabricator.wikimedia.org/T174342#3790202 (10Mholloway) I've reached out to Partnerships about getting in touch with Maroc and INWI for IP range updates. [20:53:50] 10Traffic, 10Operations, 10Phabricator, 10Zero: Missing IP addresses for Maroc Telecom - https://phabricator.wikimedia.org/T174342#3790296 (10Keegan) >>! In T174342#3790202, @Mholloway wrote: > I've reached out to Partnerships about getting in touch with Maroc and INWI for IP range updates. Great, thank you! [21:27:04] 10Traffic, 10Operations, 10Phabricator, 10Zero: Missing IP addresses for Maroc Telecom - https://phabricator.wikimedia.org/T174342#3790430 (10Dzahn) Sounds like it would be useful if Partnerships could be added in the Phabricator-based workflow for future updates. [21:28:49] 10Traffic, 10Operations: Puppetize LVS interface IP sets per-DC for easy use in ferm rules - https://phabricator.wikimedia.org/T179027#3790444 (10Nuria) [21:38:30] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3790483 (10Gilles) I've looked for that particular request from the HAR file in the varnish logstash data you've linked to @ema and couldn't find it. [21:52:13] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3790555 (10Tgr) If varnish saw this request as taking long, that would be reflected in the `Backend-Timing` header, right? So this must be happening inside v... [22:29:37] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3790663 (10Samat) >>! In T181315#3788787, @ema wrote: > We've recently started logging requests taking longer than 60 seconds (from varnish's point of view)... [22:36:54] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3790707 (10Samat) @Tgr I think I experienced this on multiple connections, but I usually use my laptop to edit Wikipedia. The only other place where I edit s... [22:54:28] 10Traffic, 10Operations, 10Performance-Team: load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3790749 (10Gilles) >>! In T181315#3790555, @Tgr wrote: > If varnish saw this request as taking long, that would be reflected in the `Backend-Timing` header,...