[09:52:55] 10Traffic, 06Analytics-Kanban, 06Operations, 06Wikipedia-iOS-App-Backlog, and 2 others: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#3046056 (10elukey) Today I tried to do the following: * Bump the maximum Apache connections allowed (hence the number of processes due... [13:38:12] lag on cp1074 again, no errors so far [14:51:29] 10Traffic, 06Operations, 07Mobile: Samsung Internet's desktop mode getting redirected to mobile site - https://phabricator.wikimedia.org/T158599#3046781 (10ema) p:05Triage>03Normal [15:52:30] cp1074 recovered, now it's cp1049 and cp1072's turn [15:52:45] interestingly, their lag started growing roughly at the same time [15:53:19] 15:32ish [15:57:34] I'd like to find the exact point at which they start throwing 503s to tune the icinga alert heh [15:58:05] clearly now the alert is too loud, we got several criticals with no errors whatsoever [16:11:31] and they both recovered at the same time, interesting [16:12:37] * volans provides ema a mirror for a more interactive chat :-P [16:20:09] :) [16:20:51] I'm of course curious, but too busy to look at it today [16:21:12] varnish upgraded to 4.1.5 on cp3006 and cp4019 (maps) [16:21:40] not expecting any issues, but I'll wait a bit and see if they're doing fine before upgrading the rest of the cache hosts [16:25:37] mailbox lag didn't cause errors and managed to fix itself up to ~336k, we could probably bump the alerting threshold to 350k or so [17:04:26] 10Traffic, 06Operations: Name Asia Cache DC site - https://phabricator.wikimedia.org/T156028#3047116 (10BBlack) [17:04:29] 10Traffic, 06Operations: Select location for Asia Cache DC - https://phabricator.wikimedia.org/T156029#3047111 (10BBlack) 05Open>03Resolved a:03BBlack Singapore approved by Legal and selected, which is pretty much our ideal candidate on a range of issues. [17:08:45] likely asia cache dc names at this point: cosin, insin, gssin, iosin [17:08:58] kinda hoping for one of the first two, but if the others are better vendors we'll deal :) [17:09:42] oh "drsin" too, that's one of the options [17:13:22] oh, Singapore is decided ? [17:13:25] nice [17:13:36] Changi is an amazing airport, heh [17:14:11] butterfly sanctuary , showers [17:15:47] yeah I love SG in general (not that that was really a factor in this) [17:16:02] I lived there for ~5 years (high school age) [17:18:16] 07HTTPS, 10Traffic, 06Operations, 10fundraising-tech-ops: update SSL certificate for benefactorevents.wikimedia.org by 2017-03-02 - https://phabricator.wikimedia.org/T158684#3043892 (10CaitVirtue) Added Danny as a subscriber, as he's the event lead. [17:30:20] 10Traffic, 06Discovery, 06Operations, 10Wikimedia-Portals: https://www.wikipedia.org/ portal doesn't have any text - https://phabricator.wikimedia.org/T158782#3047244 (10ema) [17:31:18] #traffic added to T158782 as we'll probably have to ban /portal/wikipedia.org/assets/js/index-d1cc91a7f4.js after the problem is fixed [17:31:18] T158782: https://www.wikipedia.org/ portal doesn't have any text - https://phabricator.wikimedia.org/T158782 [17:37:33] 10Traffic, 06Discovery, 06Operations, 10Wikimedia-Portals: https://www.wikipedia.org/ portal doesn't have any text - https://phabricator.wikimedia.org/T158782#3047254 (10ema) And the redirect comes from mediawiki: ``` < HTTP/1.1 301 Moved Permanently < Date: Wed, 22 Feb 2017 17:33:13 GMT < Content-Type:... [17:46:42] 10Traffic, 06Discovery, 06Operations, 10Wikimedia-Portals: https://www.wikipedia.org/ portal doesn't have any text - https://phabricator.wikimedia.org/T158782#3047301 (10Paladox) Is this a mediawiki core change causing this problem? Should #mediawiki-general-or-unknown be added? [17:46:47] 10Traffic, 06Discovery, 06Operations, 10Wikimedia-Portals: https://www.wikipedia.org/ portal doesn't have any text - https://phabricator.wikimedia.org/T158782#3047125 (10Dzahn) also see T128546 [17:49:15] 10Traffic, 06Discovery, 06Operations, 10Wikimedia-Portals: https://www.wikipedia.org/ portal doesn't have any text - https://phabricator.wikimedia.org/T158782#3047125 (10Dzahn) @Paladox Seems more likely it's related to the last deploy on T128546, but that's possible too. [17:49:26] 10Traffic, 06Discovery, 06Operations, 10Wikimedia-Portals: https://www.wikipedia.org/ portal doesn't have any text - https://phabricator.wikimedia.org/T158782#3047125 (10Dereckson) There is a .jsl10n CSS rule with `visibility: hidden`. I guess JS code should show it, but it's not the case. [18:28:23] 10Traffic, 06Operations, 07Mobile: Samsung Internet's desktop mode getting redirected to mobile site - https://phabricator.wikimedia.org/T158599#3047490 (10elukey) Ran some hive queries on webrequest data with @ema, our understanding is that the "Mobile" keyword in the UA should indicate that the TV is askin... [18:29:01] ema --^ [18:30:58] thanks elukey [19:10:00] 10Traffic, 06Analytics-Kanban, 06Operations, 06Wikipedia-iOS-App-Backlog, and 2 others: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#3047643 (10elukey) After the changes done this morning, something *seems* to have changed, namely I don't see anymore `FetchError no... [19:19:51] 07HTTPS, 10Traffic, 06Operations, 10fundraising-tech-ops: update SSL certificate for benefactorevents.wikimedia.org by 2017-03-02 - https://phabricator.wikimedia.org/T158684#3047685 (10EWilfong_WMF) @Jgreen @RobH - Checking in on our next steps here. I would love to get the new cert in place this week so w... [19:22:37] 07HTTPS, 10Traffic, 06Operations, 10fundraising-tech-ops: update SSL certificate for benefactorevents.wikimedia.org by 2017-03-02 - https://phabricator.wikimedia.org/T158684#3047689 (10RobH) @EWilfong_WMF: Just wanted to check, we'll be generating the private key, csr, and ordering the certificate. In t... [19:24:41] 07HTTPS, 10Traffic, 06Operations, 10fundraising-tech-ops: update SSL certificate for benefactorevents.wikimedia.org by 2017-03-02 - https://phabricator.wikimedia.org/T158684#3047695 (10EWilfong_WMF) Perfect. ewilfong@trilogyinteractive.com [19:58:37] 07HTTPS, 10Traffic, 06Operations, 10fundraising-tech-ops: update SSL certificate for benefactorevents.wikimedia.org by 2017-03-02 - https://phabricator.wikimedia.org/T158684#3047812 (10RobH) a:05Jgreen>03RobH I'll handle key/csr generation and cert ordering. Then I'll pgp encrypt and email the key ove... [20:26:09] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 15 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#3047910 (10Gilles) You might not have access to the file contents when you generate a URL. What I'm saying is that you can't just drop that information fr... [20:27:31] 10Traffic, 10ArchCom-RfC, 06Commons, 10MediaWiki-File-management, and 15 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#3047915 (10Gilles) Ah nevermind, it's only the md5 of some form of the filename, not the file contents? [21:44:39] 10Traffic, 06Discovery, 06Operations, 10Wikidata, 10Wikidata-Query-Service: Set up backend per-IP limits on varnish for WDQS - https://phabricator.wikimedia.org/T119917#3048221 (10Smalyshev) 05Open>03Resolved a:03Smalyshev I think this is done. [22:15:36] ema: (and whoever else is lurking) - XioNoX == Arzhel btw, he's been lurking in the channel the past day or so :) [22:15:58] he's not started yet officially, so don't bombard with netops tasks yet :) [22:16:55] hey! [22:22:58] XioNoX: welcome here [23:20:11] hello XioNoX [23:33:44] hi XioNoX :) [23:33:59] also see #wikimedia-netops :) [23:34:17] I'm in there too ;) [23:34:24] oh heh [23:34:27] (I'm Faidon) [23:36:02] Yay, another channel for me to join :)