[00:38:28] Krinkle: did you end up filing a task for the mwdebug timeout issue? [00:38:34] tgr: nope. [00:39:12] will do that then, seems to happen rarely on non-debug requests as well (might or might not be the same issue) [00:39:38] it remains a concern, especially because it didn't used to happen, and now happens pretty much consistently whenever touching a mwdebug/hhvm after a sync. Could be debug specific, or something that it never happens on a non-idle server; but as of yet, no reason to believe it doesn't affect prod, and could be what we're seeing in prod after every deployment. [00:50:58] tgr: this https://phabricator.wikimedia.org/T203664 ? [00:51:35] greg-g: yeah I was wondering if it's the same issue [00:51:58] just filed T215368 for it [00:51:59] T215368: First request after a MediaWiki sync times out on mwdebug - https://phabricator.wikimedia.org/T215368 [00:52:50] that sounds like how HHVM behaves generally [00:53:33] maybe, but it has started not so long ago [00:55:58] tgr: I also feel like it was recent, and I know for sure it didn't used to be the case N months ago. [00:56:17] But. it's possible this started when we changed mwdebug from app server to VM. [00:56:35] We kept the hostname as-is, so I don't quite know when that happened, but feels like that was longer ago. [00:57:17] but yeah, once someone has time, should be possible to profile it and see where the time is spent. [00:59:45] so we've got 4 symptoms: 1) cdb rebuild on mwdebug is slower than app servers during scap deploy; 2) first few XWD reqs to mwdebug in a browsing session is slow (times out after 60s), 3) the monitoring req by scap to mwdebug sometimes times out during a deploy (after 10s); 4) we have a spike of 60s timeout reached on prod app servers shortly after major deploys. [01:00:30] Issue 2 and 4 both started relatively few months ago and due to 4 seems unrelated to it being an underresourced VM. [01:00:53] and hence why it's potentially worrying, but could be a separate issue. [01:04:44] I'd like your feelings on that task, Krinkle :) [03:01:16] greg-g: OK. So the 4 issues I mentioned above, are actually three issues, and they all have tasks now: 1) T203625; 2) T215368 thanks tgr ; 3/4) T204871 [03:01:17] T203625: mwdebug1001 and mwdebug1002 are reliably the last two hosts to finish scap-cdb-rebuild - https://phabricator.wikimedia.org/T203625 [03:01:17] T215368: First request after a MediaWiki sync times out on mwdebug - https://phabricator.wikimedia.org/T215368 [03:01:18] T204871: Investigate the spikes of "web request took longer than 60 seconds and timed out" during deployments - https://phabricator.wikimedia.org/T204871 [04:20:18] Krinkle: right, I've been paying attention to those. _joe_ knows about them to, who's my best guess for helping investigate (and or say "just wait to see if php7 fixes it") (which is what I thought his opinion was before re the first two tasks you listed, afair) [05:00:53] https://phabricator.wikimedia.org/T215324 feels dupey [05:06:12] Guessing it's because I remembered https://phabricator.wikimedia.org/T210937 [05:36:12] <_joe_> Krinkle, greg-g regarding 1) and 2) above, we said with alex we'd pump up a bit mwdebug's CPU [05:36:28] <_joe_> regarding 3), that's going to be fixed by php7, at least partially [05:36:38] <_joe_> given on php7 our timeouts don't work [05:59:41] _joe_: timeouts don't work? [05:59:53] I thought TimStarling had a patch to make them work via excimer [06:00:21] it's not reviewed yet [06:00:28] <_joe_> we briefly talked about it in SF, I didn't know Tim had a patch already [06:00:47] <_joe_> Can I take a look? I might help testing it [06:00:51] https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/487069/ [06:00:59] or check your gerrit dashboard [06:01:01] <_joe_> TimStarling: good evening :) [06:01:33] <_joe_> TimStarling: heh, I literally started working since 20 minutes, I got back home on monday evening and took yesterday off [06:02:34] ok [06:02:53] I've got to go, time for linux user group [06:05:44] <_joe_> see you in 14 hours I guess :) [06:06:06] <_joe_> err 15, I really still am jetlagged [07:22:59] legoktm: do you think you'll be able to make it to that code coverage meeting on thursday? [19:30:20] TimStarling: fyi, looking into sampling profiler issue. https://performance.wikimedia.org/xenon/logs/daily/ looks like something broke Dec 13. [19:30:28] Thanks AaronSchulz [19:51:30] Hello, just started exploring MediaWiki APIs. How do I get the translated languages of a particular page? [19:51:47] ? [19:52:30] You want a list of all the pages that exist in other languages? [19:53:08] no, the other way around. List of all languages a particular page is translated in. [19:54:00] https://en.wikipedia.org/w/api.php?action=query&prop=langlinks&titles=Paris [19:54:17] probably with a limit=max [19:54:18] https://en.wikipedia.org/w/api.php?action=query&prop=langlinks&titles=Paris&lllimit=max [19:55:50] oh, I should been more specific, talking about meta.wikimedia.org [19:55:55] say https://meta.wikimedia.org/w/api.php?action=parse&format=jsonfm&page=Library_Card_platform [19:56:51] how do I get the language codes of this page's translations [19:59:14] I don't know if we actually expose that via the API [19:59:19] You can use the prefix/subpages... [19:59:20] https://meta.wikimedia.org/w/api.php?action=query&list=allpages&aplimit=max&apprefix=Library_Card_platform [19:59:25] But has some pages that aren't translations [20:00:34] although not ideal, this will do for now [20:00:43] thank you for the help! [20:01:05] * Reedy files a bug [20:01:14] Ha [20:03:15] https://phabricator.wikimedia.org/T215456 filed for now [20:03:55] *fingers-crossed* [20:04:00] thanks, bye for now