[16:50:42] jbond42: _/\_ for the `pcc.py last parse_commit` thing [16:53:33] :) np [22:36:24] Would anyone know where I might be able to find more details about a Varnish timeout error: https://phabricator.wikimedia.org/T269029#6685404? Specifically I'm trying to pin down where the timeout is happening, and wondered if there might be a log I can search [22:38:58] Tchanders: https://logstash.wikimedia.org/app/kibana#/dashboard/Varnish-Webrequest-50X might be able to help you some, although we can also get much more detailed logs from the traffic nodes themselves if there's an easy reproduction [22:41:46] there's also a variety of appserver-side logs available, which probably will be more helpful in this case -- if Varnish is persistently timing out on the same request, the problem is more likely in the appserver layer [22:43:49] yeah, although as cdanis often correctly points out, we might not have as much logging information at the appserver if the response never completes [22:44:24] if it's reproducible to the extent that we can fire the request at an mwdebug host and see what happens, we might be able to get some more data out [22:45:09] is tallying votes a mutating operation? [22:45:41] the 2 min timeout sounds like it's hitting the excimer timeout? [22:46:23] cdanis rzl Thanks - I've managed to find it in logstash. Good to know the other options are available too [22:47:10] cdanis: I'm not sure if it's mutating or not - I'm not familiar with this extension and largely going from circumstantial evidence [22:48:20] But got hard proof now from logstash :) [22:48:40] it might be worth trying to run that codepath via the shell.php maintenance script which doesn't have a wall clock timeout [22:52:56] legoktm: logstash exception message says "the execution time limit of 200 seconds was exceeded", which I guess is the timeout you mean? [22:54:58] yep [22:55:34] poor securepoll. nobody ever looks at it except when it is broken [22:55:45] Cool. Thanks everyone [22:55:52] Tchanders: web requests can only take so long. I'd suggest just running it via the CLI for now to avoid the timeout and then later on figuring out how to make it faster or just turn it into a proper CLI script [22:56:26] yeah, if something needs to run for that long we should make it asynchronous one way or another [22:56:30] bd808: My team (AHT) will be looking at it for the next quarter! [22:56:48] but from an extremely uninformed perspective, it sounds like it could probably be more efficient [22:58:27] I think SecurePoll stores votes as gpg encrypted blobs in the db and that slows it all down at atally time [22:59:04] legoktm, rzl: We've been asked to make it startable from the web interface even for slow cases, so I thought we'd look into either making it more efficient or, if that's not possible, running a job that sends an alert when it's done [22:59:39] bd808: Thanks. That sounds consistent with what I've skim-read [23:00:46] using the job queue will avoid the timeout too AIUI [23:01:23] Tchanders: you probably know this already, but Tim Starling and Aaron Schultz are likely the folks still around who have memory of SecurePoll design decisions and hidden traps. [23:02:46] bd808: Oh great - I only knew half of that! [23:04:18] that extension was a yearly hot potato in the mediawiki-core team of yore. Whoever touched it last would be the first one offered up when a new bug report came in. [23:10:38] I've probably touched it a few too many times [23:10:40] * Reedy coughs [23:30:06] Reedy: you blew your own cover. :) [23:30:24] Brad did some work in there somewhere too [23:41:55] Reedy: You're going to regret admitting that... [23:42:18] I'm sure git blame would've outed me eventually