[00:00:23] If we put it in logstash then MW isn't slowing down to do it [00:01:07] since it would be derived from logs we are already sending [00:01:43] we could make stats like MediaWiki.debugLog.$CHANNEL.$LEVEL or something [00:01:55] as part of AaronSchulz's "defer $foo" series of commits, it is either the case already that metrics are sent after the response is flushed to the user, or there exists some patch to make them so [00:02:04] *nod* [00:02:05] but yes, doing it from logstash LGTM [00:02:39] Nicely from logstash we can do hhvm and apache2 as well [00:02:59] right, that's a good argument [00:03:09] want me to file a task? [00:03:22] yeah that would be awesome [00:04:16] ori, https://gerrit.wikimedia.org/r/#/c/212485/ really needs CR [00:04:21] forceprofile doesn't work atm [00:05:00] AaronSchulz: ok, on it [00:10:07] bd808: done https://phabricator.wikimedia.org/T100735 [00:10:46] perfect [00:18:12] AaronSchulz: instead of adding a doPostSendShutdownInternal, I think that it'd be better to define an anonymous function inside doPostSendShutdown. In fact, you already create an anonymous function to pass as the callback to register_postsend_function. If you move the code from doPostSendShutdownInternal into that anonymous function, and assign that function a name (so that it can be called from the else block as well), that'd be a lo [00:18:12] t cleaner, IMO [00:26:59] ori, amend :) [00:40:08] AaronSchulz: ok [05:26:49] AaronSchulz: https://gerrit.wikimedia.org/r/#/c/214512/ [06:13:45] AaronSchulz: is the absence of forceprofile especially urgent? I think that patch could be a bit better [06:14:17] it can wait a few days [08:39:18] 10MediaWiki-Core-Team: Provide English-only MediaWiki installer - https://phabricator.wikimedia.org/T100768#1319994 (10Bennylin) 3NEW [14:09:06] legoktm: https://gerrit.wikimedia.org/r/#/c/214610/ could use a review as soon as you're around. [14:13:21] * marktraceur doesn't see any likely ^d nicks in this channel [14:13:23] Is he a ghost? [14:14:37] Anyway, my question: Special:LinkSearch still exists, does ElasticSearch have/plan to have the same functionality, and if so, are we deprecating S:LS? [14:15:18] I doubt we could deprecate Special:LinkSearch since ElasticSearch isn't a core requirement. [14:16:04] Fair enough [14:29:37] legoktm: anomie: does one of you have free bandwidth to review some trivial patches on jobrunner? Setting up .gitignore and a basic composer entry point :} [14:29:53] I would like to add a Jenkins job that runs composer test so AaronSchulz can expand on it as needed [14:30:00] hashar: I don't know anything about composer [14:30:10] ah that is bd808 so :) [14:30:26] I could not remember who did the initial run on cdb liberalization I was assuming it was you :D [15:10:31] marktraceur: ^d is vacationing in France still. I think he will be back on Monday [15:12:59] anomie: I just want to say, I love you :) [15:13:16] marktraceur: chad's on vacation in France still [15:13:27] greg-g: For fixing that bug? [15:13:32] anomie: for all of it :) [15:14:59] hashar: I can review jobrunner patches for you [15:18:56] bd808: would be nice :} .gitignore update https://gerrit.wikimedia.org/r/#/c/214612/ [15:19:06] oh got comments there [15:19:59] .gitignore bikeshedding. perfect [15:21:04] always useful [15:21:21] * bd808 steps away from the comment button [15:21:32] Never read the commentns. [15:21:50] especially on patches you've been asked to review ;) [15:24:42] anomie: made the gitignore .*.swp :-}  https://gerrit.wikimedia.org/r/#/c/214612/2/.gitignore,unified [15:25:10] bd808: there is some basic composer at https://gerrit.wikimedia.org/r/#/c/214613/ based on what we have in cdb [15:26:25] hashar: Looks good to me. There's probably no need to worry about .swo and so on that was mentioned there, that only happens if you open the same file in multiple instances, which tends to be trouble. [15:26:55] I wouldn't bother at all but now that there's a patch up it seems harmelss to merge [15:31:55] bd808: and I rebased https://gerrit.wikimedia.org/r/#/c/214613/ which adds the basic composer [15:32:08] I should add the jenkins job right now [15:33:02] hashar: merged [15:33:08] :-) [15:39:58] greg-g: curious how the new deployment schedule will work out ;) [15:40:16] * aude thinks it's worth a try but somewhat skeptical [15:47:40] :) good to be skeptical, but I have hopes it'll work [15:47:45] if it doesn't, we'll change it again [15:48:58] we've been doing good, but i found an issue this morning with our deployed version of wikibase + wmf8 core [15:49:02] on test.wikidata [15:49:51] finally have patches and am not panicked since this doesn't go to wikidata yet [15:53:01] :) [16:06:18] bd808: Does mediawiki-vagrant use a constant value for $wgSecretKey, or does it generate something random? [16:07:22] anomie: it lets the cli installer generate one [16:07:37] bd808: Ah, then my comment on https://gerrit.wikimedia.org/r/#/c/210036/ wasn't quite correct. [16:07:59] that's probably why I got errors though [16:08:20] it needs to be computed locally right? [16:08:29] it == the hash [16:08:45] Well, you could set $wgOAuthSecretKey to a constant to be able to precompute it. [16:08:51] But otherwise yeah. [16:09:06] *nod* [16:09:28] I added you to that review so you'd find a problem like that :) [16:09:42] Happy to help [16:18:32] sooo - how does job undelaying work now? I've lost it [16:18:43] I can't find any code the interacts with z-delayed any more [16:18:46] other than reads it [16:18:57] it _does_ work because jobs aren't stuck in prod [16:21:36] AaronSchulz: ^^ can you point me in the right direction on job queue undelaying? [16:34:18] ok - found it - can ignore now [16:34:39] ori: It doesn't look like the mysql connect timeout did anything to help :( [16:35:14] i haven't followed this problem closely. how is the problem manfesting? [16:35:23] https://logstash.wikimedia.org/#/dashboard/elasticsearch/wfLogDBError [16:35:25] making the configuration values agree seems like the right thing to do regardless [16:36:01] jynus (new DBa, supporting springle) has been doing some database work in the past few days and he's still quite fresh -- is it possible he's doing something wrong? [16:36:06] worth asking springle [16:36:50] the logstash page is still loading [16:36:55] jynus took a look at the problem on his first/second day and couldn't find anything on the mysql side other than disconnects [16:37:18] huh. loads fast for me [16:40:28] ori: would hhvm have automatically restarted when that puppet change hit or would we need to salt a hup of the fcgi process? [16:40:46] * bd808 sees that was a lot of jargon [18:10:54] bd808, legoktm: any of you have time to troll through https://gerrit.wikimedia.org/r/#/q/owner:%22Aaron+Schulz%22+status:open,n,z ? [18:16:34] AaronSchulz: maybe this afternoon [18:16:48] AaronSchulz: also should we be freaked out about https://phabricator.wikimedia.org/T100815 ? [18:17:34] not for that job type [18:17:39] https://gerrit.wikimedia.org/r/#/c/212485/ should fix those afaik [18:42:03] AaronSchulz: in https://gerrit.wikimedia.org/r/#/c/212762/1/CheckUser.hooks.php,cm, don't DeferredUpdates already run in a transaction? what's the point of the explicit commit? [18:55:23] AaronSchulz: based on the phpdoc it seems like doPreSendCommit() should be called before getOutput()->output(). why is it the other way around in https://gerrit.wikimedia.org/r/#/c/212485/6/includes/MediaWiki.php,unified [18:55:52] that would let you get rid of another call to commitMasterChanges() too [18:56:06] oh... profiling data [18:56:32] it's not really presend it's pre shutdown [18:59:41] legoktm, in case any later updates take a bit of time [19:00:02] bd808, well, profilertext is presend :) [19:01:41] but you want the comment to come after the response body right? [19:02:25] which is why you trigger output first. but the comment about wanting to commit transactions first is confusing [19:03:15] and in main() you still commit the db first, then output() and then commit again and add profiling [19:04:05] Maybe the profiler stuff could move to the start of doPostSendShutdown() ? [19:04:16] outside the callback [19:13:36] bd808, logDataPageOutputOnly has to not be in the post-shutdown callback or it wont show at all [19:14:10] right, but it could be in doPostSendShutdown() before the callback logic [19:14:20] as the "footer" action basically [19:14:31] the doPreSendCommit commit is used by the other endpoints mostly [19:14:32] the naming is just confusing me [19:15:52] the name doPostSendShutdown seems to imply "the http response is done, given content-length and everything", so it would be weird to output stuff there before registering the callback (though it would work) [19:15:58] doPreSendCommit() sounds like it is supposed to happen before the main response body but that doesn't seem to be the case anywhere [19:17:02] and it duplicates the commit in restInPeace [19:17:31] I kind of see what you are doing today but in 6 months it will confuse me [19:17:45] you have to commit again due to deferred updates and jobs [19:17:54] ah [19:18:00] which could have done new stuff [19:18:01] doPreSendCommit is names such due to buffering [19:18:02] sure [19:18:16] maybe it could be named better [19:19:01] yes the entry points do stuff like print $stuff; $mediawiki->doPreSendCommit(); ... [19:19:08] is it ever not paired with a call to doPostSentShutdown()? [19:19:11] bd808, is that what was confusing? [19:19:16] yeah [19:20:48] bd808, in run()/main() [19:21:30] I guess we can use the buffering assumption to remove the old wfGetLBFactory()->commitMasterChanges(); in main() [19:22:30] I'm just wondering if you could combine the 2 functions into one called something like MediaWiki::shutdown() [19:23:12] then again, output() calls print(), so if the commit failed you'd have the text and the exception text [19:24:35] maybe not, I'll need to try that [19:25:10] I think that's right, but part of my confusion. [19:25:55] main() ends with commit, output, presendcommit [19:26:19] followed immediately by doPostSendShutdown [19:27:17] doPreSendCommit is more like doPrePostSendShutdown [19:27:53] which makes me want to instead see doShutdown that combines the two of them into one call [19:36:26] yeah throwing an error after output() causes duplicated output [19:37:09] so it has to be commit, pre-flush stuff, flush, post-flush stuff, commit [19:46:54] MatmaRex: I just noticed that OO.ui.InputWidget.prototype.getInputElement is marked as @private, even though all the subclasses have to override it. [20:11:12] AaronSchulz: the WAN cache keys for spamblacklist and gadgets are appearing at the top of the most frequently accessed keys on mc1001 (a random server I checked) [20:14:06] setting a key in APC with a short expiry to indicate whether or not memcached should be checked might be a good idea [20:14:24] i don't think 1-5 minute delay in gadget definition updates is anything to worry about [20:14:53] ori, is it the :t: key? [20:15:14] yeah [20:16:00] WANCache:t:ptwiki:gadgets-definition:8, 171.05 reqs/sec, 1174.10 kbps [20:56:29] ori, is it causing any problem? [20:59:53] AaronSchulz: it's just unnecessary and ripe for optimization imo [21:00:36] memkeys aggregates by bandwidth, so it takes the size of the value into account, and the gadget definition key is still near the top [21:03:45] I guess you can play with https://gerrit.wikimedia.org/r/214764 [21:09:37] AaronSchulz: amended, see what you think [21:33:59] ori, the APC set()s are already randomized, so it shouldn't matter, though I guess it won't hurt [21:34:48] AaronSchulz: oh, right. No, if that's the case then we shouldn't duplicate that logic. [21:34:58] I'll rebase your PS1 and +2 [21:35:12] also, YGPM [22:05:18] AaronSchulz: check out `curl localhost:9002/check-sql` on mw1041 [22:14:02] ori, are you going to use that xml to make some sort of query frequency leaderboard? [22:32:21] AaronSchulz: yeah we ought to. I was going to a file a task for that certainly. [22:32:35] ishmael used to do that :( [22:32:45] but the monitoring tool put too much load on the db [22:33:44] did it rank by frequency? or did it take execution time into consideration as well? [22:34:31] you could sort by either [22:36:32] AaronSchulz, we should have just put all the MySQL performance data into MongoDB! [23:34:02] AaronSchulz: can you remind me -- why did you decide against using mcrouter?