[00:06:50] * Note, this is a standalone component. Please don't mix MediaWiki-specific [00:06:50] * code or library calls into this file. [00:06:55] if ( class_exists( 'FormatJson' ) && function_exists( 'wfEscapeWikiText' ) ) { [00:06:57] * Reedy squints [00:18:18] SMW? [00:18:51] Nope [00:19:05] Json classes in EventLogging [00:19:28] I'm guessing that wfEscapeWikiText is pretty MW centric [00:19:42] You'd think so, right? [00:19:47] bd808: WDYT about adding the canonical special page name to the things automatically collected by structured logging? [00:20:16] "automatically"? how would that happen? [00:20:28] There's 3 copies of JsonUtil and roblas other code in all our hosted extensions [00:22:18] I've only found one regression while doing MW dev on PHP7 for the past two weeks [00:22:19] from a monolog processor I suppose [00:23:38] MW could store it in some global variable if it is not easily accessible otherwise [00:23:58] * bd808 readies a trout for a mention of a new global [00:24:18] tgr: what are you trying to "fix"? [00:24:36] legoktm: that's pretty awesome [00:25:16] bd808: that logstash url still not working with Fx 45.2 (Fx ESR) :/ I'll look more tomorrow [00:25:24] (same errors) [00:25:30] greg-g: lame [00:25:43] (not working on it later; it not working) [00:26:05] I'm trying to filter out DB errors caused by signup and urls like /w/index.php?title=%D8%AE%D8%A7%D8%B5:%D8%A5%D9%86%D8%B4%D8%A7%D8%A1_%D8%AD%D8%B3%D8%A7%D8%A8 are not super helpful [00:28:15] One feature that monolog is missing that I loved from log4j is a global diagnostic context. That's basically a thread local dict that any code can grab and stick key=value data into that then ends up in all log events [00:29:01] should be trivial to add manually tho? [00:29:06] I wrote one once but didn't bring it over to the MW logging stack because I was trying to keep it PSR3 clean [00:29:59] yeah, in php it's just a static method or two to implement [00:30:06] but its not PSR3 clean at all [00:30:24] what do you mean by PSR3 clean? [00:30:44] alternative PSR3 implementations would simply ignore that information [00:31:00] they would because they wouldn't know about it [00:31:24] like they would ignore the things we currently have monolog processors for, like URL and DB name and whatnot [00:31:35] does not seem problematic to me [00:31:47] at $DAYJOB-1 we tagged logs with lots of things as we found out about them in the app stack. customer id, databases used, services called, etc [00:32:19] the processors we have now are passive though, not active [00:32:28] but yeah [00:32:29] we could introduce it [00:32:58] I'd like to think about making monolog the default logger for core too [00:33:05] it could simplify some things [00:33:17] and would still let libs be independent via PSR3 [00:36:59] but anyway, a static logging context like that would let you tag things like canonical special page name without some really one off hack [00:37:58] and we could expose it via LoggerFactory [00:39:31] log4j (and some academic papers I can't find right now) call this thing a "diagnostic context" -- https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/MDC.html [00:41:08] http://c2.com/cgi/wiki?PatternsForLoggingDiagnosticMessages [00:42:24] bd808: Could you do one more merge + release (minor I think)? - https://gerrit.wikimedia.org/r/300202 [00:42:37] I think we've got them all now :) [00:45:30] Krinkle: this simple code keeps getting more complex :) [00:46:06] If I had been better with writing more tests, I would've found this earlier. [00:46:47] bd808: Yeah, it's still pretty straight forward though. I'd be happy to have feedback about how it can be simpler. [00:48:43] its not scary yet but the cyclomatic complexity of compact() is pretty high (lots of loops and conditions) [00:49:59] anyway. looks good for now. I'll merge and tag for you [00:51:35] Krinkle: v2.1.2 ? [00:52:06] bd808: v2.2.0 maybe, since it's not a bugfix. [00:52:12] Or maybe it is. [00:52:22] * Krinkle is very sparingly with patch releases. [00:52:33] up to you [00:52:38] Okay, 2.2.0 then :) [00:52:43] Thanks! [00:55:13] https://github.com/wikimedia/WrappedString/releases/tag/v2.2.0 [00:56:10] bd808: Cool [13:05:56] * anomie has been running MW on PHP7 for a little while too. Only one bug so far: apparently "Error" is a class now, and Exception->getPrevious() might return one, and something somewhere blew up because of an Exception typehint. [13:50:33] James_F: https://gerrit.wikimedia.org/r/#/c/138840/ could use a review by someone in Editing. Could you add an appropriate reviewer, or add it to a queue, or whatever would be needed so someone looks at it? Thanks. [16:41:30] anomie: Will do. [18:08:37] all -- schana is Nathaniel Schaaf, a developer on the Research team. Since Research is not a typical engineering team schana has been a bit isolated from the rest of engineering. So I suggested this & a couple of other channels as having a good vibe and signal-to-noise ratio. Say hello and introduce yourself if you get the chance :) [18:09:30] thanks ori! hello everyone [18:09:52] o/ schana. Welcome to the random channel that the 2015 reorg couldn't kill :) [18:10:58] schana: I'm https://www.mediawiki.org/wiki/User:BDavis_(WMF) . You can bug me about jsut about anything, but I'm most likely to be helpful with things related to MediaWiki-Vagrant, Labs and Tool Labs [18:12:28] * bd808 -> lunch [18:44:52] ori: I had a fun idea for a metric we could track. At the end of a request, see how many (and what) of files we include/require() that did *not* get loaded via the autoloader. Would help us track down nasty non-lazy loaded code. [18:45:37] is that common? [18:45:44] I haven't encountered that, but I hadn't thought to look [18:46:14] There's a bunch in Setup.php [18:46:20] I'm pretty sure some extensions do similar things [18:46:46] a bunch? :P [18:46:46] eg in core: require_once "$IP/includes/compat/normal/UtfNormalUtil.php"; [18:47:32] require_once "$IP/includes/OutputHandler.php"; [18:48:54] require_once __DIR__ . '/compat/normal/UtfNormalDefines.php'; [18:49:02] Global defines make me sad. [18:49:12] Especially ones in class files like DiffEngine. [18:49:31] (true story, if you try to use MW_DIFF_VERSION prior to loading the class, it'll blow up. Luckily no code path does) [18:51:01] It may be useless. It may not. Just seemed interesting to track. [18:51:12] It's a messy code pattern we should avoid these days. [18:51:18] E_ITSALREADY2016 [18:51:56] heh [18:52:43] I think it's probably worth tracking [18:52:47] And slowly cleaning them up [18:53:03] Some are obviously going to have to stay: require_once "$IP/includes/AutoLoader.php"; [19:02:25] Yeah. And we can have a list of ones to skip tracking for now perhaps. [19:02:34] DefaultSettings, WebStart, Setup, Autoloader. [19:02:40] Those all get included by hand for now [19:02:50] Others are suspect! [19:04:00] Mmm [19:23:27] bd808: can you look at https://phabricator.wikimedia.org/T141020 ? [19:23:39] accounts being unattached after global rename [19:23:53] fun [19:24:25] my changes would have hit commons yesterday right? [19:24:39] yeah [19:25:00] do the stewards start the renames from meta? [19:25:14] yes, always [19:25:16] * bd808 trying to think about how to debug this [19:25:51] https://meta.wikimedia.org/w/index.php?title=Special%3ACentralAuth&target=Fuck+your+pussyhole also has an unattached account but wasn't renamed [19:25:54] idk if that's related [19:26:15] lol. that's the NSFW link in the bug? [19:26:19] yeah [19:28:18] I have to run for lunch sorry, bbl [19:57:14] Wikibase\Repo\Store\WikiPageEntityStore::updateWatchlist: Automatic transaction with writes in progress (from DatabaseBase::query (LinkCache::addLinkObj)), performing implicit commit! [19:57:20] Remains spammy in wmf.11 as it was in 10 [20:39:44] legoktm: is there a maintenance script that will attach accounts to CA? [20:41:04] maybe migrateAccount will work... [20:46:21] duploktm: I think this will work -- mwscript extensions/CentralAuth/maintenance/migrateAccount.php --userList /home/bd808/projects/T141020-rename-unattached/renamed.txt --attachmissing [20:50:40] does this look plausible? https://graphite.wikimedia.org/render/?width=586&height=308&_salt=1469134210.255&target=scale(MediaWiki.site.users.rate%2C%2060)&from=-12months [20:51:08] that should be showing sign-ups per minute, but the way the metric is reported may have changed [20:53:15] AuthManager/SessionManager almost certainly changed it. [20:53:51] the first big drop would be SessionManager. The second looks like the right time for AuthManager [20:58:29] sign-ups are down a lot year-over-year but that could be due to number of reasons [20:58:51] Has anyone other than maybe duploktm / legoktm run migrateAccount.php before? [20:59:24] I'm wondering if I should use the --suppressrc option when cleaning up some unattached account poop [20:59:42] the first drop is T125054 [20:59:42] T125054: Sharp dip in MediaWiki.site.users metric since wmf11 roll-out - https://phabricator.wikimedia.org/T125054 [21:00:48] don't know about the drop around start of March but that seems temporary [21:01:33] other than that it seems like seasonal change to me [21:02:20] note that this includes autocreations (which are probably the majority), not just signups [21:03:00] * bd808 heading home, back in 15m or so [21:03:11] I don't see any dip around the time AuthManager was turned on. [21:07:04] i'll calculate month-by-month signups for the past two years via select count(*) from globaluser where gu_registration like '201606%', etc. [21:07:09] will post later, gotta run [21:09:30] The March drop looks like about the time of 1.27.0-wmf.16. No SessionManager or AuthManager things during that timeframe, I don't think. [21:10:47] i'm not doing an *Manager witchhunt btw, the context for the question is me trying to come up with good thresholds for alerting on rate of sign-ups [21:19:23] bd808: doesn't really matter if you use --suppressrc, but it's preferred. it was mostly so SULF didn't destroy the IRC feed [21:30:02] * legoktm waves to schana [21:34:49] legoktm: *nod* I guess I should just give the script a shot then [21:38:19] bd808: give it a user list with one user in it to start with and if that goes well give it the entire list? [21:38:34] seems like a good idea [21:41:36] hmmm... it doesn't look like it really does anything [21:41:49] https://commons.wikimedia.org/w/index.php?title=Special%3ACentralAuth&target=Leefeni+de+Karik [21:43:29] re-reading the script it looks like it's only going to help if email is confirmed on all the nattached wikis [21:43:34] which seems unlikely [21:43:54] try --attachbroken ? [21:44:40] it did attach some accounts for that user, just not all of them [21:44:42] I tihnk I usually had a local copy of the script with the && $local['editCount'] === '0' part of --attachbroken removed [21:44:57] that seems like it might work [21:51:52] what's the trick to run a maintenace script outside of the $IP tree? I know I've done it before but can't remember the magic [21:53:41] just need to replace the inclue Maintenance.php bit [21:53:53] *include Maintenance.php* [21:57:44] nah [21:57:45] MW_INSTALL_PATH=/var/www/wiki/mediawiki/core [21:57:55] MW_INSTALL_PATH=/var/www/wiki/mediawiki/core php /path/to/script.php [21:58:19] * bd808 tracked down https://raw.githubusercontent.com/wikimedia/operations-puppet/3218df6/modules/admin/files/home/ori/.binned/mw [21:58:41] you don't even need MW_INSTALL_PATH [21:58:55] just mwscript ../../../../../../home/bd808/whatever.php [21:59:14] IF you use mwscript [21:59:15] I don't :P [21:59:20] ah yeah. that's the hack I've used before [22:01:56] ok, with --attachbroken and the hack to ignore edit count things are better, but still not quite right -- https://commons.wikimedia.org/w/index.php?title=Special%3ACentralAuth&target=Leefeni+de+Karik [22:02:02] 2 wikis still unattached [22:05:34] pipe dream: rewrite maintenance into fewer actual scripts. [22:05:47] Just have a few small entry points with related subcommands. [22:06:08] o.ri wrote an RfC for doing that I think [22:07:01] T99268 [22:07:19] poor stashbot :/ [22:07:41] did it crash because of no comment? [22:07:53] T99268 [22:07:57] "Carriage returns not allowed in privmsg(text)" [22:09:26] .replace('\r', ' ') ? [22:10:07] yeah [22:12:21] T99268 [22:12:44] T99268: RfC: Create a proper command-line runner for MediaWiki maintenance tasks - https://phabricator.wikimedia.org/T99268 [22:13:19] live hacks for the win [22:13:28] * bd808 goes back to the other live hack [22:16:41] bd808: Wow, kibana's advanced settings page gives you all kinds of fun ways to shoot yourself in the foot. [22:16:50] oh yes [22:16:53] and they are global [22:17:09] so a change is a change for everyone [22:17:40] * bd808 assumes kibana4 will be hosed pretty regularly [22:18:08] you can of course lock it down if you just by the payware perms plugin for Elasticsearch [22:18:13] Also it's slow as all hell.... [22:18:35] really? it's been way faster than kibana3 for me [22:18:39] And now it 503s. [22:18:43] gj ostriches [22:18:47] It's been unusably slow [22:18:49] For me [22:19:19] you crashed the nodejs service :) [22:19:20] or someone did [22:19:57] I was only trying to make a new visualization [22:20:06] Guess that's a little much to ask of kibana :p [22:20:24] better upgrade node [22:21:02] Jul 21 12:46:12 logstash1001 kernel: [2603396.900408] node[18588]: segfault at 2fcb25f00009 ip 0000000000ad9846 sp 00007ffe526bbb40 error 4 in node[400000+1383000] [22:21:53] I found out about a day before the upgrade that there is a config switch that makes kibana3 horribly slow [22:21:59] helpfully on by default [22:23:30] How much do they charge to tell you what to change to fix it? [22:24:56] bd808: is this on node 4 / jessie? [22:24:58] I agreed to send ori mail about https://gerrit.wikimedia.org/r/#/c/296699/ in yesterday's ArchCom meeting [22:25:05] gwicke: yes [22:25:21] well, jessie at least [22:25:21] are there any binary dependencies? [22:26:01] I wonder if it bundles it's own node? [22:26:43] is it installed from debs? [22:27:21] its from an upstream deb made by elastic.co. seems to bundle it's own node 4.4.7 [22:27:39] generally, segfaults in node itself (without weird binary modules) are rather rare, and are considered a security issue [22:27:40] or so says /opt/kibana/node/bin/node --version [22:28:33] that's a bit ugly [22:29:31] if they bundle their own, then they probably did some funky hacks / binary modules [22:29:55] https://phabricator.wikimedia.org/P3542 -- I think those are the modules [22:30:22] also elastic.co makes "interesting" debs. The fpm guy works there [22:31:02] or at least contracts for them regularly [22:32:08] * bd808 goes back to starting a attach fixing script [22:32:15] in the module list, I don't see anything obvious that stands out [22:32:35] they are using cutting edge js & compile it down to ES5 [22:33:16] bd808: attachbroken only works if they have no local password, meaning that the account was created by autocreation...if it predates SUL, it probably won't work. Might be easiest to just do it manually (eval.php) or write a one-off script for this [22:33:58] legoktm: yeah. I was thinking similarly. If CA says it's unattached then just attach it [22:34:14] grep -r 'gyp' node_modules would show binary modules [22:34:35] or should, rather [22:35:24] that finds a lot of files with the string "Egypt" [22:35:46] those are not.. binary [22:35:59] it's not a very crude technique [22:36:16] err, a rather crude technique [22:36:54] grep -r -l 'gyp' . -- nothing interesting looking [22:40:35] bd808: walk like an egyptian [22:58:17] anomie: legoktm: Thoughts on https://gerrit.wikimedia.org/r/#/c/300439/ ? [22:58:25] legoktm: do you have a minute to look at P3543 before I try running it? [22:58:26] P3543 attachAccount.php - https://phabricator.wikimedia.org/P3543 [23:00:00] The other thing that could be done is just `$central->attach( $wiki, 'login', false );` for all of them I suppose [23:00:19] tgr: you may have thoughts too ^ [23:00:52] yeah, that seems much simpler to me [23:00:58] but the script looks fine [23:01:26] legoktm: just have it always say 'login' for the reason? [23:01:33] that works for me. [23:01:49] yeah [23:03:09] ok. I'm going to try it on the test user and see what happens [23:04:02] https://commons.wikimedia.org/w/index.php?title=Special%3ACentralAuth&target=Leefeni+de+Karik -- all attached now [23:04:19] * bd808 picks a new test account [23:05:54] looks good to me -- https://commons.wikimedia.org/w/index.php?title=Special%3ACentralAuth&target=MartyRus [23:06:29] the only thing that may piss someone off is that if their home wiki was a group2 wiki (very very likely) they won't be marked as having a home wiki anymore [23:06:49] is that worth trying to do something about legoktm ? [23:10:36] bd808: LGTM [23:10:46] home wiki is not really recoverable, is it? [23:11:00] it depends on which account pressed the merge button [23:11:24] yeah. best bet would be to find highest edit count [23:11:39] or wait for the user to notice and file a bug [23:11:54] it's just a value in the localuser table [23:12:58] there is a getHomeWiki method which falls back to edit count [23:13:01] * bd808 runs the script for the remaining users [23:13:40] 2016-07-21 23:13:27 processed 52 usernames (3.2/sec), 52 (100.0%) fully migrated [23:13:42] so a home wiki will probably still be shown on Special:CA, it will just probably be a bit slower [23:13:49] bam [23:16:00] oh, there is a globaluser.gu_home_db [23:26:01] greg-g: do you think that T141020 merits an incident report? As far as we know at the moment 18 users were affected [23:26:01] T141020: Users with detached local accounts after rename during train deployment of 1.27.0-wmf.11 - https://phabricator.wikimedia.org/T141020 [23:29:10] bd808: A quick one yes, it looks like you all have it under control? Are the renames fixed or is that still TBD?) [23:29:14] to be done [23:29:24] the ones I know about are fixed [23:29:49] I think for good hygeine we should, but we probably don't need a retrospective meeting :) :) [23:29:49] I'll check again tomorrow to see if there are any that sneaked by my search [23:30:19] :) I'll save the write up for tomorrow. beer:30 here [23:31:04] word [23:39:07] bd808: homewiki doesn't really matter. [23:44:58] legoktm: good because I didn't fix that :)