[00:00:59] legoktm: https://gerrit.wikimedia.org/r/#/c/205787/ [00:02:18] evillllll [00:05:22] legoktm: So.. status update before you disappear? [00:05:49] csteipp: heh, was just about to write you an email [00:06:11] legoktm: Even better! Do that :) [00:06:12] csteipp: current status is that 142k renames left on enwiki, and about 20k on mediawiki.org [00:06:18] oh, IRC is easier :/ [00:06:37] I *think* it'll finish overnight [00:06:42] legoktm: cool, go for it here :) [00:06:52] Ok, on terbium? [00:06:56] yes [00:07:09] it's running in a screen named "rename" (I'm not sure if you can attach to my screens?) [00:07:10] Anything I should check on tomorrow to make sure it finished? [00:07:28] you can look at /home/legoktm/fru-last2.log [00:07:55] the non-debug output of the script is also sent to fluorine as "centralauthsulrename.log" [00:09:23] after all the page moves finish (check showJobs.php --group and grep for "LocalPageMoveJob"), there's another script that has to be queued up to send out the post-rename notifications [00:10:43] * randomcat wonders if any sapi stuff abuses $wgCommandLineMode [00:11:08] you have to prepare the lists of users who were renamed, I have a script in my home on fluorine that does that: /home/legoktm/get_renamed_users.py [00:11:16] copy those lists over to terbium [00:11:29] specifically /home/legoktm/fru-lists/ [00:11:43] and then /home/legoktm/postNotif.sh $dbname [00:11:50] legoktm: So you need to to run the script to send out the notifications tomorrow? [00:12:38] I might be able to start it depending on when enwiki finishes, otherwise you'll have to start it [00:13:02] legoktm: Cool, I think I can do that. Just email me tonight either way. [00:13:51] legoktm: And enwiki is the only wiki left right? [00:13:59] and mediawiki.org :/ [00:14:13] https://phabricator.wikimedia.org/T96489 is why I skipped it [00:14:27] I tried it again last night and it was still being slow (relatively) so I put it after enwiki [00:14:36] Oh, alright... so does that need to get started up with the commands on https://etherpad.wikimedia.org/p/sulf ? [00:15:17] Assuming the task either gets resolved, or we just don't care that it's slow since it's the last one. [00:15:34] no, I currently have the script running in a loop so it should start on mw.o whenever enwiki finishes [00:15:47] Ah much better :) [00:15:52] aaron didn't want to merge it so I think it's just going to be slow [00:16:37] legoktm: please send me the numbers before you sign off for the night if you can for the daily update :) [00:16:39] So if I keep tailing /home/legoktm/fru-last2.log, I'll see it switch to mediawikiwiki soonish, right? [00:17:02] csteipp: if by soonish you mean 5 or 6 hours :P [00:17:18] /home/legoktm/left.sh $dbname will tell you how many renames are left [00:17:32] Yeah, I meant soon relative to the next few days :) [00:18:26] legoktm: Nice, very useful [00:19:46] legoktm: I need to start heading to the train, but email me if you send out the notifications, or if you need me to do them, and I'll hopefully not hit any issues in that process. I'm guessing it's basically generate the list of names, then feed that into the mailer, right? [00:20:26] csteipp: I'll send you an email with the specific commands I run, but yeah, that's basically it [00:20:33] Oh, have you had mail runs fail? Do I just restart it, or would that re-email people? [00:20:45] no, MassMessage doesn't fail :) [00:21:04] Ah, cool then :) [00:21:12] oh but the script has no resume capabilities or anything, so only run it once :P [02:32:12] legoktm: So, whar be we stand? [02:45:26] Keegan: It was "less than 100k on enwiki " 90 minutes ago. [02:49:22] James_F: Thanks :) [02:49:33] Keegan: Presumably, "more" now. [03:17:12] bd808: what's the status of logstash these days? [03:17:37] ori: new hardware being racked [03:17:46] I need to update puppet stuff to use it [03:18:13] and then I need to get the config changes approved to get MW logs flowing back into it [03:18:16] like, as we speak? [03:18:38] I think so. Or really soon [03:19:25] 03:18:02 sync-file failed: /srv/mediawiki-staging/hhvm-fatal-error.php has content before opening indeed it does [03:19:49] ugh [03:20:25] that's the check we added to try and stop the BOM problem from a couple of months ago [03:21:09] Keegan|Away, James_F|Away: 16k left on enwiki! (and 20k on mww) [03:21:24] legoktm: are those the last wikis? [03:21:31] yep [03:21:37] w00t! [03:22:46] ori: cmjohnson made the ticket for racking today -- https://phabricator.wikimedia.org/T96692 [03:22:53] oh cool [03:24:09] If I can keep from being sidetracked by reorg madness tomorrow I should be able to get the puppet config setup to split the frontend/backend roles [03:24:31] * bd808 makes a ticket to track that [03:26:02] * ori watches http://graphite.wikimedia.org/render/?width=586&height=308&_salt=1429673140.711&target=MediaWiki.errors.fatal [03:26:31] i just synced that change [03:26:34] to add that metric [03:26:40] look at it go [03:26:59] hhvm fatals? [03:27:14] yes [03:27:18] I hope most are just the known OOM problem [03:28:06] does it manifest as a segfault in HPHP::GlobStreamWrapper::opendir ? [03:28:24] (serious question) [03:32:16] hmmm.. I haven't seen that, I've seen hhvm actually reporting the OOM in various places where largish output would be buffered [03:32:56] like streaming a pdf or a huge page of watchlist links or something [03:52:29] Fatal error: Class SomeJob contains abstract method (run) and must therefore be declared abstract [03:52:37] bd808: does vagrant give you that with unit tests? [03:53:04] There was a bug for it. [03:53:10] I thought it got fixed? [03:53:55] not in master at least [03:54:11] super annoying :/ [03:55:12] randomcat: https://phabricator.wikimedia.org/T95864 [03:55:18] not fixed apparently [03:55:28] * bd808 thought he looked at that [03:57:01] was it broken recently? [03:58:31] maybe? Jan noticed a week ago or so [03:59:22] The problem is in tests/phpunit/includes/jobqueue/JobTest.php [03:59:31] getMockJob() [04:00:40] I wonder if it is a phpunit change? [04:01:48] "The getMockForAbstractClass() method returns a mock object for an abstract class. All abstract methods of the given abstract class are mocked." -- https://phpunit.de/manual/current/en/test-doubles.html [04:02:14] and yet the error says run() didn't get mocked [04:04:28] I guess we could add an explicit stub for the run() method in there easily [04:18:22] Keegan: enwiki is done [04:18:40] https://en.wikipedia.org/wiki/Special:UsersWhoWillBeRenamed :D [04:19:23] :o [04:19:30] * Keegan hugs legoktm [04:20:38] whoa [04:20:42] congrats d00ds [04:20:43] ! [04:21:04] now just 19k left on mw.o :) [04:21:52] by the way.... <_< [04:22:00] is it too late to ask for a name-change? [04:22:13] not SUL-related, just a generic one. Ori.livneh => Ori Livneh [04:22:25] https://en.wikipedia.org/wiki/Special:GlobalRenameRequest [04:22:35] you guys have a special page for everything [04:22:55] now we just need an api for everything [04:23:17] that's my one and only special page [04:23:27] Oh and the backend side of it I guess [04:23:56] and Special:SulRenameWarning! [04:25:50] heh. I just saw my name on https://en.wikipedia.org/wiki/Special:Version for the first time. I'm internet famous :) [04:27:10] https://wikiapiary.com/wiki/Special:RunQuery/Extension_by_Author?Extension_by_Author[Author]=Bryan%20Davis&wpRunQuery=true ! [04:29:04] :) [04:29:08] https://wikiapiary.com/wiki/Special:RunQuery/Extension_by_Author?Extension_by_Author[Author]=Kunal%20Mehta&wpRunQuery=true [04:29:13] more impressive [04:50:07] legoktm: I'm not getting anomie's concern here: https://gerrit.wikimedia.org/r/#/c/202800/1/XAnalytics.class.php -- could you explain it? [04:55:23] ori: the API used set a made up title "Dwimmarliak" or something as $wgTitle, now it sets Special:BadTitle/Something, so if something calls $out->getTitle() there's a good chance it will get a nonsense title [04:55:29] used to* [05:34:05] congrats legoktm [05:34:19] :) [06:40:46] https://lists.wikimedia.org/pipermail/wikimedia-l/2015-April/077623.html tl;dr: {{done}} [06:45:51] :) [06:57:05] legoktm: the slave are rejoicing that the job is done https://tendril.wikimedia.org/host/view/db1072.eqiad.wmnet/3306 :) [06:57:08] *slaves [06:57:43] haha wow [06:57:58] ahaha "took us over a decade but oh well, Well Done Mr Pretzel :D " [06:58:04] i love wikimedia-l [07:00:28] :) [07:01:16] * randomcat hands legoktm https://gerrit.wikimedia.org/r/#/c/205824/3 [07:03:33] randomcat: sorry, I'm on vacation now :) [13:05:54] ori: (re https://gerrit.wikimedia.org/r/#/c/202800/1/XAnalytics.class.php) Basically what legoktm said, yes. More specifically WikimediaEventsHooks::onXAnalyticsHeader() is probably going to be expecting to get a title related to the API request, but they might really be getting that special page. Even if it doesn't blow up (which it seems that it won't), it probably won't give them the analytics they're hoping for. [13:09:39] legoktm: Re 205716, if you don't want to backport the logging patches you could just replace the calls to ApiResult::setIndexedTagName() and ApiResult::setIndexedTagNameOnSubarrays() in ApiQueryLogEvents with one call to ApiResult::setIndexedTagNameRecursive(). Apparently I missed fixing that one in the last patchsets of I7b37295e8862b188d1f3b0cd07f66ac34629678f. [14:25:42] ^d: some easy stuff at https://gerrit.wikimedia.org/r/#/q/owner:%22Aaron+Schulz%22+status:open,n,z [14:44:18] * anomie grumbles about T88393 still not being fixed [15:52:35] 6MediaWiki-API-Team, 10MediaWiki-extensions-OAuth, 5Patch-For-Review: OAuth: Authorisation should not fail because you don't have an account on central wiki - https://phabricator.wikimedia.org/T74469#1228057 (10Tgr) a:3Tgr [16:01:31] legoktm: can you declare victory on https://phabricator.wikimedia.org/T49918 now? [Rename of global (attached) users to existing global usernames] [18:34:28] ^d: https://gerrit.wikimedia.org/r/#/c/184115/ [18:34:57] <^d> I removed myself from that :p [18:35:57] <^d> Hmm, soft deprecation. [18:35:58] did you read timo's response? [18:36:03] right, and that :) [18:36:06] that's why i'm pinging you :) [18:36:48] <^d> I guess depending on your logging levels it'll actually let you know it's broken [18:37:37] also, re: what problem we're trying to fix (valid question) -- partly it's that first impressions count, and a repo root with .php5 files doesn't impress me as modern [18:40:22] anomie or troncat, would either of you be able to look at T96267? [18:40:32] * anomie looks [18:40:34] <^d> ori: That's a much more valid opinion to hold than that it causes a maintenance burden to me. [18:40:44] <^d> Seeing as they're 1 line + a bunch of comments. I don't consider that a burden ;-) [18:41:36] <^d> Adding a new if() block to Setup always makes me squeamish but I guess it can't be avoided here [18:41:53] ...so we still have unattached accounts, ones that have redirects to renamed users [18:42:03] https://en.wikipedia.org/w/index.php?title=Special%3ACentralAuth&target=Teke [18:42:12] My old username. [18:42:16] Crap. [18:42:27] <^d> ori: Also, a followup to this change would be to start deprecation of $wgScriptExtension [18:43:56] Keegan: So Teke@enwiki isn't a part of the global account? [18:44:13] Keegan: Well, I fully expected there to be tons of unattached accounts left after the finalisation. :-) [18:44:19] csteipp: nope, it should have been renamed to Teke~enwiki [18:44:31] Keegan: My advice is to get some data on exactly how many accounts are left unattached, so you know the scope of the problem. [18:44:40] <^d> ori: I left a nitpick on your release notes. Otherwise I'll merge [18:45:29] Deskana: Thanks, yeah. Who'd like to do an audit? I can't :) legoktm is on vacation until Monday [18:46:45] Deskana: See, when we starting contacting accounts to be renamed it hit those redirects and contacted the new account name, making people think their new account was going to be renamed. We altered the script to skip those redirects, and so it looks like the renaming script followed that. [18:46:50] Anywho. [18:49:25] Keegan: Right. [18:49:37] Keegan: I suspect it can wait until Kunal's back. [18:49:54] Keegan: We've had tons of unattached accounts for years, a few days won't hurt. ;-) [18:49:59] Hmmm [18:50:00] mysql:wikiadmin@db1033 [centralauth]> select * from users_to_rename where utr_name='Teke' AND utr_wiki='enwiki'; [18:50:00] +---------+----------+----------+------------+ [18:50:00] | utr_id | utr_name | utr_wiki | utr_status | [18:50:01] +---------+----------+----------+------------+ [18:50:01] | 3666405 | Teke | enwiki | 5 | [18:50:03] +---------+----------+----------+------------+ [18:50:23] Keegan: Although it seems like you've got csteipp curious. ;-) [18:52:17] Hmmm indeed. [18:54:24] So yeah, Keegen, you were right. Redirects are skipped. $rows = $updates->findUsers( $wiki, UsersToRenameDatabaseUpdates::NOTIFIED, $this->mBatchSize ); [18:55:39] ^d: done [18:55:48] <^d> MatmaRex left a comment too [18:55:55] Keegan: So 1808 from enwiki were skipped [18:55:57] <^d> Ah, I see your response to him [18:55:58] <^d> nvm [18:56:07] Keegan: Oh, that was all wikis. [18:56:15] So 1808 totall [18:56:46] A drop in the water! [18:56:53] csteipp: The only thing I can think of for T96267 is if the second submission gets an empty array from $this->oldCAUser->listAttached(), it'll happily do no local renames but still add another log entry. [18:57:24] Keegan: I'd rather wait on kunal to do them, but let me know if there's a rush and I can probably safely update his scripts to run them. [18:57:41] There's no rush, no [19:00:13] ori: want to look at https://gerrit.wikimedia.org/r/#/c/202929/ ? [19:00:21] csteipp: Thanks for checking. :-) [19:00:42] <^d> troncat: So out of curiosity, what can I do to avoid that ptwiki namespace snafu next time? Should I run namespaceDupes prior to actually removing the NS? [19:00:56] bd808: do your tabs look broken in vagrant in vector? [19:00:56] <^d> Because that was nasty [19:01:21] ^d: the lag or the manual queries? [19:01:32] <^d> Manual queries to cleanup [19:01:37] <^d> (lag sucked, but it happens) [19:04:01] troncat: sure [19:05:34] 6MediaWiki-API-Team, 10MediaWiki-extensions-CentralAuth: Possible to globally rename the same user twice (race condition) - https://phabricator.wikimedia.org/T96267#1228702 (10csteipp) csteipp: The only thing I can think of for T96267 is if the second submission gets an empty array from $this->oldCAUs... [19:05:40] troncat: My tabs look ok -- http://serious-musk-4001.vagrantshare.com/wiki/Main_Page [19:06:00] ^d: isn't the problem just the script not handling logs? [19:06:29] bd808: mine have grey boxes as backgrounds [19:06:57] * bd808 runs vagrant git-update [19:09:45] * troncat has to reread https://phabricator.wikimedia.org/T91401 [19:10:02] <^d> troncat: The script did jack-shit in terms of cleanup after the NS didn't exist. [19:10:16] <^d> I think what I needed to do was force move them to the mainspace with namespace dupes first [19:10:21] <^d> And /then/ remove the NS [19:12:42] did you use source-pseudo-namespace ? [19:13:59] ah, same problem [19:20:22] <^d> troncat: Right, which is what I started trying to tackle in the commit linked from the task but I didn't get it working right yet. [19:23:04] bbl [19:23:15] <^d> l8r [19:46:49] so my deployment is gonna go way long because I'm still waiting for my trivial patches to merge [19:47:21] just a heads up, haven't started scap yet [19:47:39] twentyafterfour: see -operations; might want to hold off until the root cause of the 5xx spike has been isolated [19:51:23] ^d: I want to cleanup the logstash puppet code and move things that are using $::realm switches into hira. Problem is that these thing are settings for ::elasticsearch and that won't play nice with the way that hieradata/labs/deployment-prep/common.yaml reads. [19:51:33] ^d: Any ideas on a nice way to fix that? [19:52:06] <^d> twentyafterfour: If zuul/jenkins are still sick that's why we have the ability to say "screw you" and merge anyway [19:52:32] In mw-vagrant I would just add params to the role::logstash class, but I'm afraid that _j.oe_ would not like that in ops/puppet [19:52:47] <^d> Yeah we tried to avoid that layer of abstracting in ops/puppet [19:53:09] <^d> Might have been a mistake though... [19:53:18] It should be fine in prod because of the hieradata/role/* setup I think [19:53:41] which AFIK picks hiera config based on node role [19:53:58] <^d> yep [19:54:20] <^d> (role then common) [19:54:43] Can I make hieradata/role/labs/deployment-prep/* stuff too? [19:55:29] Or should I just step away from the edge and do what I need to do now without adding hiera into the mix? [19:55:43] <^d> You can do deployment-prep/host/* I know [19:55:48] <^d> But I dunno about up one level [19:55:56] oh [19:56:02] I can do it that way [19:56:15] only one host so not a big deal [19:57:49] <^d> We just hiera-ized all this, so let's try to keep using it if we can :) [20:04:08] ^d: so that script seems to just take any title with a valid prefix in NS_MAIN and move it to the NS (stripping the dbkey). You want to take pages registered in a bad NS and update the NS to 0 and force them to have the old prefix in the dbkey. Seems like a --former-namespaces flag could be added (erroring if found in $spaces). [20:04:34] <^d> Something like that yeah [20:04:35] * troncat finds that commit [20:04:54] <^d> https://gerrit.wikimedia.org/r/#/c/194570/ [20:07:57] maybe --former-namespace= and --dest-pseudo-namespace= [20:10:03] <^d> Removing namespaces seems generally hairy. How many tables have namespace/title tuples? How many extensions add similar columns? [20:16:30] ^d: very few things care about redlinked pages, so most just use page id [20:17:12] ^d: Echo, for example, uses page id instead of (ns, title), while it should use the latter. (and so you get mildly broken UI when someone deleted a page you've gotten a notification about) [20:24:52] <^d> MatmaRex: What about recentchanges? [20:24:55] <^d> Broken for 30 days? :p [20:25:51] php maintenance/rebuildRecentChanges.php [20:25:53] there, fixed it [20:26:03] no one looks at recent changes past the first 50 entries, anyway. [20:26:30] various link tables, too. [20:26:54] (but these contain all kinds of stupid junk anyway) [20:57:06] anomie: any other bits on https://gerrit.wikimedia.org/r/#/c/200325 ? [20:59:14] troncat: Looks ok, no time to test it at the moment. [21:05:10] ^d: https://gerrit.wikimedia.org/r/#/c/195088/ would just break the smw tests right? [21:05:36] * ^d shrugs [21:06:20] so says grep [21:17:15] ^d: want to take over https://gerrit.wikimedia.org/r/#/c/205967/ ? [21:32:39] bd808: ahh, the tab issue is due to 404s... [22:08:35] bd808: this was a new failure mode: https://gerrit.wikimedia.org/r/#/c/205988/ <-- that [22:08:46] that's what caused scap fail [22:09:08] well specifically, that's what fixed it [22:09:31] o_O a return in the extension load point? [22:10:45] heh legoktm did it [22:10:58] There is no need at all for a return there [22:12:11] I'd flip that all around and write `if not exists, die` and then the bootstrap [22:13:32] bd808: any reason vector/images/ stuff would 404 lately? [22:13:38] twentyafterfour: `} elseif ( !( include_once $fileName ) ) {` that's what got broken [22:13:44] in vagrant [22:13:49] bd808: yes [22:14:05] that's how I figured it out .. I read the calling code and realized that "return" is not true [22:14:13] troncat: not that I know of. I was working for me when you asked earlier. Have you looked for logs? [22:15:10] maybe ... elseif (include_once $file == false ) would be better [22:15:18] er === [22:15:32] wouldn't be any different [22:15:40] include doesn't work like that really [22:15:41] void === false? [22:15:52] it should require instead of include [22:16:01] and if that fails it would go boom! [22:16:31] true [22:16:47] the return value of include is anything returned or printed to stdout by the file being included [22:17:18] if there is no explict return or echo then it will return true/false [22:17:48] wow I didn't know include would return output [22:17:48] one of those goofy PHP bits that comes in handy once in a while but is mostly werid [22:18:25] That maint script is relying on "If the file can't be included, FALSE is returned and E_WARNING is issued. " [22:18:32] I usually prefer to wrap things in functions and explicitly call the function... [22:18:42] https://php.net/manual/en/function.include.php -- after example #5 [22:18:44] rather than including raw code in top scope [22:20:29] I's switch it to use require and call it a day personally [22:20:47] The fatal that pops out will give the file name [22:20:53] I think [22:21:01] easy enough to test [22:22:03] Fatal error: require(): Failed opening required 'not_found.zzz' [22:22:05] yup [22:23:52] bd808: Quick one: Any concrete plans (or actionable things) to get back backtraces for fatals? [22:24:10] hoo: we have to patch hhvm :/ [22:24:17] (why?) [22:24:23] hoo, you mean publicly? [22:24:32] they are on fluorine, on /a/mw-log/hhvm-fatal.log [22:24:35] ori: On fluorine [22:24:58] ori: because hhvm fatals do raise a catchable signal, but the signal is raised after the stack has unwound [22:25:01] Hi TimStarling. (Were you able to liberate your parents?) There is a simple and straightforward patch that could do with a +2 from you: [22:25:50] hoo, ori: https://phabricator.wikimedia.org/T89169#1199157 [22:26:19] bd808: aha, useful. Thanks. [22:26:24] hhvm-fatal seems to only cover hhvm dying [22:26:52] ^d: https://gerrit.wikimedia.org/r/205998 [22:27:55] what about error_document500 => '/srv/mediawiki/hhvm-fatal-error.php', [22:28:04] does that get invoked by handleFatalError()? [22:28:22] have we attempted to log( debug_get_backtrace() ) from there? [22:28:26] hoo: yeah. that log file is captured by the init script wrapping hhvm startup [22:29:28] ori: I don't think we have tried that. I doubt it is called in the same thread though. If the fatal is for OOM the fcgi process wouldn't want to reuse it [22:29:58] There's a reason we had a custom PHP extension to do this in php5 [22:30:47] all we really need is to add a "php_stack_trace_on_fatal" thingy to hhvm though I think [22:31:17] hhvm know when it's going bad, raises a fatal signal and starts cleaning up [22:31:40] we just need to have it be able to spit out a PHP stacktrace before it cleans up [22:32:25] There may be some fatal cases where that won't work but they wouldn't have worked for us in php5 either [22:32:58] If it works about as well as the zend thing did I'm fine [22:34:02] hoo: the trouble is finding someone with the time and inclination to work on it [22:34:36] Yeah, that's the problem we have with most stuff that's in phab [22:34:42] essentially [22:34:46] pretty much [22:36:05] the channel name in the wfErrorLogFormat is folllowed by a space or a tab? [22:36:32] space [22:36:39] I think... [22:36:41] let me look [22:39:09] ori: "[{$channel}] {$message}" is that what you mean? [22:39:20] Of do you mean in the udp packet? [22:39:43] in the udp packet [22:40:17] space there too [22:40:27] $text = preg_replace( '/^/m', $this->prefix . ' ', $text ); [22:41:59] Is ^d around someone? [22:42:13] I'm about to kill his script [22:56:49] ori: I don't think we have tried that. I doubt it is called in the same thread though. If the fatal is for OOM the fcgi process wouldn't want to reuse it [22:56:52] your guess was correct [22:56:59] well, educated guess :) [22:57:05] :) [22:57:27] your rational estimation [22:57:52] * bd808 read lots of C books in the olden days [22:59:04] like back when they were talking about making an ISO standard for it olden days [23:07:22] My dad made me read http://www.amazon.com/The-Programming-Language-Brian-Kernighan/dp/0131103628 when I was like 8 years old [23:08:15] Nice. I think I was in Jr High when I read it [23:09:00] I think I should read it again. I rememebr it being really actually very interesting (even though I didn't understand most of it at the time) [23:10:15] I have a pile of CS textbooks that I keep planning on reading but never seem to get to [23:11:09] mostly stuff that my small liberal arts college CS degree didn't cover -- neural networks, lisp, that kind of stuff [23:11:55] What I really should do is take a bunch of stats classes though if I wanted to learn something really useful