[00:07:26] !log bots danmichaelo: bots installed libxml2-dev, libxslt-dev on bots-3 [00:07:28] Logged the message, Master [00:11:58] petan or any bots, admin, can you install "python-virtualenv" on bots-nr1 please? [01:35:11] ugh. I think I just broke virt0 [01:40:15] fucking iptables [01:43:02] bah. typo [08:44:21] petan: could you install python-virtualenv on bots-nr1 [08:44:37] it would mean i no longer have to bug you for any other installs :P [08:59:17] !logs [08:59:17] logs http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-labs [08:59:39] duh: sure [09:02:59] !log bots petrb: nr1 only 100mb of free ram, needs fix [09:03:01] Logged the message, Master [09:03:13] !log bots petrb: nr1 installing python-virtualenv [09:03:15] Logged the message, Master [09:04:08] thanks :) [09:09:40] @notify Beetstra [09:09:41] I will notify you, when I see Beetstra around here [09:10:35] we HAVE to switch to btrfs soon [09:38:55] duh: are you actually running something on nr1? [09:39:02] I don't see a single process from you [09:39:05] right [09:39:12] because [09:39:17] it may be because the machine is out of ram [09:39:27] beetstra eats all [09:39:36] apergos said my crontab was generating a ton of cronspam [09:39:42] oh really? [09:39:44] which i realized was because virtualenv wasnt installed [09:39:47] so i disabled it [09:39:50] aha [09:39:52] and havent been able to renable it yet [09:39:59] /had the time to [09:40:02] ok [09:40:05] no problem [10:10:53] !tunnel [10:10:53] ssh -f user@bastion.wmflabs.org -L :server: -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000 [14:00:40] Am I crazy if I'm itching to start on the tools project even though I'm only slated to start on the 25th? :-) [14:01:16] * Coren is already testing OGS deployment scenarios. [14:13:03] haha, I got the message about cronspam too :P [14:21:17] !log bots petrb: addshore bot eats about 90% of ram on bnr1 0.o [14:21:19] Logged the message, Master [14:22:01] petan: /me checks [14:23:12] petan: fixed? [14:23:15] eh, yea, cronspam. we are getting mail from Nagios [14:23:35] Cron test -d /root/labsnagiosbuilder/ && ((cd /root/labsnagiosbuilder/ && git pull); /root/labsnagiosbuilder/labsnagiosbuilder/build.py) &> /dev/null [14:23:42] mutante: do you have still the cronspam coming from a beta instance ? [14:24:06] ... there's nobody listed as admin of the Bots project? o_O [14:24:18] Coren: the list on the bots project page is broken [14:24:33] Ah. [14:24:33] !log bots addshore: fixed addbot process in TASK_UNINTERRUPTIBLE state [14:24:35] Logged the message, Master [14:24:50] hashar: nope, i dont see it [14:24:57] i see another one: [14:25:03] Cron /data/project/legoktm/py2/bin/python /data/project/legoktm/supersimplemediawiki/super_purge.py [14:25:19] hm? [14:25:34] duh just add MAILTO="" at the top of your cron [14:25:37] is there something wrong with my process? [14:25:47] ho [14:25:47] or mail it to you :P [14:25:56] oh* [14:26:01] mutante: will fix that, sorry [14:26:23] You guys are going to /love/ the new architecture. :-) [14:26:26] :D [14:26:36] Oh btw, congrats on getting hired Coren :) [14:26:47] duh: no problem, we are used to a lot of mails, just trying to reduce them [14:27:02] * duh needs to figure out which server that one is running on... [14:27:05] duh: Thanks. Perhaps condoleances will be more indicated in a few months when I have no hair left. :-) [14:27:27] <^demon> mutante: apergos pinged me about the gerrit-dev one yesterday, that should be fixed now. [14:27:28] addshore: so MAILTO="legoktm@blah.com" ? [14:27:39] yep :P [14:27:55] duh: i-000000e5 [14:27:57] duh, it was great when my process in cron had php warnings and it ran every 5 mins ;p [14:28:08] for about 12 hours :), root got allot of rmails :P [14:28:10] *emails [14:28:13] duh: Hey, incidentally, I'll need victi^H^H^H^H^Hvolunteers for deployment in the OGS-driven environment. :-) [14:28:14] mutante: yeah…no idea what that means but i figured out its bots-3 [14:28:21] Haha [14:28:28] Well, I'm always open to new things [14:28:39] OGS? [14:28:48] and none of my bots are crucial, and it seems like the only person who complains when my bots dont run is addshore so :P [14:28:54] ^demon: thank you. looks like i got the last one about 7 hours ago Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) [14:29:00] HAHA! [14:29:02] Open Grid Engine. Open source fork of SGE [14:29:11] nice, Coren ill help [14:29:12] OGS* [14:29:18] <^demon> mutante: Yeah, I told him I dunno what that run-parts cron is, it's nothing I setup. [14:29:22] Scheduler. [14:29:24] <^demon> Was wondering if it was a generic labs thing. [14:29:28] I keep forgetting they changed the name. [14:29:29] I could slowly split my work load onto it [14:29:31] Coren: I guess you're also the person I can ask when database replication is coming to labs? [14:29:37] s/ask/bug/ [14:29:52] duh: I *will* be that person. Right now, I'm just standing by with no more information than anyone else. :-P [14:30:03] !instance i-000000e5 [14:30:04] need help? -> https://labsconsole.wikimedia.org/wiki/Help:Instances want to manage? -> https://labsconsole.wikimedia.org/wiki/Special:NovaInstance want resources? use !resource [14:30:14] !resource i-000000e5 [14:30:15] https://labsconsole.wikimedia.org/wiki/Nova_Resource:i-000000e5 [14:30:17] Although I talked to Ryan about it and he's still saying that mid-March is a realistic goalpost. [14:30:20] !log deployment-prep doing some git maintenance: cd /home/wikipedia/common/php-master/extensions ; git submodule foreach 'git gc --aggressive && git repack -a' [14:30:23] Logged the message, Master [14:30:38] duh: see above, you can use the !resource command and you get https://labsconsole.wikimedia.org/wiki/Nova_Resource:I-000000e5 [14:30:44] and that tells you the server name [14:30:49] <^demon> hashar: git gc aggressive and git repack are kind of redundant. [14:30:58] ahh [14:31:00] I should have asked [14:31:03] so just repack -a ? [14:31:07] mutante: Ah ok. I just wrote down my crontab on a wikipage so I can figure out what server everything is running on :P [14:31:09] i think I need to make something to auto kill my processes if they go into TASK_UNINTERRUPTIBLE :/ [14:31:16] <^demon> You can accomplish both with something like `git repack -a -d --depth=250 --window=250` [14:31:36] haha, i am just re doing my cron also [14:31:47] duh: Part of the point of using OGS is to get rid of that nonsense with crontabs and having to know where stuff runs. [14:32:22] !log bots root: shutting down sql1 which is not being used to upgrade it to mariadb and convert fs [14:32:25] Logged the message, Master [14:32:36] Coren: give me a ping whenever you get OGS up and ill give it some processes to play with ;p [14:32:43] !log deployment-prep git maintenance override. Now running: git submodule foreach 'git repack -a -d --depth=250 --window=250' [14:32:45] Logged the message, Master [14:32:47] ^demon: thx :) [14:32:47] Well right now I have jobs running on clematis, willow, ircbots on nightshade, wikibots on bots-3, random scripts on bots-nr1 so it gets confusing... [14:32:55] duh: documentation is always good, but there are specific wiki pages for it ,like https://labsconsole.wikimedia.org/wiki/Nova_Resource:I-000000e5/Documentation [14:33:34] https://labsconsole.wikimedia.org/w/index.php?search=Documentation&title=Special%3ASearch [14:33:46] thanks :D [14:33:47] addshore: I'd start working on it now, actually, if I had about a half dozen instances to deploy in parallel with the current stuff. I'll talk to Ryan once he's up so I can hit the ground running on the 25th if not earlier. [14:34:01] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots/Documentation [14:34:39] duh: The objective is that you'll have bots running "on the cluster" with no need to know which execution host is actually running it at any specific time. [14:34:44] duh: probably here is best: https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots/Documentation#bots-3 [14:34:58] yup, my bot is listed there :) [14:35:13] sounds good :) [14:35:17] duh: Which has the positive side effect that if the execution host breaks, it'll just run on another automagically. [14:35:28] LGTM, +2. :) [14:36:05] duh: wanna link to the actual source code ? [14:36:13] * duh is good :P [14:36:19] I have to finish up my wikidata scripts [14:36:38] duh: eh, i mean, i see now there is a github link, would be cool though if you could switch to gerrit.wm [14:36:40] Coren: if you have nothing to do in the time being could you set a reasonable memory limit on bots-bnr1 for php ? [14:36:47] i.e. smaller than all of the memory on the instance [14:37:03] addshore: I dunno if I own the bits to do that. Lemme check. [14:37:12] mutante: advantage of github is its something i can stick on my resume, and all my code is in one place. [14:37:43] duh: why can't you stick it on your resume if it is in WMFs git? [14:37:56] because then its not all in one place [14:37:58] you have code review that way [14:38:03] addshore: Nope. I'm a mere mortal with no sudo on -bnr1 [14:38:07] it's not? what is the other place? [14:38:08] i doubt wmf git wants to host my non-wmf projects :P [14:38:14] oh well :P [14:38:24] on github i can host all my random projects, wmf and non-wmf related [14:38:26] oh, i see [14:38:27] gerrit can only have wmf ones [14:38:43] addshore: I am, after all, not yet wearing a WMF hat. :-) [14:38:53] sigh, i understand, but .. then there is no code review, and we can't see the bot changes [14:38:54] * duh sticks a WMF hat on Coren  [14:38:56] {{done}} [14:39:02] and stuff on labs is all over the place [14:39:31] im ready to add my new 100 line cron to bots-bnr1 :/ [14:39:53] duh: On the 25th [14:40:24] duh: Though, honestly, if there are no legal obstacles (read: paperwork, etc) I'll bug Ryan to ease in earlier as a volunteer. :-P [14:40:36] :D [14:40:52] hopefully all the omg arbcom people dont give you too much trouble [14:41:09] ill be back later :) [14:41:23] duh: It seems not. Frankly, right now, the only "real" caveat is one of the Arbs. I think she's just being overly cautious. :-) [14:41:50] mutante: the problem is that there never has been any sort of code review for bots, except for probably enwiki which doesnt even technically require it [14:42:16] Coren: Yeah I skimmed it. Personally I couldn't care less as long as you did the job you're getting paid for :P [14:42:37] duh: that's why we want to fix that ?:) [14:42:41] I think the worry is around the job I /don't/ get paid for. :-) [14:44:02] duh: I've seen nobody worry about "OMG an arbitrator is going to be working in engineering" so much as "OMG someone paid by the WMF is on ArbCom" :-) [14:44:54] People need to get their priorities in order :P [14:45:01] I mean, after all, everyone knows the WMF is evil and ArbCom is evil, right? So that'd make be double-evil or something. I think. I'm not sure I ever understood that kind of reasoning. :-) [14:45:28] duh: good that you use git at all, maybe mirroring is an option some day, so it is just on both.. shrug [14:45:55] Don't look at my code :/ [14:46:07] * duh loves git [14:46:21] i remember when i thought svn was the best thing after using cvs for a while [14:47:06] I still prever svn to git. [14:47:17] prefer* [14:47:30] try having a symlink, then deleting it and replacing with a file, then commit to SVN [14:47:51] you'll get some error message about that file having "mysteriously disappeared" or so [14:48:07] running into that when making DNS changes :p [14:48:13] mutante: I don't do things like that. :-) I tend to write code that needs to build on many platforms, so symlinks are verbotten anyways. [14:49:04] de-centralized > centralized [14:49:38] mutante: For many, but not all, use cases. In most, de-centralized ~= centralized in practice. [14:49:56] i give up. it's like OS wars :) [14:50:22] I'm old school. I use the least minimal tool for the job at hand. :-) [14:50:34] ... "least minimal"? [14:50:41] so CVS?:) [14:52:13] mutante: I can think of a couple of scenarios where CVS would suffice. Hell, even RCS. :-) I wouldn't /bother/ downgrading if there's a svn in place though, and I'm pretty sure the maintenance on those has become spotty. [14:54:48] Coren: but.. it's also about giving others the option to send patches.. wiki way .. right [14:55:13] mutante: I didn't say there weren't upsides to the git model either. [14:55:28] ok:) [14:57:10] mutante: It's original use case is the perfect example of where that model is very beneficial (and, indeed, needed) [14:57:18] Its* [14:57:22] Gah, I type like an ass today. [14:58:35] The mediawiki code base is also a good example. [15:02:47] Coren \o/ welcome, cool! [15:14:46] [bz] (8RESOLVED - created by: 2Antoine "hashar" Musso, priority: 4Normal - 6normal) [Bug 36996] [OPS] exim config points to mchenry.wmflabs.org - https://bugzilla.wikimedia.org/show_bug.cgi?id=36996 [15:15:36] [bz] (8RESOLVED - created by: 2Antoine "hashar" Musso, priority: 4Normal - 6normal) [Bug 37079] deployment-prep root tracking bug (tracking) - https://bugzilla.wikimedia.org/show_bug.cgi?id=37079 [15:15:49] [bz] (8RESOLVED - created by: 2Antoine "hashar" Musso, priority: 4Normal - 6normal) [Bug 37081] admin tools on beta (tracking) - https://bugzilla.wikimedia.org/show_bug.cgi?id=37081 [15:16:01] [bz] (8RESOLVED - created by: 2Antoine "hashar" Musso, priority: 4Normal - 6normal) [Bug 37080] upload, thumbnails and transcoding on beta (tracking) - https://bugzilla.wikimedia.org/show_bug.cgi?id=37080 [15:16:50] [bz] (8RESOLVED - created by: 2Jasper Deng, priority: 4Normal - 6normal) [Bug 38433] AbuseFilter and TitleBlacklist do not work on beta wikis - https://bugzilla.wikimedia.org/show_bug.cgi?id=38433 [15:17:15] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 38995] [OPS] udp2log prevents udp2log-mw from starting - https://bugzilla.wikimedia.org/show_bug.cgi?id=38995 [15:17:41] I just converted my svn repository (from the toolserver) to git on gerrit, with lots of help. Should I write a howto in https://labsconsole.wikimedia.org/wiki/Help:Git ? [15:18:17] <^demon> For other toolserver users? Sounds like it could be useful :) [15:19:05] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Normal - 6enhancement) [Bug 41530] setup redis on beta - https://bugzilla.wikimedia.org/show_bug.cgi?id=41530 [15:19:21] I'll just need someone to check what I write, as I'm still new to git. [15:23:11] Silke_WMDE: Yeay! Thanks. [15:37:47] Darkdadaah: That would be VERY much appreciated. [15:38:16] Darkdadaah: And don't hesitate to document what problems you might have had doing so. [15:38:54] Silke_WMDE: Was there a public announcement made already, or did you hear internally? [15:39:25] I read it right here. :) [15:39:46] Ah. You're removing all the mistique. :-) [15:40:11] But yeah, I signed the contract two days ago, effective on the 25th [16:00:47] Here it is (took me longer than I thought): https://labsconsole.wikimedia.org/wiki/Help:Git#Migration_from_svn [16:05:56] Darkdadaah: You win 1.7 brownie points. Full point for the doc, and .7 bonus of appreciation. :-) [16:08:32] :-) [16:08:45] http://mysqlgame.com/ [16:09:27] \o/ [16:09:56] Good timing, time for a snack [16:58:39] Change on 12mediawiki a page Wikimedia Labs/Account creation improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=646399 edit summary: [+274] [17:00:10] Change on 12mediawiki a page Wikimedia Labs/Account creation improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=646402 edit summary: [+33] /* User renaming */ [17:01:27] Change on 12mediawiki a page Wikimedia Labs/Account creation improvement project was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=646403 edit summary: [+58] /* User renaming */ [19:11:02] do you have some news for me, re. E:OpenID [19:11:05] oop [19:11:07] ok [19:11:16] ping me, if you have time [19:11:17] what news would I have? [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:11:40] regarding your "ChooseName" problem - where you sent me the debug log [19:12:27] Change on 12mediawiki a page Wikimedia Labs/Toolserver features needed in Tool Labs was modified, changed by Kolossos link https://www.mediawiki.org/w/index.php?diff=646494 edit summary: [+138] /* OSM */ [19:12:31] what news would I have regarding it? [19:12:34] it's not working [19:12:40] I thought I was waiting on you [19:12:47] Now I am back [19:12:57] and can resume the work [19:13:08] Question: [19:13:26] Can I (how?) also check this effect with a second instance ? [19:13:40] use a socks proxy [19:13:46] uh. [19:13:49] and use the pmtpa.wmflabs addresses [19:14:01] but I need two instance, or ? [19:14:07] ...ces [19:14:19] I'd imagine you can do this with one instance and two wikis [19:14:53] too much work [19:15:11] if you set this up, please, then I do [19:15:12] I don't have time for that [19:15:24] perhaps in the next days ? [19:15:44] I tell you why: [19:15:48] I think, that you [19:15:54] have same framework [19:16:01] for settign up >1 wikis on 1 instance [19:16:47] s/same/some( [19:16:50] arrgh [19:17:18] not my day, today [19:17:21] bye [19:35:30] would some root / ops be kind enough to reboot i-00000576 integration-contintrefactor.pmtpa.wmflabs ? [19:35:39] does not work from the labsconsole :D [19:41:58] Is the gluster outage mentioned on the mailing list affecting ssh login to bastion, or is that a different outage? [19:43:00] stwalkerster: I have no problem sshing to bastion atm. [19:43:15] hmm... [19:43:33] ok. I'll try and figure out what's going on at my end then [19:43:38] ty Coren :) [19:51:07] I'm having problems with `git review`, it prompts for my username then "Password for 'https://spage@gerrit.wikimedia.org':" but then "fatal: Authentication failed". I can login to the gerrit web site from a browser and ssh to it OK. [19:53:36] I'm getting problems logging into bastion.wmflabs.org - server is definitely refusing my key, and I know it is the right key. [19:54:29] Coren, I don't think it's an issue on my end, cos I can log into other systems with the same key no problems. [19:54:51] * stwalkerster will leave it for a while and see what happens [19:55:41] stwalkerster: I don't have the magic bits on bastion to look at the relevant logs. Can you ssh -v to see where it hangs? [19:56:03] I.e.: before, during or after auth? [19:56:16] it doesn't hang, the server actively refuses my key [19:56:42] Huh. What's your username? [19:56:47] stwalkerster [19:57:30] I see your key is accessible from bastion, so that's not it either. [19:58:19] stwalkerster: Sorry I can't do more for you atm, I'm a mere mortal on bastion [19:59:30] yeah, it's weird cos I have a saved putty session that I always use with pageant, with two keys always loaded from the same files, and that's just stopped working, but I can log into other systems which use those keys. [19:59:49] meh, I just figured I'd look to see if it was a "just me" or "everyone" problem [20:00:16] Looks like a "just you" problem right now, which doesn't mean the cause isn't on bastion. [20:01:11] But my diagnostic abilities right now are limited. [20:02:08] hashar, what happens when you reboot from labsconsole? And, did you try rebooting from the instance console? [20:03:50] yeah, don't worry too much about it. :) here's log files in case anyone finds them useful: https://gist.github.com/stwalkerster/2419ee50e4d6b9fc8994 [20:04:02] spagewmf: can you fetch? [20:09:54] andrewbogott: how sorry. So Ryan told me about some Gluster outage, that might be why I can't connect. [20:09:54] it is [20:09:54] andrewbogott: anyone rebooting via the labsconsole says "reboot failed" :-] [20:09:54] andrewbogott, yes I can fetch [20:09:54] hashar: hm, weird. Well, best to let Ryan finish up before we dig too deep. [20:09:54] Ryan_Lane andrewbogott: apparently the instance is back up hehe [20:09:55] spagewmf: OK… I had a similar problem the other day but sounds like what's happening to you is different. [20:10:14] Coren: What's OGS? [20:10:48] andrewbogott, good news I can git review in other projects, so I'll re-init that one. [20:10:52] scfc_de: Open Grid Scheduler, the open source half-brother of the Sun Grid Engine. [20:12:00] scfc_de: The nutshell "I have task X to perform. Ask OGS to run it 'somewhere there is room'." [20:12:37] Coren: Ah, okay. That's the grid-engine (?) package in Ubuntu? [20:12:40] Where 'somewhere' is any one of an arbitrary number of execution nodes that can be flexibly added to or removed without breaking things. [20:13:57] scfc_de: No, that's a different one (but it has similar function). [20:14:46] scfc_de: IIRC, though, gridengine is more geared towards actual parallel job scheduling. [20:16:10] Coren: I have no particular taste in this regard, I just want to avoid the present situation on toolserver where there's effectively no upstream to discuss patches, fixes, etc. :-). [20:16:32] scfc_de: I'm thinking OGS for three primary reasons: it is /very/ well supported by a large community, it is more flexible than gridengine to mold to our needs but also it is drop-in compatible with SGE making the transition easier from the TS [20:17:12] scfc_de: So yeah, there's a robust upstream to talk with. :-) [20:18:26] It's well suited for long-running tasks like bots too: you can actually set things up so that if the execute node running a bot dies, OGS will just restart the task somewhere else. [20:18:44] Doesn't SGE support that already? [20:19:01] scfc_de: It does; it's the same codebase. :-) [20:19:21] scfc_de: But the TS didn't have the resources to deploy it in a true cluster configuration. [20:20:58] scfc_de: Whereas the Labs architecture allows us to do so. [20:32:41] Coren: But don't you have add support for that in your jobs? Also, in Labs CPU and memory is distributed much more fine-grained than any grid engine could do, so an instance idling doesn't mean that CPU is wasted :-). [20:34:20] scfc_de: Obviously, the distribution isn't made on resources which are not at issue. :-) [20:35:13] scfc_de: And no, any job that can be killed-and-restarted will work (most bots). If you /want/, you can add support for cleaner checkpoints (OGS will send you a SIGUSR2 if you're about to go away, which you can use to save state) [20:40:19] Coren: Yeah, but if resources aren't an issue, why set up an elaborate scheme to distribute jobs? :-) The -notify bit is interesting, though on toolserver I'm more often bitten by the query killer. [20:43:05] scfc_de: Reliability. If you have 4 execution nodes (i.e. labs instances) and one dies, only 25% of things get interrupted. [20:43:42] scfc_de: And I expect we don't want to have unbounded instances in the first place, so some distribution will always be necessary (even if it's just a 'n-task-by-node' scheme) [20:45:57] scfc_de: To wit: we have 3 bots instances atm, plus some experimental, plus one just for ClueBot, etc. [20:47:10] scfc_de: From a reliability standpoint, being able to move ClueBot from one instance to another at need, for instance, is a godsent. [20:48:01] Coren: Okay, you made your point :-). [20:54:39] Damianz, ping? [21:31:09] andrewbogott: PONG! [21:35:04] Is gluster still down? [21:37:12] still? [21:38:05] Damianz: I saw the email about it dying earlier, and I was wondering if it was still down [21:38:31] Damianz: automount is throwing wonderful segfaults, and I'm getting lots of "Invalid argument" errors when trying to do anything on the filesystem of the instances I'm on [21:39:52] Finished reading email - unless Ryan_Lane says otherwise right now, yeah it's borked (again) (yay gluster) [21:40:04] still down, yes [21:40:18] that said, not all projects should be affected [21:40:21] most should be up [21:41:22] andrewbogott: Gonna make pizza or maybe rice or maybe noodles, but some form of food - then around a bit if you want salty stuff, then gotta read about esso [21:45:10] Anyone know who Dr. Jeffrey Carver is? [21:48:35] <^demon> Someone who didn't do a good job of eliminating duplicates before sending e-mails. [21:48:42] <^demon> I've already gotten two. [21:50:37] me too [21:50:46] Damianz, lemme know when you're back and fed. [21:52:34] lol [22:24:01] * Damianz sneaks up behind andrewbogott and makes bunny ear signs [22:25:28] Damianz: For starters, I'm just trying to get keystone eauth configure [22:25:36] Probably making dumb syntax mistakes [22:29:50] Ok - how far did you get? Does using salt -T -a keystone * test.ping work? (assuming you configured an acl to allow that) [22:29:57] bastion totally isn't allowing me in atm [22:30:26] atm I can't get salt-master to start, because of http://dpaste.org/5UbaM/ [22:30:52] um… because it's ignoring my venv and trying to operate in /etc [22:31:05] can anybody else not connect to bastion or bots? [22:31:13] I can't [22:31:15] * Damianz frowns at Ryan_Lane [22:31:27] Gluster is flaking out again [22:31:27] now I can [22:31:30] * Damianz frowns [22:31:36] I had to restart all the processes again [22:31:38] I still can't [22:31:45] one min [22:32:07] andrewbogott: --config ./youconfig? [22:32:09] wait lol nevermind [22:32:26] Ryan_Lane: Restarting is never a valid solution [22:32:37] Damianz: says you [22:32:46] Damianz, yeah, it seems like eauth isn't using the pki_dir setting from the config [22:32:47] I was connecting to the wrong username again [22:33:00] I got rid of the damn upstarts [22:33:05] I replaced them with init scripts [22:33:13] and you know what? they actually fucking work [22:33:17] :) [22:33:21] upstart sucks [22:33:21] I hate upstart [22:33:42] andrewbogott: Hmm, let me update my local copy and see [22:33:51] petan around? :) [22:33:56] oh, dammit, it was choking on a tab in the config file. [22:33:57] Damianz, the best solution is the one that works: http://thedailywtf.com/Articles/ITAPPMONROBOT.aspx [22:34:12] andrewbogott: Oh yeah - it's yaml, no tabs [22:34:39] MaxSem: Temp fix [22:34:50] * Ryan_Lane groans [22:35:00] almost every single project has at least one brick down [22:35:09] :D [22:35:27] so, yeah, we're going to be looking for a gluster replacement really soon now [22:35:44] Damianz: ok, ping success! Now let's see if I can make this work via the api [22:36:25] I'm going to try to steal the netapp. not sure I'll be successful there, but we'll see [22:36:41] I like how when authentication fails it returns 0 but doesn't do anything. *scowl* [22:37:14] Damianz, you use curl or the chrome REST console? [22:37:33] curl since it gives you a token back [22:37:39] in theory a browser should work [22:38:14] ok. force starting all volumes [22:38:19] * Ryan_Lane grumbles [22:39:38] Damianz: OK, I can authenticate via the API as well. Guess I was closer than I thought… [22:39:50] awesome [22:39:53] mostly due to salt's weird error handling (e.g. passing a bad password gets me a 500) [22:40:10] yeah - there's a bug about that [22:40:15] ie it should be a 403 [22:42:32] OK, so now I have a token… that's the part returned from curl as X-Auth-Token: right? [22:42:39] mhm [22:43:09] so pass X-Auth-Token back to a call with test.ping and it should work [22:43:40] Hm… nothing on stdout, returns 35 [22:43:47] Same as if I pass a bogus token [22:44:30] curl -si https://localhost:8000 -H "Accept: application/x-yaml" -H "X-Auth-Token: 2774c09901958665934d2232a9677a1bfd3bcb97" -d client=local -d tgt='*' -d fun='test.ping' -d arg [22:45:49] Hmm [22:45:53] Try getting rid of client=local [22:46:06] and arg [22:46:17] I think just Accept/Token header and tgt/fun args are required [22:46:56] So… curl -si https://localhost:8000 -H "Accept: application/x-yaml" -H "X-Auth-Token: 2774c09901958665934d2232a9677a1bfd3bcb97" -d tgt='*' -d fun='test.ping' [22:46:58] same [22:48:04] You don't even get headers back? [22:48:47] andrewbogott: "it was choking on a tab in the config file" YAML? [22:48:59] andrewbogott: Oh yeah - it's yaml, no tabs [22:49:27] YAML is the format Satan rejected as "too evil" [22:49:35] <3 yaml [22:49:46] trying one more thing before spending ages restating all the volumes [22:49:54] Coren: Some WMF coding styles require tabs, otherwise I'd have tabs disabled entirely in vim [22:50:07] spaces ftw [22:50:19] As it is I guess I'd have vim turn it on and off depending on file extension if I were smart [22:50:26] I'm a unix geek. To me, all whitespace is meant to be the same. :-) [22:50:41] Anyway, damianz, yeah, 35 is ssl failure [22:50:47] so not getting far enough for headers [22:50:51] lol [22:50:54] ah yeah [22:50:56] you're using -s [22:51:00] so no warning [22:51:02] -k should work [22:51:13] ok. bringing it all back up [22:52:01] My curl lines are mostly c/p from http://salt-api.readthedocs.org/en/latest/ref/netapis/all/saltapi.netapi.rest_cherrypy.html [22:53:53] wow. that did actually help quite a bit [22:53:57] \o/ [22:54:08] I just cut down the volume force start time by 1/2 [22:55:17] What should we do if an instance isn't appearing on Ganglia? [22:55:22] cry [22:56:09] Damianz: okay [22:56:24] Damianz, curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol [22:56:33] Maybe that just means that my apache isn't write? [22:56:36] Ganglia is a bit dodgy - scripts need tweaking [22:56:59] FastLizard4, I spent some time fixing ganglia the other day… got at least a couple of instances to report correctly. [22:57:30] FastLizard4: I'm happy to investigate more when I have a minute. [22:57:37] I'm betting most of the ones that are not reporting are using puppetmaster::self [22:57:39] andrewbogott: Hmm, that normally means it's expecting http or etc - did you get your token from the command line or the api to start with? [22:57:52] Just try curling / with no args, should get a json response back [22:58:44] andrewbogott: Thanks! The one I'm not seeing is accounts-database (i-00000505); the other two in the project (accounts-application and accounts-puppetmaster) show up fine on Ganglia. [22:59:16] Damianz, I got the token via the api. But you'll see that the example on the doc page for POST doesn't use https [22:59:43] hmm ok - that's weird then, they should both use https [22:59:44] FastLizard4, does puppet run on accounts-database? [22:59:57] andrewbogott: I believe so, lemme quickly run it and see [22:59:58] was the last error trying to use non https? that would explain the message [23:00:05] could you try https and I'll fix the docs if it's borked [23:00:19] andrewbogott: Motd says: The last Puppet run was at Thu Feb 14 22:31:10 UTC 2013 (28 minutes ago). [23:00:43] And it runs cleanly? [23:00:51] Is ganglia running on your instance? [23:01:13] Damianz, so, starting at the beginning… you want me to get a new token but use https? [23:02:01] FastLizard4: is your instance using puppetmaster::self? [23:02:05] FastLizard4: which instance is this? [23:03:41] hm [23:03:45] not using puppetmaster::self [23:04:06] using port 50088 [23:05:11] andrewbogott: What does curl -ik https://localhost:8000 -H "Accept: application/x-yaml" -H "X-Auth-Token: 2774c09901958665934d2232a9677a1bfd3bcb97" -d tgt='*' -d fun='test.ping' do? [23:05:30] Damianz: authenticating, here's how it behaves: http://dpaste.org/OiKvd/ [23:05:58] Basically anything with https fails [23:06:25] I don't think this should make a difference but does -k work on the top line? [23:06:58] nah, same [23:07:26] hmm [23:07:33] openssl s_client -connect localhost:8000 [23:07:41] does it look like ssl [23:08:35] andrewbogott: Sorry, connection wonked out. Yes, puppet does run, and a manual run just finished [23:08:41] hm… no certs but looks ok… http://dpaste.org/A1bEY/ [23:08:46] FastLizard4: Ryan_Lane is on the case [23:08:55] andrewbogott: Ahh, okay. Thansk :) [23:08:57] *Thanks [23:12:27] andrewbogott: stupid question - the cert/key paths are right? valid certs? [23:12:30] digging into the code atm [23:13:16] Damianz, it's perfectly possible that the certs aren't right; I don't really know how to set that up. [23:13:36] salt-call tls.create_self_signed_cert [23:13:49] Ah, yeah, that I've done. [23:14:17] * Coren idly worders who uses saltstack and puppet simultaneously? [23:14:22] wonders* [23:14:33] andrewbogott: Just do curl localhost:8000 and paste the output? [23:14:35] Coren: Me [23:14:38] Coren: us! Soon, you! [23:14:54] Clearly. I suppose the better question would be "Why?" [23:15:13] Puppet for state management and puppet for remote execution [23:15:20] andrewbogott: My salt-api totally isn't acting like yours =\ [23:15:29] Damianz, {"status": "401 Unauthorized", "message": "Please log in"} [23:15:37] ... [23:15:39] ok [23:15:42] I don't get that [23:15:46] I get that over https [23:15:54] but http I get The client sent a plain HTTP request, but this server only speaks HTTPS on this port. [23:16:03] hmmph [23:16:11] Running master? [23:16:12] " Puppet for state management and puppet for remote execution" doesn't leave much room for salt. :-P [23:16:14] So sounds like I have the reverse, port 8000 only speaking http? [23:16:27] Coren: um… 'and salt for remote execution' [23:16:29] Only way to do re with puppet is like mc.... and that sucks [23:16:44] andrewbogott: Oh you're reverse proxying this? [23:16:46] fuck that shit [23:17:15] No, not 'reverse proxy' [23:17:29] Just, the reverse meaning 'opposite' [23:17:49] oh [23:17:55] I was like that's gonna make this hard [23:18:34] ah I wonder [23:18:41] andrewbogott: Are you running debug: true still? [23:18:46] that runs a different server [23:18:52] 'Does not use SSL and ignores the certificate configuration options.' [23:18:56] would explain this shit [23:18:59] * andrewbogott checks [23:19:58] Damianz: I was. [23:20:05] Could you try without :D [23:20:07] So. [23:20:26] Now it's rejecting my cert, and telling me about it. [23:20:34] progress [23:23:38] So I guess I can just -k for now [23:24:29] Bah, there's always a new failure under every failure [23:26:42] *snort* [23:26:48] yeah -k should work [23:26:48] Damianz: with -k and -d client=local it seems to work [23:27:02] awesomeness [23:27:12] Yep, that should get me to the interesting part. [23:27:18] Thanks for talking me through [23:27:30] Totally not helpful heh [23:35:01] very helpful!