[05:15:33] [bz] (8NEW - created by: 2Michelle Grover, priority: 4Unprioritized - 6normal) [Bug 53335] When I try to access Special Versions on beta labs I get an error - https://bugzilla.wikimedia.org/show_bug.cgi?id=53335 [05:22:03] [bz] (8NEW - created by: 2orenbochman, priority: 4High - 6major) [Bug 50222] tool-labs is not configured correctly to run phpunit tests - https://bugzilla.wikimedia.org/show_bug.cgi?id=50222 [12:06:24] would someone with German fluent knowledge please read https://meta.wikimedia.org/wiki/Grants:IEG/LaTeX_Export and its talk page, and ask the guy what he actually needs? a sysadmin? I got the impression everything can be done for free and requested [12:27:41] [bz] (8PATCH_TO_REVIEW - created by: 2Nemo, priority: 4Unprioritized - 6normal) [Bug 48358] Set up Arab wiki in beta - https://bugzilla.wikimedia.org/show_bug.cgi?id=48358 [12:47:39] [bz] (8RESOLVED - created by: 2Amir E. Aharoni, priority: 4Normal - 6enhancement) [Bug 52222] fill http://he.wikipedia.beta.wmflabs.org/ with some useful data from he.wikipedia.org - https://bugzilla.wikimedia.org/show_bug.cgi?id=52222 [12:53:20] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 53339] migrate beta databases to MariaDB - https://bugzilla.wikimedia.org/show_bug.cgi?id=53339 [12:53:21] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Lowest - 6enhancement) [Bug 45122] migrate all beta instance from Lucid to Precise - https://bugzilla.wikimedia.org/show_bug.cgi?id=45122 [12:53:38] [bz] (8NEW - created by: 2Johannes Kroll (WMDE), priority: 4Highest - 6critical) [Bug 52500] Broken Disk Controller is Broken - https://bugzilla.wikimedia.org/show_bug.cgi?id=52500 [13:17:42] YuviPanda: We can make it an alias is nothing else. [13:21:42] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Low - 6enhancement) [Bug 45525] beta: set $wgCategoryCollation for languages - https://bugzilla.wikimedia.org/show_bug.cgi?id=45525 [13:24:56] [bz] (8NEW - created by: 2Chris McMahon, priority: 4High - 6enhancement) [Bug 53061] support Flow on beta cluster - https://bugzilla.wikimedia.org/show_bug.cgi?id=53061 [13:45:50] Coren: I inserted tools.wmflabs.org to hosts on -dev and it works, but I have no idea how to do that in puppet [13:46:10] I know how to rewrite whole file using puppet, but no idea how to check if it contains some line or not [13:46:31] petan: We [13:46:44] we? [13:47:04] we'd have to write a provider; this should be in DNS anyways, but I don't think we're set for split horizon. [13:47:22] * Coren goes back to figuring out why the hell the controllers have so much trouble. [13:47:26] hmm I mean as a workaround until it's fixed [13:48:11] Coren: there is some guy running "yasbot" on -login, it always eat as much RAM as it can and then it get slowly killed by terminator... (it's hundreds of small processes) [13:48:24] typicaly uses 1600 resident mb of ram on login [13:49:57] it's kind of swapping right now :/ [13:50:57] "some guy"? [13:51:25] well that some guy is called "yasbot" [13:51:32] it's their login user [13:51:39] they don't use service acc [13:51:51] Hm. Goess medieval on his backside. [13:52:56] @seenrx reza [13:52:56] petan: Last time I saw reza they were quitting the network with reason: Quit: Page closed at 6/20/2013 2:14:17 PM (66d23h38m39s ago) (multiple results were found: reza1615, reza1615_) [13:56:45] they died o.o [14:00:58] petan: https://wikitech.wikimedia.org/w/index.php?title=User_talk:%DB%8C%D8%A7%D8%B3&diff=81321&oldid=79842 [14:06:27] Tools-login should feel better now. [14:07:09] Ah, same thing with reza [14:23:54] Coren: how you resolve wiki name from shell name? [14:24:11] getent? [14:24:13] petan: getent passwd is the easiest way [14:24:16] that probably would work... [14:27:03] uhh, anyone else having trouble ssh'ing to a wmflabs instance? i'm trying to ssh to wikistream-web and it is just hanging [14:27:06] [bz] (8PATCH_TO_REVIEW - created by: 2Ryan Lane, priority: 4Normal - 6enhancement) [Bug 43309] Add nagios check to ensure global nfs shares are shared properly from labstore1-4 - https://bugzilla.wikimedia.org/show_bug.cgi?id=43309 [14:28:02] edsu: Works for me; but maybe that project has a stuck gluster. I'll be able to check in a few minutes if you can hold until then? [14:32:44] petan: Trying something on you. [14:33:18] huh [14:33:22] what u mean [14:33:38] petan: Look at your shells on tools-login. :-) [14:33:57] aha [14:33:58] I see [14:34:03] cool [14:34:12] but not sure how are you going to use it? [14:34:22] most of people who run their bots on -login do that using cron [14:34:28] so they probably wouldn't see that [14:34:37] after I killed a bot running in screen [14:34:46] for cron jobs, I disable the crontab and put that in comments. [14:35:01] ah ok [14:35:11] how you do that? [14:35:15] !toolsadmin [14:35:15] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Documentation/Admin [14:35:48] warn-screen [14:38:32] ok [14:54:07] Coren: you can ssh ok? [14:54:21] Coren: sure I'll wait ... [15:01:42] sc [15:03:25] YuviPanda: what do you think about shutting off wikistream-web where wikistream.wmflabs.org points at currently? and point it at your tools instance instead? [15:03:54] YuviPanda: it seems wikistream-web froze up again for reason i am unable to diagnose, and i would rather not have to keep an eye on it like this [15:04:10] edsu: I would love that, yes :) but I can't guarantee uptime until the patches are merged, which I expect to be done this week [15:04:20] edsu: I was hoping to convince you of that, actually :) [15:05:08] YuviPanda: yeah, it's not terribly important that it runs all the time, so downtime is to be expected, as long as things come up again [15:05:15] toollabs is good because the underlying infra is taken care of by other people. [15:05:27] YuviPanda: yeah, i'm starting to see the value in that :) [15:05:51] :) edsu I'll switch it next week. The data for the current routing tables is in a temp. redis instance, and I don't want to guarantee anything [15:05:51] i didn't realize that labs was this unstable [15:06:16] YuviPanda: ok sounds good [15:06:29] edsu: labs in general is good, it is gluster that's fucked up. Usually people just use NFS instead. [15:06:52] YuviPanda: i would gladly use NFS instead, but i don't remember having much control over it [15:07:16] edsu: you can easily do that, but it is sortof not too well documented (I think?) [15:07:34] edsu: however, since wikistream doesn't really need full control over its underlying platform, I always thought toollabs was a better bet [15:07:55] edsu: That instance looks quite dead to me, and it doesn't look like a gluster issue. I think you're stuck having to reboot it. [15:14:34] any smart mind around to fix up integration-jobbuilder.pmtpa.wmflabs which does not let me in ? :D [15:14:40] ssh hang from bastion.wmflabs.org :( [15:15:49] maybe /home is dead for good although I rebooted it a couple times [15:16:03] probably gluster again [15:16:07] ? [15:19:37] Coren: i rebooted it, and it's doing the same thing [15:20:49] edsu: I logged in as root okay, so /that/ seems to be a gluster issue. I'm guessing your userland died because of gluster making it completely hosed and needing a reboot. [15:21:29] Yeah, gluster home volume is hosed. Gimme a min to kick it in the pants. [15:23:39] * Coren loves gluster sooo much. [15:24:06] I love it the same way I love beef: dead, cut into pieces, and placed on an open flame. [15:26:16] edsu: Should be working now. [15:26:35] Coren: while you are cooking GlusterFS related issues, would you mind cooking integration-jobbuilder.pmtpa.wmflabs as well ? [15:26:49] wont let me in although it is up and puppet does run on it (according to console output) [15:27:22] Fun times. [15:27:27] Yeah, looks like. [15:28:12] I did reboot it though [15:30:27] hashar: gluster kicked. [15:30:51] While kicking gluster is an enjoyable activity, it gets old fast. [15:33:59] Coren: we need a new button in openstack manager [kick gluster] [15:34:09] logged in! merci :-] [16:30:04] chrismcmahon: you asked a while ago about betalabs and cirrussearch - I'm not 100% sure right now. I'll have a look in a bit and get back to you. [16:40:47] manybubbles: I'm just getting caught up after taking Friday off, no hurry [16:41:21] chrismcmahon: yeah, I had friday and half of thursday off. We hit an amusement park and drove back from Maine. [16:45:46] Coren: is it easy to switch my homedir over to nfs? [16:55:16] edsu: Relatively so. Technically, all it involves is adding a class to your puppet definitions and rebooting. (Specifically, role::labsnfs::client) [17:04:58] Coren: would that mean my existing gluster homedir would go away, or would the contents of my home directory be available? [17:06:08] edsu: It'd go away. You may want to copy things from one to the other beforehand, though, by mounting the labsnfs filesystem by hand (labnfs.pmtpa.wmnet:/wikistream/home) [17:47:04] Coren: do you know what timezone scfe_de is in? :) [17:47:06] !log deployment-prep rebuilding search indecies to unbreak CirrusSearch.... [17:47:09] Logged the message, Master [17:47:27] YuviPanda: AFAIK, he's at or arount UTC [17:48:07] hmm, okay [17:55:13] [bz] (8RESOLVED - created by: 2Ryan Lane, priority: 4Low - 6enhancement) [Bug 43890] Allow instance resizes - https://bugzilla.wikimedia.org/show_bug.cgi?id=43890 [17:56:56] [bz] (8RESOLVED - created by: 2silke.meyer, priority: 4Low - 6minor) [Bug 44035] Instance failed to build but stays in list after deletion - https://bugzilla.wikimedia.org/show_bug.cgi?id=44035 [18:36:51] Coren: output = 'Unable to run job: denied: host "tools-webserver-03.pmtpa.wmflabs" is no submit host.\nExiting.\n' <-- is this something new? I haven't had issues before submitting jobs from a webserver [18:37:28] legoktm: Oh, hm. Yes, it's new since I didn't make -03 a submit host. Fixing now. [18:37:45] thanks [18:38:18] Done. [18:38:40] yay :D [18:38:40] * Coren wishes gridengine had a proprly puppetizable interface. [18:38:55] Coren not on puppet? [18:42:48] YuviPanda: Not obvious to puppetize; adding things like "X is a submit host" isn't done on the submit host itself, and since we don't have exported resources... [18:43:06] some day I'll read up on wtf exported resources mean :P [18:44:30] NFS exported? [18:47:19] [bz] (8RESOLVED - created by: 2Ryan Lane, priority: 4Low - 6normal) [Bug 46817] Upgrade OpenStack to the Folsom release (tracking) - https://bugzilla.wikimedia.org/show_bug.cgi?id=46817 [20:24:24] Added wikitech to wm-bot's interwiki list today... should be available within a couple days... got a little more other patches/fixes I want to do before compiling, uploading, and restarting... [21:36:52] [bz] (8NEW - created by: 2Chris McMahon, priority: 4High - 6enhancement) [Bug 53061] support Flow on beta cluster - https://bugzilla.wikimedia.org/show_bug.cgi?id=53061 [21:51:42] I'm not able to log into my instance within bastion [21:52:17] I keep getting a 'Permission Denied (publickey) [21:53:27] petan: https://gerrit.wikimedia.org/r/#/c/53145/ you can use that file now [21:54:20] pragunbhutani: what is your key configuration? 2048 SHA2-RS?? [21:54:32] T13: I don't know [21:54:33] which instance, pragunbhutani? [21:54:36] how can I find out? [21:54:36] it might just be gluster [21:54:40] needing a kick [21:54:46] YuviPanda: wikidata-mobile-tester [21:54:51] pragunbhutani: did it work before? [21:54:59] as I recall, yes [21:55:12] that was a few weeks ago though [21:55:54] pragunbhutani: does 'ssh gerrit.wikimedia.org -p29418' work for you? [21:56:13] lemme check [21:56:55] I did successfully connect over ssh [21:57:09] but then connection was then closed [21:57:18] 'Interactive shells are disabled' [21:58:29] pragunbhutani: that's okay. this is 99% probably glusterfs :) [21:58:36] pragunbhutani: but then you'll need Coren or Ryan_Lane [21:58:53] YuviPanda: okay I'll wait for them then :) [21:58:57] ? [21:59:05] hello Ryan_Lane [21:59:10] which instance? [21:59:16] wikidata-mobile-tester [21:59:17] pragunbhutani: might also try rebooting [21:59:19] on bastion, type this: ssh-add -l [22:00:10] I can ssh into it [22:00:24] and I can add files into the filesystem [22:00:35] so I think this is likely due to a key not being forwarded [22:00:59] says 'the agent has no identities' [22:01:06] that's your problem [22:01:12] so, exit from bastion [22:01:19] on your local system, type: ssh-add -l [22:01:42] same message [22:01:53] hmm, I do remember adding my ssh key to it [22:03:47] Ryan_Lane: should I try adding it again? [22:04:15] still no scfe_de :( [22:04:16] yes, add your key to it [22:04:29] pragunbhutani: then make sure it shows up when you do ssh-add -l [22:04:39] then ssh -A bastion.wmflabs.org [22:04:41] then ssh-add -l [22:04:48] to see if it's there [22:05:17] Ryan_Lane: okay I'll go do that and see if it works [22:07:32] Ryan_Lane: it worked! Thanks :) [22:07:37] yw [22:09:46] Ryan_Lane: does the instance come with mediawiki pre-installed? [22:10:00] @mediawiki [22:10:06] @search mediawiki [22:10:06] Results (Found 14): morebots, labs-home-wm, labs-nagios-wm, labs-morebots, gerrit-wm, extension, revision, info, bots, labs-project, openstack-manager, wl, deployment-prep, coren, [22:10:20] -_- [22:10:44] https://wikitech.wikimedia.org/wiki/Help:Single_Node_MediaWiki [22:10:52] !mediawiki-instance is https://wikitech.wikimedia.org/wiki/Help:Single_Node_MediaWiki [22:10:52] Key was added [22:11:10] if you configure the instance to have mediawiki, as shown above, yes [22:11:38] I'll go read, thanks [22:11:53] Ryan_Lane: talking about single node mw.. https://gerrit.wikimedia.org/r/#/c/79955/ [23:04:29] [bz] (8PATCH_TO_REVIEW - created by: 2spage, priority: 4High - 6normal) [Bug 51580] configure beta labs for SUL2 - https://bugzilla.wikimedia.org/show_bug.cgi?id=51580 [23:12:37] argh. I locked my ssh keys in the car. How do I find the Nova instance management panel, from this: [23:12:40] grep certname /etc/puppet/puppet.conf [23:12:43] certname = i-00000459.pmtpa.wmflabs [23:18:29] awight: https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000459 [23:18:39] you append it to Nova_Resource [23:18:44] and that gets you more details [23:19:39] Actions->Configure [23:22:31] mutante: thanks! [23:23:52] The full list of project is very... not discoverable [23:24:54] https://wikitech.wikimedia.org/wiki/Category:Nova_Resource