[06:25:43] * jeremyb waves paravoid [06:27:35] 11 18:17:09 < jeremyb> i think i broke something in labs... (but probably just effects this one new user) [06:27:38] 11 18:17:41 < jeremyb> got a deadlock timeout try restarting transaction error on labsconsole during account creation [06:27:41] 11 18:18:02 < jeremyb> account exists with the right name and email in LDAP and labsconsole says the account doesn't exist at all [06:27:44] 11 18:25:38 < jeremyb> here's the broken account: https://www.mediawiki.org/wiki/Project:Labsconsole_accounts?diff=571117&oldid=570852 [06:27:52] paravoid: do you wnat to look at it or we should wait for ryan? [06:28:25] also, saw this via #-release: http://www.linux-dev.org/2012/08/glusterfs-wheezy-status/ [06:28:37] want to* [06:32:57] labsconsole just threw me a nova trace [06:33:47] would be really nice if it printed the whole error msg instead of just the first 2.1 words! [06:35:35] opened in a new tab and it worked ok. trace: http://dpaste.com/785501/plain/ [06:35:57] i guess could just be more perf issues? but this was fast i think. not a slow 500 white screen [06:42:43] would be really nice to get more insight into the state of the system [06:43:09] e.g. i just booted an instance and have no idea what's happening with scheduler etc. [06:43:24] or any way to be notified of progress (so i have to poll) [06:50:19] well, i'm seeing nothing. instance is i-000003ad [06:50:22] bbl [07:00:15] are we out of space for new instances? [07:01:33] why is virt2 getting load??? [07:01:44] i thought we were all on 6,7,8 now? [07:02:24] huh, maybe it's not. but weird graph [07:13:35] at least it does exist in LDAP. still no sign on wiki [12:45:59] wtf, that instance *still* never showed up?! [12:49:06] got a stack again, looks the same as last night [12:49:28] andrewbogott_afk: last night's trace: trace: http://dpaste.com/785501/plain/ [12:49:41] gah, "trace: trace:" ;P [12:51:38] hey Jyothis [12:52:48] this is getting to be a regular stack trace... [13:05:57] so, this is getting to be a problem i think. i've now creating 2 instances that are up and running (one still puppetrunning and one done) and both are still redlinks on wiki [13:06:09] tim created one in between mine and it's not a red link [13:06:23] i-000003a[df] [13:06:58] btw, none of those stacks were while making an instance. they were all while just trying to read [[special:novainstance]] [13:09:54] jeremyb: my instance is still a redlink [13:10:10] created on august 3 [13:10:15] ... [13:10:41] btw, the manage instances page works when i reloaded it, but saw lovely error [13:10:44] http://dpaste.com/785657/ [13:11:10] looks about the same as mine [13:11:20] i linked to a stack above [13:11:26] 13 12:52:48 < jeremyb> this is getting to be a regular stack trace... [13:30:41] jeremyb: ? [13:31:03] Jyothis: just hi is all ;) [13:34:25] Ah! Morning morning [14:00:54] morning andrewbogott [14:01:00] 'morning! [14:02:10] andrewbogott: lots of stack traces and other weirdness in labs [14:03:03] I'm still catching up on my email. Can you give me an example? [14:04:04] http://dpaste.com/785501/plain/ [14:04:13] that's happened at least 5 times to me [14:04:20] and also aude at least once [14:05:46] oh, stack traces on labsconsole! I will look. [14:20:22] jeremyb: That happens every time, or intermittently? [14:20:34] (It's similar to an intermittent problem that I've been seeing for a couple of weeks.) [14:34:37] http://dpaste.com/785657/ [14:34:48] my stacktrace [14:37:47] andrewbogott: there's other problems too. sorry, had an outage to take care of (still working on it even) [14:38:17] aude: Intermittent, or every time? [14:38:31] intermittent [14:45:46] yay, pslit be gone! [14:45:46] andrewbogott: intermittent [14:48:42] andrewbogott: other issues too though. instance creation doesn't create nova resource page for instance and no symbolic DNS for instance (only the instance ID DNS) and I created a labs user that somehow ended up with an LDAP account but no mediawiki account [14:48:48] i'm probably missing something even [14:49:18] andrewbogott: lots of issues, just thought the stack trace was closest to your area [14:49:36] ok. I'm pretty sure some of those problems have separate causes. I'm going to look at nova a bit, but probably you'll want to pass that list on to Ryan. [14:49:51] And probably he will say something like "I'm not interested in debugging until after I upgrade later this week." [14:51:11] hah [17:10:29] Ryan_Lane: having a bunch of problems with labs. is there a particular time that's good for you or should i just dump them all on you? [17:10:46] just let me know now [17:11:24] Can't believe it took like 30 attempts to get into ishmael :( Damn typing fail [17:12:17] Ryan_Lane: http://dpaste.com/785501/plain/ has become a common (but intermittent) occurrence. i think i got it 5+ times in an hour? [17:12:39] yeah, that's going to happen until the upgrade [17:12:47] patching isn't easy [17:12:50] ok, it was new to me [17:12:53] so i didn't know [17:13:17] from a few days back: [17:13:17] 11 18:17:09 < jeremyb> i think i broke something in labs... (but probably just effects this one new user) [17:13:20] 11 18:17:41 < jeremyb> got a deadlock timeout try restarting transaction error on labsconsole during account creation [17:13:23] 11 18:18:02 < jeremyb> account exists with the right name and email in LDAP and labsconsole says the account doesn't exist at all [17:13:26] 11 18:25:38 < jeremyb> here's the broken account: https://www.mediawiki.org/wiki/Project:Labsconsole_accounts?diff=571117&oldid=570852 [17:13:43] is it someone that already had svn? [17:13:49] no [17:14:07] 99% sure [17:14:11] and he said no [17:15:27] separate issue: i created 2 new instances last night and no resource page was ever created for either one. tim made one in between my two and his did get a page. aude says she has one from ~10 days ago with no page [17:15:42] i think both mine and tim's have broken arecord [17:15:43] jeremyb: that's also due to the 500s [17:15:56] ok [17:16:32] yeah, arecord still broken [17:16:43] I can fix that. give me a little bit [17:16:47] sure [17:16:51] you can always access the instance by the instance id [17:16:56] right, i know [17:16:57] i- [17:17:03] and i did ;) [17:17:41] btw, fyi, http://www.linux-dev.org/2012/08/glusterfs-wheezy-status/ [17:19:17] when someone gets a chance please pull data (at least mysql data, maybe /etc and /var/www too, or just copy the whole thing and I'll sort it out) from corrupted wep (i-000000c2) and put it on project storage (?) via wep2 (i-000003ad) [17:19:29] that's Jeroen's box ^^^ [17:20:09] You scared him away! [17:23:21] jeremyb: can someone just create an instance in the project? [17:23:23] please? [17:23:40] it'll make my life a million times easier [17:23:52] Ryan_Lane: i think i said i already did that? [17:23:57] 2 even! [17:23:59] what's the instance name? [17:24:14] > via wep2 (i-000003ad) [17:33:20] jeremyb: it's at /mnt/wep.tar.gz [17:33:24] on wep2 [17:33:32] I'll the the arecords soon [17:38:11] danke, /me takes a look [17:38:25] i assume i can then destroy wep at will [17:39:38] I'd wait [17:39:45] actually [17:39:49] you can kill it [17:39:55] the original files are still on gluster storage [17:40:02] wait [17:40:07] no they aren't heh [17:40:14] stupid block migrate deleted them [17:40:27] might be good to check the data first [17:43:40] right, i meant once i'm satisfied ;) [18:24:14] <^demon> Yay, my new project links are already viewable on gerrit-review :) [18:25:27] ^demon: what do you mean? [18:25:33] <^demon> https://gerrit-review.googlesource.com/#/ [18:25:39] <^demon> New top level menu item -> "Projects" [18:25:43] <^demon> So you don't need to go to "Admin" [18:26:50] i think i'm not a fan [18:26:58] both have just one link under them now [18:27:38] <^demon> Well, if you're an admin you have 2 links. [18:27:38] <^demon> "Create new project" [18:27:40] <^demon> Also, when the new "request queue" stuff lands, you could get a "Request new repo" button [18:28:46] <^demon> The *main* point was to get it out of "Admin" so anons could find it, as well as normal people (I mean who looks under Admin for this...really) [18:29:17] It's annoying trying to get to gitweb that way tbf [18:29:31] Would be nice if /r/gitweb just showed all projects in the index or was just generally less lame [18:29:40] <^demon> gitweb sucks. [18:29:46] <^demon> How many times have I said I'm getting rid of it? [18:29:47] <^demon> Really. [18:30:04] gitweb works, the way it's integrated into gerrit sucks for projects :) [18:30:06] why not just make the groups listing available to anons and give the whole tab a new name? [18:30:08] <^demon> Also, I want to add gitweb links to that table, linking to summary or HEAD or something sane. [18:30:16] <^demon> Damianz: No, it sucks. [18:30:20] <^demon> Full stop. [18:30:41] ^demon: well to default branch [18:30:46] Github has a nice browser ;) [18:30:50] or summary [18:31:01] <^demon> jeremyb: Yeah, I'm thinking summary. [18:31:15] i think fedora (and drupal?) just moved to cgit [18:31:21] ^demon: you're home? [18:31:27] <^demon> Unfortunately. [18:31:32] what's happening with the gerrit decision? [18:31:52] cgit makes me think of the mercurial viewer =/ [18:31:54] <^demon> brion was supposed to send out an e-mail friday. He got distracted by [something shiny, I dunno]. [18:32:00] <^demon> Damianz: I'm not using cgit either. [18:34:06] <^demon> https://demo-gitblit.rhcloud.com/ - GitBlit :) [18:34:32] looks sorta nice [18:35:06] <^demon> It's java, so we won't need to install new stuff to the gerrit box for it. [18:35:20] <^demon> Also, some guy is working on a plugin to integrate it more deeply with gerrit, so that'd be nice, long-term. [18:36:19] Has a sorta fish eye feelish [18:36:37] ^demon: you know drupal.org wrote their own sshd (twisted i think) and php<->VCS abstraction layer [18:37:32] <^demon> I didn't. Don't know what I'd use it for though :) [18:38:37] they wanted to be able to have a huge number of users pushing, etc. without a huge number of unix system accounts [18:38:46] I hate the idea of the gerrit ssh server, it's half decoupled and lacks stupid features. [18:39:02] <^demon> What do you mean by that? [18:39:02] isn't lacking stupid features a good thing? [18:39:05] usually? [18:39:32] Damianz: how is the ssh in gerrit a problem? [18:39:33] Things like being able to pull user info (keys) from ldap should be a basic feature if they support ldap in the webside. [18:39:45] that has nothing to do with ssh [18:40:28] the only downside to gerrit implementing ssh is that it's slower [18:40:38] but, I *like* the fact that it's not using the system ssh [18:42:14] <^demon> For most operations it's not all that much slower anyway. The ssh daemon is a small part of your overall load during a request to gerrit. [18:42:20] <^demon> Most of the overhead is gerrit itself. [18:42:44] <^demon> Not that this is all that bad, since we fixed the db latency issues. [18:44:37] yeah [19:28:48] jeremyb: any luck with the mysql dump on that instance? [19:28:56] jeremyb: I fixed the arecords, btw [19:29:06] i looked at the contents but didn't unpack [19:29:12] yeah, i saw the arecords, thanks [19:29:28] Ryan_Lane: can we get /etc and /var/www too ? [19:29:52] sure. gimme a bit [19:29:52] or is that harder than mysql? [19:29:56] not harder [19:30:33] Ryan_Lane: how about that half created account? [19:30:54] did it get created in ldap? [19:33:05] yes [19:33:59] 13 17:13:17 < jeremyb> 11 18:17:41 < jeremyb> got a deadlock timeout try restarting transaction error on labsconsole during account creation [19:34:02] 13 17:13:17 < jeremyb> 11 18:18:02 < jeremyb> account exists with the right name and email in LDAP and labsconsole says the account doesn't exist at all [19:34:05] 13 17:13:17 < jeremyb> 11 18:25:38 < jeremyb> here's the broken account: https://www.mediawiki.org/wiki/Project:Labsconsole_accounts?diff=571117&oldid=570852 [19:36:27] that's really odd [19:36:41] i was thinking maybe it was a race [19:36:51] hm. not too sure how that could happen [19:37:04] * jeremyb creating 2 accounts too fast [19:37:05] gimme a sec [19:37:09] oh [19:37:13] one for him, one for someone else [19:37:21] one worked, one failed [19:37:44] definitely something that needs fixing before anon reg [19:37:47] well... [19:37:47] no [19:37:50] if it's a race [19:38:07] the only race condition I could think of would be due to uid uniqueness [19:38:11] and their uids are fine [19:38:30] and that would also result in their ldap account being rejected [19:38:56] ahhhh, no that's it [19:39:00] they both have the same uid [19:39:09] they do? [19:39:12] yes [19:39:20] ah, well, crap [19:39:26] tom and kaboomki [19:39:36] yeah. they do [19:39:39] damn [19:39:50] i should have checked that earlier [19:39:56] I must not have uid number uniqueness properly configured [19:41:28] jeremyb: ok, he's good to go now [19:41:39] I fixed his uid and he now has a mediawiki account [19:41:49] he'll need to reset his password via "mail me a password" [19:42:15] Ryan_Lane: ahh, but he's special like he was made from SVN, no [[special:log]] entry ;P [19:42:45] yes [19:42:51] because that's how I made him ;) [19:43:08] haha, apergos just got a labs account today?! [19:43:21] anyway, danke [19:43:50] race condition will need to be fixed before registration, yes [19:47:54] ugh, i just realized the link i've been sending people to (which I just copied from past people that already had accounts) is a broken anchor. someone renamed a section [19:48:26] heh [19:52:24] Ryan_Lane: do you understand https://labsconsole.wikimedia.org/w/index.php?title=Help:Access&diff=5026&oldid=5024 ? [19:53:58] no [19:54:09] I really wish people would stop fucking with that [19:54:28] this part should be left removed: This will probably be your common name (first and last name). [19:54:40] k [19:54:55] hm [19:55:05] well, technically the other part isn't wrong either [19:55:11] since now we create accounts via email [19:55:13] i think part of the problem (or at least the reason why i once changed it) is it's not the same UI for everyone [19:55:16] yes [19:55:31] I can't wait till I can enable self-registration [19:55:40] i.e. if you're logged in, you have different hoops to jump through to do a passwd reset [19:55:51] so people try it, it looks wrong and they "fix it" [19:58:56] yeah [19:59:46] andrewbogott: ping [20:00:02] whatup? [20:00:36] nothing, me being silly and not seeing you replied to me on the other channel [20:00:45] sorry. [20:21:22] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=571657 edit summary: /* User:TomT0m */ done completely [20:21:37] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=571658 edit summary: /* User:TomT0m */ [20:21:55] @help [20:21:55] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.2.6 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [20:24:13] @RC+ labsconsole Help:Access [20:24:13] Unable to insert the string to the list because there is no such wiki site known by a bot, contact some developer with svn access in order to insert it [20:24:32] @recentchanges+ labsconsole [20:24:32] There is no such a wiki in list of wikis [20:24:38] ohhhh [20:24:58] Ryan_Lane: can we get labsconsole on irc.wm.o ? [20:26:13] @RC+ meta.wikimedia foo [20:26:13] Unable to insert the string to the list because there is no such wiki site known by a bot, contact some developer with svn access in order to insert it [20:26:20] @RC+ meta_wikimedia foo [20:26:20] Unable to insert the string to the list because there is no such wiki site known by a bot, contact some developer with svn access in order to insert it [20:26:35] @recentchanges+ meta_wikimedia [20:26:35] There is no such a wiki in list of wikis [20:26:42] @recentchanges+ meta.wikimedia [20:26:42] There is no such a wiki in list of wikis [20:26:45] huh [20:26:47] Wow mothers can talk for england [20:32:19] jeremyb: it isn't sending an RC feed [20:32:30] I think there's an open bug for this [20:32:32] right, i thought so [20:32:40] it isn't terribly high on the priority list right now [20:32:45] i thought the bug was about beta labs wikis [20:33:03] that's already done [20:33:03] getting it working for labsconsole would be more trivial i think [20:33:07] but i'm not sure [20:33:09] why would we want that going there? [20:33:19] hm [20:33:26] I guess it'll be more needed when registration is open [20:34:32] anyway, will figure it out some other day [20:35:26] * jeremyb wonders about puppetizing irc anyway [20:35:30] * jeremyb runs away [20:36:01] it's also on the agenda [20:36:06] but not very high priority [21:21:00] Change abandoned: Bhartshorne; "on obsolete test branch" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/9392 [22:09:05] Hello, I'd like to learn how to get an account. [22:09:36] !account [22:09:36] in order to get an access to labs, please type !account-questions and ask Ryan, or someone who is in charge of creating account on labs [22:09:41] Helpful ^^ [22:09:46] !account-questions [22:09:47] I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your preferred email address. 3. Your SVN account name, or your preferred shell account name, if you do not have SVN access. [22:09:50] Hmm [22:09:55] I thought we had a wiki page for this, didn't we [22:10:19] What would #3 be? [22:10:29] https://www.mediawiki.org/wiki/Developer_access [22:10:33] Yes, that page [22:10:40] They said either go there, or come here. [22:10:59] Going there should work [22:11:25] !help [22:11:25] !documentation for labs !wm-bot for bot [22:11:33] !documentation [22:11:33] https://labsconsole.wikimedia.org/wiki/Help:Contents [22:12:02] Nathan2055: assuming you do not have an SVN account, we will need what is listed here https://labsconsole.wikimedia.org/wiki/Help:Access#Giving_users_Labs_access.2C_if_they_don.27t_already_have_SVN_access [22:24:18] I assume you can run a bot on this server? [22:24:56] if it's related to Wikimedia in any way, yea [22:24:58] Yes, there is a bots project for running bots [22:25:20] Yeah, it would be editing Wikipedia. [22:25:48] yes, just join the existing project then as Roan just said [22:27:41] Okay, thanks everyone!