[00:12:08] Damianz: haven't gotten a chance yet [00:12:25] np [01:16:11] Ryan_Lane: I think I remember the answer to this from before, but… the manage-volumes is just a stray file, right? Not in git or puppet? [01:16:23] it's in git [01:16:25] wait [01:16:27] or is it in svn? [01:16:30] it's in one of the two [01:16:39] I think svn [01:18:45] OK… time for me to learn svn! [01:18:52] What repo? [01:19:55] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by 216.38.130.167 link https://www.mediawiki.org/w/index.php?diff=589404 edit summary: Deleting something that was striked out. [01:21:05] screw svn [01:21:09] git-svn ftw [01:25:54] * RoanKattouw hugs git-svn [01:26:14] * legoktm loves git-svn [01:26:43] Ryan_Lane: Is there a single svn repo that has everything opsy in it? [01:26:52] I mean, everything-opsy-not-get-in-git? [01:26:56] *yet [01:29:57] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=589410 edit summary: [01:30:04] * andrewbogott finds the time period between bugfix and checkin to be a great source of anxiety [01:30:56] heh [01:31:02] it won't get overwritten [01:31:07] since it's in svn [01:31:14] lemme find it in svn [01:31:47] andrewbogott: svn.wikimedia.org/svnroot/mediawiki/trunk/tools/subversion/user-management [01:32:09] then, in the puppet repo, we tell puppet which version of the tools to deploy [01:32:31] we used to have the tools directly in the puppet repo, but they were moved so that the devs could modify them [01:32:48] they could likely be moved back, since it's possible now [01:32:48] Wait, so that file is puppetized? I don't follow, quite. [01:32:59] no. the file is in svn [01:33:09] Puppet does an svn checkout onto labstore2? [01:33:14] puppet says "use svn version " [01:33:20] yeah [01:33:39] so puppet might indeed clobber my file then. [01:33:49] although I haven't seen evidence of that so far [01:33:57] I don't really see the benifit of an svn co to a known version over having them in puppet, the later seems a better idea [01:33:57] nah [01:34:05] it won't clobber [01:34:16] Damianz: I prefer it be in puppet [01:34:21] who broke enwiki [01:34:21] like I said, it was originally there [01:34:22] :( [01:34:32] shit [01:34:44] actually who broke wiki, apparently not just en [01:35:05] * andrewbogott stands back [01:35:11] * Damianz looks at andrewbogott [01:35:25] * andrewbogott keeps his hands where everyone can see them [01:35:56] I think some bits in puppet really need to go away tbh because it makes things a) impossible to use outside of this setup and b) generally confusing. [01:36:27] heh [01:36:35] was a bad deploy [01:38:01] * Damianz mutters something about ci and staging changes quietly in the corner [01:42:11] well... [01:42:21] it was that php5-redis was installed, but apache hadn't been restarted [01:42:41] Shouldn't the package do that when it installs itself? [01:42:44] tasty [01:42:50] RoanKattouw: no [01:43:24] I'm not really a fan of most packages doing service restarts to begin with [01:43:25] mod_php I assume then? [01:43:35] Ryan_Lane: OK, so, given that you gave me the svn path to the /file/ rather to a container… does that mean that SVN lets me check out and commit one standalone file like that? [01:43:49] I thought the php5-apache package or whatever it's called gracefuls Apache in postinst [01:43:55] Oh, wait, I guess it was a dir and not a file, nm [01:44:00] andrewbogott: I gave you a full directory :) [01:44:04] heh [01:44:14] RoanKattouw: that may [01:44:17] * andrewbogott skipped straight from cvs to git [01:44:18] RoanKattouw: but.... [01:44:29] RoanKattouw: remember that php5-redis may be used from cli [01:44:42] without any web server package around [01:44:49] in fact, it can also be used with nginx, etc [01:44:53] After a brief flirtation with tcss. The first s is for 'fictional' [01:44:54] so, which package does it restart? [01:45:05] Aaah [01:45:20] RoanKattouw: basically the only one that should do a restart (and I don't even like this one doing one) is the apache mod for php [01:45:47] if you want to put that dependency in, it's possible for puppet to do so, though ) [01:45:48] :) [01:46:01] See if this was python it would happily just byte compile the source out of sys.path and go on with its day =D [01:46:22] this is a C module [01:46:33] Though I'm just as bad for php as I run php-fpm with multiple pools. [01:49:15] ok. I'm heading home [01:50:46] 10/02/2012 - 01:50:46 - User jasonspriggs may have been modified in LDAP or locally, updating key in project(s): bots,bastion,testing,wikibits,wlmjudging [01:50:58] 10/02/2012 - 01:50:58 - Updating keys for jasonspriggs at /export/keys/jasonspriggs [01:51:40] * Damianz wonders off towards bed [01:52:15] is it just me or is it odd that i can get into bastion with a key but not my instances... [01:52:26] "Permission denied (publickey)" [01:52:57] have you forwarded your key [01:53:50] ya; it was working earlier today; but now it isnt [01:53:57] hmm [01:54:11] * Damianz wonders if andrewbogott is cleaning up stuff still [01:54:42] Damianz: could you be more specific re: 'stuff'? [01:54:59] Pretty sure the access failure is not me. [01:55:55] well that bot/script(s) does dirs and key shizzle :P [01:59:04] Um… true. [02:00:07] JasonDC: What instance/username also paste the ssh output with -vv options [02:00:55] all instances i have access to, specificly wlm-apache1, and user jasonspriggs [02:00:58] and 1 moment [02:01:48] fpaste.org/9VKU [02:01:48] wlm-apache1 seems to be down [02:01:51] its not [02:02:05] its an issue with the monitoring system [02:02:14] no it's not [02:02:18] >> http://wlm.wmflabs.org/ [02:02:22] ryan just blocked pings when he changed the security groups [02:02:41] Damianz: Once you're done with JasonDC, want to coach me on doing an SVN commit? [02:02:47] its been an issue for all of the instances in the project [02:03:31] andrewbogott: If I can remember [02:03:41] weird, the instance is up but ganglia things its down as well as ping failing [02:03:56] your membership is correct and it looks like it's trying the keys properly [02:04:12] try going to https://labsconsole.wikimedia.org/wiki/Special:NovaKey and re-saving your key [02:04:24] im though having the same issue on bots as well [02:04:26] sure 1 sec [02:04:56] * andrewbogott looks forward to learning that he has broken all home directories everywhere [02:05:39] * JasonDC twiddles thumbs for puppet [02:05:48] 10/02/2012 - 02:05:48 - User jasonspriggs may have been modified in LDAP or locally, updating key in project(s): bots,bastion,testing,wikibits,wlmjudging [02:05:59] 10/02/2012 - 02:05:59 - Updating keys for jasonspriggs at /export/keys/jasonspriggs [02:06:52] now im getting Server rufused our key when connecting to bastion [02:07:56] I blame andrewbogott [02:08:28] works for me™ [02:08:40] works for me too funny enough [02:09:03] So, the thing with SVN is that commit gives me a 403. [02:09:15] But it should be using my key from gerrit, right? [02:10:45] 10/02/2012 - 02:10:45 - User jasonspriggs may have been modified in LDAP or locally, updating key in project(s): bots,bastion,testing,wikibits,wlmjudging [02:10:51] assuming you have commit access and you cloned the repo (checked it out with) svn checkout svn+ssh://andrewbogott@svn.wikimedia.org/svnroot/mediawiki/ or such you should be able to commit [02:10:55] 10/02/2012 - 02:10:55 - Updating keys for jasonspriggs at /export/keys/jasonspriggs [02:10:59] You can't commit against http as it's read only [02:11:38] well i can log back into bastion now [02:11:43] not into instances tho [02:11:50] hmm [02:12:18] Damianz: OK, that's my first mistake then. [02:12:28] JasonDC: Try and ssh to bots-1 [02:12:44] same error [02:13:00] well that put a grand total of piss all into the log [02:14:25] fuck [02:14:26] I see why [02:14:45] andrewbogott: I might need you [02:14:57] seems nfs is being crap [02:15:03] getting rpc timeouts from the instances for the key share [02:15:40] Pretty sure I don't know what to do about that :( [02:15:44] Do you? [02:16:07] grrr svn hates me [02:16:45] errr [02:16:53] http://ganglia.wmflabs.org/latest/?c=testlabs&h=labs-nfs1&m=load_one&r=hour&s=by%20name&hc=4&mc=2 < load :( [02:17:16] this is why we need to kill nfs [02:18:18] JasonDC: Try bots-1 again now? [02:18:35] negative [02:19:28] Hmm strange, I actually got it to list the mount's contents without totally lagging [02:23:34] want me to generate a new key pair and see if that fixes anything? [02:24:56] I don't think it will [02:25:09] your keys and user is defintly there let me try 1 think quick [02:25:35] try bots-1 again now [02:27:21] Damianz, would you expect my svn username to be my gerrit account name or my gerrit 'user name'? [02:27:25] That is, web name or shell name? [02:27:33] nope [02:27:44] shell I think [02:27:46] it's ssh so shell [02:27:48] and damn [02:27:53] no idea what's up with this [02:28:38] As far as I can tell it should work and the reason it's not doing so is related to nfs being a douche. Your keys are there and I've just restarted the caches and ls'd the dir so it's all mounted and happy and it doesn't work *frown* [02:29:10] The load of labs-nfs1 does seem rather higher than usual and ~5 instances I've just checked all have rpctimeouts in the logs and laggy mounting.... [02:29:46] The fact that it's putting nothing in the logs and will let me login fine is rather annoying and confusing however [02:32:13] * Damianz thinks it's 3.30am and he's going to bed for a while [02:32:45] nite :) [03:27:02] !log metavidwiki Successfully installed MetavidWiki rev115744, works as expected on MW 1.18.5 with minor fixes. The mvpcf skin currently a no-go: at a minimum, there are fatal makeKnownLink issues (related to Bug 28875 ?) [03:27:17] !ping [03:27:17] pong [03:29:28] labs-morebots is away [03:29:39] damn ir [03:29:41] *it [03:29:44] I really need to fix that bot [06:18:43] test [07:27:04] hello [07:54:45] hi [07:55:13] !ping [07:55:13] pong [07:59:36] test [08:11:50] !ping [08:11:50] gsg [08:11:50] pong [08:17:15] @comm [08:17:15] @help [08:17:15] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.23.2 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [08:17:20] !ping [08:17:21] pong [08:17:37] woot [08:17:41] we have html log now :D [08:18:06] !htmllogs is experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [08:18:06] Key was added [08:19:22] yay :D [08:23:59] hashar hey [08:24:11] petan: good morning :) [08:42:25] The labs instance I was planning to use today doesn't seem to exist. [08:42:40] preilly set it up for me last night. [08:42:47] 18:27 < preilly> jerith: Created instance i-0000047a with image a84558b0-ffaa-4dcd-a020-281b45a87af5 and hostname i-0000047a.pmtpa.wmflabs. [08:42:48] 18:27 < preilly> jerith: it's called vumi-metrics and should be live in a few minutes [08:43:55] Oh, I just found https://labsconsole.wikimedia.org/wiki/Nova_Resource:I-0000047a which claims it's "building". [08:44:56] Is there someone around who can poke it with a stick or something, please? [10:18:56] ajerith: I guess the easiest would be to destroy it and create a new one [10:19:42] hashar: I'm not sure if I have privs to do that. How do I find out? [10:20:03] ajerith: have you created the instance ? [10:20:15] No, preilly did. [10:22:43] ajerith: well it is not listed in the list of instances :/ [10:23:01] and I am not an admin there hehe [10:23:23] hashar: Yeah, I think that's because it's still building or something. [10:24:02] paravoid: could you possibly delete / recreate I-0000047a ? It is stalled in 'building' state on virt5. Informations for creation are at https://labsconsole.wikimedia.org/wiki/Nova_Resource:I-0000047a . [10:24:24] ajerith: should have been build already. Anyway paravoid would be able to help if he is available. [10:24:42] hashar, paravoid: Thanks muchly. [10:28:48] I see it in an error state [10:28:55] no idea why [10:31:37] ajerith: can I (try to) delete the instance? [10:31:44] are you sure you have no useful data there? [10:32:06] paravoid: got created by preilly so there should be nothing on it [10:32:09] paravoid: preilly set it up for me last night, and I haven't been able to get to it at all. [10:33:08] okay, deleted [10:42:16] paravoid: Can you set up a new one the same for me? [10:43:34] * ajerith is out for lunch for a bit. [12:38:36] !log bots petrb: upgrading bot [12:38:38] Logged the message, Master [12:39:00] !log bots petrb: rebooted wm-bot [12:39:02] Logged the message, Master [12:39:13] @ping [12:39:14] @help [12:39:14] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.23.8 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [12:39:22] !ping [12:39:22] pong [12:40:09] :) [12:40:10] cool [12:50:47] @seenrx sumanah [12:50:48] petan: Last time I saw sumanah they were quiting the network N/A at 10/2/2012 2:05:52 AM (10:44:55.4404590 ago) [13:29:22] Change on 12mediawiki a page Developer access was modified, changed by DAndreescu link https://www.mediawiki.org/w/index.php?diff=589701 edit summary: [14:01:09] @seenrx chrismcmahon [14:01:09] chrismcmahon: are you really looking for yourself? [14:01:23] petan++ thanks for @seenrx! [14:11:38] @seenrx Damian. [14:11:38] Damianz: Last time I saw Damianz they were talking in the channel, they are still in the channel #wikimedia-labs at 10/2/2012 2:11:38 PM (00:00:00.1778290 ago) (multiple results were found: Damianz_) [14:11:41] heh [14:11:44] dumb bot [15:30:49] <^demon> Trying to update packages on an instance, getting, "Could not connect to brewster.wikimedia.org:8080" [16:06:27] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=589762 edit summary: /* User:DAndreescu */ not done [16:06:55] ^demon: you should mention that to leslie [16:07:30] <^demon> Leslie is traveling today I believe. [16:13:15] <^demon> jeremyb: Seems to be working now. mark did something to brewster. [16:13:31] ok [16:13:53] ^demon: i was just thinking about the labs can't ssh to TS issue she fixed [16:14:01] but i guess not relevant [16:14:26] <^demon> Yeah unrelated I think :) [16:15:17] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=589773 edit summary: /* User:Luis Felipe Schenone */ done [16:17:37] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=589779 edit summary: /* User:Das Schäfchen */ not done [16:17:51] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=589780 edit summary: /* User:Atrawog */ done [16:18:18] <^demon> I hate you maven. [16:18:22] <^demon> You lie. [16:18:46] ^demon: we've now had a second user request an account again with the same name as their past request which was already granted ;-) [16:19:36] {{fact}} (re lies) [16:20:11] <^demon> I wrote a plugin for gerrit that uses the github api. So I added it to the dependencies list, and it compiled. [16:20:18] <^demon> But trying to load the plugin gives me ClassNotFound. [16:21:01] so you hate maven or the whole java ecosystem? [16:22:26] <^demon> Not sure. [16:22:30] <^demon> Could be the latter. [16:24:13] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=589784 edit summary: /* User:Quentinv57 */ done! [16:25:21] jeremyb, thank you :) [16:25:24] ^demon: do we have a stance on umlaut in gerrit name? [16:25:35] 02 16:17:37 <+wm-bot> Change on mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=589779 edit summary: /* User:Das Schäfchen */ not done [16:25:38] <^demon> Should be fine, we fixed utf8 ages ago. [16:25:44] Quentinv57: sure ;) [16:25:48] <^demon> Just not in shell name, obviously. [16:25:56] ^demon: and spaces? [16:26:05] <^demon> Also fine, and common. [16:26:13] <^demon> (again, not for shell) [16:26:17] right, but i thought we were avoiding spaces [16:26:49] <^demon> I don't see why. It allows you to be "John Doe" instead of "JohnDoe" or something. [16:27:03] i thought there were a bunch of confirmed, working spacey people and some degree of known problem with some spacey people (maybe fixed somehow) [16:27:09] right, sure [16:31:10] jeremyb, can you help me to configure my labs account ? [16:31:20] idk... [16:31:23] you can ask the wind [16:31:32] the wind ? :P [16:31:39] maybe i have time, maybe i don't [16:31:40] yes [16:32:58] jeremyb, but now that I have a WMF Labs account, how can I connect to SSH, etc ? [16:33:09] there are docs... [16:33:14] did you follow my link? [16:33:32] do you mean the one on the mail ? [16:33:45] no [16:33:54] the link in the wiki change i made [16:34:45] Quentinv57, what is your labs username? I may need to set you up with shell access. [16:35:08] hello andrewbogott, it's quentinv57 [16:35:16] ok, just a moment... [16:35:44] jeremyb, okay, but that just helps to log in... [16:35:48] andrewbogott: me too me too [16:35:53] username:Mardetanha :) [16:36:04] andrewbogott: i'll need to get some labsconsole debug logs... [16:37:25] andrewbogott: shell username length limit is what? can we triple check it? [16:37:31] hey Mardetanha ! :) [16:37:41] hey Quentinv57 :) [16:37:50] Quentinv57, Mardetanha: OK, you should be all set. First step is to see if you can ssh to bastion.wmflabs.org [16:38:24] jeremyb: No idea what the length limit is… what's the context for this? [16:38:33] andrewbogott: acct creation fail [16:38:43] I get this [16:38:43] If you are having access problems, please see: https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [16:38:43] [16:39:02] the error msg is familiar from when i tried to create an account for someone that already had one. i think [16:39:11] > There was either an authentication database error or you are not allowed to update your external account. [16:39:25] i'm very certain in this case though that the account doesn't exist already [16:39:29] Mardetanha, that's a good place to start :) Do you have keypairs set up &c? [16:39:31] andrewbogott, but there's no need for a public key ? [16:39:44] andrewbogott: yes, it is already uploaded to wikilabs [16:39:45] the docs at the top of the acct req page say 16 chars, this one is 15 chars [16:39:52] Quentinv57: Yes, bastion will be checking your key. [16:40:06] jeremyb: What's the name? [16:40:19] andrewbogott: https://www.mediawiki.org/wiki/Developer_access#User:Hemant_wikikosh [16:40:26] andrewbogott, yep, but do you have my public key ? [16:40:36] it rejected me because of the key [16:40:43] 10/02/2012 - 16:40:43 - Created a home directory for mardetanha in project(s): bastion [16:40:51] jeremyb: I'm not feeling confident about that underscore. [16:40:56] andrewbogott: i am now logged in [16:41:07] Mardetanha: great! [16:41:08] andrewbogott: the docs say it's ok i think [16:41:32] jeremyb: Docs may be wrong. I can look up the actual regex but in the meantime maybe try without? [16:41:36] well sure [16:41:43] i can try with a - instead [16:42:18] andrewbogott, I have the error "server refused our key" [16:42:57] Quentinv57: Two things: 1) you'll need to upload your key to labsconsole /and/ to gerrit (I'm not sure which one the shell uses) 2) It'll take several minutes after you upload it for projects to actually know about it. [16:44:11] shell uses the one in labsconsole unless it's the gerrit ssh server then it uses gerrit one [16:44:19] andrewbogott, I'm sorry but I don't know how to upload it (I don't know if I have a gerrit access btw) [16:44:53] Quentinv57: For starters, here: https://labsconsole.wikimedia.org/wiki/Special:NovaKey [16:45:11] …and… here https://gerrit.wikimedia.org/r/#/settings/ssh-keys [16:45:13] "There were no Nova credentials found for your user account. Please ask a Nova administrator to create credentials for you. " [16:45:14] :D [16:45:21] And, you /do/ have gerrit access, or you wouldn't have gotten this far. [16:45:34] quentinv57: Log out of labsconsole and back in again and try again. [16:45:49] 10/02/2012 - 16:45:49 - User mardetanha may have been modified in LDAP or locally, updating key in project(s): bastion [16:45:51] Labsconsole accounts derive from gerrit accounts, sort of. [16:46:12] Aha! Quentinv57, see that message from labs-home-wm? You'll see one of those once your key has percolated through. [16:46:35] :D [16:46:41] * Damianz goes back to moving sites around [16:47:56] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=589807 edit summary: /* User:Hemant wikikosh */ done with some changes [16:48:11] ok… jeremyb, if the user name with a - works then I'm not sure I want to dig into code at the moment. Do you mind filing a bug about the _ issue? [16:48:25] That or you could just change the docs to discourage underscores :) [16:48:35] right [16:48:48] andrewbogott: i can later, thanks for talking it through [16:48:55] (should do both) [16:49:25] jeremyb: Pretty sure this isn't the first time I've seen that the docs and the web-validation and the backend-validation all disagree about things. So hopefully it won't fall through the cracks this time :/ [16:53:04] andrewbogott: well i've certainly seen it! [16:53:20] i've even seen apps say that 10003 isn't a valid zip code! [16:53:39] (and all other 5 digit ones i try that i know exist are not valid either) [16:53:56] Quentinv57: http://blog.gerv.net/2010/01/from_the_wind/ [17:21:46] !log metavidwiki Successfully installed MetavidWiki rev115744, works as expected on MW 1.18.5 with minor fixes; Bug #37621 still outstanding. mvpcf skin currently a no-go: at a minimum, there are fatal makeKnownLink issues (likely related to Bug 28875 ?) [17:22:01] !log metavidwiki test [17:22:02] Logged the message, Master [17:22:18] !log metavidwiki Successfully installed MetavidWiki rev115744, works as expected on MW 1.18.5 with minor fixes; Bug #37621 still outstanding. [17:22:19] Logged the message, Master [17:22:31] !log metavidwiki mvpcf skin currently a no-go: at a minimum, there are fatal makeKnownLink issues (likely related to Bug 28875 ?) [17:23:20] !log metavidwiki mvpcf skin currently a no-go: at a minimum, there are fatal makeKnownLink issues (likely related to Bug 28875) [17:23:33] !log metavidwiki mvpcf skin currently a no-go. at a minimum, there are fatal makeKnownLink issues (likely related to Bug 28875 ?) [17:23:59] !log metavidwiki mvpcf skin currently a no-go. at a minimum, there are fatal makeKnownLink issues, related to Bug 28875? [17:24:21] !log metavidwiki mvpcf skin currently a no-go. fatal makeKnownLink issues, poss. related to Bug 28875 [17:24:36] !log metavidwiki mvpcf skin currently a no-go. makeKnownLink issues, poss. related to Bug 28875 [17:24:48] it's one of these words... [17:24:57] * Damianz kicks labs-morebots in the face [17:25:09] !log metavidwiki mvpcf skin currently a no-go, poss. related to Bug 28875 [17:25:15] GChriss: It randomly gets an error when it tries to edit the page [17:25:23] possibly due to us using the login in multiple places [17:25:38] ah [17:25:53] so the following should fail: [17:25:57] !log metavidwiki test [17:25:59] Logged the message, Master [17:26:03] yup [17:26:26] !log metavidwiki mvpcf skin currently a no-go, poss. related to Bug 28875 [17:26:37] !log metavidwiki mvpcf skin currently a no-go [17:26:38] Logged the message, Master [17:26:44] yeah dunno [17:26:49] lol [17:26:52] the error back is an edit page issue [17:27:02] only url stuff is blocked as far as I know [17:27:14] * Damianz wonders if we have some edit filters [17:27:31] !log metavidwiki mvpcf skin currently a no-go, poss. related to Bug #28875 [17:59:57] Change on 12mediawiki a page Developer access was modified, changed by Hemant wikikosh link https://www.mediawiki.org/w/index.php?diff=589832 edit summary: /* User:Hemant wikikosh */ [18:15:49] !log wikistats added wikimedias_csv, fixed links in wikimedias_html, updates WMF wiki versions, ... [18:15:50] Logged the message, Master [18:22:34] !log wikistats fixed updates for "vep" and "nso" wikipedias (did not try API method). [18:22:35] Logged the message, Master [18:28:11] !log wikistats fixed language name for arc.wp "Assyrian Neo-Aramaic" -> "Aramaic" [18:28:12] Logged the message, Master [18:30:49] 10/02/2012 - 18:30:49 - Created a home directory for mardetanha in project(s): auth-gateway [18:35:47] 10/02/2012 - 18:35:47 - User mardetanha may have been modified in LDAP or locally, updating key in project(s): auth-gateway [18:45:47] 10/02/2012 - 18:45:47 - User jasonspriggs may have been modified in LDAP or locally, updating key in project(s): bots,bastion,testing,wikibits,wlmjudging [18:45:58] 10/02/2012 - 18:45:58 - Updating keys for jasonspriggs at /export/keys/jasonspriggs [18:53:45] Ryan_Lane, hi :D [18:54:22] hey, great to see you back :) [18:54:50] adminxor: would you mind pushing in a change to send some logs to yourself? [18:54:57] have you used puppet yet? [18:55:36] How do I go about pushing in a change? [18:55:42] !git [18:55:42] For more information about git on labs see https://labsconsole.wikimedia.org/wiki/Help:Git [18:55:56] you need to check out the repo, then add in some config for syslog [18:56:00] then push it in to gerrit [18:56:06] have you logged into gerrit yet? [18:56:16] ahh okay. Thanks. [18:56:32] Nope. I dont know if I have an account there too. [18:57:25] I will start with puppet very soon. [18:58:31] if you have a labs account you have a gerrit account [18:59:34] adminxor: yeah, you'll need to log into gerrit and upload your ssh key [18:59:41] I can't wait till that's pulled from LDAP [18:59:55] it's actually less secure to have in both places [19:00:04] Ryan_Lane: Did you see my grumbling about nfs being a pile of meh and lagging on the key export last night that seemed to be causing login failures for someone. [19:00:10] ah [19:00:12] crappy [19:00:21] yeah. that should be replaced soon [19:00:28] it's really really bad sometimes :( [19:00:29] it shouldn't be horrible, but I need to schedule it [19:02:15] also I fixed you whitespacenessish and things in the osm change [19:16:43] <^demon> Ryan_Lane: hi :) [19:22:00] !log glam installed apache2, php5, mysql5, phpmyadmin with apt-get [19:22:01] Logged the message, Master [19:22:55] !log glam in instance glam-gwtoolset-apt [19:22:56] Logged the message, Master [19:24:03] ^demon: howdy [19:24:12] Damianz: cool. I'll review and merge that change in soon [19:24:24] <^demon> Ryan_Lane: https://gerrit.wikimedia.org/r/#/c/26329/ :) [19:24:28] <^demon> I think we're finally ready [19:24:45] :o [19:28:40] ^demon: sweet [19:28:44] does this require 2.5? [19:29:05] <^demon> Nope, will work with both [19:29:14] ^demon: this ensures that the private repo won't be replicated, right? [19:29:28] <^demon> Right. The authGroup only has READ for mediawiki/core. [19:29:33] oh [19:29:34] <^demon> DENY'd everything else. [19:29:37] no extensions? [19:29:42] <^demon> Not yet. Baby steps :) [19:29:44] heh [19:29:56] merged [19:32:10] did gerrit need to be restarted for that? [19:32:24] because it didn't restart [19:32:47] <^demon> Yeah, it needs to. I should tweak the service definition to also watch replication.config [19:33:08] restarting [19:34:08] ^demon: failed to start [19:34:12] really need a less ugly 503 page [19:34:42] Caused by: org.eclipse.jgit.errors.ConfigInvalidException: remote.github.url "git@github.com:mediawiki/core.git" lacks ${name} placeholder in /var/lib/gerrit2/review_site/etc/replication.config [19:34:47] <^demon> Pfft. [19:34:49] <^demon> Stupid gerrit. [19:34:50] <^demon> Fixing. [19:35:01] what's the fix? [19:35:03] I'll add manually [19:35:17] oh, or you can :) [19:36:49] <^demon> Blah, can't write. [19:36:56] <^demon> The url line should be: "url = git@github.com:${name}" [19:36:58] <^demon> (No quotes) [19:37:42] url = gerrit2@formey.wikimedia.org:/var/lib/gerrit2/review_site/git/${name}.git [19:37:58] <^demon> That one's fine. [19:37:59] wee [19:38:00] err [19:38:03] <^demon> This is in the github section [19:38:10] url = git@github.com:mediawiki/core.git [19:38:48] failed [19:39:01] <^demon> Wtf. [19:39:07] address in use [19:39:11] it likely half-started [19:39:44] Caused by: java.net.BindException: Address already in use [19:39:53] I don't see anything running there... [19:40:05] <^demon> Neither do I. [19:40:18] 8080? [19:40:20] is that its port? [19:40:29] <^demon> Yeah, jetty's on 8080. [19:40:31] <^demon> Apache proxies that. [19:40:39] apache is running on that port for some reason [19:40:48] <^demon> Um... [19:40:51] <^demon> That's not right [19:41:20] tcp 0 0 127.0.0.1:8080 127.0.0.1:8080 TIME_WAIT - [19:41:22] * Ryan_Lane sighs [19:41:51] yay for hung sockets [19:42:10] closed [19:42:29] back up [19:42:42] ^demon: please push in your change before puppet runs again :) [19:42:51] <^demon> k [19:44:14] <^demon> https://gerrit.wikimedia.org/r/#/c/26353/ [19:45:02] <^demon> Well, it's not replicating. Github's denying the key. [19:45:08] <^demon> But hey, at least it's getting this far :) [19:45:09] shouldn't that be mediawiki/${name}? as the paths aren't going to match or are you just going to replicate mediawiki and create other users for other stuff? [19:45:41] <^demon> That's just for replicating mediawiki/* projects. [19:45:46] <^demon> Other stuff may go elsewhere. [19:46:20] <^demon> Plus, direct replication isn't going to work, since github doesn't let you put /'s in repo names. [19:46:32] <^demon> Gonna have to modify replication plugin to let you rewrite repo names. [19:46:34] <^demon> :\ [19:46:48] <^demon> So yeah, can only do core right now :( [19:47:29] probably a good idea to be able to do to longer term, though then you're split to manual replication setups rather than group based... unless you could do regex based renaming [19:47:48] * Damianz thinks one day we might auto create the repo using the gh api and let magic flow automatically =D [19:48:02] <^demon> I actually started a hacky plugin to do that :) [19:48:11] <^demon> Having trouble getting it to run [19:48:47] Kind of a shame you can't push to /random name and have it auto created on gh's side [19:58:57] <^demon> Ryan_Lane: Can gerrit2 write to /var/lib/gerrit2/.ssh/known_hosts? [19:59:11] doubtful [19:59:30] I take that back [19:59:30] yes [19:59:32] it can [19:59:38] <^demon> :( [19:59:40] <^demon> Dammit. [20:01:29] <^demon> Are there keys stored as ecdsa-sha2-nistp256 instead of ssh-rsa? [20:06:59] ^demon: nope [20:07:20] <^demon> Well gerrit's just being retarded then. [20:07:28] <^demon> Meh, it's harmless for now. [20:07:29] did you look in the error log? [20:07:33] <^demon> Yep. [20:07:34] replication to formey is failing too [20:07:55] Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 52e7dd5c76cd4c6e673a9aa6cc7cadee9ad77fb7 [20:08:08] <^demon> That's known. [20:08:13] ah [20:08:13] ok [20:08:14] <^demon> Missing object still from downtime. [20:08:46] * Ryan_Lane nods [20:19:59] andrewbogott: you may add me to auth-gateway [20:23:38] <^demon> Ryan_Lane: Can you grant me sudo on manganese for a bit so I can poke around ~gerrit2/.ssh? [20:32:26] ^demon: sure [20:32:37] Mardetanha: I already added you there [20:32:54] Ryan_Lane: thanky :) [20:32:58] yw [20:52:16] <^demon> Ryan_Lane: Don't worry about it actually, I'll poke tomorrow. Harmless to leave as-is for tonight. [20:52:37] ok [20:53:37] andrewbogott: oh, had you fixed gluster on labstore? [20:53:56] the rmdir and removing the attrs on /a fixed it? [20:54:02] Should be fixed, yeah. [20:54:05] cool [20:54:27] I can't access svn to check in the fix though. Should use the same key as gerrit, right? [20:54:57] ldap probably *shrug* [20:54:59] <^demon> Need to be added to svn's authz. [20:55:34] ^demon: That could be it! How do I do that? [20:55:59] <^demon> http://wikitech.wikimedia.org/view/Svn#Access_Control [20:56:23] <^demon> `/usr/local/bin/svn-group add mediawiki` is probably what you need [20:56:29] <^demon> (Do that on formey) [20:56:41] <^demon> Actually, I'll go ahead I'm logged in [20:56:47] thanks! [20:57:11] <^demon> andrewbogott is your shell user, right? [20:57:51] andrewb in production and andrew in labs, so… 'andrew' for svn I think. [20:57:59] <^demon> andrew it is. [20:58:13] <^demon> K, you should have write access to SVN now. [20:58:21] <^demon> 'cept stuff that's readonly cuz I moved it to git. [20:58:39] thanks! [20:59:12] <^demon> yw [21:00:03] So, for example… svn checkout svn+ssh://svn.wikimedia.org/svnroot/mediawiki/trunk/tools/subversion/user-management [21:00:08] ? [21:00:16] (this I ask because it doesn't work :( ) [21:00:49] 10/02/2012 - 21:00:48 - User andrew may have been modified in LDAP or locally, updating key in project(s): testlabs,gluster,openstack,bots,bastion,gerrit,globaleducation,deployment-prep,phabricator,mwreview,signwriting,pdbhandler,wikibits,translatesvg,conventionextension,wlmjudging,account-creation-assistance,watchlist-groups,utrs [21:00:51] <^demon> Yeah, should be it. [21:00:59] 10/02/2012 - 21:00:59 - Updating keys for andrew at /export/keys/andrew [21:01:37] argh [21:01:44] fucking toolserver is having issues again [21:02:24] * Damianz thinks he really should patch this code to fallback on the api [21:02:27] ^demon: Any ideas how to troubleshoot? I've recently talked to gerrit from this system so pretty sure my key is set up right... [21:02:47] <^demon> What's the error you're getting? [21:02:57] svn: Network connection closed unexpectedly [21:03:05] oh [21:03:08] this is from labs? [21:03:14] we still block ssh to that, I think [21:03:17] Oh, wait... [21:03:17] you need to do so outside of labs [21:03:24] <^demon> Yeah, that's ssh to the cluster. [21:03:33] <^demon> Makes sense now. [21:03:37] <^demon> Yeah not gonna work :\ [21:03:37] My mistake, I accidentally ssh'd to labs because I was testing my key :) The actual message is... [21:03:46] svn: To better debug SSH connection problems, remove the -q option from 'ssh' in the [tunnels] section of your Subversion configuration file. [21:03:47] svn: Network connection closed unexpectedly [21:03:49] <^demon> Gerrit only works because it's 29418 and not 22 ;-) [21:03:51] That's from my normal development VM [21:04:32] <^demon> andrewbogott: That's a key auth issue, not an authz issue. [21:04:46] <^demon> Ryan_Lane: Does he have to be in the svn group? [21:04:48] <^demon> Or something. [21:05:29] <^demon> We could just move the path to git instead and pretend svn doesn't exist :p [21:06:02] I strongly support any solution that allow me to pretend that svn doesn't exist [21:06:05] Or just move the file to puppet where it should be [21:06:16] Since it does a svn co currently [21:06:37] ^demon: yes. needs to be in mediawiki core group [21:06:43] and in svn group [21:06:49] <^demon> I added him to svn's authz. [21:06:54] <^demon> Not ldap group--I can't do that. [21:07:01] <^demon> (ldap group script dies when I try) [21:07:04] ah [21:07:06] right [21:07:27] he's in svn group in ldap [21:07:33] <^demon> Hm. [21:07:34] he needs to be in mediawiki core [21:07:43] <^demon> Doesn't need to be in core, just 'mediawiki' [21:08:07] noi [21:08:16] hm [21:08:21] I could have sworn he needs to be in core [21:08:21] <^demon> Plus, if it was svn's authz he'd be getting errors on commit, not co. [21:08:26] true [21:08:37] andrewbogott: are you trying this outside of labs? [21:08:38] It's very likely that I'm doing something dumb. [21:08:40] inside of labs ssh is blocked [21:08:58] Not in labs. [21:09:34] Hm… what's something else I can do that will verify that I"m sending my key properly? [21:11:19] ssh directly to the box [21:11:44] That's allowed? [21:12:09] That key comes from ldap which comes from the gerrit interface, yes? [21:12:21] andrewbogott: add -vvv to the ssh command line [21:13:16] <^demon> Key comes from ldap which comes from labsconsole. [21:13:26] <^demon> gerrit ssh keys are separate. [21:13:57] ok. [21:14:03] gerrit really needs to use ldap long term :( [21:14:07] (It should be there as well, but I will now triple-check) [21:14:13] oh [21:14:13] <^demon> Damianz: Yes, I know. It's stupid and I've complained. [21:14:15] <^demon> Loudly. [21:14:18] it doesn't come from ldap [21:14:31] ^demon: we need to manually update users' keys on formey [21:14:37] <^demon> We do? [21:14:37] I didn't want to open a security hole [21:14:38] yes [21:14:40] <^demon> I forgot that. [21:15:17] surly having multiple keys around where one potentially gets comprimised and upon replacing the keys the user misses one it's a greater security issue. [21:15:25] <^demon> I can't wait until the rest of SVN is r/o and I can forget about it. [21:16:11] r/o? just rm -rf it =D [21:16:18] ok, this sounds promising! Where on formey should I stick my key? [21:16:45] <^demon> /home/$user/.ssh/authorized_keys [21:17:11] Damianz: well, ideally we'll be turning off svn write access eventually [21:17:21] Oh! But… is that dir shared with svn.wikimedia.org? Or are they the same machine? [21:17:33] I bet there will be at least 1 guy who refuses to move and makes a huge fuss. [21:17:35] <^demon> formey == svn.wikimedia.org [21:18:24] *light bulb on* [21:18:54] * Damianz watches the light bulb explode [21:19:37] Damianz: too bad for that person [21:19:44] Damianz: we aren't supporting svn forever [21:20:42] It will end in a 40+ long thread on the mailing list documenting the 500 ways in which everyone sucks and someone will probably get burnt from the flames and I'll probably choke on ice cream while reading it. [21:20:46] Fun times. [21:22:36] <^demon> Well hopefully we can get the last of the active extensions out soon. [21:22:48] <^demon> Should do an inventory of non-extension stuff and see what the plan is for that. [21:23:29] ok, and… 'svn commit' actually pushes the patch upstream, right? [21:23:37] Or does that just create a patch which I have to subsequently push? [21:24:03] <^demon> No, commit also pushes. [21:24:08] <^demon> Ryan_Lane: btw, I managed to get a sane *.war built today of 2.5. We should schedule an upgrade window soonish. [21:24:11] <^demon> Anyway, calling it a day. [21:24:16] sounds good to me [21:24:20] ^demon: later [21:24:56] svn doesn't really do 'push' you're working off one repo which really fucking sucks [21:25:11] Damianz: OK, same as cvs, I can dig it. [21:25:14] Everything is working now, thanks all. [21:25:45] cvs... *shdder* [21:26:00] I sorta like cvs. [21:27:43] I should say: cvs is a big improvement over rcs, which I have also used in a production environment [21:28:30] Yeah, svn is a small improvement over cvs and git kicks all their asses [21:31:33] What does svn do that cvs doesn't? I never understood why it was necessary in the first place. [21:52:56] A little quicker due to the use of a db for stuff, some meta data bits, transactions (I think) and some newer ideas for internals. Usage wise really not that much. [22:40:57] 10/02/2012 - 22:40:57 - Creating a home directory for atrawog at /export/keys/atrawog [22:45:53] 10/02/2012 - 22:45:53 - Updating keys for atrawog at /export/keys/atrawog [23:45:16] Damianz: any reason we can't add this package to the bot's puppet class? https://labsconsole.wikimedia.org/wiki/Help:Locale_errors [23:45:29] we have one of those, right? :) [23:45:44] nope [23:45:50] we should be encouraging people to document stuff they are installing, and we should install it on all bots nodes [23:45:58] we can add that, we need a labs::bots::application class though [23:46:13] let's just make a bots module [23:46:25] then we can have a role::bots::labs [23:46:25] it's all hit and miss currently as it's dev and prod together, we should really make bots-{1..4} 'prod' and -dev dev [23:46:32] yeah [23:46:43] and bots-sql{1..3} need fixing [23:46:48] not puppetized, one is running mariadb [23:46:50] Ryan_Lane: Where should someone be documenting what they've installed to bots? [23:46:52] one is out of ram [23:46:57] !log at least [23:46:57] at is not a valid project. [23:47:06] yeah. !log at minimum [23:47:11] !project Bots [23:47:11] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots [23:47:19] but, you can also edit the docs directly on that page [23:47:41] It's kinda difficult to enforce since we should be moving towards puppet but it's so wonky currently it's hard to push people that way. [23:47:54] that page is out of date btw [23:47:56] well, it's good to have dependencies documentated [23:48:02] whether they are in puppet or not [23:49:19] I want to reshuffle project storage around and start using the scripts dir more [23:49:29] * legoktm has to figure out what he's installed now -.- [23:49:34] Though I'd like to know if generally people liked my idea the other day, had 1 person yes and 0 others opinoin [23:49:56] which idea? [23:50:28] Damianz: Ryan_Lane: we have this in generic-definitions.pp class generic::locales::international [23:50:53] it installs all locales listed in puppet:///files/locales/local_int and execs locale-gen [23:51:04] !log bots Installed python3-minimal, python-virtualenv, python-pyexiv2, git-core, git-svn, git-doc on bots-3 [23:51:06] Logged the message, Master [23:51:10] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots/Documentation#bots_instances [23:51:18] mutante: Awesome, we can include that in a role class and standardise these m'fckers [23:51:21] there, added a section for package dependencies [23:51:26] yep [23:51:41] it would be really nice for the bot instances to be standardized, so that we can quickly spin up extras [23:52:04] Damianz: i use that for planet hosts and also applied it on the pdf servers [23:52:04] Ryan_Lane: Creating a common scripts/tools repo for labs for cross project tools (like mysql backup scripts etc) that people can clone into project storage and bang in crontab to stop time wasting on re-inventing the wheel. [23:52:18] yeah [23:52:19] would be nice [23:52:20] Like bots have scripts to create public_html dirs for users in the project, dump mysql dbs out to project storage etc [23:52:33] we need to replicate the repositories off to labs [23:52:48] that would be nice [23:53:10] on to globally shared storage would be really nice [23:53:41] having 1 share that's global and read only would be nice long term [23:53:45] bang loads of stuff under there [23:54:30] yeah [23:54:46] btw is /data/homes going to contain all user dirs? [23:54:51] (when we move to gluster) [23:55:29] each project will have their own [23:55:35] yeah [23:55:36] like they do now [23:55:45] it'll be exactly like right now [23:55:52] except the home dirs will be on gluster [23:55:56] I was thinking about someone requested symlinks to in ~ for public_html to the relevant dir in /data/project/public_html on bots but doing it now due to automount is a pita, one dir automount from gluster is simples. [23:56:00] and we'll use pam_mkhomedir to create them [23:56:15] ah [23:56:17] right [23:56:21] we'll be mounting /home directly [23:56:29] rather than individual home dirs per user [23:56:31] that makes it simples then heh [23:56:41] and less painful on the network for lots of users [23:56:44] poor crappy nfs box [23:56:45] I'm considering doing a direct mount for /data/project, too [23:56:54] since people get very confused by it [23:57:09] On a lot of instances /data/project is perm mounted anyway just due to crons/running procs etc [23:57:12] yeah [23:57:17] The automount just becomes overhead every now and then [23:57:23] yeah [23:57:42] I was thinking that in the future we could allow people to create additional volumes [23:57:48] but I think that's probably just overkill [23:58:15] we can just increase quotas anyway [23:58:27] that could be interesting from the point of saying dumps makes a share and marks it as globally read only then others could read the data. That would be overkill though heh [23:58:37] yep [23:58:45] though that's not a bad idea... [23:58:49] hm. [23:59:04] we can handle that one at a time [23:59:23] It's a nice idea but I can see the implimentation of acls a nightmare at scale [23:59:58] yep