[00:04:24] andrewbogott_afk: how do I enable extensions in essex? [00:04:30] there's no freaking docs for it :( [03:36:37] jesusaurus: and ::self ? [07:09:36] If it's not too much trouble, could someone add me to the Bots and Testing projects on wmflabs? [07:12:28] or...maybe not :o [07:32:09] Fastily you are there [07:32:24] Yes, I am [07:32:33] in bots, I mean [07:32:39] :D [07:32:51] ah, one second, let me check [07:33:00] 08/01/2012 - 07:32:59 - Created a home directory for fastily in project(s): bots [07:33:19] yes I am! [07:33:30] Thanks petan! :) [07:34:02] 08/01/2012 - 07:34:01 - User fastily may have been modified in LDAP or locally, updating key in project(s): bots [08:01:01] 08/01/2012 - 08:01:01 - Created a home directory for hashar in project(s): wikidata-dev [08:02:01] 08/01/2012 - 08:02:01 - User hashar may have been modified in LDAP or locally, updating key in project(s): wikidata-dev [08:03:01] 08/01/2012 - 08:03:00 - User robla may have been modified in LDAP or locally, updating key in project(s): wikidata-dev [08:04:02] 08/01/2012 - 08:04:02 - User tstarling may have been modified in LDAP or locally, updating key in project(s): wikidata-dev [08:04:47] !putty | Fastily [08:04:47] Fastily: official site: http://www.chiark.greenend.org.uk/~sgtatham/putty/ | how to tunnel - http://oldsite.precedence.co.uk/nc/putty.html [08:05:00] !forwarding [08:05:03] 08/01/2012 - 08:05:03 - Created a home directory for csteipp in project(s): wikidata-dev [08:05:04] !ssh [08:05:04] https://labsconsole.wikimedia.org/wiki/Help:SSH [08:05:09] !accesss [08:05:15] !access [08:05:15] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [08:05:20] @seach forward [08:06:03] 08/01/2012 - 08:06:02 - User csteipp may have been modified in LDAP or locally, updating key in project(s): wikidata-dev [08:07:42] @search ssh [08:07:42] Results (Found 4): pageant, ssh, socks-proxy, git-puppet, [08:08:13] Fastily you either need to create key on bastion or use forwarding [08:08:40] !forwarding is https://labsconsole.wikimedia.org/wiki/Access#Using_agent_forwarding [08:08:41] Key was added [08:08:53] petan: Okay, I'm ssh'd into bastion using "'ssh -A fastily@bastion.wmflabs.org" [08:09:04] I just ran ssh-keygen [08:09:19] insert your public key to console [08:09:22] !keys [08:09:22] http://bots.wmflabs.org/~petrb/db/ list of infobot keys [08:09:25] eh [08:09:26] !ssh [08:09:26] https://labsconsole.wikimedia.org/wiki/Help:SSH [08:09:27] left all fields blank and pressed enter [08:09:41] Fastily you need to insert public key to console [08:09:44] !labsconsole [08:09:44] 208.80.152.32 [08:09:47] o.o [08:09:53] !labsconsole del [08:09:53] Unable to find the specified key in db [08:09:54] how do I do that? :o [08:10:16] !labsconsole https://labsconsole.wikimedia.org/wiki http://208.80.152.32/ [08:10:17] 208.80.152.32 [08:10:24] oh wait [08:10:29] one sec [08:10:34] !labsconsole is https://labsconsole.wikimedia.org/wiki http://208.80.152.32/ [08:10:34] Key was added [08:10:37] !labsconsole [08:10:37] https://labsconsole.wikimedia.org/wiki http://208.80.152.32/ [08:10:40] ok [08:10:45] !ssh | Fastily [08:10:45] Fastily: https://labsconsole.wikimedia.org/wiki/Help:SSH [08:10:47] here is guide [08:11:17] also you want to use bots-4, not bots-1 [08:11:26] bots-1 is already full [08:13:46] petan: I've been using dsa all this time...should be using rsa? [08:13:59] dsa is ok [08:14:01] 08/01/2012 - 08:14:01 - User fastily may have been modified in LDAP or locally, updating key in project(s): bastion [08:15:00] okay, so I've copied my local public key into the authorized_keys files on bastion [08:16:00] and I now have a public key on bastion, which I just created using 'ssh-keygen' [08:18:31] Fastily don't copy anything, it's done by a bot [08:18:43] insert it to labsconsole [08:18:44] !ssh [08:18:44] https://labsconsole.wikimedia.org/wiki/Help:SSH [08:18:46] read it [08:18:57] bot will insert it to all instances for you [08:19:00] so that you can ssh there [08:19:01] 08/01/2012 - 08:19:01 - User fastily may have been modified in LDAP or locally, updating key in project(s): bots,bastion [08:19:24] 08/01/2012 - 08:19:24 - Updating keys for fastily at /export/keys/fastily [08:19:54] Oh, I see! That makes sense [08:20:01] 08/01/2012 - 08:20:01 - User fastily may have been modified in LDAP or locally, updating key in project(s): bots,bastion [08:20:24] 08/01/2012 - 08:20:24 - Updating keys for fastily at /export/keys/fastily [08:22:00] petan: Okay, bots-4 worked! :D [08:22:08] Thanks! [08:22:14] good [08:22:24] don't forget to log stuff [08:22:29] when you change something in system [08:22:35] !logging [08:22:35] To log a message, use the following format: !log [08:22:44] okay, will do [08:22:48] oh [08:22:55] two more quick questions [08:23:02] 08/01/2012 - 08:23:02 - Created a home directory for demon in project(s): wikidata-dev [08:24:02] 08/01/2012 - 08:24:02 - User demon may have been modified in LDAP or locally, updating key in project(s): wikidata-dev [08:24:20] do executing bot scripts I run on bots-4 quit when I exit ssh? [08:24:34] when you run it in screen then yes [08:25:07] I'm guessing I'd have to use crontab? [08:28:03] 08/01/2012 - 08:28:03 - User fastily may have been modified in LDAP or locally, updating key in project(s): bots,bastion [08:28:25] 08/01/2012 - 08:28:25 - Updating keys for fastily at /export/keys/fastily [08:32:36] petan: I think I figured it out! Thanks for all your help! :)) [08:33:00] Fastily you can use whatever you like [08:33:41] Yeah, I figured that much :P crontab will work best for my bots :D [10:41:11] hi :) [10:50:01] hashar: hi [10:50:26] paravoid: hello :-D [10:50:50] paravoid: I came across a nice issue with Gluster that would not honor renaming a file (throw a permission denied error :/) [10:51:01] ryan is going to upgrade Gluster later tonight [11:00:16] yeah I saw that... [11:00:31] it'll probably solve this problem and create another two or something [11:56:06] paravoid: what could we do to make deployment-dbdump instance to stop sending syslog to remote ? https://gerrit.wikimedia.org/r/#/c/16661/ [11:56:28] I would love to finally have some syslog on the instance :-D [11:57:57] hm [11:58:21] at one point I thought about having a rsyslog::disabled class that would simply ensure that rsyslog is uninstalled :-D [11:58:24] so, wait, it sends *only* to remote? [11:58:34] and apply that to the nfs1/nfs2/deployment-dbdump machines [12:00:48] paravoid: I am not sure what you are asking :) [12:00:56] $ cat 30-remote-syslog.conf [12:00:56] *.info;mail.none;authpriv.none;cron.none @deployment-dbdump.pmtpa.wmflabs [12:01:07] but there is still the local logging to various files / xconsole [12:01:42] right [12:02:03] the syslog-ng conf apparently handle the default files too destination df_kern { file("/var/log/kern.log"); }; [12:02:09] # kern.* -/var/log/kern.log [12:02:10] log { [12:02:11] source(s_all); [12:02:13] filter(f_kern); [12:02:14] destination(df_kern); [12:02:15] }; [12:02:16] and so on [12:03:21] anyway my two other solutions are 1) ensure rsyslog package is absent 2) introduce a new global variable like $remote_syslog and have it set to false for a set of hosts / instance names [12:03:29] but 2) is still has tacky as the current class :/ [12:03:49] let me have a better look [12:03:54] and if I don't find something more elegant [12:04:03] I promise I'll merge that awful hack :) [12:04:58] ;) [13:52:09] hi labs crowd! [13:52:31] I got my labs account a while back (at Wikimania), but I'm a bit stuck right now [13:52:59] I've browsed the docs, but at the most basic level it says find a project admin to add you to a project [13:53:11] I don't see how I can start a project [13:58:12] dschwen: what are you willing to work on ? [13:58:28] dschwen: you might get a new project or be added to an existing one :) [14:00:42] I think labs is not ready to support the WikiMiniAtlas yet, from what I heard at wikimania [14:00:56] the Commons Zoomviewer however would be a better starter project [14:01:07] it is much simpler, and widely used tool [14:01:30] it generates multiresolution pyramids of images, and serves them through a flash or javascript interface [14:02:52] dschwen: I can't find out where to ask for project creations :/ [14:02:54] hi dschwen ! [14:03:05] hi! [14:03:26] labs is experimental (e.g. not to be relied on) [14:03:35] but if you want to test, we have a maps project and we can add you [14:03:35] sure [14:03:39] there you can create instances [14:03:42] ok [14:03:57] it's not like the toolserver can be relied on ;-) oops! [14:04:00] or if you want your own project that can be done too [14:04:16] maps project is fine [14:04:25] do you have osm db access? [14:04:43] and access to all the wikipedia/wikimedia wiki dbs? [14:05:20] no we don't [14:05:44] not wikipedia yet [14:06:03] ok, but it might be good for me just to get started on labs [14:06:37] for the zoomviewer I would recommend a separate project though [14:06:51] have you seen the latest WikiMiniAtlas? [14:07:11] latest as in "yesterday" [14:07:17] http://toolserver.org/~dschwen/wma/index_dev.html [14:07:44] dschwen: added you [14:07:53] dschwen: yes, it's very cool [14:07:55] thanks [14:08:01] 08/01/2012 - 14:08:01 - Created a home directory for dschwen in project(s): maps [14:08:20] ok, let me see Iif I can figure out how to log in [14:08:20] i'm trying to learn puppet, and setting up osm rails + api with puppet [14:08:30] so we have an instance for that and i'm using test3 [14:08:40] apmon is working on the others and tim is working on tilemill [14:09:02] 08/01/2012 - 14:09:01 - User dschwen may have been modified in LDAP or locally, updating key in project(s): maps [14:20:45] hm [14:21:02] it denies my key. I added a new one just after I was added to the maps project [14:21:15] it does not update the keys automatically? [14:22:53] let me see.... [14:23:46] ryan lane or someone might be better to help with that but maybe i can figure it out [14:25:19] jeremyb: can you help us? [14:25:30] too early for ryan to be here [14:27:33] yeah [14:27:42] what timezone are you in? [14:28:14] i am in germany now [14:28:27] jeremyb is in nyc and might be able to help [14:28:43] ryan would be in SF and who knows if he's on vacation.... [14:29:07] * jeremyb looks up [14:29:11] paravoid: can you help? [14:29:18] what's up? [14:29:18] * aude not sure who ryan's deputy is for labs stuff [14:29:19] oh, right, I didn't make the connection, but we've met at Wikimania! [14:29:20] with? [14:29:26] dschwen: yep [14:29:34] morning paravoid [14:29:38] evening :) [14:29:45] paravoid: dschwen ssh keys for dschewn [14:29:46] no! [14:29:47] dschwen: [14:29:55] i added him to the maps project and he can't login yet [14:30:09] do keys get automaticall updated or what? [14:30:10] i added a new key in gerrit after she added me [14:30:20] i see 2 keys for dschwen [14:30:26] let me try again [14:30:29] in ldap. idk about in gerrit [14:30:35] they don't share keys [14:30:40] gerrit's ssh key management is totally separate from labs [14:30:43] unfortunately [14:30:45] oh, darn [14:30:49] sorry, my bad then [14:30:50] you need to add keys in the wiki [14:30:59] paravoid: but he does have 2 in ldap already [14:31:10] it's probably the cache again [14:31:13] which instance is that? [14:31:20] I was on that page and saw two keys, but didn't check which ones [14:32:24] i had two in the ldap before [14:32:33] added the new one which makes it 3 [14:32:58] cannot login yet though, I'll give it a few more minutes [14:33:01] 08/01/2012 - 14:33:01 - User dschwen may have been modified in LDAP or locally, updating key in project(s): maps [14:33:07] ha! [14:33:24] 08/01/2012 - 14:33:23 - Updating keys for dschwen at /export/keys/dschwen [14:33:29] good good [14:33:51] now i see 3 keys for him! [14:34:15] and I ged a new and different ssh error: Connection closed by 208.80.153.207 [14:34:36] (rather than permission denied) [14:34:48] what is 208.80.153.207 ? [14:34:54] bastion? [14:34:57] what are you running? [14:35:23] it's bastion indeed [14:35:27] linux [14:35:38] I followed the ProxyCommand instructions [14:36:34] dschwen: can you try "ssh bastion.wmflabs.org" for me? [14:37:40] yeah, just did [14:37:43] same error [14:38:00] connection closed by ... (same ip) [14:38:21] who is dschoon? IMPOSTOR!! [14:38:30] ;-) [14:39:19] dschwen: try again? [14:39:41] sorry, same thing [14:39:45] although now maybe cached... [14:39:51] paravoid: dump the cache on bastion? [14:39:52] I just flushed the cache [14:39:55] again? [14:39:58] so, you weren't on project-bastion [14:40:02] right [14:40:07] * jeremyb added him [14:40:13] bastion works [14:40:15] and I flushed the cache, so that should work [14:40:16] $ groups dschwen [14:40:16] dschwen : svn project-bastion project-maps [14:40:18] great [14:40:25] the proxycommand thing to maps doesn't [14:40:29] new error though [14:40:29] which instance is that? [14:40:39] channel 0: open failed: administratively prohibited: open failed [14:40:51] maps [14:41:02] 08/01/2012 - 14:41:02 - Created a home directory for dschwen in project(s): bastion [14:41:15] there is no maps [14:41:18] which instance? [14:41:29] !resource maps | dschwen [14:41:29] dschwen: https://labsconsole.wikimedia.org/wiki/Nova_Resource:maps [14:42:03] 08/01/2012 - 14:42:02 - User dschwen may have been modified in LDAP or locally, updating key in project(s): bastion [14:42:27] aude: which instance can I play on? [14:42:45] ah, can i add people to bastion? [14:42:49] or should I roll a new one? [14:42:53] dschwen: you can make a new one [14:43:01] ok [14:43:09] * aude has to delete one of the osm rails ones (had problems with labs this past weekend) [14:43:12] aude: yes, but don't do it much... [14:43:15] i'm also playing with test-3 [14:43:19] aude: can has maps ? ;-) [14:43:20] jeremyb: ok [14:43:33] tim is working on tilemill and has stuff to see i think [14:43:49] apmon is working on the others and not sure what to play with [14:43:53] aude: what problem? was it corrupt? [14:44:01] think he's importing osm database and testing import settings [14:44:19] jeremyb: ldap was not working so i was never able to login until ryan fixed it [14:44:25] huh [14:44:30] jeremyb: all fixed [14:44:38] but that didn't require nuking an instance? [14:44:55] jeremyb: i created extra instances, thinking the second one might work ;) [14:45:08] is it possible that I do not have permission to create a new instance? [14:45:09] now everything works and i don't need duplicates [14:45:19] dschwen: hmmm.... [14:45:26] paravoid: danke [14:45:54] dschwen: are you in? [14:46:09] jeremyb: paravoid do i have to add dschwen as sysadmin for him to create instances? [14:46:15] yes [14:46:34] aude: is he in the office? [14:46:44] ok [14:47:00] yes [14:47:04] I feel useless [14:47:06] dschwen: try again [14:47:10] jeremyb is so much quicker than me [14:47:11] heh [14:47:20] paravoid: what time zone are you in? [14:47:27] aude: apergos [14:47:56] paravoid: now you know how i feel when someone needs banning. almost always someone else gets it first [14:48:01] aude: EEST == EET + DST == GMT+3 [14:48:04] yeah! [14:48:13] paravoid: no, UTC! [14:48:31] * aude met a labs person in berlin  [14:48:36] hi paravoid ! :) [14:48:39] who is that? [14:48:54] aude: we definitely met in Berlin [14:48:55] he was sort of ryan's deputy [14:49:03] andrew bogott? [14:49:07] paravoid: that's you? [14:49:09] nick=andrewbogott [14:49:12] probably met andrew also [14:49:18] * aude terrible with names [14:49:19] Andrew was not in Berlin [14:49:27] * andrewbogott waves hello [14:49:36] heya Andrew :) [14:50:15] aude: my name is Faidon, we met in Berlin, I'm a staffer and a labs person [14:50:21] :-) [14:50:29] and I live in Athens, Greece. [14:51:00] paravoid: ah, yes [14:51:06] i remember now [14:51:13] jeremyb is confusing me [14:51:23] * aude knows apergos [14:51:25] there's also a picture of me in the staff & contractors page :-) [14:51:27] and he was not in berlin [14:51:31] aude: Handy face:name correlator http://wikimediafoundation.org/wiki/Staff_and_contractors [14:51:39] aude: his *timezone* is apergos [14:51:45] paravoid: yes [14:51:45] jeremyb meant that we both live in the same city [14:51:49] jeremyb: ok :) [14:52:18] * aude dislikes the cloak of invisibility [14:52:18] aude: so, feel free to ping me for labs-related troubles. [14:52:27] paravoid: ok, thanks [14:55:05] aude: http://i.imgur.com/PvMg6.jpg [14:55:20] aude: from http://www.flickr.com/photos/wouterverhelst/7565045766/in/photostream/ [14:55:37] aude: go to asheesh at the bottom, over one to the left and then up [14:56:44] jeremyb: I'm in the staff page, no need for that :) [14:56:45] jeremyb: yes, i remember now [14:56:59] paravoid: heh, i like my method ;) [14:57:03] hehe [14:57:03] paravoid: and you were not at wikimania? [14:57:08] I wasn't [14:57:08] nope! [14:57:11] aw :( [14:57:14] I was at the picture above :) [14:57:15] hong kong! [14:57:22] paravoid is just omnipresent [14:57:23] DebConf12, Nicaragua [14:57:33] paravoid: then how was paulproteus in the picture? ;-) [14:57:49] jeremyb: with the power of gimp ;-) [14:58:05] bdale wasn't happy ;P [14:58:08] hehehehe [14:58:50] I just created a new instance and it is shown as running. Attempting to log in gives a permission denied. [14:59:04] has puppet run yet? [14:59:13] dschwen: check the log. sometimes there's a delay before puppet's done. you can't log in immediately [14:59:22] (console) [14:59:46] c_rehash should improve that a little [15:00:03] aude: 01 14:46:34 < jeremyb> aude: is he in the office? [15:00:22] ok, yeah, I'm looking at the console utput [15:00:32] paravoid: who's the guy with no shirt looking away? [15:00:38] sorry for being impatient ;-) [15:00:45] I'm just excited [15:00:52] paravoid: did the venue have a shoe requirement policy? [15:01:04] dschwen: we understand ;) [15:01:23] dschwen: would be so nice to have a live stream of the console... [15:01:33] jeremyb: I don't know, I always wear shoes when I'm outside :-) [15:01:44] jeremyb: who? he? [15:01:49] paravoid: i mean inside. remember taffit? [15:01:55] aude: dschwen [15:01:59] just a bit of socket.io [15:02:03] jeremyb: no [15:02:10] he was at wikimania [15:02:19] I wore shoes! [15:02:22] and a shirt [15:02:25] we don't do maps at WMDE [15:02:25] haha [15:02:29] * aude does maps as a volunteer [15:02:37] oh. how odd [15:03:05] I don't see anything odd about wearing shirts [15:03:14] believe me, it is for the best of you all [15:03:23] dschwen: no, the not doing maps [15:03:39] oh [15:03:56] aude: can I reboot three of your instances? [15:04:02] paravoid: you may [15:04:17] * aude will delete one of the osm rails ones but can't remember which one i was using [15:04:29] so, aude, I was totally excited about the SOTM-US in Portland [15:04:53] unfortunately I have to be in Germany for a wedding on the 15th (SOTM is 13,14th of Oct) :-( [15:05:12] :( [15:05:14] I have to interact more with OSM people [15:05:31] i see rain [15:05:35] dschwen: maybe some mapping parties? [15:06:08] well, yeah, I'm mapping by myself, or when I'm hiking with my kids [15:06:14] ah, okay [15:06:15] I live fairly remotely [15:06:22] * aude will be at state of the map tokyo this year [15:06:33] I don't think there is a big OSM comminity here in northern New Mexico [15:06:38] last time i attended was 2009 [15:06:38] nice [15:06:41] dschwen: right [15:07:21] try to sum up all the maps and geo stuff that wikimedians are doing and would like to give a lightning talk [15:07:34] * aude about mobile maps and apps [15:07:36] and other stuff [15:07:50] so, how stable are instances. Can or should I copy large amounts of data onto my instance? (map tiles), import a big mysql database, etc. [15:08:02] or would that be a waste of time and space [15:08:04] not too stable but it depends [15:08:10] can instances be resized? [15:08:11] the wikidata one seems stable [15:08:25] don't think they can be resized but there is additional storage available i think [15:08:38] dschwen: you're in NM? [15:09:01] yeah [15:09:03] https://labsconsole.wikimedia.org/wiki/Help:Shared_storage [15:09:05] near Snata Fe [15:09:14] in Los Alamos to be precise [15:09:29] like the toolserver but more space availalbe [15:10:07] why? do you know any OSM,WM etc. people here? [15:10:21] * aude doubts it [15:10:24] hah [15:10:27] there's someone in kansas [15:10:42] oh, sure, that's right around the corner [15:10:49] only a 12h drive [15:10:49] and, of course, martijn is in utah and there are a few people in colorado [15:10:54] heh [15:10:55] like apmon [15:11:01] yes [15:11:06] people spread apart pretty far [15:11:07] that is still a 7h drive :-) [15:11:32] I miss the days I spent in Goettingen, with the monthy meetups. 10+ people [15:11:52] oh, and off topic, but would you be interested in helping with wiki loves monuments? [15:12:06] how can I help? [15:12:17] we need judges (and you can take photos too) [15:12:29] i know you've done many featured pictures [15:12:41] judges = for the US part of the competition [15:13:10] * aude can email you info  [15:14:53] ok [15:15:19] if you're too busy or not interested, that's okay (but hope you'd like to help) [15:15:53] ok, i should get back to coding [15:16:08] the console of my instance only shows dhcpd activity (successful) [15:16:13] but I cannot log in [15:16:17] permission denied [15:16:24] hmmmm.... [15:16:28] too soon? [15:16:39] some ldap stuff has to run [15:16:57] it was broken during the weekend and i couldn't login (it should be fixed now) [15:18:08] dschwen: what's your instance number? [15:18:32] i don't see it in the maps project yet, though it can take some time for the instance to be fully setup (even when ldap works) [15:21:27] i-0000038c [15:21:37] I saw it in the list [15:22:34] now I don't see it anymore. huh?! [15:22:41] checking [15:23:11] how did I get to the console output [15:23:18] man, I'm so confused [15:23:21] https://labsconsole.wikimedia.org/wiki/Nova_Resource:I-0000038c [15:23:26] you can check the console output [15:23:38] * aude sees a puppet error but different than what i experienced [15:23:57] you should be able to see mine: https://labsconsole.wikimedia.org/wiki/Nova_Resource:I-00000373 [15:24:48] Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Package[apache2] is already defined... [15:24:53] aude: can has maps? [15:25:38] hah dschwen [15:25:40] dhclient stuff is normal [15:25:47] the catalog stuff seems a problem with puppet [15:25:56] as much as i know about puppet (not a lot yet) [15:26:01] did you start out with just a vanilla instance or did you add puppet classes? [15:26:22] stupid me added puppet classes. webserver, php, php-mysql, git [15:26:42] bad idea [15:26:53] yeah, the page warned me [15:26:57] heh [15:26:58] why does it even allow it? [15:27:05] but I haaaad to do it anyways [15:27:10] aude: if you're an expert? ;) [15:27:11] heh [15:27:38] * aude doesn't understand which circumstances it's okay to add some puppet when creating instances [15:27:52] shouldn't show these choices to non-experts [15:27:55] you can add puppet later after the instance is running [15:27:59] right [15:29:17] jeremyb: is there any way to fix the instance? [15:29:27] or should he create a new one and somehow delete this one? [15:30:05] Change on 12mediawiki a page Wikimedia Labs/Agreement to disclosure of personally identifiable information was modified, changed by MZMcBride link https://www.mediawiki.org/w/index.php?diff=567880 edit summary: minor fixes [15:30:59] i unchecked all puppet options under configure [15:31:12] no idea if that fixes things [15:32:08] delete instance gives me "The requested host does not exist." [15:32:18] right [15:32:25] i got that [15:32:50] no idea how to fix an instance that didn't start up correctly [15:32:59] superpowers maybe [15:33:25] what's the issue? [15:33:42] my newly created instance is stuck in limbo [15:33:49] cannot delete, cannot login [15:33:56] https://labsconsole.wikimedia.org/wiki/Nova_Resource:I-0000038c [15:34:04] I screwed up by adding puppet options on instance creation [15:34:09] what do we do in such situations? [15:35:20] delete and recreate [15:35:30] he can't delete [15:35:34] see above [15:35:40] 15:32 < dschwen> delete instance gives me "The requested host does not exist." [15:35:49] that's what i got too when i had problems over the weekend [15:35:50] hrm [15:35:52] that sounds like a bug [15:36:04] you can't delete unless it's fully and successfully created [15:36:32] maybe unless you have superpowers or something (no idea) [15:37:35] I can't [15:37:40] from the web intf at least [15:43:04] huh [15:45:50] time for virsh ;) [15:46:40] virsh destroy jeremyb [15:51:48] i managed to log in [15:52:23] yay [15:52:34] I'm in too [15:52:44] probably pupppet was taking ages because of all the classes [15:52:45] good [16:04:56] so now I do regular apt-get install? [16:06:40] dschwen: i think you can [16:06:59] if you're really interested, you can try puppet but probably not necessary for what you're doing [16:07:17] and your sudo password should be same as labsconsole password [16:08:18] dschwen: The advantage of using puppet vs. hand configuration is that when Labs suffers a crisis and destroys your instance, you can just build a new one. (It also depends on whether you think anyone else will ever want to configure a server in the same way.) [16:11:21] andrewbogott: not just anyone else, also if you do yourself [16:11:32] * andrewbogott nods [16:13:01] dschwen: Labs users often make the mistake of assuming that they can 'duplicate' an instance, but that's not supported. Puppetizing a config is the only real path to producing multiple similar systems. [16:14:28] Clone support would be awesome [16:16:04] * aude nods [16:16:23] Damianz: Cloning encourages the propagation of mystery systems, i.e. "This works but I don't know how or why." Better to have the knowledge of where a system came from externalized someplace. [16:16:54] Otherwise you can't learn from/share parts of your work. [16:17:14] It's useful for development boxes that don't justify 6months of work puppetizing it and submitting for merging etc [16:17:36] Yeah, I'm not saying I wouldn't use cloning if it were available :) [16:17:48] <^demon> I'd love a "wipe the disk and reinstall" option though. [16:18:02] If we could have submodules pulled in per project and just push manifests to "production" earlier that would be doable - self hosted puppet is fine but that's a lot of hassle for day to day work over working on puppetizing feature x [17:02:31] one last question for now, are there per project git repos? [17:04:53] dschwen: other than gerrit, not sure [17:05:58] stuff can go on gerrit (git) but it can be a bit annoying at times [17:29:26] <^demon|lunch> Ryan_Lane: Quick question, are the values in openstack::nova_config available to labs? [17:29:37] what do you mean? [17:30:02] <^demon|lunch> Right now we get some LDAP settings for gerrit from openstack::nova_config::foovar for production. [17:30:06] you /can/ use them but it's layering violation [17:30:11] yeah, that's just plain wrong imho [17:30:17] it is indeed [17:30:20] <^demon|lunch> Yeah, it's being rewritten :) [17:30:48] gerrit / ldap / openstack are three completely separate things [17:31:10] and now with essex nova is completely decoupled from ldap too, as far as I understand it [17:31:19] not really [17:31:26] isn't this being moved to keystone? [17:31:36] the DNS stuff will require it [17:31:43] isn't that a plugin? :) [17:31:46] yes [17:31:59] and the plugin requires configuration [17:32:07] it writes to LDAP [17:32:35] completely OT: but why did we make it write to LDAP instead of doing signed DNS updates? [17:32:54] what do you mean? [17:33:49] why is the nova plugin doing ldap writes instead of doing "echo ... | nsupdate" [17:33:55] meh, easier I guess [17:34:18] well, this allows it to not be coupled with the dns server [17:34:21] if we wanted to move it later [17:34:48] it? [17:34:56] the hook? [17:35:04] nsupdate works over signed DNS queries [17:35:20] and is server-agnostic (it's part of the protocol) [17:36:42] <^demon|lunch> Ryan_Lane: Anyway, hashar and I did a lot of the work to get gerrit refactored in puppet :) [17:36:46] <^demon|lunch> Not 100%, but getting there [17:37:23] paravoid: ah [17:37:40] not sure if powerdns supports it though, so I may be talking shit here :) [17:38:40] heh [17:38:43] I haven't looked into it [17:38:46] ok. gotta go for a bit [17:38:52] good luck :P [17:46:07] <^demon|lunch> paravoid: Anyway, the reason for asking was https://gerrit.wikimedia.org/r/#/c/13484/ [17:59:29] !log deployment-prep manually running puppet on -bastion [17:59:41] grmblb [18:00:46] petan: the labs bot is dead again :-D [18:00:58] hm [18:01:30] maybe put it in a while loop with a 180 seconds sleep? [18:02:35] debug1: channel_connect_stdio_fwd deployment-bastion.pmtpa.wmflabs:22 [18:02:35] debug1: channel 0: new [stdio-forward] [18:02:36] debug1: getpeername failed: Bad file descriptor [18:02:37] yeahhh [18:05:39] where are you trying to connect? [18:06:04] labs DNS lacks an entry for deployment-bastion.pmtpa.wmflabs [18:06:15] https://bugzilla.wikimedia.org/show_bug.cgi?id=38846 [18:06:29] would need some op to fix it up :-D [18:06:58] can't that be done from labsconsole? [18:07:04] I have no idea [18:08:04] I am out for dinner :-D [18:09:18] hashar that's complicated [18:09:23] problem is not with a bot but server [21:36:20] Hi guys, I'm getting https://labsconsole.wikimedia.org/wiki/Special:NovaKey [21:37:15] Can one of the "Nova administrators" please add my key again? [21:41:17] Ryan_Lane: ^^ [21:43:47] multichill: I'm not quite following. Are you not able to upload your own key? [21:43:59] My key is gone [21:44:22] Ah, ok. The link that you pointed at is a per-user page so it looks different for me than for you. [21:44:36] The key that was up there came from your local system originally, didn't it? [21:44:58] Imported from svn. Worked a couple of weeks ago [21:46:07] So you're saying that you no longer have a copy of your own public key? [21:46:16] It's probably visible in gerrit, if you visit https://gerrit.wikimedia.org/r/#/settings/ssh-keys [21:46:39] o.0 [21:46:43] andrewbogott: It's empty. Where the hell did my key go? [21:46:47] who doesn't have a copy of their own key!? [21:47:13] Though tbf I have lost too many pgp keys... but I blame hard drive failures and lack of caring about email [21:47:38] Of course I have a copy, it's the labs side that lost my key [21:48:18] They're stored in ldap so shouldn't just 'dissapear' [21:48:44] I don't know where it went, but… you can just paste it back into that page can't you? [21:53:19] WHy did it dissapear in the first place? [21:53:32] This is even worse than the toolserver [21:53:46] rather hard to believe :P [21:54:01] 08/01/2012 - 21:54:00 - User fastily may have been modified in LDAP or locally, updating key in project(s): bots,bastion [21:54:23] 08/01/2012 - 21:54:23 - Updating keys for fastily at /export/keys/fastily [22:00:06] gah, no mutlichill ;( [22:00:38] * Damianz yawns [22:01:14] * petan slaps Damianz [22:01:36] I just thought it's a good idea to do that [22:01:41] * Damianz bites petan [22:01:56] * petan eats Damianz [22:02:07] iztasty? [22:02:10] no [22:02:22] lot of fat [22:02:24] :D [22:02:31] * Damianz sadface... [22:02:46] Actaully true, I got into a lift today as was like gah my ass is fat [22:03:18] well, I switched from using bus to walking to work every day [22:03:22] it doesn't help [22:03:28] but I feel better XD [22:03:51] it resulted in being more hungry [22:03:51] Not walking 8miles to the office [22:03:53] so I eat more [22:04:09] ok, it's like 2 km in my case [22:04:10] I'd take the bike but road bike + potholes == bad idea, need to buy a new bike [22:04:29] my teamlead is using bike [22:04:33] even when it's raining :D [22:04:47] Hmm, it's not raining at the moment it's pouring [22:04:54] heh [22:04:57] Your Amazon.co.uk order has dispatched < Woow, must remember to go to the office earlier tomorrow [22:20:11] andrewbogott_afk: you're not doing the ceph benchmarks in virt1004 by any chance, are you? [22:20:26] andrewbogott_afk: one of its disks has failed, it sends mails every now and then [22:20:36] andrewbogott_afk: that would probably mess with performance benchmarks [22:31:43] paravoid: around? [22:31:49] yes. [22:31:53] cool [22:31:54] hi :) [22:32:07] I'm soon going into full-blown ignore everyone for a week or two mode [22:32:17] so that I can finish the OpenStackManager upgrades [22:32:18] hahahaha [22:32:20] how come? [22:32:21] ah [22:32:26] okay :) [22:32:28] it's more work than I expected [22:32:29] feel free to ignore me [22:32:35] I'll just continue with the migrations [22:32:45] well, I was wondering if you and andrewbogott_afk could work on the migrations together [22:32:59] also, it would be good to migrate all of virt5 first [22:33:05] then gluster can be removed from it [22:33:11] and it can be used as a migration host [22:33:14] Ryan_Lane: Buuut we love you [22:33:16] :P [22:33:48] Damianz: heh [22:33:52] virt5 is 14 vms [22:33:59] cool. not too many [22:34:04] easy enough [22:34:10] btw, you have the most VMs out of everyone :) [22:34:15] :D [22:34:18] that's not surprising [22:34:23] I should probably delete some [22:34:33] yes please [22:34:37] we need to delete the corrupted instances at some point too [22:34:55] good [22:35:03] there's another thing that would help with stability issues right now [22:35:03] what's your plan re: openstackmanager? [22:35:21] openstack keeps the deleted instances around for "auditing" reasons [22:35:30] if we purge those, then things will become much faster [22:35:31] you said the new version would help with that though. [22:35:39] and the 500's we're getting would also go away [22:35:46] it will, but it doesn't hurt to purgw [22:35:47] *purge [22:35:52] but.... [22:35:57] but? [22:35:58] I'd also like to keep statistics [22:36:13] so, if we purge, we just need to keep track of how many we purged [22:36:25] if the upgrade is a week or two away I don't see the point of purging tbh [22:36:28] I have a lot of thoughts on statistics I'd like to start collecting [22:36:45] but, that's a discussion for later [22:37:01] yeah, it may be better to just wait [22:37:11] but the 500's kind of suck [22:37:30] re: openstackmanager.... [22:37:36] I added keystone support [22:37:52] now I'm working on switching to the openstack api [22:38:18] unfortunately, the instance ids are different in nova's api than they are in ec2'd api [22:38:35] so, I also need to switch over to using nova for DNS, rather than osm doing it [22:38:36] wait [22:38:50] is the new api a prerequisite for essex? [22:39:16] not completely, but it's more difficult using the ec2 api with keystone [22:39:34] and I already added all the needed rest components for keystone [22:39:43] so switching to nova's api isn't very hard [22:40:14] okay [22:40:32] after I get the DNS stuff in, and the openstack api in, I'm done [22:40:48] my changes are backwards incompatible ;) [22:40:56] oh that's nice [22:41:03] getting rid of ec2 is also very very nice [22:41:07] that said, if we need to rollback, we just switch OSM back to the older version [22:41:19] yes. I hate the ec2 api [22:41:53] so wait, we can do the DNS stuff now too? [22:41:57] yep [22:42:08] nice, right? :) [22:42:13] yeah, that's great [22:42:19] I thought we were waiting for essex for that [22:50:14] paravoid: well, essex upgrade will allow us to switch [22:50:31] paravoid: so, since I'm making the rest of the changes now, I might as well do this too [22:50:44] it's going to be hard not to, honesrly [22:50:47] *honestly [22:55:41] openstack's api returns such a saner set of data [22:55:46] except for the ip addresses [22:55:51] it sticks all the fucking IPs together [23:27:18] Could someone install jdk/jre 6 (preferably 7) on bots-4? I'd do it myself but I don't know the sudo password :o [23:27:41] petan: ? [23:28:03] Ryan_Lane: ? [23:28:38] Fastily, have you tried your own password? [23:28:47] you know, the one you use at labsconsole [23:29:01] you mean...that's my sudo password...? -_- [23:29:17] yes [23:29:43] that's your account password [23:29:52] okay, thank you :) [23:29:57] and sudo will require you to use that [23:30:51] !log bots Installed jdk/jre 6 on bots-4 [23:30:53] Logged the message, Master. [23:31:19] Master? neat :3