[00:35:33] * spagewmf wonders why some interwiki links on labsconsole.mediawiki.org are broken, https://labsconsole.wikimedia.org/wiki/User:Spage#Tests [00:37:18] lol [00:38:25] Works fine [00:38:41] If you want ot link to an interwiki, prefix it with : [00:39:51] Any link without that will show "blank" and appear in the "In other languages" on the left hand side at the bottom [00:42:18] Reedy, thanks. Makes sense, an interlanguage link is a special kind of interwiki link, but only on wikis that have it. [12:16:33] Hi there, what's going on with the labsconsole web interface??? [12:19:42] Seems to be fine here [12:26:54] Damianz_: OK thx, mine is okay, too, now. [12:30:53] okay. [12:31:05] so on https://labsconsole.wikimedia.org/w/index.php?title=Special:NovaInstance&action=consoleoutput&project=wikidata-dev&instanceid=i-00000259 [12:31:33] instance won't reboot, logfile ends with 12:30:43 [69/100]: url error [timed out] [12:31:37] any hints? [12:32:08] ^^error message comes from "cloud init" [12:35:06] hashar: got a minute? [12:35:18] Jens_WMDE: sure [12:35:23] instances are blocked [12:35:26] at boot [12:35:41] what happen is that some init script need to grab data from the OpenStack metadata server [12:35:43] that timeout [12:35:50] meaning? [12:35:52] so the instance can not continue booting and remains blocking [12:35:56] blocked [12:36:09] what can we do? [12:36:10] got a bug opened somewhere [12:36:13] will we ever see our poor little instance again? [12:36:38] ah the bug is at https://bugzilla.wikimedia.org/show_bug.cgi?id=38473 [12:36:44] the instances are safe as far as I know [12:36:59] that is "just" a timeout problem with some openstack brick [12:37:18] once it timed out it will boot normally? [12:37:19] I queried Ryan Lane about it yesterday night (eu time) [12:37:24] he told me he was looking at it [12:37:31] must be some nasty issue [12:37:45] once timed out, instance remain blocked I think. Let me check one of mines [12:38:01] as to what can we do ? <--- nothing :/ [12:38:09] need Ryan to fix ti [12:38:15] Oh dear. [12:38:29] indeed :/ [12:38:48] So, basically, I just broke the public demo system. m( [12:39:40] 3 days ago rebooting still worked alright... [12:40:17] http://nooooooooooooooo.com/ !!!!! [12:41:07] so the symptom is [12:41:07] cloud-init running: Thu, 19 Jul 2012 09:00:42 +0000. up 2.26 seconds [12:41:08] waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id [12:41:25] yes yes yes [12:41:25] yep [12:42:00] then it tries 100 times and can boot because there is no cloud configuration :/ [12:42:11] so I am pretty sure the instance is safe [12:45:07] paravoid: do you know anything about openstack system ? [12:45:39] paravoid: instances are locked on boot cause the metadata service is slow as hell [12:45:46] Whooops - it's online again! [12:45:52] paravoid: bug is https://bugzilla.wikimedia.org/show_bug.cgi?id=38473 [12:48:57] it.. it fixed itself... [12:49:28] and it... hid all console output... [12:49:57] hashar: I read Ryan's mail on -labs [12:50:22] bah gmane overloaded :D [12:53:09] reading too [12:53:28] and that's the full extent of my knowledge about the problem [12:54:17] that is an openstack bug anyway [12:54:30] so nothing to do beside getting a fix with them and have it deployed on labs :D [12:54:49] Or trying to make stuff faster by killing history as a short term fix [14:46:40] hiyaaaaa [14:46:47] I'm getting this error when running puppet on my labs instance [14:46:47] err: Failed to apply catalog: Could not find dependent Package[sudo-ldap] for User[root] at /etc/puppet/private/manifests/passwords.pp:6 [14:47:00] I just included the class misc::package-builder via labsconsole [14:47:07] that won't work. [14:47:20] i'm following this wiki [14:47:20] http://wikitech.wikimedia.org/view/Pbuilder [14:47:48] hrm. [14:48:10] i can just run dpkg-buildpackage on my local vm, no problem, but I think there must be more official ways of doing this [14:48:27] so I'm trying to do it on a labs instance the way it is supposed(?) to be done [14:48:45] i'm sure these need signed with some wikimedia key that I don't have [14:48:54] nope [14:49:12] nope? [14:49:16] don't need signed? [14:49:40] no [14:49:46] but you need root to include it to the repo [14:50:02] in apt? yeah I think i'll need to get someone else to do that part for me [14:50:10] just trying to do as much ahead of time so that ops people have less do to [14:50:11] and with great power comes great responsibility [14:50:11] todo* [14:50:15] hehe yeah [14:50:19] :) [15:54:03] 07/19/2012 - 15:54:02 - User neilk may have been modified in LDAP or locally, updating key in project(s): bastion [15:54:24] ^ that was me [15:54:25] 07/19/2012 - 15:54:25 - Updating keys for neilk at /export/keys/neilk [15:55:02] 07/19/2012 - 15:55:02 - User neilk may have been modified in LDAP or locally, updating key in project(s): bastion [15:55:24] 07/19/2012 - 15:55:24 - Updating keys for neilk at /export/keys/neilk [20:28:01] 07/19/2012 - 20:28:01 - Creating a project directory for echo [20:28:01] 07/19/2012 - 20:28:01 - Created a home directory for werdna in project(s): echo [20:29:03] 07/19/2012 - 20:29:03 - User werdna may have been modified in LDAP or locally, updating key in project(s): echo [20:32:46] hey Ryan_Lane… [20:32:47] werdna@bastion1:~$ ssh echo-xmpp [20:32:47] Permission denied (publickey). [20:32:54] agent is fwd'd [20:33:00] ssh key in question is added [20:33:05] you need to wait for it to fully build [20:33:11] look at the console log [20:33:13] can ssh to ee-prototype [20:33:14] ah [20:33:22] it needs to say the puppet run finished [20:35:08] spamming with these [20:35:08] [0;36mnotice: /Stage[main]/Base::Apt/File[/etc/apt/sources.list.d/wikimedia.list]/ensure: defined content as '{md5}8b3477eee9956d3ca6d4c9a1dc6a59c1'[0m [20:35:13] yeah [20:35:16] so, it's still running [20:35:31] k [20:39:18] <^demon> Ryan_Lane: I wrote a gerrit extension :D [20:39:32] <^demon> Er, plugin. Whatever :p [20:41:53] <^demon> See kids, it can be done. We have the technology [20:43:35] in java? :o [20:43:47] <^demon> yup [20:44:05] boo :P [20:44:30] <^demon> java's easy to write [20:44:34] ^demon: cool, to do what? [20:44:45] Little verbose but yeah [20:44:53] <^demon> Nothing so far. Just tied in my own httpservlet to serve over a custom /url/ in gerrit. [20:44:55] <^demon> And print "test" [20:45:05] <^demon> Mainly just a proof of concept to see if I could do it [20:45:18] ah [20:45:55] <^demon> http://p.defau.lt/?OYhykNfRTSInLLCKzgQOlA - that's it [20:46:04] <^demon> Eventually it'll do our "stats dashboard" type stuff [21:26:29] ^demon|away: it doesn't matter, cause we'll be switching away anyway [21:26:32] :D [21:29:28] <^demon|away> Ryan_Lane: If we do, I might just lock myself in my room and cry for a month straight. [21:29:35] hahaha [21:29:49] github man, github [21:30:11] <^demon|away> I hate github. [21:30:15] <^demon|away> Yes, I know that makes me a heathen [21:30:45] <^demon|away> Anyway, dinner reservations. Bye. [21:33:06] One can't simply hate github [21:33:38] I think github is a weird model for things like puppet... yeah pull requests are more social but when you're trying to inter-link issues with bz it would be a nightmare [21:34:01] * Damianz puts some food in the room for chad [21:35:45] Damianz: i think ryan / chad are objecting to the 'zomg unicorns' school of github advocacy [21:36:05] no. I'm objecting to the fact that it's hosted and proprietary [21:36:14] and that we need a private repo to store private things in labs [21:36:32] Ryan_Lane: ah, fair point [21:36:33] and the fact that github occasionally has vulnerabilities that let people write to your repos [21:36:33] NOTHING WRONG WITH UNICORNS [21:36:38] I <3 unicorns [21:36:42] have you *seen* labs? :) [21:37:11] ah right. i stand corrected :P [21:37:18] Yeah, the reason why all private works code is deployed from fab scripts and not CD style stuff is due to 3rd party hosting issues [21:37:28] I'm totally cool replicating to github, though [21:37:32] we really need to set that up [21:37:42] I like github, but it's more suited to opensource stuff... not so much private never gonna open stuff [21:39:04] I wish github would impliment hooks for issues/prs so you can do unit testing on pull requests, I know travisci does it but they use polling and don't/can't enforce not merging it and master in a state of broken is bad for life