[00:12:35] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [00:42:35] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [00:54:43] the problem was that the ssh connection was closing when the object went away, and a new connection wasn't being opened [00:54:57] I changed it to always create and destroy a connection for every single command [00:55:03] seems gerrit only supports that anyway [00:55:25] well, that might not be true. I may have just had really shitty ssh code :) [00:55:57] bah. wrong channel [01:12:35] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [01:37:26] removing template from sitenotice on labsconsole... [01:37:32] now that issues are resolved [01:42:35] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [02:12:35] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [02:42:35] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [03:12:35] PROBLEM host: driver-dev is DOWN address: driver-dev CRITICAL - Host Unreachable (driver-dev) [08:23:55] PROBLEM Total Processes is now: CRITICAL on hugglewa-db hugglewa-db output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:24:35] PROBLEM dpkg-check is now: CRITICAL on hugglewa-db hugglewa-db output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:25:15] PROBLEM Current Load is now: CRITICAL on hugglewa-db hugglewa-db output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:26:05] PROBLEM Current Users is now: CRITICAL on hugglewa-db hugglewa-db output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:26:45] PROBLEM Disk Space is now: CRITICAL on hugglewa-db hugglewa-db output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:27:25] PROBLEM Free ram is now: CRITICAL on hugglewa-db hugglewa-db output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:37:21] !log hugglewa petrb: installing sql server [08:37:22] Logged the message, Master [08:41:46] !log hugglewa petrb: created s1 vm [08:41:47] Logged the message, Master [08:49:27] !log hugglewa petrb: installing a nfs server for project [08:49:28] Logged the message, Master [08:50:32] !log hugglewa petrb: installing new kernel [08:50:33] Logged the message, Master [08:52:25] Ryan_Lane: can I request a public ip [08:52:34] morning :) [08:52:36] sure [08:52:42] well "morning" [08:52:43] cool [08:52:45] it's 1am here :) [08:52:57] we really need a place for this... [08:53:05] we have liquid threads [08:53:09] we can make a page for it [08:53:21] problem is that we can't echo that to irc [08:53:27] we can't? [08:53:27] because lq doesn't support that [08:53:37] mediawiki standard page does [08:53:39] but lq not [08:53:39] it's just mediawiki pages [08:53:42] oh [08:54:00] it doesn't log anything to feed [08:54:02] it doesn't necessarily need to hit IRC [08:54:14] yes, probably not [08:54:16] people that have it on their watchlist can get email notifications [08:54:22] that's better for me than IRC [08:54:23] but I really want lq to support this [08:54:39] since I can get requests from people when I'm not online [08:54:49] ok [08:58:15] !log testswarm installed 'testswarm' debian package on swarm-specialpage and applied some additional fixes (see [[Nova_Resource:I-0000017b#Config]]). [08:58:17] Logged the message, Master [08:58:48] !log testswarm commented out apache alias on swarm-specialpage for /testswarm for now to avoid abuse in the mean time [08:58:49] Logged the message, Master [08:58:57] Ryan_Lane: let me know when we get it :) [08:59:01] the ip [08:59:08] what project is it for? [08:59:12] hugglewa [08:59:18] can I get 2 domains? [08:59:23] hg and hugglewa [08:59:27] both to same ip [09:00:07] hg? [09:00:13] oh [09:00:13] yeah [09:00:15] it's shortcut of huggle [09:00:19] just add another hostname [09:00:24] ok [09:00:30] ok, you can allocate an IP [09:00:59] !log hugglewa petrb: registered 208.80.153.229 for project [09:01:00] Logged the message, Master [09:02:03] !log hugglewa petrb: rebooting web server [09:02:04] Logged the message, Master [09:02:38] !log hugglewa petrb: domain hugglewa.wmflabs assigned to ip [09:02:39] Logged the message, Master [09:03:56] PROBLEM dpkg-check is now: CRITICAL on hugglewa-w1 hugglewa-w1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:04:06] !log hugglewa petrb: ip was assigned to i-0000018d [09:04:07] Logged the message, Master [09:04:36] PROBLEM Current Load is now: CRITICAL on hugglewa-w1 hugglewa-w1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:05:21] PROBLEM Current Users is now: CRITICAL on hugglewa-w1 hugglewa-w1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:06:06] PROBLEM Disk Space is now: CRITICAL on hugglewa-w1 hugglewa-w1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:06:46] PROBLEM Free ram is now: CRITICAL on hugglewa-w1 hugglewa-w1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:07:28] !log hugglewa petrb: sudo puppetd -tv [09:07:30] Logged the message, Master [09:08:16] PROBLEM Total Processes is now: CRITICAL on hugglewa-w1 hugglewa-w1 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:08:29] how do I list users in irssi [09:09:44] btw Ryan_Lane you know that puppet fails on some instances? [09:10:02] there are instances where it never started [09:10:04] I think [09:10:15] RECOVERY Current Users is now: OK on hugglewa-w1 hugglewa-w1 output: USERS OK - 1 users currently logged in [09:10:22] because the package fails to install? [09:10:23] if you check nagios there are 2 instances which are whole red, so it probably never updated [09:11:05] RECOVERY Disk Space is now: OK on hugglewa-w1 hugglewa-w1 output: DISK OK [09:11:07] I don't know why it fails, it's not my vm :) [09:11:11] I just know it's red in nagios [09:11:20] it looks like nrpe is down [09:11:24] or something [09:11:45] RECOVERY Free ram is now: OK on hugglewa-w1 hugglewa-w1 output: OK: 92% free memory [09:13:15] RECOVERY Total Processes is now: OK on hugglewa-w1 hugglewa-w1 output: PROCS OK: 91 processes [09:13:55] RECOVERY dpkg-check is now: OK on hugglewa-w1 hugglewa-w1 output: All packages OK [09:14:35] RECOVERY Current Load is now: OK on hugglewa-w1 hugglewa-w1 output: OK - load average: 0.42, 0.28, 0.12 [09:14:46] which instance? [09:14:49] oh [09:14:51] it's in nagios [09:14:52] sec [09:14:52] heh [09:15:05] oh no, it's gone [09:15:08] after you restarted labs [09:15:22] it's patchtest [09:15:38] prefixexport [09:15:58] these [09:19:01] well, they still show having issues [09:20:49] I think it's because of nrpe [10:21:59] hellp needed: http://translatewiki.net/wiki/Thread:Support/About_MediaWiki:Openstackmanager-removehost-action/fi [10:28:51] Nikerabbit: where is the string you need to help with [10:29:29] original [10:41:15] petan|wk: where as in ...? [10:41:47] as in http://translatewiki.net/wiki/MediaWiki:Openstackmanager-removehost-action/en ? [10:44:06] ok this is action to remove a dns record [10:44:25] !addresses [10:44:27] !address [10:44:27] https://labsconsole.wikimedia.org/wiki/Help:Addresses [11:16:19] petan|wk: so it is like a button but link instead? [11:17:10] why do I get this on labsconsole "There were no Nova credentials found for your user account. Please ask a Nova administrator to create credentials for you." ? [11:19:31] Nikerabbit: logout [11:19:33] then back [11:19:40] Nikerabbit: yes it's a "button" [11:20:15] open the nova page you will see it there [11:20:19] !address [11:20:19] https://labsconsole.wikimedia.org/wiki/Help:Addresses [11:20:35] it's in side bar [11:20:59] We probably could add a note to say about logging out and in until we fix that bug [11:21:12] @search log [11:21:12] Results (found 9): morebots, labs-morebots, credentials, logging, terminology, newgrp, initial-login, requests, hyperon, [11:21:20] !credentials [11:21:20] when you see No Nova credentials found for your account just relog to wiki and should be ok [11:21:26] :o [11:23:49] aha [11:24:08] Damianz: unfortunatelly someone blanked sitenotice insted of template so we can't [11:24:20] D: [11:24:22] mutante: can you restore it :D [11:24:55] point of that was to allow non sysops to change site notice [11:25:06] petan|wk: in your wikitech message you forgot to says what is Huggle :-] [11:25:12] hehe [11:25:25] I hope it's popular enought that everyone knows :D [11:25:39] but probably nor [11:25:40] not [11:25:53] it's mysterious sw [11:28:28] hmm seems to be a reimplementation from scratch of some windows app [11:28:33] can't you make it a gadget on the site ? [11:28:41] cause you will probably have to reinvent a lot of stuff [11:29:09] when a gadget will already have everything it needs (javascript, user authenticated ) [11:29:17] hashar: yes it's a web version of application I develop [11:29:33] I am not really a dev of this web version, I am a dev of application itself [11:29:51] just some people had an idea to start this, so I wanted to announce it in case someone wanted to join us [11:30:32] hashar: I also support a gadget a bit, however I am afraid it can't support so many features as application has [11:30:48] gadget can't connect to irc.pmtpa.net etc [11:31:00] gadget can't connect to multiple wikis in same time [11:31:06] it can't work in multiple threads [11:31:09] etc etc [11:31:45] I don't even think that web server based version will have the speed of application that's why I want to work on both [11:33:00] yeah you are probably right [11:33:08] just the multiple wikis is probably enough [11:33:35] IRC feed need to be migrated to something else. Not going to happen anytime soon though :-\ [11:33:51] I don't really have troubles with irc :) [11:33:54] it's simple [11:33:59] we do [11:34:04] petan|wk: so that is a button except that it is a link [11:34:19] cause the messages depends on l10n messages generated by mediawiki and they changes! [11:34:24] anyway, off for lunch now [11:34:25] is there screenshot of that view anywhere? [11:34:26] see you later [11:35:26] Nikerabbit: I can make it [11:36:02] petan|wk: if it is on commons/translatewiki.net I can include that in the message documentation [11:37:15] api server is a bit down atm [11:37:20] it doesn't open [11:41:15] Nikerabbit: what license should I use [11:41:18] I hate this [11:41:29] why c't I pick no license :/ [11:41:48] WTFPL [11:41:53] {{I don't give a fuck about it}} [11:42:05] that's not valid :/ [11:42:10] WTFPL? [11:42:10] ok [11:42:15] sounds good [11:42:22] I like WTF part [11:42:23] WTFPL - Do What The Fuck You Want To Public License :D [11:42:27] ah [11:42:27] ok [11:42:37] :D [11:42:40] does it exist? [11:42:43] I will use it [11:42:45] http://sam.zoy.org/wtfpl/ [11:42:50] It's in v2! [11:43:48] http://commons.wikimedia.org/wiki/File:Wikimedia_labs_address_admin.png [11:43:54] here we go [11:43:57] petan|wk: yay thanks [14:08:08] 03/07/2012 - 14:08:08 - Updating keys for hashar [14:08:13] 03/07/2012 - 14:08:12 - Updating keys for hashar [14:08:20] 03/07/2012 - 14:08:20 - Updating keys for hashar [15:01:22] !docs [15:01:22] View complete documentation at https://labsconsole.wikimedia.org/wiki/Help:Contents [15:19:12] !labs [15:19:12] https://labsconsole.wikimedia.org/wiki/$1 [15:19:56] !accountreq [15:19:56] in case you want to have an account on labs, please contact someone who is in charge of doing that: Ryan.Lane, m.utante or ssmolle.tt [15:21:05] ssmollett: around? [15:21:48] DeltaQuad: no [15:21:55] she is away like 2 months [15:22:03] oh ok. [15:22:04] I never heard her to speak :D [15:22:14] * DeltaQuad will wait then [15:22:19] or ask mutante :P [15:22:27] mutante: away with message “zzz” [15:22:36] he's never away :P [15:22:45] lol [15:22:48] people from operation only pretend to sleep [15:22:56] they are watching the servers [15:23:02] :o [15:23:54] DeltaQuad: you want account I guess [15:23:59] !account-questions [15:23:59] I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your preferred email address. 3. Your SVN account name, or your preferred shell account name, if you do not have SVN access. [15:24:04] you can prepare this [15:24:28] done :P [18:44:15] petan: around? [18:45:04] petan|wk: around? [20:51:44] hexmode: :o [21:02:42] petan: I was going to ask you if you could document the changes made to deployment-prep, but Ryan_Lane told me werdna did those [21:04:32] petan: I asked Ryan_Lane to set up the cluster with a seperation between shell users and root users, like production. We're going to start using het-deploy, etc more so that it will provide a production-like environment for any dev to use [21:06:08] cool [21:06:16] hexmode: I implemented better log system [21:06:23] werdna: u can type log in shell now [21:06:40] Wibbles [21:06:43] just type a command, then go to history and type it again prefixed with log [21:06:45] :D [21:06:50] petan: ah, nice... where is that documented [21:06:51] that's ok [21:06:55] hexmode: in my head [21:07:04] petan: "log !!" would do what you want [21:07:05] Don't look in his head, it's all confusing [21:07:07] I wanted Ryan to help me get it in puppet [21:07:12] no retyping [21:07:27] hexmode: in shell? [21:07:40] this work in shell [21:07:49] so you don't need to be in two windows [21:07:52] petan: yes, !! justg copies the previous line in history [21:08:00] ah ok [21:08:02] in bash? [21:08:04] try it out [21:08:04] that's cool [21:08:05] yes [21:08:07] wow [21:08:09] damn [21:08:20] that's cool [21:08:38] there are lots of other history shortcuts like that [21:08:42] ok [21:08:52] however the tool is simple [21:09:00] just ype "log message" and that's all [21:09:19] I don't know if we need a documentation, source code has 5 lines [21:09:23] :D [21:09:38] with comments I think [21:09:39] petan: people need to know the command exists and what it does [21:09:53] that is the sort of documentation we need [21:09:59] yes I know, but I don't believe that these people I made this for, would read a docs [21:10:39] petan: I know what you mean, but this week I'm sitting by chrismcmahon and he is letting me know we need lots of documentation [21:10:51] because these people I mean, were not logging stuff :D so they probably didn't read doc where is written log it! [21:10:55] <^demon> Docs exist solely for developers to go "omg you didn't read the docs!" when people ask common questions. In practice, nobody reads docs before asking questions. [21:11:10] hehe [21:11:13] hehe [21:11:25] so true [21:11:48] !demon is <^demon> Docs exist solely for developers to go "omg you didn't read the docs!" when people ask common questions. In practice, nobody reads docs before asking questions. [21:11:48] Key was added! [21:11:51] ^demon: there is another reason: so QA people don't ask "Where are the docs?!" [21:11:53] That's what the phrase 'RTFM' is for [21:12:05] hexmode: I will quip it, can I [21:12:12] :) [21:12:12] sure [21:12:41] * chrismcmahon always looks for the M so I can RTF it. [21:12:59] <^demon> Please don't rich-text format our docs :( [21:13:05] lol [21:13:13] heh [21:13:19] Read the markdown formatted README file :D [21:15:54] mostly I don't want to be surprised when looking at http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version. beyond that, it would be nice to have that updated in a sane and rational way on a regular basis. [21:21:18] so, who wants gluster project storage? :) [21:21:45] I'm almost done with a script to share it out [21:21:46] heh [22:10:53] hmm, Ryan_Lane after your maintenance yesterday it seems that my crontabs stopped working (they were set to run hourly), please can you tell me how to restart them? [22:11:03] oh? which instance? [22:11:07] bots-3 [22:11:13] as your user? [22:11:15] yep [22:11:25] lemme see [22:11:28] it should run... [22:11:30] thanks [22:12:05] they really aren't running? [22:12:22] running them manually seems to work [22:14:58] well, I reinstalled your crontab [22:15:04] hopefully that'll help [22:15:46] let me know if it still doesn't work [22:15:48] I can't see why it wouldn't [22:15:52] a reboot shouldn't affect that [22:17:37] okay thanks Ryan_Lane [22:17:40] yeah they're not running [22:17:52] if you check pywikipediabot/logs commands [22:17:59] last one executed yesterday [22:18:06] and I replied to your message on meta [22:20:52] ah. thanks [22:21:10] Thehelpfulone: well, it'll try again when the hour hits [22:21:19] if it doesn't run then, I'll take another look [22:21:22] wait [22:21:46] there's another problem, then [22:21:53] because I just ran them manually [22:22:00] I meant that it wasn't running [22:22:07] between yesterday and today [22:22:11] oh [22:22:13] ok [22:22:48] Thehelpfulone: Meta:Babel seems to be about language issues [22:23:12] oh [22:23:13] wait [22:23:18] no. it's general purpose [22:23:22] so confusing :) [22:23:31] heh [22:23:35] yeah, "This is Meta's multilingual general and policy discussion page" [22:23:49] you'd think it would be language issues, given it's name [22:23:53] its* [22:25:38] yep [22:54:05] I wonder if I'm insane [22:54:14] yes [22:54:19] wonder no longer! [22:55:30] Thinking about compiling some bots to see if their performance increases :D [22:55:38] heh [22:56:17] * addshore sneezes [22:56:36] * Damianz wipes addshore's snot on ChanServ [22:56:44] * addshore thanks Damianz [22:58:22] i drive an asphalt boat and it's called a town car [22:58:25] ' [22:58:34] luckily not all the time [22:58:42] oh wait [22:59:02] sorry guys, this was meant for another channel [22:59:10] o.0 [22:59:15] * Damianz gives hyperon her drugs [23:00:14] haha [23:05:50] dschoon: seems you install the command line tools through xcode [23:05:57] ahh. [23:05:59] that's new. [23:06:00] and lame. [23:06:02] now I have gcc [23:06:07] actually, it's kind of good [23:06:13] oh? [23:06:26] because I only installed that, and not the other 809082390809234GB of crap [23:06:26] (who installs a dev toolkit without wanting the shell tools??) [23:06:30] ahh. [23:06:32] well, yes. [23:06:40] the SDKs should each be an optional addition [23:06:50] but i mean, everything depends on having a compiler :P [23:07:00] agreed, though, that's dumb to not ship a compiler [23:07:18] i'm pretty sure it has one [23:07:21] Ryan_Lane: the crontab is working now, thanks again :) [23:07:26] Thehelpfulone: yw [23:07:32] installing the CLI stuff just links things into /usr/bin or something [23:07:33] Thehelpfulone: all I did was crontab -e [23:07:34] then save [23:07:46] hmm [23:07:48] dschoon: nah, it actually installed 130MB of stuff [23:07:52] and how did you manually run the crontab? [23:07:53] huh [23:08:01] ls /Developer/usr [23:08:04] Thehelpfulone: I did crontab -l [23:08:11] Thehelpfulone: then ran the commands manually [23:08:17] ah okay [23:08:40] I did that to make sure the commands weren't erroring out [23:08:40] i was under the impression the normal unix toolchain is always installed there, and then linked into place [23:08:42] *shrug* [23:08:53] yeah, it's weird [23:09:07] !account-questions [23:09:07] I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your preferred email address. 3. Your SVN account name, or your preferred shell account name, if you do not have SVN access. [23:31:55] PROBLEM Free ram is now: CRITICAL on deployment-web deployment-web output: Critical: 3% free memory [23:31:55] PROBLEM Free ram is now: CRITICAL on deployment-web5 deployment-web5 output: Critical: 1% free memory [23:34:35] PROBLEM Current Load is now: CRITICAL on deployment-web deployment-web output: CRITICAL - load average: 65.14, 51.88, 24.35 [23:35:35] PROBLEM Current Load is now: CRITICAL on deployment-web5 deployment-web5 output: CRITICAL - load average: 60.97, 54.76, 27.42 [23:36:55] PROBLEM Free ram is now: WARNING on deployment-web3 deployment-web3 output: Warning: 11% free memory [23:39:35] PROBLEM Current Load is now: WARNING on deployment-web3 deployment-web3 output: WARNING - load average: 2.16, 9.54, 6.09 [23:41:56] RECOVERY Free ram is now: OK on deployment-web deployment-web output: OK: 71% free memory [23:41:56] RECOVERY Free ram is now: OK on deployment-web5 deployment-web5 output: OK: 67% free memory [23:41:56] RECOVERY Free ram is now: OK on deployment-web3 deployment-web3 output: OK: 66% free memory [23:44:36] RECOVERY Current Load is now: OK on deployment-web3 deployment-web3 output: OK - load average: 0.24, 3.61, 4.45 [23:49:36] PROBLEM Current Load is now: WARNING on deployment-web deployment-web output: WARNING - load average: 0.19, 5.65, 14.68 [23:50:36] PROBLEM Current Load is now: WARNING on deployment-web5 deployment-web5 output: WARNING - load average: 0.22, 5.47, 15.38