[00:00:07] New review: Lcarr; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2002 [00:00:08] Change merged: Lcarr; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2002 [00:00:11] 01/22/2012 - 00:00:11 - Updating keys for dale [00:03:00] !account-questions | jpostlethwaite [00:03:00] jpostlethwaite : I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [00:04:50] Ryan_Lane: jpostlethwaite for all accounts and jpostlethwaite@wikimedia.org [00:05:15] Ryan_Lane: when you get a second can you walk me through the quilt process for the nginx project? [00:06:34] RECOVERY Disk Space is now: OK on fwserver1 fwserver1 output: DISK OK [00:06:39] hi [00:06:48] hi [00:06:50] :o [00:06:54] RECOVERY Free ram is now: OK on fwserver1 fwserver1 output: OK: 92% free memory [00:07:11] 01/22/2012 - 00:07:11 - Updating keys for zaran [00:07:32] Ryan_Lane: you should write a guide how people are supposed to call instances or we will have a lot of mess :) [00:07:44] tons of names of instances we wouldn't clearly see where they belong to [00:08:14] RECOVERY Total Processes is now: OK on fwserver1 fwserver1 output: PROCS OK: 83 processes [00:08:30] hexmode: did you setup the http://commons.wikimedia.beta.wmflabs.org ? [00:08:54] RECOVERY dpkg-check is now: OK on fwserver1 fwserver1 output: All packages OK [00:09:07] ok, so I have a labs account. Now what ? :D [00:09:43] petan: i think we also need djvu installed for wikisource [00:09:44] RECOVERY Current Users is now: OK on fwserver1 fwserver1 output: USERS OK - 1 users currently logged in [00:09:53] thedj: ok [00:09:57] mdale: hi [00:10:01] mdale: I did it [00:10:04] RECOVERY Current Load is now: OK on fwserver1 fwserver1 output: OK - load average: 0.01, 0.06, 0.03 [00:10:10] 01/22/2012 - 00:10:10 - Updating keys for dale [00:10:10] 01/22/2012 - 00:10:10 - Updating keys for dale [00:10:11] diederik: https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [00:10:17] mdale: is there any problem? [00:10:42] small issues [00:11:09] and I have other questions ... we are trying to setup a test instance for mediaHandler and need a copy of labs commons [00:11:21] copy of config? [00:11:28] or what exactly you mean? [00:12:15] 01/22/2012 - 00:12:14 - Creating a home directory for tstarling at /export/home/deployment-prep/tstarling [00:12:15] 01/22/2012 - 00:12:14 - Creating a home directory for hartman at /export/home/deployment-prep/hartman [00:12:15] 01/22/2012 - 00:12:14 - Creating a home directory for zaran at /export/home/deployment-prep/zaran [00:12:32] trying to figure out the best way forward.. I think a copy of config will help... essentially we need a ~near-as-possible to cluster labs set of machines [00:12:33] New patchset: Lcarr; "fixing $title" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2003 [00:12:37] thedj: it's already installed on apache [00:12:45] thedj: is there anything else to install? [00:12:46] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2003 [00:13:06] thedj: can you please check if it works ok and if not, let me know what needs to be configured or installed [00:13:14] 01/22/2012 - 00:13:13 - Updating keys for hartman [00:13:14] 01/22/2012 - 00:13:13 - Updating keys for tstarling [00:13:14] 01/22/2012 - 00:13:13 - Updating keys for zaran [00:13:34] New review: Lcarr; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2003 [00:13:34] Change merged: Lcarr; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2003 [00:13:35] Ryan_Lane: it's saturday, what's going on, so many people here :) [00:13:44] hackathon! [00:13:46] oh [00:13:50] heh [00:13:54] didn't know [00:13:59] petan, this is for the TimedMediaHandler extension [00:14:04] I've added a note here: http://labs.wikimedia.beta.wmflabs.org/wiki/Global_requests#Extensions [00:14:10] ok [00:14:12] petan: http://en.wikisource.beta.wmflabs.org/wiki/File:Wind_in_the_Willows_(1913).djvu [00:14:14] will handle it right now [00:14:20] Error creating thumbnail: Unable to fetch XML for DjVu file [00:14:28] We'll probably need a copy of the commons.beta.wikimedia.wmflabs.org unless you're OK to screw with the existing commons test site [00:14:44] I am ok, dunno if Mar k H. is ok [00:14:56] because he is de-facto project lead of beta [00:15:06] hexmode: ^ [00:15:12] mdale, do you have a list of the main brokenness with commons.beta.wikimedia.wmflabs.org somewhere, as well as the dependencies for TMH? [00:15:52] yes... let me copy it to a wiki [00:16:29] petan: what are you wondering if hexmode is okay with? [00:16:46] robla: deploying some extensions which is not being used on prod now to beta [00:16:55] I checked with him earlier generally about mucking with it, so he's been warned [00:17:04] djvu isn't in production? [00:17:10] ah, ok and did he agree with that? [00:17:13] yeah [00:17:27] PROBLEM host: ubuntu1 is DOWN address: ubuntu1 CRITICAL - Host Unreachable (ubuntu1) [00:17:35] that surely is, I meant the extension you wanted to install, btw Eloquence I can't find that on request page :| [00:17:39] New patchset: Lcarr; "fixing title again" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2004 [00:17:40] http://labs.wikimedia.beta.wmflabs.org/wiki/Global_Requests#Extensions [00:17:54] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2004 [00:18:01] New review: Lcarr; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2004 [00:18:01] Eloquence: which extensions you want to install there? [00:18:02] Change merged: Lcarr; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2004 [00:18:14] TimedMediaHandler? [00:18:28] oh I see now [00:18:51] :D [00:19:01] sorry everyone, being bombarded [00:19:05] heh I see [00:19:09] * Ryan_Lane reads backscroll [00:19:56] jpostlethwaite: you already have an account [00:20:48] petan: djvu seems to require setup of the djvu binaries... [00:21:38] !deployment-prep installed timedmediahandler (trunk) to commons [00:21:44] !log deployment-prep installed timedmediahandler (trunk) to commons [00:21:45] Logged the message, Master [00:21:59] thedj: I believe all are installed on apache server [00:22:08] at least apt say it [00:22:23] !initial-login jpostlethwaite [00:22:23] https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [00:22:32] !initial-login | jpostlethwaite [00:22:32] jpostlethwaite: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [00:24:13] PROBLEM Total Processes is now: CRITICAL on ubuntu1-pgehres ubuntu1-pgehres output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:24:30] Ryan_Lane: Thanks, login works - how do I deploy my wiki? [00:24:53] PROBLEM dpkg-check is now: CRITICAL on ubuntu1-pgehres ubuntu1-pgehres output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:25:11] 01/22/2012 - 00:25:11 - Updating keys for jpostlethwaite [00:25:33] Ryan_Lane: $ git clone ssh://jdz@gerrit.wikimedia.org:29418/labs/private.git [00:25:36] Cloning into private... [00:25:39] fatal: '/labs/private.git': not a Gerrit project [00:25:43] PROBLEM Current Load is now: CRITICAL on ubuntu1-pgehres ubuntu1-pgehres output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:26:13] PROBLEM Current Users is now: CRITICAL on ubuntu1-pgehres ubuntu1-pgehres output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:27:03] PROBLEM Disk Space is now: CRITICAL on ubuntu1-pgehres ubuntu1-pgehres output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:27:12] btw, enjoy hackaton :) [00:27:43] PROBLEM Free ram is now: CRITICAL on ubuntu1-pgehres ubuntu1-pgehres output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:29:53] PROBLEM host: patchtest is DOWN address: patchtest PING CRITICAL - Packet loss = 100% [00:30:16] Eloquence: http://commons.wikimedia.beta.wmflabs.org/wiki/Special:Version [00:30:54] !log deployment-prep updating /live to head [00:30:56] Logged the message, Master [00:31:10] Ryan_Lane: i'm here on and off but will respond much faster (or at all) if you say my name. something i can do about the bombarding? [00:31:38] petan, thanks :-) [00:31:45] New patchset: Lcarr; "fixing title" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2006 [00:31:54] jesse__: private may be inaccessible right now [00:31:59] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2006 [00:32:08] New review: Lcarr; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2006 [00:32:09] Change merged: Lcarr; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2006 [00:32:43] jesse__: it probably isn't necessary for most things you need right now [00:33:21] New patchset: Lcarr; "fixing file name" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2007 [00:33:56] New review: Lcarr; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2007 [00:33:57] Change merged: Lcarr; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2007 [00:34:22] is jesse staff? (private is staff only? or i'm confused) [00:34:32] or maybe s/staff/ops/g [00:34:40] private for labs is open to all in labs [00:34:52] Hi, I'm just trying to figure out how to put mediawiki on my instance [00:34:55] there's a private for production that's only accessible to ops, and isn't in gerrit [00:34:55] aha [00:35:07] jesse__: it's unfortunately a fairly manual process right now [00:35:21] ok. is it even on a git server? or just locally on fenari or something [00:35:23] so how would i get access to http://en.wikisource.beta.wmflabs.org/w/index.php?title=Special:Import [00:35:26] jesse__: I have 30 minutes, if you needed help, let me know [00:35:31] thedj: moment [00:35:49] apt-get install apache2 mysql mysql-server php5 php5-mysql php-apc php5-memcache [00:35:49] i suppose it's not so bad to lose it because it's related to the other private [00:36:01] there's probably more you need [00:36:16] it'd be awesome if someone made a turnkey-mediawiki puppet class :) [00:36:21] OK do I need sudo or something? [00:36:24] thedj: what is your acc name? [00:36:27] TheDJ [00:36:29] jesse__: you have sudo already [00:36:35] probably [00:36:41] jesse__: you have sudo. the password is your wiki password [00:36:45] > User account "Jesse" is not registered. [00:36:49] jesse__: which project are you trying to do this in? [00:36:57] jeremyb: It's Jdk [00:36:59] thedj: I gave you steward try nwo [00:37:01] * now [00:37:01] ah [00:37:10] it should work on all wikis [00:37:11] would help if i logged in [00:37:12] bastion allows nothing [00:37:13] petan: I guess we should add j^ and me to that project group [00:37:24] btw thedj you may need to relog [00:37:35] and we should move that to a separate commons instance / wiki [00:37:40] hey, everyone, is someone waiting on me for something? [00:37:42] mark: which project? [00:37:46] oh sorry [00:37:48] I'm forgetting what people are pinging me about :) [00:37:48] mark: nvm [00:37:52] mdale: which project? [00:38:04] preilly: I know you need an IP [00:38:40] preilly: which project? [00:38:56] petan: looking [00:39:08] Ryan_Lane: I don't have a project, is there anything I can read about this? [00:39:23] PROBLEM host: ubuntu1-pgehres is DOWN address: ubuntu1-pgehres CRITICAL - Host Unreachable (ubuntu1-pgehres) [00:39:33] petan: Hi thanks, seems to be the problem is I don't have sudo on my node [00:39:48] jesse__: you surely have [00:39:55] jesse__: which instance it is? [00:39:56] New patchset: Lcarr; "trying test class 2" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2008 [00:40:13] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2008 [00:40:16] petan: i-000000ba [00:40:29] oh that is bastion :) [00:40:35] that's ok you don't need sudo there [00:40:41] !bastion [00:40:41] http://en.wikipedia.org/wiki/Bastion_host; lab's specific bastion host is: bastion.wmflabs.org; see !access [00:40:43] jesse__: what will you be working on? [00:40:51] New review: Lcarr; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2008 [00:40:51] Change merged: Lcarr; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2008 [00:41:08] I either need to create a project for you, or need to add you to an existing one [00:41:23] Ryan_Lane: I'm testing a couple of extensions to generate RSS feeds from pages [00:41:29] ah. ok [00:41:37] let me create a project for you [00:41:43] I'll call it rss [00:41:45] or [00:41:46] Ryan_Lane: maybe we could create a general sandbox project with configured apache mysql memcached where people can easily create wikis for now? [00:41:47] feeds? [00:41:49] petan: jdz is not in the sudoers file. This incident will be reported. [00:42:00] petan: that would be ideal, but it's not terribly easy [00:42:10] as everyone would need root, right now [00:42:12] jesse__: you don't need to sudo on bastion [00:42:35] jesse__: bastion is an instance that lets you access other ones [00:42:36] Ryan_Lane: I don't think it would be problem to have root on sandbox project, at least if there was a backup [00:42:49] I'd prefer to just make separate projects for now [00:42:52] ok [00:43:00] I mean some people may have troubles installing mysql etc [00:43:04] petan: what did you want your project to be called again? [00:43:11] 01/22/2012 - 00:43:11 - Updating keys for pgehres [00:43:13] Ryan_Lane: deployment-prepbackup [00:43:16] we don't have an automated way of installing most of the mediawiki related things [00:43:17] ok [00:43:17] 01/22/2012 - 00:43:17 - Updating keys for pgehres [00:43:36] thanks [00:44:17] 01/22/2012 - 00:44:16 - Creating a project directory for deployment-prepbackup [00:44:17] 01/22/2012 - 00:44:17 - Creating a home directory for laner at /export/home/deployment-prepbackup/laner [00:44:17] 01/22/2012 - 00:44:17 - Creating a home directory for petrb at /export/home/deployment-prepbackup/petrb [00:44:28] petan: ^^ [00:44:32] nice [00:45:03] RECOVERY Disk Space is now: OK on client2-lcarr client2-lcarr output: DISK OK [00:45:13] RECOVERY Current Users is now: OK on client2-lcarr client2-lcarr output: USERS OK - 1 users currently logged in [00:45:16] !log deployment-prepbackup creating a backup instance for online backup of mysql from deployment project + fs backup [00:45:16] deployment-prepbackup is not a valid project. [00:45:18] 01/22/2012 - 00:45:17 - Updating keys for petrb [00:45:18] 01/22/2012 - 00:45:17 - Updating keys for laner [00:45:41] petan: OK thanks, my bad [00:45:49] !log deployment-prep creating a backup instance in -prepbackup project for online backup of mysql from deployment project + fs backup [00:45:50] Logged the message, Master [00:46:03] RECOVERY Free ram is now: OK on client2-lcarr client2-lcarr output: OK: 78% free memory [00:46:03] jesse__: no problem [00:46:17] 01/22/2012 - 00:46:17 - Creating a project directory for feeds [00:46:17] 01/22/2012 - 00:46:17 - Creating a home directory for laner at /export/home/feeds/laner [00:46:17] 01/22/2012 - 00:46:17 - Creating a home directory for jdz at /export/home/feeds/jdz [00:46:37] does wm-bot need boot on new project creation? [00:46:49] wm-bot: surely not [00:46:55] morebots, who knows [00:47:16] yeah... sameish thing [00:47:18] 01/22/2012 - 00:47:18 - Updating keys for jdz [00:47:18] 01/22/2012 - 00:47:18 - Updating keys for laner [00:47:33] RECOVERY Total Processes is now: OK on client2-lcarr client2-lcarr output: PROCS OK: 81 processes [00:47:33] labs-morebots: reboot [00:47:33] wm bot doesn't need reboot at all :) there is a @reload command [00:48:01] I hate programs which needs restart on config update [00:48:13] RECOVERY dpkg-check is now: OK on client2-lcarr client2-lcarr output: All packages OK [00:48:17] 01/22/2012 - 00:48:17 - Updating keys for pgehres [00:48:43] labs-home-wm should say which project it's talking about [00:49:11] 01/22/2012 - 00:49:10 - Updating keys for pgehres [00:49:12] jeremyb: fix it! [00:49:13] RECOVERY Current Load is now: OK on client2-lcarr client2-lcarr output: OK - load average: 0.00, 0.06, 0.02 [00:49:25] petan: where does it live? where is version control? [00:49:32] in tools [00:49:35] I will check it [00:49:41] svn.wikimedia [00:50:02] most of labs utils are there [00:50:06] including nagios [00:50:11] 01/22/2012 - 00:50:11 - Updating keys for pgehres [00:50:11] 01/22/2012 - 00:50:11 - Updating keys for gicode [00:50:17] 01/22/2012 - 00:50:16 - Updating keys for gicode [00:50:17] 01/22/2012 - 00:50:17 - Updating keys for pgehres [00:51:30] petan: i don't have commit there (i don't think) [00:51:38] you don't have svn? [00:51:43] but i could make a patch [00:51:43] ah, ok [00:52:06] that's weird you don't because I have very restricted svn access and I can commit there [00:53:59] PROBLEM Total Processes is now: CRITICAL on deployment-backup deployment-backup output: Connection refused by host [00:54:28] !load is http://ganglia.wikimedia.org/2.2.0/graph_all_periods.php?h=virt2.pmtpa.wmnet&m=load_one&r=hour&s=by%20name&hc=4&mc=2&st=1327006829&g=load_report&z=large&c=Virtualization%20cluster%20pmtpa [00:54:28] Key was added! [00:54:49] RECOVERY host: ubuntu1-pgehres is UP address: ubuntu1-pgehres PING OK - Packet loss = 0%, RTA = 74.76 ms [00:54:59] PROBLEM dpkg-check is now: CRITICAL on deployment-backup deployment-backup output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:55:17] !load-all is http://ganglia.wikimedia.org/2.2.0/?c=Virtualization%20cluster%20pmtpa&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [00:55:17] Key was added! [00:55:19] PROBLEM Current Load is now: CRITICAL on deployment-backup deployment-backup output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:56:09] PROBLEM Current Users is now: CRITICAL on deployment-backup deployment-backup output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:57:09] PROBLEM Disk Space is now: CRITICAL on deployment-backup deployment-backup output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:57:49] PROBLEM Free ram is now: CRITICAL on deployment-backup deployment-backup output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:57:59] ok folks I need to go, anyone need anything I could help with :) [00:58:03] ? [00:58:28] if you wanted to set up anything on beta, please ping me and I will read it later or list it on global requests [00:58:57] thedj: if you needed to import large dump, send me a list of pages on pastebin [00:59:59] PROBLEM host: patchtest is DOWN address: patchtest PING CRITICAL - Packet loss = 100% [01:01:28] petan: could I get added to deployment-prep ? [01:01:37] !log deployment-prep configured firewall for backup instance [01:01:39] Logged the message, Master [01:02:40] mdale: hm, you should probably ask hexmode, anyway I don't have problem with that... [01:02:46] or... hmm do we want to relay some commands? to get it working there? or how do we want to move forward? [01:03:47] mdale: site is for testing of software, so actually if you request any config change or extension, just list it on request page and I do that, or once you get shell access, you can do it too [01:04:03] petan: maybe i do. i haven't actually tried. certainly i haven't applied for it or seen an announcement that i've been added [01:04:04] PROBLEM Total Processes is now: CRITICAL on feeds feeds output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:04:05] but you should probably ask hex [01:04:54] PROBLEM dpkg-check is now: CRITICAL on feeds feeds output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:05:14] PROBLEM host: patchtest2 is DOWN address: patchtest2 CRITICAL - Host Unreachable (patchtest2) [01:05:44] PROBLEM Current Load is now: CRITICAL on feeds feeds output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:06:03] petan: ok ... well right now its just run update.php ... by "the page" do you mean: https://labsconsole.wikimedia.org/w/index.php?title=Nova_Resource_Talk:Deployment-prep&action=edit ? [01:06:14] PROBLEM Current Users is now: CRITICAL on feeds feeds output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:06:43] Ryan_Lane: How long do you expect I need to wait for my ssh key to work on a fresh instance after ssh is running? [01:06:44] mdale: http://labs.wikimedia.beta.wmflabs.org/wiki/Global_Requests [01:06:48] update.php is running now [01:06:54] PROBLEM Disk Space is now: CRITICAL on feeds feeds output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:06:54] great thanks [01:07:44] PROBLEM Free ram is now: CRITICAL on feeds feeds output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:07:56] gicode: are they not working yet? what msg are you getting? [01:08:24] gicode: there's a bot that logs here when keys are sync'd. but maybe that doesn't happen for new nodes (only existing nodes) [01:08:36] gicode: you should check the node's console output (onwiki) [01:10:22] jeremyb: My keys are showing in my home directory on another instance and I haven't changed them recently [01:10:34] 22 01:08:36 < jeremyb> gicode: you should check the node's console output (onwiki) [01:10:50] jeremyb: I have that up now, but I don't know what to look for [01:11:16] gicode: "get console output"? [01:12:16] jeremyb: Yes, I see the output. it is quite extensive though [01:12:24] In any case it started working :-) [01:14:37] having trouble ssh'ing into bastion.wmflabs.org [01:14:55] already took a look at https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [01:15:17] 01/22/2012 - 01:15:17 - Creating a home directory for laner at /export/home/selenium/laner [01:15:35] gicode: the output should tell you if it's still doing stuff or it's done (is there a login prompt) [01:15:37] getting permission denied ( public key ) [01:16:05] mdale: sure you didn't accidentally have a carriage return in your pubkey (happens t me when i copy/paste sometimes) [01:16:17] 01/22/2012 - 01:16:17 - Updating keys for laner [01:16:23] jeremyb: If there is a login prompt in there it is buried deep [01:16:29] will try to replace the public key on the wiki one sec. [01:17:11] 01/22/2012 - 01:17:11 - Updating keys for dale [01:17:12] 01/22/2012 - 01:17:11 - Creating a home directory for aaron at /export/home/bastion/aaron [01:17:12] 01/22/2012 - 01:17:11 - Updating keys for dale [01:17:23] jeremyb: patchtest > i-000000fd if you are curious, but it is working in any case [01:17:28] gicode: also sometimes, it just never works and you have to boot it [01:17:36] no luck [01:17:38] gicode: i surely don't have access to view it's log [01:17:56] jeremyb: OK [01:18:00] mdale: setting teh right username? [01:18:11] 01/22/2012 - 01:18:11 - Updating keys for aaron [01:18:24] running: ssh mdale@bastion.wmflabs.org [01:18:56] and my id_rsa.pub file is the same up on the site ... [01:19:19] /.ssh/id_rsa.pub [01:20:37] btw folks if someone want account on nagios before I fix ldap auth [01:20:39] let me know [01:20:46] !nagios [01:20:46] http://nagios.wmflabs.org/nagios3 [01:23:01] RECOVERY Free ram is now: OK on deployment-backup deployment-backup output: OK: 82% free memory [01:24:01] RECOVERY Total Processes is now: OK on deployment-backup deployment-backup output: PROCS OK: 84 processes [01:25:01] RECOVERY dpkg-check is now: OK on deployment-backup deployment-backup output: All packages OK [01:26:11] RECOVERY Current Users is now: OK on deployment-backup deployment-backup output: USERS OK - 1 users currently logged in [01:26:51] RECOVERY Current Load is now: OK on deployment-backup deployment-backup output: OK - load average: 0.64, 0.72, 0.43 [01:27:11] RECOVERY Disk Space is now: OK on deployment-backup deployment-backup output: DISK OK [01:27:24] we really need to set up ganglia [01:27:31] RECOVERY Total Processes is now: OK on ubuntu1-pgehres ubuntu1-pgehres output: PROCS OK: 123 processes [01:27:51] RECOVERY Free ram is now: OK on ubuntu1-pgehres ubuntu1-pgehres output: OK: 89% free memory [01:28:19] hee ryan: abe just told me about logging http requests as serialized json [01:28:35] would that be something that we could do because that would like life much easier :) [01:28:38] :D [01:29:05] eh? [01:29:11] RECOVERY Current Load is now: OK on ubuntu1-pgehres ubuntu1-pgehres output: OK - load average: 0.03, 0.12, 0.07 [01:29:31] RECOVERY Current Users is now: OK on ubuntu1-pgehres ubuntu1-pgehres output: USERS OK - 2 users currently logged in [01:29:31] I don't think that would help :) [01:29:51] RECOVERY dpkg-check is now: OK on ubuntu1-pgehres ubuntu1-pgehres output: All packages OK [01:29:54] why not? [01:30:01] PROBLEM host: patchtest is DOWN address: patchtest PING CRITICAL - Packet loss = 100% [01:30:11] RECOVERY Disk Space is now: OK on ubuntu1-pgehres ubuntu1-pgehres output: DISK OK [01:30:53] because udp2log expects it in a very different format? [01:36:21] PROBLEM host: patchtest2 is DOWN address: patchtest2 CRITICAL - Host Unreachable (patchtest2) [01:39:16] 01/22/2012 - 01:39:16 - Creating a project directory for wikisource-dev [01:39:17] 01/22/2012 - 01:39:16 - Creating a home directory for robla at /export/home/wikisource-dev/robla [01:39:17] 01/22/2012 - 01:39:16 - Creating a home directory for hartman at /export/home/wikisource-dev/hartman [01:39:17] 01/22/2012 - 01:39:16 - Creating a home directory for zaran at /export/home/wikisource-dev/zaran [01:40:18] 01/22/2012 - 01:40:18 - Updating keys for hartman [01:41:18] 01/22/2012 - 01:41:18 - Creating a home directory for tstarling at /export/home/wikisource-dev/tstarling [01:42:18] 01/22/2012 - 01:42:18 - Updating keys for tstarling [01:45:22] diederik: src/core/ngx_string.c [01:45:26] diederik: ngx_escape_uri [01:45:40] ryan: AWESOME! [01:47:43] \o/ [01:51:54] RECOVERY host: patchtest2 is UP address: patchtest2 PING OK - Packet loss = 0%, RTA = 2743.61 ms [01:52:14] PROBLEM HTTP is now: CRITICAL on patchtest2 patchtest2 output: Connection refused [01:52:49] PROBLEM Free ram is now: CRITICAL on patchtest2 patchtest2 output: Connection refused by host [01:52:49] PROBLEM Current Load is now: CRITICAL on patchtest2 patchtest2 output: Connection refused by host [01:53:24] PROBLEM dpkg-check is now: CRITICAL on patchtest2 patchtest2 output: Connection refused by host [01:54:14] PROBLEM Free ram is now: CRITICAL on wikisource-web wikisource-web output: Connection refused by host [01:54:54] PROBLEM Total Processes is now: CRITICAL on patchtest2 patchtest2 output: Connection refused by host [01:55:54] PROBLEM Total Processes is now: CRITICAL on wikisource-web wikisource-web output: Connection refused by host [01:56:24] PROBLEM dpkg-check is now: CRITICAL on wikisource-web wikisource-web output: Connection refused by host [01:56:54] heh. I just broke some stuff [01:57:04] PROBLEM Current Load is now: CRITICAL on wikisource-web wikisource-web output: Connection refused by host [01:57:05] (sorry) [01:57:54] PROBLEM Current Users is now: CRITICAL on wikisource-web wikisource-web output: Connection refused by host [01:57:55] deployment-prep might be having issues with its floating IP [01:58:25] Ryan_Lane: ... one more question... whats are default passwords? [01:58:34] PROBLEM Disk Space is now: CRITICAL on wikisource-web wikisource-web output: Connection refused by host [01:58:39] or is there anything like that? [01:58:53] no defaults [01:58:59] just empty? [01:59:20] same as wiki? [01:59:26] wait [01:59:29] default for what? [01:59:34] PROBLEM Current Users is now: CRITICAL on patchtest2 patchtest2 output: Connection refused by host [01:59:44] PROBLEM Disk Space is now: CRITICAL on patchtest2 patchtest2 output: Connection refused by host [01:59:52] ok. deployment's public IP should be good again [01:59:57] I guess it's possible it never broke [02:00:04] PROBLEM host: patchtest is DOWN address: patchtest PING CRITICAL - Packet loss = 100% [02:00:28] mdale: password for what again? [02:00:49] like for the instance if I need to install something? [02:01:03] for root? [02:01:07] you use sudo [02:01:09] yes [02:01:13] the password is your wiki password [02:01:14] it asks for a password [02:01:15] ok [02:01:17] perfect thanks :) [02:06:14] PROBLEM host: patchtest2 is DOWN address: patchtest2 PING CRITICAL - Packet loss = 100% [02:25:32] !log testlabs deleting some unused instances [02:25:34] Logged the message, Master [02:27:41] !log membase deleting instances, will recreate when we actually start work on this [02:27:42] Logged the message, Master [02:30:04] PROBLEM host: patchtest is DOWN address: patchtest PING CRITICAL - Packet loss = 100% [02:30:54] RECOVERY Total Processes is now: OK on wikisource-web wikisource-web output: PROCS OK: 78 processes [02:31:24] RECOVERY dpkg-check is now: OK on wikisource-web wikisource-web output: All packages OK [02:32:04]