[00:21:14] Ryan_Lane: Can nova-production1 actually launch instances? [00:21:33] And, if so, what are you using as hosts? [00:32:18] yes, but they aren't accessible [00:32:30] andrewbogott: it's an all in one instance [00:32:45] it's also a compute node [00:33:08] How did you arrange that? Presumably you didn't tweak the nova.conf since it gets clobbered by puppet. [00:33:28] look at the configuration for it [00:33:39] it's kind of a pain, overall [00:33:57] the puppet config says if labs => do crazy things [00:34:13] in openstack.pp [00:34:22] oh... so that means that I should already be getting all that on my labs instance. [00:34:32] yeah [00:34:43] When you say they aren't accessible, does that mean that when you create an instance with euca it remains in the 'building' state forever? That's what I see. [00:34:58] I think it should say running [00:35:18] Ok, so something is still broken. I will persevere. [00:35:37] did a puppet run ever finish? [00:35:58] I think so. [00:36:05] It hangs every other time [00:37:42] did pdns install properly? [00:37:51] that's what usually causes the problem [00:38:25] andrewbogott: ps for the pdns processes [00:38:27] and kill them [00:38:36] then the puppet run will finish [00:40:04] ok, I'll give that a try [00:48:02] RECOVERY dpkg-check is now: OK on nova-essex-test i-000001f9 output: All packages OK [01:52:12] PROBLEM Puppet freshness is now: CRITICAL on puppet-lucid i-00000080 output: Puppet has not run in last 20 hours [03:02:14] RECOVERY Puppet freshness is now: OK on deployment-web4 i-00000163 output: puppet ran at Thu Apr 12 03:02:05 UTC 2012 [03:39:44] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 16% free memory [03:44:44] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 16% free memory [03:59:46] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 4% free memory [04:00:46] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 15% free memory [04:00:46] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 16% free memory [04:04:46] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 5% free memory [04:04:46] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:09:46] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [04:20:46] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 3% free memory [04:25:54] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory [04:25:54] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 4% free memory [04:25:54] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 11% free memory [04:30:54] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 93% free memory [04:30:54] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [06:39:48] RECOVERY Disk Space is now: OK on login-test4 i-000001f4 output: DISK OK [06:52:13] PROBLEM Puppet freshness is now: CRITICAL on kirke i-000001fd output: Puppet has not run in last 20 hours [06:52:13] PROBLEM Puppet freshness is now: CRITICAL on kripke i-000001fc output: Puppet has not run in last 20 hours [07:04:33] RECOVERY Disk Space is now: OK on aggregator1 i-0000010c output: DISK OK [09:22:37] PROBLEM Disk Space is now: CRITICAL on login-test4 i-000001f4 output: DISK CRITICAL - free space: / 197 MB (2% inode=93%): [09:57:38] PROBLEM Disk Space is now: WARNING on aggregator1 i-0000010c output: DISK WARNING - free space: / 399 MB (4% inode=93%): [10:02:37] PROBLEM Disk Space is now: CRITICAL on aggregator1 i-0000010c output: DISK CRITICAL - free space: / 148 MB (1% inode=93%): [11:22:16] !petan [11:22:24] !labs [11:22:24] https://labsconsole.wikimedia.org/wiki/$1 [11:53:13] PROBLEM Puppet freshness is now: CRITICAL on puppet-lucid i-00000080 output: Puppet has not run in last 20 hours [12:46:16] you know bots-3 has 1% free ram? [12:46:30] whoever is running bot there, it needs to be fixed [12:47:39] ACKNOWLEDGEMENT Puppet freshness is now: CRITICAL on kirke i-000001fd output: Puppet has not run in last 20 hours [12:47:39] ACKNOWLEDGEMENT Puppet freshness is now: CRITICAL on kripke i-000001fc output: Puppet has not run in last 20 hours [12:48:26] * Thehelpfulone hides [12:48:48] most likely me and my inefficiency petan [12:49:18] I'm running 3 screens with 3 different interwiki.pys [12:49:41] is there a way to see how much RAM I'm using petan? [12:50:28] Thehelpfulone: no it's beetstra [12:50:35] your bot doesn't eat much [12:50:43] heh [12:50:46] not yet anyways [12:50:47] he takes like 90% of resources [12:50:53] how can you tell that? [12:51:01] I see it in process list [12:51:17] his bot is written in perl, no wonder then [12:51:26] how do I view the process list? [12:51:30] ps [12:51:31] man ps [12:54:48] system information gives me this [12:54:49] Usage of /: 10.1% of 9.84GB Users logged in: 1 [12:54:50] Memory usage: 28% IP address for eth0: 10.4.0.54 [12:54:58] is that my own usage or in general? [12:55:10] Thehelpfulone: you may want to move it to bots-4 which isn't used at all [12:55:16] !nagios [12:55:16] http://nagios.wmflabs.org/nagios3 [12:55:24] that's far better to check stats [12:55:34] Critical: 3% free memory [12:55:37] how do I get out of this manual? [12:55:43] press q [12:55:49] or ctrl c [12:55:49] oh [12:55:55] I tried ctrl c, didn't work [12:55:58] then q [12:56:00] :P [12:56:03] yep [12:58:03] is it easy to migrate to bots-4? [12:58:24] or would it just be better if Beestra moved [13:00:08] depends where your bot is, if it's in home you could just switch instance and start it again [13:00:22] if it's someone on local storage you need to copy [13:00:27] somewhere [14:41:43] ACKNOWLEDGEMENT Puppet freshness is now: CRITICAL on puppet-lucid i-00000080 output: Puppet has not run in last 20 hours [16:45:55] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 1.57, 14.45, 10.06 [17:00:52] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.27, 1.34, 4.33 [18:02:03] hi Ryan_Lane [18:02:09] howdy [18:02:14] if I want to get an extension installed on deployment-prep, who do I need to ask? [18:02:19] umm [18:02:24] maybe petan [18:02:42] is this an extension that is about to be deployed, or is already deployed on the sites? [18:02:51] okay, it's related to http://en.wikipedia.org/wiki/Wikipedia_talk:ACC#Account_Request_extension.2C_revisited -> I'm thinking it might be a good idea for people to see what the extension is as it's been a few years since we last saw it [18:03:14] deployment-prep isn't the right place for it then [18:03:23] you should add it to a development wiki of some variety [18:03:31] ok [18:03:38] deployment-prep is for extensions that are being tested for deploymenyt [18:03:41] *deployment [18:03:48] are there any development wikis for that purpose at the moment? [18:06:49] I was thinking deployment-prep as it's already got all the current configuration for enwiki Ryan_Lane, so it would be a bit more realistic to see how it looks [18:07:28] it would make deployment tests flawed, though [18:09:25] hmm a valid point. I'll see if petan can install it on a new wiki then maybe [18:44:15] PROBLEM Puppet freshness is now: CRITICAL on nova-production1 i-0000007b output: Puppet has not run in last 20 hours [18:48:15] PROBLEM Puppet freshness is now: CRITICAL on nova-gsoc1 i-000001de output: Puppet has not run in last 20 hours [19:25:22] Hello, I see, sombody created an account for me on labsconsole? [19:26:57] robiH: were you not expecting that? [19:27:02] did you apply for gerrit access? [19:28:14] Actually, I made a task for DZahn, to do it ;-) [19:28:42] It applies to the wikistats on wmflabs. [19:29:10] New patchset: Sara; "Yet another iteration of adding ganglia for labs." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4799 [19:29:18] so far, they also can be maintained using a special console application, the WSA. [19:29:20] robiH: I'm confused about your question, then [19:29:23] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4799 [19:30:08] The confusion ist, the account has been created, but I still am waiting for the credentials. [19:30:49] New review: Sara; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4799 [19:30:57] Change merged: Sara; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4799 [19:33:25] !initial-login | robiH [19:33:25] robiH: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [19:33:33] robiH: make it send you another password [19:34:12] Ryan_Lane: i'd like to delete some of the login-test* instances i created the other day. are there any that you've made changes to that i should not delete? [19:35:07] ssmollett: nope. have it it [19:35:09] *at [19:35:33] Got it. See you later. [19:39:55] PROBLEM host: aggregator2 is DOWN address: i-000001e8 check_ping: Invalid hostname/address - i-000001e8 [19:43:45] PROBLEM Current Load is now: CRITICAL on ganglia-test i-00000202 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:44:25] PROBLEM Current Users is now: CRITICAL on ganglia-test i-00000202 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:45:05] PROBLEM Disk Space is now: CRITICAL on ganglia-test i-00000202 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:45:45] PROBLEM Free ram is now: CRITICAL on ganglia-test i-00000202 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:46:55] PROBLEM Total Processes is now: CRITICAL on ganglia-test i-00000202 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:47:35] PROBLEM dpkg-check is now: CRITICAL on ganglia-test i-00000202 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:51:55] RECOVERY Total Processes is now: OK on ganglia-test i-00000202 output: PROCS OK: 94 processes [19:52:35] RECOVERY dpkg-check is now: OK on ganglia-test i-00000202 output: All packages OK [19:52:55] RECOVERY Disk Space is now: OK on login-test4 i-000001f4 output: DISK OK [19:53:45] RECOVERY Current Load is now: OK on ganglia-test i-00000202 output: OK - load average: 0.09, 0.35, 0.36 [19:54:25] RECOVERY Current Users is now: OK on ganglia-test i-00000202 output: USERS OK - 1 users currently logged in [19:55:05] RECOVERY Disk Space is now: OK on ganglia-test i-00000202 output: DISK OK [19:55:45] RECOVERY Free ram is now: OK on ganglia-test i-00000202 output: OK: 85% free memory [19:58:08] Ok, it didn't work. [19:59:08] I tried to send myself the password by email, but it didn't show up, not even in my spam folder. Obviously, RobH got assigned a wrong email address. [20:16:13] 04/12/2012 - 20:16:13 - Updating keys for yaron [20:22:31] robiH: what's your correct one? [20:22:45] and what's your shell account name? [20:30:20] !account-questions robiH [20:30:20] I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your preferred email address. 3. Your SVN account name, or your preferred shell account name, if you do not have SVN access. [20:30:25] !account-questions | robiH [20:30:25] robiH: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your preferred email address. 3. Your SVN account name, or your preferred shell account name, if you do not have SVN access. [20:30:34] robh is taken [20:41:54] robiH: sent your account info via email [20:42:33] err, now I did [20:43:07] got it. [20:46:04] How do I assign myself to a certain project? [20:47:02] you can't [20:47:08] you need to get a project member to assign you [20:47:39] Ok, I will contact DZahn. [20:47:49] ok [20:48:24] * Damianz assigns Ryan_Lane to /dev/null [21:16:08] dschoon: if you get the no-credentials bug in labsconsole, can you let me know? [21:16:17] I made some changes I hope will help [21:16:26] Ryan_Lane: will do [21:17:16] Ryan_Lane: Guess who's in town next week and available to chat about OpenStack conference planning [21:17:47] karen? [21:17:48] :) [21:18:45] Ryan_Lane: your least favorite volunteer development coordinator, of course [21:18:54] :D [21:18:54] (also possibly your most favorite, not sure how many you know) [21:19:09] I know about 3 [21:19:18] then perhaps I am in the middle [21:19:23] regardless [21:19:24] openstack's, canonical's, and ours [21:19:36] ;) [21:19:39] this sets up a Goldilocks scenario [21:19:43] but anyway [21:19:48] heh [21:19:55] is there a gameplan already? did I perhaps miss an email? [21:20:04] no gameplan yet [21:20:08] also, ok to put off talking about it if you are fighting a fire [21:20:10] we need some mech [21:20:12] *merch [21:20:15] mech! [21:20:18] ok, merch [21:20:27] mechs would be awesome too [21:20:28] launchpad sucks [21:20:33] about how many people you think would pass by the table? [21:20:47] like, 100 stickers + 30 badges would be easy to come by [21:20:51] we're sharing it with some others, so I'd imagine a decent amount [21:20:58] we should likely bring a bunch [21:21:16] we'll be able to know the first day if we've brought enough [21:21:48] Ryan_Lane: ok, can I ask you to ping James Alexander about it? [21:30:43] pinged him [21:31:35] Thanks Ryan_Lane [21:32:02] Ryan, my reflexes tell me that we'll need: flyers (which we can just print at the office), people, a bit of merch, and something to talk about [21:32:10] Ryan_Lane: sounds like we're set on the first three [21:32:18] Ryan_Lane: what are our talking points? what are we aiming to offer or get? [22:59:14] PROBLEM Puppet freshness is now: CRITICAL on nova-essex-test i-000001f9 output: Puppet has not run in last 20 hours [23:21:20] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5 output: Critical: 4% free memory [23:41:20] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5 output: Warning: 6% free memory [23:46:20] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5 output: Critical: 3% free memory