[00:26:44] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [00:34:36] RECOVERY Free ram is now: OK on deployment-web2 i-00000125 output: OK: 20% free memory [00:58:46] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [01:16:16] RECOVERY Free ram is now: OK on deployment-web6 i-000001d9 output: OK: 20% free memory [01:28:46] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [01:29:16] PROBLEM Free ram is now: WARNING on deployment-web6 i-000001d9 output: Warning: 19% free memory [01:58:46] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [02:28:46] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [02:39:19] RECOVERY Free ram is now: OK on deployment-web6 i-000001d9 output: OK: 23% free memory [02:58:54] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [02:58:55] RECOVERY Puppet freshness is now: OK on mobile-feeds i-000000c1 output: puppet ran at Sun Apr 22 02:58:45 UTC 2012 [03:02:24] PROBLEM Free ram is now: WARNING on deployment-web6 i-000001d9 output: Warning: 19% free memory [03:26:06] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-2 i-0000020a output: Puppet has not run in last 20 hours [03:29:16] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [03:29:42] PROBLEM Free ram is now: CRITICAL on mobile-feeds i-000000c1 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:34:37] PROBLEM Free ram is now: WARNING on mobile-feeds i-000000c1 output: Warning: 7% free memory [03:37:36] PROBLEM Free ram is now: WARNING on deployment-web2 i-00000125 output: Warning: 19% free memory [03:43:55] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 12% free memory [03:43:55] PROBLEM Free ram is now: WARNING on deployment-web4 i-00000163 output: Warning: 19% free memory [03:47:56] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 14% free memory [03:49:02] RECOVERY Free ram is now: OK on deployment-web4 i-00000163 output: OK: 23% free memory [03:49:28] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 14% free memory [03:59:16] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [04:03:21] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 5% free memory [04:05:05] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:05:37] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:08:07] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:09:07] PROBLEM Current Load is now: WARNING on bots-sql3 i-000000b4 output: WARNING - load average: 6.83, 8.05, 6.00 [04:10:07] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 17% free memory [04:10:07] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory [04:11:07] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 0.45, 5.69, 5.42 [04:13:47] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 92% free memory [04:16:07] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.38, 2.34, 4.03 [04:19:07] RECOVERY Current Load is now: OK on bots-sql3 i-000000b4 output: OK - load average: 2.48, 3.85, 4.64 [04:30:07] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 5% free memory [04:30:07] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [04:40:07] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 96% free memory [04:43:47] PROBLEM Free ram is now: WARNING on deployment-web4 i-00000163 output: Warning: 19% free memory [05:00:07] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [05:30:07] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [05:42:38] RECOVERY Free ram is now: OK on deployment-web2 i-00000125 output: OK: 20% free memory [06:00:11] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [06:00:38] PROBLEM Free ram is now: WARNING on deployment-web2 i-00000125 output: Warning: 19% free memory [06:30:14] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [06:45:47] RECOVERY Free ram is now: OK on mobile-feeds i-000000c1 output: OK: 80% free memory [06:48:39] PROBLEM Current Load is now: WARNING on mobile-feeds i-000000c1 output: WARNING - load average: 0.81, 7.61, 5.07 [06:53:06] RECOVERY Current Load is now: OK on mobile-feeds i-000000c1 output: OK - load average: 0.07, 2.85, 3.70 [07:00:42] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [07:30:51] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [08:01:01] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [08:31:01] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [09:00:11] PROBLEM Puppet freshness is now: CRITICAL on nova-production1 i-0000007b output: Puppet has not run in last 20 hours [09:01:01] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [09:04:11] PROBLEM Puppet freshness is now: CRITICAL on nova-gsoc1 i-000001de output: Puppet has not run in last 20 hours [09:31:01] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [10:01:01] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [10:31:06] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [10:56:54] hey Ryan_Lane [10:57:08] should I recreate all web servers with 8gb? [10:57:10] or only some [11:01:06] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [11:31:07] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [11:45:03] petan|wk: well, how many are there? [11:45:06] i'd say all of them [11:45:15] if we have too many, we can remove some later [11:45:22] they have way too little ram right now, though [12:01:10] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [12:15:27] New patchset: Dzahn; "use the project- labs groups to allow puppet-created user to execute cron jobs" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5555 [12:15:41] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/5555 [12:17:35] New review: Dzahn; "without it my cron jobs are stuck" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/5555 [12:17:38] Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5555 [12:31:10] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [13:01:18] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [13:22:39] mutante, if you can take a look to grail instance when you have time... [13:26:08] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-2 i-0000020a output: Puppet has not run in last 20 hours [13:26:30] !log gareth adding myself to be able to reboot grail instance [13:26:32] Logged the message, Master [13:26:38] 04/22/2012 - 13:26:37 - Creating a home directory for dzahn at /export/home/gareth/dzahn [13:27:16] were you able to log in? [13:27:31] I had already rebooted it yesterday, but didn't help [13:27:39] 04/22/2012 - 13:27:38 - Updating keys for dzahn [13:27:47] <-- need keys first.. hold on [13:28:06] need keys? [13:28:34] the instance needs my keys or i cant login anyways [13:29:00] is some special step needed to give the keys to the instance? [13:29:09] I thought they fetched them automatically from ldap [13:29:15] that one i just did, added myself to project [13:29:15] (once you were in the project) [13:29:40] well, grail does not let me login [13:29:47] doesn't let me either [13:29:54] it rejects the public key [13:29:59] same here [13:30:08] also, the dhcp looks wrong [13:30:14] see the frequency at the console log [13:30:25] it's renewing the lease every 50 seconds or so [13:30:36] and this was an issue that happened before and was fixed by restart? [13:30:55] no [13:31:13] no action was done there after creating the instance [13:31:18] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [13:31:28] is it possible that it got broken by the security rules? [13:31:29] but you dont mind if i restart it , right [13:31:36] of course I don't [13:32:02] /me uses "get console output" on wiki [13:32:47] oh, i see the DHCP issues, yep [13:32:59] but eventually it bound to an IP in the end [13:33:05] yes, it gets bound [13:33:16] and we are able to connect with sshd [13:33:23] yea [13:33:45] look at the security group [13:33:53] that's the only piece I could have done wrong [13:33:56] security rules would also let us connect ..or not.. but not refuse the key [13:34:02] looking [13:34:22] I thought that they might be blocking some check to the ldap, or something similar [13:34:33] if it's ok, I'm out of ideas, then [13:34:38] true, that could be [13:35:07] I don't know what is needed for that, though [13:35:21] where is the ldap server? [13:35:45] afaik sanger [13:35:50] which had recent issues :p [13:36:00] but i can use my instance just fine.. so [13:36:20] i suggest: reboot without changes,, then reboot after removing security group and compare [13:36:36] I thought you can't remove a security group from an instance? [13:36:56] users in "netadmin" should be able to [13:37:05] but also regarding this there are/were changes [13:37:17] i'll try [13:37:48] ok, try [13:37:52] !log gareth rebooting grail [13:37:53] Logged the message, Master [13:39:26] puppet-agent[700]: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Package[apache2] is already defined in file /etc/puppet/manifests/webserver.pp at line 69; cannot redefine at /etc/puppet/manifests/webserver.pp:29 [13:39:29] note how console output now looks normal and good [13:39:30] could this be the cause? [13:39:35] but it still does not let us log in [13:39:39] the DHCP stuff is gone though [13:39:58] it's there again [13:40:43] yea, if puppet run is broken... and that is supposed to add the keys [13:40:50] it explains i cant login [13:41:44] !log gareth removing webserver classes from grail instance temp. puppet breakage [13:41:45] Logged the message, Master [13:42:09] Platonides: where did you just get that output ? [13:42:21] it's in the console log [13:42:35] just at the right of "i-00000210 login:" [13:42:44] ah, i see [13:43:03] can we also get the output of the next run without rebooting again and no login? [13:43:12] i was wondering [13:43:31] I don't follow you [13:43:39] when you reboot, the console output is cleared [13:43:46] well, we'll have to restart again to remove the classes from the instance [13:43:49] you can however copy & paste the old one before rebooting [13:44:48] Hey all. I had previously bookmarked http://orgcharts.wmflabs.org/, but now it times out. [13:45:37] Platonides: i meant the output of puppet runs..after the first one at boot, don't think you can get them when not being able to login, but we need to restart instance anyways after changing the config which classes it is supposed to use [13:46:15] I still can't login [13:46:17] 2012-04-22 13:45:13,261 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: socket timeout [timed out] [13:46:25] that might be interesting? [13:46:53] seems a zeroconf address [13:46:59] perhaps it's expected :/ [13:47:37] hmm, but right after that it says "found data source" anyways [13:48:14] automount[2404]: syntax error in nsswitch config near [ syntax error ] [13:49:36] did it work before with a specific set of puppet classes until classes were added? [13:49:47] this instance never worked [13:49:48] or was it just like "create instance, apply classes" and did not work since then [13:50:08] ok, then i guess there never was a successful puppet run on it [13:50:09] I created the instance trying to set the right classes [13:50:15] probably not [13:50:16] which is needed to add our keys and users [13:50:48] I don't know why did the webserver classes conflict [13:51:06] I just wanted to give it a basic lamp config :s [13:51:28] it's probably better to create the instance without addding classes, [13:51:35] starting it up, and logging once [13:51:41] trying [13:52:53] "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED" hehe [13:53:34] now it refuses the connection :S [13:54:48] still rejecting the private key [13:54:50] give me a minute.. looking for docs [13:55:10] I'm going now [13:55:15] will be back in a few hours [13:55:26] play freely with it [13:59:00] ok, i am a little, but also going soon again.. sunday after all.. cya [14:01:23] PROBLEM host: grail is DOWN address: i-00000210 PING CRITICAL - Packet loss = 100% [14:19:20] !log gareth deleted non-default security group and instance, created new instance with same name, settings and default security group [14:19:21] Logged the message, Master [14:21:23] RECOVERY host: grail is UP address: i-00000212 PING OK - Packet loss = 0%, RTA = 0.69 ms [14:22:53] PROBLEM Current Load is now: CRITICAL on grail i-00000212 output: Connection refused by host [14:22:53] PROBLEM Current Users is now: CRITICAL on grail i-00000212 output: Connection refused by host [14:24:13] PROBLEM Free ram is now: CRITICAL on grail i-00000212 output: Connection refused by host [14:29:13] PROBLEM Total Processes is now: CRITICAL on grail i-00000212 output: Connection refused by host [14:29:13] PROBLEM dpkg-check is now: CRITICAL on grail i-00000212 output: Connection refused by host [14:29:13] PROBLEM Disk Space is now: CRITICAL on grail i-00000212 output: Connection refused by host [14:29:37] How can I request an account on wikilabs? [14:31:34] !accountreq [14:31:34] in case you want to have an account on labs, please contact someone who is in charge of doing that: Ryan.Lane, m.utante or ssmolle.tt [14:32:50] Shujen: https://labsconsole.wikimedia.org/wiki/Help:Access#Access_FAQ [14:34:13] !accountreq is case you want to have an account on labs please read here: https://labsconsole.wikimedia.org/wiki/Help:Access#Access_FAQ [14:34:13] Key exist! [14:34:23] !del accountreq [14:34:33] !accountreq is case you want to have an account on labs please read here: https://labsconsole.wikimedia.org/wiki/Help:Access#Access_FAQ [14:34:34] Key exist! [14:35:02] thx [14:36:33] you're welcome, via wiki request is the best way [14:37:50] requested [14:37:52] http://www.mediawiki.org/wiki/Developer_access#User:Shujenchang [14:38:25] Is there any admin? [14:40:44] !accountreq del [14:40:44] Successfully removed accountreq [14:40:59] !accountreq is case you want to have an account on labs please read here: https://labsconsole.wikimedia.org/wiki/Help:Access#Access_FAQ [14:40:59] Key was added! [14:44:10] PROBLEM Free ram is now: CRITICAL on deployment-web5 i-00000213 output: Connection refused by host [14:44:10] PROBLEM dpkg-check is now: CRITICAL on deployment-web5 i-00000213 output: Connection refused by host [14:44:50] PROBLEM Total Processes is now: CRITICAL on deployment-web5 i-00000213 output: Connection refused by host [14:45:42] PROBLEM Current Users is now: CRITICAL on deployment-web5 i-00000213 output: Connection refused by host [14:45:42] PROBLEM Current Load is now: CRITICAL on deployment-web5 i-00000213 output: Connection refused by host [14:45:42] PROBLEM Disk Space is now: CRITICAL on deployment-web5 i-00000213 output: Connection refused by host [14:46:40] ACKNOWLEDGEMENT Current Load is now: CRITICAL on grail i-00000212 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:47:10] ACKNOWLEDGEMENT Free ram is now: CRITICAL on grail i-00000212 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:47:25] ACKNOWLEDGEMENT Current Users is now: CRITICAL on grail i-00000212 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:47:25] ACKNOWLEDGEMENT Disk Space is now: CRITICAL on grail i-00000212 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:47:30] ACKNOWLEDGEMENT Total Processes is now: CRITICAL on grail i-00000212 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:47:40] ACKNOWLEDGEMENT dpkg-check is now: CRITICAL on grail i-00000212 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:48:58] !bots [14:48:59] http://www.mediawiki.org/wiki/Wikimedia_Labs/Create_a_bot_running_infrastructure proposal for bots [14:50:00] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 2.86, 11.94, 9.44 [14:51:37] 04/22/2012 - 14:51:36 - Creating a home directory for petrb at /export/home/gareth/petrb [14:52:36] 04/22/2012 - 14:52:36 - Updating keys for petrb [14:53:43] PROBLEM Current Load is now: CRITICAL on deployment-web4 i-00000214 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:54:10] !log gareth - added petrb to project netadmin to fix nagios for the instance [14:54:12] Logged the message, Master [14:54:23] PROBLEM Current Users is now: CRITICAL on deployment-web4 i-00000214 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:55:13] PROBLEM Disk Space is now: CRITICAL on deployment-web4 i-00000214 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:55:43] PROBLEM Free ram is now: CRITICAL on deployment-web4 i-00000214 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:56:53] PROBLEM Total Processes is now: CRITICAL on deployment-web4 i-00000214 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:57:33] PROBLEM dpkg-check is now: CRITICAL on deployment-web4 i-00000214 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:04:53] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.29, 0.91, 3.82 [15:08:02] ok whoever is owner of instance grail it should work now [17:41:07] RECOVERY Free ram is now: OK on deployment-web2 i-00000125 output: OK: 20% free memory [17:49:07] PROBLEM Free ram is now: WARNING on deployment-web2 i-00000125 output: Warning: 18% free memory [18:09:17] RECOVERY Disk Space is now: OK on grail i-00000215 output: DISK OK [18:09:17] RECOVERY dpkg-check is now: OK on grail i-00000215 output: All packages OK [18:09:17] RECOVERY Total Processes is now: OK on grail i-00000215 output: PROCS OK: 87 processes [18:09:27] RECOVERY Free ram is now: OK on grail i-00000215 output: OK: 92% free memory [18:11:57] RECOVERY Total Processes is now: OK on deployment-web4 i-00000214 output: PROCS OK: 146 processes [18:12:27] RECOVERY dpkg-check is now: OK on deployment-web4 i-00000214 output: All packages OK [18:12:41] mutante, petan, what if I wanted it to be in a non.default security group? [18:13:47] RECOVERY Current Load is now: OK on deployment-web4 i-00000214 output: OK - load average: 0.33, 0.28, 0.11 [18:14:27] RECOVERY Current Users is now: OK on deployment-web4 i-00000214 output: USERS OK - 0 users currently logged in [18:15:37] RECOVERY Free ram is now: OK on deployment-web4 i-00000214 output: OK: 96% free memory [18:15:47] RECOVERY Disk Space is now: OK on deployment-web4 i-00000214 output: DISK OK [18:16:17] PROBLEM host: grail is DOWN address: i-00000215 check_ping: Invalid hostname/address - i-00000215 [19:01:15] PROBLEM Puppet freshness is now: CRITICAL on nova-production1 i-0000007b output: Puppet has not run in last 20 hours [19:05:05] PROBLEM Puppet freshness is now: CRITICAL on nova-gsoc1 i-000001de output: Puppet has not run in last 20 hours [19:28:55] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 9.58, 11.89, 5.55 [19:33:55] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.20, 4.56, 4.11 [19:44:23] PROBLEM HTTP is now: CRITICAL on deployment-web5 i-00000213 output: Connection refused [21:46:07] 04/22/2012 - 21:46:06 - Creating a home directory for vvv at /export/home/openstack/vvv [21:46:58] bind mounts! [21:47:07] 04/22/2012 - 21:47:07 - Updating keys for vvv [21:48:25] SpComb: ? [21:48:44] * SpComb set up his NFS server today [21:48:56] where? [21:48:59] in Labs? [21:49:01] nah [21:49:05] ah. heh [21:49:15] we're moving away from NFS everywhere possible [21:49:22] that "create home directory in /exports/home/..." sounds like you have your home fs mounted directly in /exports [21:49:25] oh :( [21:50:08] first thing I ran into once I rebooted the server with an NFS-mounted /home was that all files became owned by nobody:nogroup until some random kernel idmapd timeout expired [21:50:31] we're using automounts for home directories [21:50:36] in a creative way ;) [21:50:39] NFSv4 automounts? [21:50:43] no [21:50:44] 3 [21:50:49] hm [21:50:52] nfs4 kind of sucks [21:50:58] http://ryandlane.com/blog/2011/11/01/sharing-home-directories-to-instances-within-a-project-using-puppet-ldap-autofs-and-nova/ [21:51:00] I was noting the same [21:51:18] nfs4 is great, except that support for it still isn't amazing [21:53:23] I just configured my /etc/exports and client mounts using Puppet, only have an O(1) number of exports/mounts here :) [21:54:07] some fine `exec { '/etc/exports': command => 'cat /etc/exports.d/* > /etc/exports', notify => Service['nfs-server'] }` magics and it's all nice :) [21:55:22] + refreshonly and a `define nfs::export` [22:00:45] hmm, you sure seem to have a lot of configuration stored in LDAP [22:02:19] yep [22:02:28] I have per-project sudo as well [22:02:58] you are using exported resources? [22:03:29] no, heard way too much bad stuff about the SQL storage stuff [22:03:51] yep [22:04:02] Ryan_Lane, how should I make a security group for an instance [22:04:06] but I really only have a handful of servers to manage, 3-6 or so [22:04:12] Platonides: you need to make it before hand [22:04:12] so that it allows me to log in [22:04:13] ? [22:04:22] in my case, Puppet is more about managing changes to the servers over their lifetime [22:04:24] wait. what do you mean? [22:04:27] Platonides: via ssh? [22:04:45] that rule is set up by default [22:04:45] yes, the public key was rejected [22:04:45] which instance are you having issues with? [22:04:55] SpComb: yeah. my use case wouldn't let me do that, though [22:05:15] SpComb: every project needs another export, and projects can be created at will [22:05:16] it started working after !log gareth deleted non-default security group and instance, created new instance with same name, settings and default security group [22:05:45] did someone create the instance without default selected? [22:05:53] default should always be selected [22:06:00] Ryan_Lane: yeah... I just have a single monolitic puppet repo that contains everything, using just parametrized classes [22:06:04] unless the person *really* knows what they are doing [22:06:14] data in LDAP is strictly separate from configuration in Puppet [22:06:17] SpComb: yeah. it's a sane way of handling things if you can do it that way [22:06:40] it's been pretty fun working on it so far :) [22:06:41] Platonides: gimme a sec. I'll look at it for you [22:07:04] being able to plop down replicating LDAP slaves on hosts at will in under a minute is awesome :P [22:07:37] 04/22/2012 - 22:07:37 - Creating a home directory for laner at /export/home/gareth/laner [22:08:24] Platonides: there's no instances in gareth [22:08:31] yep, I deleted it [22:08:36] 04/22/2012 - 22:08:36 - Updating keys for laner [22:08:48] I can't debug something that isn't there... [22:08:48] so I wanted to create it again with a non-default SW [22:08:51] but working [22:08:59] ah [22:09:03] does it need some special rule? [22:09:07] make a new security group [22:09:21] ensure the new security group and the default one are selected [22:09:39] if you remove default you'll disable ping and ssh (and nagios) [22:09:57] I had a rule of 22 to 0.0.0.0/0 [22:10:05] we could reach the sshd [22:10:11] the problem is, it rejected our keysd [22:10:47] well, create the instance and let me know if it has the problem, then I can see why [22:11:50] hm. load is spiking on labs. I wonder why [22:11:54] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 3.96, 6.68, 3.66 [22:12:11] bots and deployment prep are spiking the most [22:12:18] http://ganglia.wmflabs.org/latest/ <— per-project ganglia ;) [22:12:29] I guess sara hasn't sent that out to the list yet [22:12:53] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 3.37, 12.70, 7.60 [22:13:27] I wonder if I had the instance in the default security group [22:14:00] also, which classes should I add so it provides a basic LAMP setup? [22:14:28] using webserver classes caused conflicts [22:14:38] hm. [22:14:43] it's not terribly easy right now [22:14:53] our apache puppet config is so beyond screwed up [22:15:08] we need puppet repo documentation [22:15:32] lemme see [22:16:53] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.30, 2.72, 2.77 [22:17:41] there's no all-in-one class for a full lamp stack [22:17:59] webserver::php5 will install apache and php [22:18:12] maybe that was the source of the conflict [22:18:20] webserver::php5-mysql will give you mysql [22:18:21] well [22:18:24] php5-mysql [22:18:33] again, our classes aren't great for this [22:18:37] I had chosen both webserver::apache2 and webserver::php5 [22:18:44] ah. that'll do it [22:18:44] since I wanted apache and php [22:18:55] we have a new class that makes a lot more sense, but it's not easy to use [22:19:10] if webserver::php5 already provides apache, that could be the source of the conflict [22:19:12] and we are constantly making changes to it [22:19:16] it is [22:19:51] why wouldn't webserver::php5 just depend on webserver::apache2, so an extra webserver::apache2 didn't break it? [22:20:09] maybe that's not even possible [22:20:13] it's far from intuitive :P [22:20:17] agreed [22:20:24] which is the point of the saner new class :) [22:21:52] btw, are the -----BEGIN SSH HOST KEY FINGERPRINTS----- output by a script made by us? [22:22:03] it'd be nice to also have the ECDSA fingerprint there [22:22:12] where do you see this? [22:22:19] in the console output [22:22:27] oh [22:22:37] no. that's what happens when the instance generates them [22:22:46] the ECDSA fingerprint is above in the noise, so not very important [22:22:50] just a nice-to-have [22:22:54] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.36, 1.94, 4.08 [22:23:10] I was planning on pulling the fingerprint in when the instance builds, so that it would be available in the web interface [22:23:30] well, grail is now in that broken mode :) [22:23:37] grail? [22:23:38] oh [22:23:44] PROBLEM Current Load is now: CRITICAL on grail i-00000216 output: Connection refused by host [22:23:55] gimme a sec [22:24:03] not actually refused, since I can connect to the host sshd [22:24:14] as confirmed by the fingerprint [22:24:24] PROBLEM Current Users is now: CRITICAL on grail i-00000216 output: Connection refused by host [22:25:04] PROBLEM Disk Space is now: CRITICAL on grail i-00000216 output: Connection refused by host [22:25:28] the puppet run isn't finished yet [22:25:39] you can't log in until puppet is totally finished running. [22:25:43] give it a couple more mins [22:25:44] PROBLEM Free ram is now: CRITICAL on grail i-00000216 output: Connection refused by host [22:25:45] brb [22:26:54] PROBLEM Total Processes is now: CRITICAL on grail i-00000216 output: Connection refused by host [22:27:34] PROBLEM dpkg-check is now: CRITICAL on grail i-00000216 output: Connection refused by host [22:28:52] are you sure? [22:28:54] Finished puppet run [22:29:39] hmm.. Sub-process /usr/bin/dpkg returned an error code (1) [22:30:37] * Ryan_Lane groans [22:30:37] Duplicate definition: Sshkey[10.4.0.2] is already defined [22:31:01] seems it did run after that, though [22:31:39] that definition is not my fault :P [22:31:55] wait [22:32:06] did you build an oneiric instance? [22:32:11] or lucid? [22:32:31] oneiric [22:32:41] is that broken? [22:32:50] yes [22:32:55] sigh [22:33:07] why would you use anything other than an LTS anyway? :) [22:33:18] it's newer? ;) [22:33:34] oneiric wasn't broken, but the per-project ganglia stuff likely put the final nail in the coffin for that version [22:33:36] you should remove those options [22:33:53] ok, I'll try with lucid tomorrow [22:33:56] thanks [22:33:58] yw [23:08:44] RECOVERY Current Load is now: OK on grail i-00000216 output: OK - load average: 0.23, 0.42, 0.34 [23:09:24] RECOVERY Current Users is now: OK on grail i-00000216 output: USERS OK - 0 users currently logged in [23:10:04] RECOVERY Disk Space is now: OK on grail i-00000216 output: DISK OK [23:10:44] RECOVERY Free ram is now: OK on grail i-00000216 output: OK: 91% free memory [23:11:54] RECOVERY Total Processes is now: OK on grail i-00000216 output: PROCS OK: 69 processes [23:27:10] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-2 i-0000020a output: Puppet has not run in last 20 hours [23:57:29] right channel [23:57:57] Ryan_Lane: clicking "add group" under deployment-prep [23:58:15] ok, adding just "bz" [23:58:27] created [23:58:44] back to puppet group list [23:59:10] clicking "delete" for "bz" group I just created [23:59:23] Ryan_Lane: "The action you have requested is limited to users in the group: Administrators."