[00:23:44] PROBLEM dpkg-check is now: CRITICAL on testing-large i-000001fe output: Connection refused by host [00:25:04] PROBLEM Current Load is now: CRITICAL on testing-large i-000001fe output: Connection refused by host [00:25:44] PROBLEM Current Users is now: CRITICAL on testing-large i-000001fe output: Connection refused by host [00:26:19] PROBLEM Disk Space is now: CRITICAL on testing-large i-000001fe output: Connection refused by host [00:26:54] PROBLEM Free ram is now: CRITICAL on testing-large i-000001fe output: Connection refused by host [00:28:14] PROBLEM Total Processes is now: CRITICAL on testing-large i-000001fe output: Connection refused by host [00:53:44] PROBLEM Current Load is now: CRITICAL on testing-large1 i-00000200 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:53:44] PROBLEM Current Users is now: CRITICAL on testing-large2 i-00000201 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:54:24] PROBLEM Current Users is now: CRITICAL on testing-large1 i-00000200 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:54:24] PROBLEM Disk Space is now: CRITICAL on testing-large2 i-00000201 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:55:04] PROBLEM Disk Space is now: CRITICAL on testing-large1 i-00000200 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:55:04] PROBLEM Free ram is now: CRITICAL on testing-large2 i-00000201 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:55:44] PROBLEM Free ram is now: CRITICAL on testing-large1 i-00000200 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:56:14] PROBLEM Total Processes is now: CRITICAL on testing-large2 i-00000201 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:56:54] PROBLEM Total Processes is now: CRITICAL on testing-large1 i-00000200 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:56:59] PROBLEM dpkg-check is now: CRITICAL on testing-large2 i-00000201 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:57:34] PROBLEM dpkg-check is now: CRITICAL on testing-large1 i-00000200 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:58:14] PROBLEM Current Load is now: CRITICAL on testing-large2 i-00000201 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:38:45] RECOVERY Current Load is now: OK on login-test5 i-000001f5 output: OK - load average: 2.61, 0.94, 0.33 [02:39:25] RECOVERY Current Users is now: OK on login-test5 i-000001f5 output: USERS OK - 1 users currently logged in [02:40:45] RECOVERY Free ram is now: OK on login-test5 i-000001f5 output: OK: 88% free memory [02:41:25] RECOVERY Disk Space is now: OK on login-test5 i-000001f5 output: DISK OK [02:41:55] RECOVERY Total Processes is now: OK on login-test5 i-000001f5 output: PROCS OK: 82 processes [02:42:35] RECOVERY dpkg-check is now: OK on login-test5 i-000001f5 output: All packages OK [02:48:15] RECOVERY Total Processes is now: OK on login-test6 i-000001f6 output: PROCS OK: 94 processes [02:48:45] RECOVERY dpkg-check is now: OK on login-test6 i-000001f6 output: All packages OK [02:50:45] RECOVERY Current Users is now: OK on login-test6 i-000001f6 output: USERS OK - 0 users currently logged in [02:50:45] RECOVERY Current Load is now: OK on login-test6 i-000001f6 output: OK - load average: 0.01, 0.05, 0.01 [02:51:25] RECOVERY Disk Space is now: OK on login-test6 i-000001f6 output: DISK OK [02:51:55] RECOVERY Free ram is now: OK on login-test6 i-000001f6 output: OK: 93% free memory [02:55:45] RECOVERY Free ram is now: OK on login-test3 i-000001f3 output: OK: 63% free memory [02:55:45] RECOVERY Disk Space is now: OK on login-test3 i-000001f3 output: DISK OK [02:56:55] RECOVERY Total Processes is now: OK on login-test3 i-000001f3 output: PROCS OK: 79 processes [02:57:35] RECOVERY dpkg-check is now: OK on testing-large1 i-00000200 output: All packages OK [02:57:35] RECOVERY dpkg-check is now: OK on login-test3 i-000001f3 output: All packages OK [02:58:45] RECOVERY Current Load is now: OK on login-test3 i-000001f3 output: OK - load average: 0.00, 0.03, 0.01 [02:58:45] RECOVERY Current Load is now: OK on testing-large1 i-00000200 output: OK - load average: 0.04, 0.06, 0.01 [02:59:25] RECOVERY Current Users is now: OK on login-test3 i-000001f3 output: USERS OK - 0 users currently logged in [02:59:25] RECOVERY Current Users is now: OK on testing-large1 i-00000200 output: USERS OK - 0 users currently logged in [03:00:05] RECOVERY Disk Space is now: OK on testing-large1 i-00000200 output: DISK OK [03:00:35] RECOVERY Free ram is now: OK on testing-large1 i-00000200 output: OK: 95% free memory [03:01:55] RECOVERY Total Processes is now: OK on testing-large1 i-00000200 output: PROCS OK: 121 processes [03:02:37] RECOVERY dpkg-check is now: OK on login-test4 i-000001f4 output: All packages OK [03:03:45] RECOVERY Current Load is now: OK on login-test4 i-000001f4 output: OK - load average: 0.49, 0.33, 0.12 [03:04:25] RECOVERY Current Users is now: OK on login-test4 i-000001f4 output: USERS OK - 1 users currently logged in [03:05:45] RECOVERY Free ram is now: OK on login-test4 i-000001f4 output: OK: 92% free memory [03:05:45] RECOVERY Disk Space is now: OK on login-test4 i-000001f4 output: DISK OK [03:06:55] RECOVERY Total Processes is now: OK on login-test4 i-000001f4 output: PROCS OK: 107 processes [03:28:45] RECOVERY Current Users is now: OK on testing-large2 i-00000201 output: USERS OK - 0 users currently logged in [03:29:25] RECOVERY Disk Space is now: OK on testing-large2 i-00000201 output: DISK OK [03:30:05] RECOVERY Free ram is now: OK on testing-large2 i-00000201 output: OK: 96% free memory [03:31:15] RECOVERY Total Processes is now: OK on testing-large2 i-00000201 output: PROCS OK: 172 processes [03:31:55] RECOVERY dpkg-check is now: OK on testing-large2 i-00000201 output: All packages OK [03:33:08] RECOVERY Current Load is now: OK on testing-large2 i-00000201 output: OK - load average: 0.02, 0.07, 0.02 [03:41:08] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 12% free memory [03:46:18] RECOVERY Disk Space is now: OK on testing-large i-000001ff output: DISK OK [03:46:58] RECOVERY Free ram is now: OK on testing-large i-000001ff output: OK: 95% free memory [03:48:08] RECOVERY Total Processes is now: OK on testing-large i-000001ff output: PROCS OK: 120 processes [03:48:48] RECOVERY Current Load is now: OK on testing-large i-000001ff output: OK - load average: 0.05, 0.11, 0.08 [03:48:48] RECOVERY dpkg-check is now: OK on testing-large i-000001ff output: All packages OK [03:50:48] RECOVERY Current Users is now: OK on testing-large i-000001ff output: USERS OK - 0 users currently logged in [03:51:09] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 16% free memory [03:51:18] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 16% free memory [04:01:03] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:06:03] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 14% free memory [04:06:03] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 93% free memory [04:11:03] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 4% free memory [04:11:23] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 3% free memory [04:16:03] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 13% free memory [04:16:03] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [04:16:23] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory [04:21:03] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 5% free memory [04:21:03] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [04:26:03] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [05:51:12] PROBLEM Puppet freshness is now: CRITICAL on puppet-lucid i-00000080 output: Puppet has not run in last 20 hours [06:50:34] RECOVERY Disk Space is now: OK on deployment-transcoding i-00000105 output: DISK OK [07:06:24] RECOVERY Disk Space is now: OK on aggregator1 i-0000010c output: DISK OK [08:49:24] PROBLEM Disk Space is now: WARNING on aggregator1 i-0000010c output: DISK WARNING - free space: / 450 MB (4% inode=93%): [08:54:24] PROBLEM Disk Space is now: CRITICAL on aggregator1 i-0000010c output: DISK CRITICAL - free space: / 265 MB (2% inode=93%): [09:08:34] PROBLEM Disk Space is now: WARNING on deployment-transcoding i-00000105 output: DISK WARNING - free space: / 78 MB (5% inode=53%): [10:18:34] RECOVERY Disk Space is now: OK on deployment-transcoding i-00000105 output: DISK OK [10:31:34] PROBLEM Disk Space is now: WARNING on deployment-transcoding i-00000105 output: DISK WARNING - free space: / 72 MB (5% inode=53%): [11:39:19] PROBLEM Disk Space is now: WARNING on login-test4 i-000001f4 output: DISK WARNING - free space: / 350 MB (3% inode=93%): [11:44:19] PROBLEM Disk Space is now: CRITICAL on login-test4 i-000001f4 output: DISK CRITICAL - free space: / 117 MB (1% inode=93%): [12:15:29] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5 output: Critical: 4% free memory [12:20:29] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5 output: Warning: 10% free memory [12:55:29] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5 output: Critical: 4% free memory [14:35:29] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5 output: Warning: 15% free memory [14:54:19] werdna: hi [14:54:39] how is it possible that http://en.wikipedia.beta.wmflabs.org/w/extensions/Vector/modules/ext.vector.footerCleanup.css return 404 and not on production? [14:54:55] how is actually het pointing apache to correct place [15:20:33] ^demon: how can I request extensions to be moved to git [15:22:03] <^demon> I'll post a timeline for the next migration window sometime this week. [15:22:08] ok [15:22:31] so I need to use svn right now? [15:22:41] <^demon> Is it a new extension or an existing one? [15:22:47] 3 existing [15:23:02] I guess svn :) [15:23:03] <^demon> Keep using svn if they're already in svn. [15:23:06] ok [15:23:13] <^demon> I just don't want *new* extensions in svn :) [15:23:51] <^demon> It takes 2 minutes to set someone's repo up in git and give them permissions for a new extension. It takes significantly longer to migrate an extension from svn -> git :) [15:24:04] right [15:52:09] PROBLEM Puppet freshness is now: CRITICAL on puppet-lucid i-00000080 output: Puppet has not run in last 20 hours [16:13:51] I'm trying to work on puppet for labs, but can't add my class on Nova [16:14:09] It says I need to be in the Administrators group [16:14:12] :( [16:18:45] hexmode: which project and class? [16:19:19] I have to commit the class, but would "bugzilla" include everything under bugzilla.pp? [16:19:35] mutante: if they all start with bugzilla:: [16:19:50] http://10.4.0.133/source/xref/mediawiki/core/api.php?a=true&h= [16:19:52] He-he [16:26:33] hexmode: there are at least 2 ways to do that. the easiest is to apply all the classes in "Configure instance", or you use puppet include or require or parameterized classes. there is no magic include because they start with the same name" though. [16:26:58] but it is better to ask Ryan about this as well.. for best practive [16:27:28] * hexmode waits for Ryan [16:28:40] how many do you have in bugzilla:: [16:29:00] * hexmode looks [16:30:07] i'd say just put it in gerrit or did you [16:30:26] ::server ::config ::crons ::database-server [16:30:28] will do [16:31:23] do you want me to add a class to the project "re: need to be in admin group"? [16:31:52] or be an admin on your project [16:32:12] mutante: yeah, think so, but I should be an admin on my project, shouldn't I? [16:32:34] what's the project name [16:32:45] deployment-prep [16:33:38] * mutante uses new project-filter .. [16:34:05] yes, that filter is awesom [16:34:07] e [16:34:09] sorry, got phone 1:1 , bbiaw [16:34:16] np [16:35:20] you are sysadmin and netadmin in your project [16:37:56] yea, you want new classes in Special:NovaPuppetGroup and thats what you need admin for. suggest a category. bbl [16:38:13] eh "group" i mean [16:41:47] mutante: was able to add a group last night (bz-dev) and just added another (bugzilla) but now I'm getting permission denied for removing them [16:42:23] Also perm denied for adding a class or variable under the group [17:27:07] 04/11/2012 - 17:27:07 - Creating a home directory for faidon at /export/home/puppet/faidon [17:27:51] yayy [17:28:08] 04/11/2012 - 17:28:07 - Updating keys for faidon [17:30:45] paravoid: so you are faidon? i'm daniel. welcome! reminded me of "paradroid" the awesome retro game [17:30:59] bbl [17:35:00] mutante: yes I am, thanks :-) [17:40:36] dschoon: faidon fixed your issue with creating instances [17:40:41] they should just build now [17:40:44] sweet! [17:40:47] <3 [17:41:04] I built two large and one xlarge without errors [17:48:29] PROBLEM host: testing-large2 is DOWN address: i-00000201 check_ping: Invalid hostname/address - i-00000201 [17:48:39] PROBLEM host: testing-large1 is DOWN address: i-00000200 check_ping: Invalid hostname/address - i-00000200 [17:49:39] PROBLEM host: testing-large is DOWN address: i-000001ff check_ping: Invalid hostname/address - i-000001ff [18:32:33] Ryan_Lane: Faidon fixed the checksum failure for large instances? [18:44:38] well, wouldn't say "fixed" [18:44:41] we worked around it [18:44:57] So no explanation as to why it was happening? [18:48:03] andrewbogott: yep [18:48:07] heh [18:48:11] right. workaround [18:48:32] it had to do with a bug in squid and parallelization of apt-get [18:48:34] That's great! But doesn't satisfy my curiosity :( [18:49:08] squid was returning 206 codes (and truncated files) for some debian files [18:50:33] We use squid in front of our ppa? [18:53:16] btw, ryan_lane, did my email about summit sessions make sense? And did I delegate too much talking to you? I realized as I was writing that you wound up with the lion's share. [18:53:28] Which works for me, but may not for you [18:54:07] yep, was about to finish reponding to that [18:54:59] ok [19:00:07] 04/11/2012 - 19:00:06 - Updating keys for faidon [19:00:13] 04/11/2012 - 19:00:12 - Updating keys for faidon [20:00:35] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5 output: Critical: 4% free memory [20:05:34] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5 output: Warning: 8% free memory [20:51:14] PROBLEM Puppet freshness is now: CRITICAL on kirke i-000001fd output: Puppet has not run in last 20 hours [20:51:14] PROBLEM Puppet freshness is now: CRITICAL on kripke i-000001fc output: Puppet has not run in last 20 hours [21:47:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-web4 i-00000163 output: Puppet has not run in last 20 hours [21:50:34] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5 output: Critical: 5% free memory [21:55:34] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5 output: Warning: 7% free memory [22:20:36] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5 output: Critical: 5% free memory [23:03:01] New patchset: Andrew Bogott; "Added a couple of essex-specific flags." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4762 [23:03:14] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4762 [23:03:18] New review: Andrew Bogott; "Right on! You are the best." [operations/puppet] (test); V: 0 C: 0; - https://gerrit.wikimedia.org/r/4762 [23:03:36] New review: Andrew Bogott; "Right on! You are the best." [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4762 [23:03:38] Change merged: Andrew Bogott; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4762 [23:15:41] New patchset: Andrew Bogott; "I needed 'sudo nova-rootwrap', not just 'nova-rootwrap'." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4764 [23:15:54] New review: Andrew Bogott; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4764 [23:15:54] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4764 [23:15:56] Change merged: Andrew Bogott; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4764 [23:36:41] New patchset: Andrew Bogott; "/usr/bin/nova-rootwrap, that is." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4765 [23:36:53] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/4765 [23:36:58] New review: Andrew Bogott; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4765 [23:37:00] Change merged: Andrew Bogott; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/4765