[00:19:56] 07/16/2012 - 00:19:56 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [01:51:57] 07/16/2012 - 01:51:57 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [02:38:56] 07/16/2012 - 02:38:56 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [03:40:57] 07/16/2012 - 03:40:57 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [04:42:55] 07/16/2012 - 04:42:55 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [05:44:54] 07/16/2012 - 05:44:54 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [06:46:57] 07/16/2012 - 06:46:57 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [07:49:05] 07/16/2012 - 07:49:04 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [08:44:05] Thehelpfulone around? [08:44:28] hashar what's the status of beta, does it work now? [08:44:49] I am still kind of recovering from vacation [08:44:57] petan: enjoy recovering :-] [08:45:00] :D [08:45:05] beta should be working more or less [08:45:16] I have replaced the corrupted Apache instances [08:45:32] ok [08:45:32] and did various other stuff [08:45:41] udp2log is still dead though [08:45:46] :/ [08:45:50] irc as well [08:50:24] Ryan_Lane hey, are you back in USA? or you still in GMT+1 [08:50:50] if I won't get reply within 10 minutes I will assume you are not :) [08:50:56] 07/16/2012 - 08:50:56 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [08:51:15] petan: I am almost sure Ryan is at Wikimania [08:51:21] aha [08:52:12] I think he flew back last week or so [08:52:23] ok, we need to fix nagios I think [08:52:29] where is the bot [08:52:30] petan: I did some documentation for beta : https://labsconsole.wikimedia.org/wiki/Deployment/Overview [08:52:35] yay [08:52:36] still need ton of additions though :-] [08:52:45] bot died a little while back [08:52:49] it's a good beginning though [08:52:56] aha [08:53:24] * Damianz gives hashar a cookie [08:59:05] it can't get to nrpe [08:59:32] PROBLEM host: deployment-cache-bits is DOWN address: i-00000264 check_ping: Invalid hostname/address - i-00000264 [09:00:02] PROBLEM host: pybal-precise is DOWN address: i-00000289 check_ping: Invalid hostname/address - i-00000289 [09:01:02] PROBLEM Disk Space is now: WARNING on nagios 127.0.0.1 output: DISK WARNING - free space: /public/keys 2304 MB (13% inode=70%): /home/petrb 2304 MB (13% inode=70%): [09:04:02] PROBLEM host: testing-virt6 is DOWN address: i-00000302 check_ping: Invalid hostname/address - i-00000302 [09:04:02] PROBLEM host: testing-virt7 is DOWN address: i-00000308 check_ping: Invalid hostname/address - i-00000308 [09:14:06] PROBLEM host: deployment-backup is DOWN address: i-000000f8 CRITICAL - Host Unreachable (i-000000f8) [09:18:18] PROBLEM Current Load is now: UNKNOWN on cvn-apache2 i-00000339 output: (No output returned from plugin) [09:18:18] PROBLEM Current Load is now: UNKNOWN on deployment-apache32 i-0000031a output: (No output returned from plugin) [09:18:18] PROBLEM Current Load is now: UNKNOWN on deployment-apache33 i-0000031b output: (No output returned from plugin) [09:18:18] PROBLEM Current Load is now: UNKNOWN on deployment-cache-bits02 i-0000031c output: (No output returned from plugin) [09:18:18] PROBLEM Current Users is now: UNKNOWN on deployment-cache-upload01 i-00000326 output: (No output returned from plugin) [09:18:19] PROBLEM Disk Space is now: UNKNOWN on deployment-jobrunner06 i-0000031d output: (No output returned from plugin) [09:18:19] PROBLEM Free ram is now: UNKNOWN on extension-review-testing i-0000033d output: (No output returned from plugin) [09:18:20] PROBLEM Free ram is now: UNKNOWN on gerrit-build i-00000335 output: (No output returned from plugin) [09:18:20] PROBLEM Total Processes is now: UNKNOWN on jesusaurus-cleanup i-00000337 output: (No output returned from plugin) [09:18:23] PROBLEM dpkg-check is now: UNKNOWN on pdbhandler-1 i-0000030e output: (No output returned from plugin) [09:18:23] PROBLEM Current Load is now: UNKNOWN on puppet-demo2 i-0000032b output: (No output returned from plugin) [09:18:24] PROBLEM Current Load is now: UNKNOWN on puppet-demo3 i-0000032e output: (No output returned from plugin) [09:18:24] PROBLEM Current Users is now: UNKNOWN on signwriting-ase10 i-00000322 output: (No output returned from plugin) [09:18:24] PROBLEM Disk Space is now: UNKNOWN on signwriting-ase9 i-00000316 output: (No output returned from plugin) [09:18:24] PROBLEM Free ram is now: UNKNOWN on sultest1 i-0000032d output: (No output returned from plugin) [09:19:30] PROBLEM HTTP is now: CRITICAL on greensmw1 i-0000032c output: CRITICAL - Socket timeout after 10 seconds [09:19:55] PROBLEM Current Users is now: UNKNOWN on cvn-apache2 i-00000339 output: (No output returned from plugin) [09:19:55] PROBLEM Current Users is now: UNKNOWN on deployment-apache32 i-0000031a output: (No output returned from plugin) [09:19:55] PROBLEM Current Users is now: UNKNOWN on deployment-apache33 i-0000031b output: (No output returned from plugin) [09:19:55] PROBLEM Current Users is now: UNKNOWN on deployment-cache-bits02 i-0000031c output: (No output returned from plugin) [09:19:55] PROBLEM Disk Space is now: UNKNOWN on deployment-cache-upload01 i-00000326 output: (No output returned from plugin) [09:19:56] PROBLEM Free ram is now: UNKNOWN on deployment-jobrunner06 i-0000031d output: (No output returned from plugin) [09:19:56] PROBLEM Total Processes is now: UNKNOWN on incubator-bot3 i-00000340 output: (No output returned from plugin) [09:20:00] PROBLEM dpkg-check is now: UNKNOWN on jesusaurus-cleanup i-00000337 output: (No output returned from plugin) [09:20:01] PROBLEM Current Load is now: UNKNOWN on puppet-demo1 i-00000329 output: (No output returned from plugin) [09:20:01] PROBLEM Current Users is now: UNKNOWN on puppet-demo2 i-0000032b output: (No output returned from plugin) [09:20:01] PROBLEM Current Users is now: UNKNOWN on puppet-demo3 i-0000032e output: (No output returned from plugin) [09:20:01] PROBLEM Disk Space is now: UNKNOWN on signwriting-ase10 i-00000322 output: (No output returned from plugin) [09:20:01] PROBLEM Free ram is now: UNKNOWN on signwriting-ase9 i-00000316 output: (No output returned from plugin) [09:20:01] PROBLEM HTTP is now: CRITICAL on sultest1 i-0000032d output: Connection refused [09:20:02] PROBLEM Total Processes is now: UNKNOWN on testing-arky i-0000033b output: (No output returned from plugin) [09:20:06] PROBLEM dpkg-check is now: UNKNOWN on testing-instancecreation1 i-00000343 output: (No output returned from plugin) [09:20:06] PROBLEM Current Load is now: UNKNOWN on tutopuppet i-00000336 output: (No output returned from plugin) [09:20:06] PROBLEM Current Users is now: UNKNOWN on varnish-precise i-00000311 output: (No output returned from plugin) [09:20:06] PROBLEM Disk Space is now: UNKNOWN on ve-parsoid-puppetization i-0000033f output: (No output returned from plugin) [09:20:06] PROBLEM Free ram is now: UNKNOWN on ve-parsoid2 i-00000344 output: (No output returned from plugin) [09:20:06] PROBLEM Total Processes is now: UNKNOWN on wikiversity-sandbox-frontend i-0000033a output: (No output returned from plugin) [09:20:08] balll [09:20:32] petan: all those checks should be made dependent of the host checker or check_nrpe :-D [09:21:03] PROBLEM Disk Space is now: UNKNOWN on cvn-apache2 i-00000339 output: (No output returned from plugin) [09:21:03] PROBLEM Disk Space is now: UNKNOWN on deployment-apache32 i-0000031a output: (No output returned from plugin) [09:21:03] PROBLEM Disk Space is now: UNKNOWN on deployment-apache33 i-0000031b output: (No output returned from plugin) [09:21:03] PROBLEM Disk Space is now: UNKNOWN on deployment-cache-bits02 i-0000031c output: (No output returned from plugin) [09:21:03] PROBLEM Free ram is now: UNKNOWN on deployment-cache-upload01 i-00000326 output: (No output returned from plugin) [09:21:03] PROBLEM Total Processes is now: UNKNOWN on gerrit-build i-00000335 output: (No output returned from plugin) [09:21:08] PROBLEM Total Processes is now: UNKNOWN on greensmw1 i-0000032c output: (No output returned from plugin) [09:21:13] PROBLEM dpkg-check is now: UNKNOWN on incubator-bot3 i-00000340 output: (No output returned from plugin) [09:21:13] PROBLEM Current Load is now: UNKNOWN on pdbhandler-1 i-0000030e output: (No output returned from plugin) [09:21:13] PROBLEM Current Users is now: UNKNOWN on puppet-demo1 i-00000329 output: (No output returned from plugin) [09:21:13] PROBLEM Disk Space is now: UNKNOWN on puppet-demo2 i-0000032b output: (No output returned from plugin) [09:21:13] PROBLEM Disk Space is now: UNKNOWN on puppet-demo3 i-0000032e output: (No output returned from plugin) [09:21:14] PROBLEM Free ram is now: UNKNOWN on signwriting-ase10 i-00000322 output: (No output returned from plugin) [09:21:14] PROBLEM Total Processes is now: UNKNOWN on sultestdb i-0000032f output: (No output returned from plugin) [09:21:18] PROBLEM dpkg-check is now: UNKNOWN on testing-arky i-0000033b output: (No output returned from plugin) [09:21:18] PROBLEM Current Load is now: UNKNOWN on testing-singer-puppetization i-00000331 output: (No output returned from plugin) [09:21:19] PROBLEM Current Users is now: UNKNOWN on tutopuppet i-00000336 output: (No output returned from plugin) [09:21:19] PROBLEM Disk Space is now: UNKNOWN on varnish-precise i-00000311 output: (No output returned from plugin) [09:21:19] PROBLEM Free ram is now: UNKNOWN on ve-parsoid-puppetization i-0000033f output: (No output returned from plugin) [09:21:19] PROBLEM Total Processes is now: UNKNOWN on wikibits-mysql i-00000341 output: (No output returned from plugin) [09:21:24] PROBLEM dpkg-check is now: UNKNOWN on wikiversity-sandbox-frontend i-0000033a output: (No output returned from plugin) [09:21:24] PROBLEM host: dumps-incr is DOWN address: i-000002bb CRITICAL - Host Unreachable (i-000002bb) [09:22:14] PROBLEM HTTP is now: CRITICAL on extension-review-testing i-0000033d output: CRITICAL - Socket timeout after 10 seconds [09:22:14] PROBLEM HTTP is now: CRITICAL on wikibits-apache i-00000342 output: CRITICAL - Socket timeout after 10 seconds [09:22:39] PROBLEM Free ram is now: UNKNOWN on cvn-apache2 i-00000339 output: (No output returned from plugin) [09:22:39] PROBLEM Free ram is now: UNKNOWN on deployment-apache32 i-0000031a output: (No output returned from plugin) [09:22:39] PROBLEM Free ram is now: UNKNOWN on deployment-apache33 i-0000031b output: (No output returned from plugin) [09:22:39] PROBLEM Free ram is now: UNKNOWN on deployment-cache-bits02 i-0000031c output: (No output returned from plugin) [09:22:39] PROBLEM Total Processes is now: UNKNOWN on deployment-jobrunner06 i-0000031d output: (No output returned from plugin) [09:22:44] PROBLEM Total Processes is now: UNKNOWN on extension-review-testing i-0000033d output: (No output returned from plugin) [09:22:49] PROBLEM dpkg-check is now: UNKNOWN on gerrit-build i-00000335 output: (No output returned from plugin) [09:22:49] PROBLEM dpkg-check is now: UNKNOWN on greensmw1 i-0000032c output: (No output returned from plugin) [09:22:49] PROBLEM Current Load is now: UNKNOWN on jesusaurus-cleanup i-00000337 output: (No output returned from plugin) [09:22:49] PROBLEM Current Users is now: UNKNOWN on pdbhandler-1 i-0000030e output: (No output returned from plugin) [09:22:49] PROBLEM Disk Space is now: UNKNOWN on puppet-demo1 i-00000329 output: (No output returned from plugin) [09:22:50] PROBLEM Free ram is now: UNKNOWN on puppet-demo2 i-0000032b output: (No output returned from plugin) [09:22:50] PROBLEM Free ram is now: UNKNOWN on puppet-demo3 i-0000032e output: (No output returned from plugin) [09:22:51] PROBLEM Total Processes is now: UNKNOWN on signwriting-ase9 i-00000316 output: (No output returned from plugin) [09:22:54] PROBLEM Total Processes is now: UNKNOWN on sultest1 i-0000032d output: (No output returned from plugin) [09:23:00] PROBLEM Total Processes is now: UNKNOWN on sultest2 i-00000330 output: (No output returned from plugin) [09:23:05] PROBLEM dpkg-check is now: UNKNOWN on sultestdb i-0000032f output: (No output returned from plugin) [09:23:05] PROBLEM Current Load is now: UNKNOWN on testing-instancecreation1 i-00000343 output: (No output returned from plugin) [09:23:05] PROBLEM Current Users is now: UNKNOWN on testing-singer-puppetization i-00000331 output: (No output returned from plugin) [09:23:05] PROBLEM Disk Space is now: UNKNOWN on tutopuppet i-00000336 output: (No output returned from plugin) [09:23:05] PROBLEM Free ram is now: UNKNOWN on varnish-precise i-00000311 output: (No output returned from plugin) [09:23:20] PROBLEM Total Processes is now: UNKNOWN on ve-parsoid2 i-00000344 output: (No output returned from plugin) [09:23:25] PROBLEM Total Processes is now: UNKNOWN on wikibits-apache i-00000342 output: (No output returned from plugin) [09:23:30] PROBLEM dpkg-check is now: UNKNOWN on wikibits-mysql i-00000341 output: (No output returned from plugin) [09:23:30] PROBLEM Total Processes is now: UNKNOWN on deployment-cache-upload01 i-00000326 output: (No output returned from plugin) [09:23:36] PROBLEM dpkg-check is now: UNKNOWN on deployment-jobrunner06 i-0000031d output: (No output returned from plugin) [09:23:36] PROBLEM dpkg-check is now: UNKNOWN on extension-review-testing i-0000033d output: (No output returned from plugin) [09:23:36] PROBLEM Current Load is now: UNKNOWN on greensmw1 i-0000032c output: (No output returned from plugin) [09:23:36] PROBLEM Current Load is now: UNKNOWN on incubator-bot3 i-00000340 output: (No output returned from plugin) [09:23:36] PROBLEM Current Users is now: UNKNOWN on jesusaurus-cleanup i-00000337 output: (No output returned from plugin) [09:23:36] PROBLEM Disk Space is now: UNKNOWN on pdbhandler-1 i-0000030e output: (No output returned from plugin) [09:23:36] PROBLEM Free ram is now: UNKNOWN on puppet-demo1 i-00000329 output: (No output returned from plugin) [09:23:37] PROBLEM Total Processes is now: UNKNOWN on signwriting-ase10 i-00000322 output: (No output returned from plugin) [09:23:41] PROBLEM dpkg-check is now: UNKNOWN on signwriting-ase9 i-00000316 output: (No output returned from plugin) [09:23:41] PROBLEM dpkg-check is now: UNKNOWN on sultest1 i-0000032d output: (No output returned from plugin) [09:23:41] PROBLEM dpkg-check is now: UNKNOWN on sultest2 i-00000330 output: (No output returned from plugin) [09:23:41] PROBLEM Current Load is now: UNKNOWN on testing-arky i-0000033b output: (No output returned from plugin) [09:23:41] PROBLEM Current Users is now: UNKNOWN on testing-instancecreation1 i-00000343 output: (No output returned from plugin) [09:23:41] PROBLEM Disk Space is now: UNKNOWN on testing-singer-puppetization i-00000331 output: (No output returned from plugin) [09:23:41] PROBLEM Free ram is now: UNKNOWN on tutopuppet i-00000336 output: (No output returned from plugin) [09:23:51] PROBLEM Total Processes is now: UNKNOWN on ve-parsoid-puppetization i-0000033f output: (No output returned from plugin) [09:23:56] PROBLEM dpkg-check is now: UNKNOWN on ve-parsoid2 i-00000344 output: (No output returned from plugin) [09:23:56] PROBLEM dpkg-check is now: UNKNOWN on wikibits-apache i-00000342 output: (No output returned from plugin) [09:23:56] PROBLEM Current Load is now: UNKNOWN on wikiversity-sandbox-frontend i-0000033a output: (No output returned from plugin) [09:24:37] PROBLEM Total Processes is now: UNKNOWN on deployment-cache-bits02 i-0000031c output: (No output returned from plugin) [09:24:42] PROBLEM dpkg-check is now: UNKNOWN on deployment-cache-upload01 i-00000326 output: (No output returned from plugin) [09:24:42] PROBLEM Current Load is now: UNKNOWN on extension-review-testing i-0000033d output: (No output returned from plugin) [09:24:42] PROBLEM Current Load is now: UNKNOWN on gerrit-build i-00000335 output: (No output returned from plugin) [09:24:42] PROBLEM Current Users is now: UNKNOWN on greensmw1 i-0000032c output: (No output returned from plugin) [09:24:42] PROBLEM Current Users is now: UNKNOWN on incubator-bot3 i-00000340 output: (No output returned from plugin) [09:24:42] PROBLEM Disk Space is now: UNKNOWN on jesusaurus-cleanup i-00000337 output: (No output returned from plugin) [09:24:42] PROBLEM Free ram is now: UNKNOWN on pdbhandler-1 i-0000030e output: (No output returned from plugin) [09:24:43] PROBLEM Total Processes is now: UNKNOWN on puppet-demo3 i-0000032e output: (No output returned from plugin) [09:24:47] PROBLEM dpkg-check is now: UNKNOWN on signwriting-ase10 i-00000322 output: (No output returned from plugin) [09:24:47] PROBLEM Current Load is now: UNKNOWN on sultest1 i-0000032d output: (No output returned from plugin) [09:24:47] PROBLEM Current Load is now: UNKNOWN on sultest2 i-00000330 output: (No output returned from plugin) [09:24:47] PROBLEM Current Load is now: UNKNOWN on sultestdb i-0000032f output: (No output returned from plugin) [09:24:47] PROBLEM Current Users is now: UNKNOWN on testing-arky i-0000033b output: (No output returned from plugin) [09:24:48] PROBLEM Disk Space is now: UNKNOWN on testing-instancecreation1 i-00000343 output: (No output returned from plugin) [09:24:48] PROBLEM Free ram is now: UNKNOWN on testing-singer-puppetization i-00000331 output: (No output returned from plugin) [09:24:49] PROBLEM Total Processes is now: UNKNOWN on varnish-precise i-00000311 output: (No output returned from plugin) [09:24:52] PROBLEM dpkg-check is now: UNKNOWN on ve-parsoid-puppetization i-0000033f output: (No output returned from plugin) [09:24:52] PROBLEM Current Load is now: UNKNOWN on wikibits-apache i-00000342 output: (No output returned from plugin) [09:24:52] PROBLEM Current Load is now: UNKNOWN on wikibits-mysql i-00000341 output: (No output returned from plugin) [09:24:52] PROBLEM Current Users is now: UNKNOWN on wikiversity-sandbox-frontend i-0000033a output: (No output returned from plugin) [09:25:07] PROBLEM Total Processes is now: UNKNOWN on deployment-apache32 i-0000031a output: (No output returned from plugin) [09:25:12] PROBLEM Total Processes is now: UNKNOWN on deployment-apache33 i-0000031b output: (No output returned from plugin) [09:25:17] PROBLEM dpkg-check is now: UNKNOWN on deployment-cache-bits02 i-0000031c output: (No output returned from plugin) [09:25:17] PROBLEM Current Load is now: UNKNOWN on deployment-jobrunner06 i-0000031d output: (No output returned from plugin) [09:25:17] PROBLEM Current Users is now: UNKNOWN on extension-review-testing i-0000033d output: (No output returned from plugin) [09:25:18] PROBLEM Current Users is now: UNKNOWN on gerrit-build i-00000335 output: (No output returned from plugin) [09:25:18] PROBLEM Disk Space is now: UNKNOWN on greensmw1 i-0000032c output: (No output returned from plugin) [09:25:18] PROBLEM Disk Space is now: UNKNOWN on incubator-bot3 i-00000340 output: (No output returned from plugin) [09:25:18] PROBLEM Free ram is now: UNKNOWN on jesusaurus-cleanup i-00000337 output: (No output returned from plugin) [09:25:19] PROBLEM Total Processes is now: UNKNOWN on puppet-demo1 i-00000329 output: (No output returned from plugin) [09:25:22] PROBLEM Total Processes is now: UNKNOWN on puppet-demo2 i-0000032b output: (No output returned from plugin) [09:25:28] PROBLEM dpkg-check is now: UNKNOWN on puppet-demo3 i-0000032e output: (No output returned from plugin) [09:25:28] PROBLEM Current Load is now: UNKNOWN on signwriting-ase9 i-00000316 output: (No output returned from plugin) [09:25:28] PROBLEM Current Users is now: UNKNOWN on sultest1 i-0000032d output: (No output returned from plugin) [09:25:28] PROBLEM Current Users is now: UNKNOWN on sultest2 i-00000330 output: (No output returned from plugin) [09:25:28] PROBLEM Current Users is now: UNKNOWN on sultestdb i-0000032f output: (No output returned from plugin) [09:25:28] PROBLEM Disk Space is now: UNKNOWN on testing-arky i-0000033b output: (No output returned from plugin) [09:25:28] PROBLEM Free ram is now: UNKNOWN on testing-instancecreation1 i-00000343 output: (No output returned from plugin) [09:25:29] PROBLEM Total Processes is now: UNKNOWN on tutopuppet i-00000336 output: (No output returned from plugin) [09:25:33] PROBLEM dpkg-check is now: UNKNOWN on varnish-precise i-00000311 output: (No output returned from plugin) [09:25:33] PROBLEM Current Load is now: UNKNOWN on ve-parsoid2 i-00000344 output: (No output returned from plugin) [09:25:33] PROBLEM Current Users is now: UNKNOWN on wikibits-apache i-00000342 output: (No output returned from plugin) [09:25:33] PROBLEM Current Users is now: UNKNOWN on wikibits-mysql i-00000341 output: (No output returned from plugin) [09:25:33] PROBLEM Disk Space is now: UNKNOWN on wikiversity-sandbox-frontend i-0000033a output: (No output returned from plugin) [09:25:33] PROBLEM dpkg-check is now: UNKNOWN on cvn-apache2 i-00000339 output: (No output returned from plugin) [09:25:34] PROBLEM dpkg-check is now: UNKNOWN on deployment-apache32 i-0000031a output: (No output returned from plugin) [09:25:34] PROBLEM dpkg-check is now: UNKNOWN on deployment-apache33 i-0000031b output: (No output returned from plugin) [09:25:35] PROBLEM Current Load is now: UNKNOWN on deployment-cache-upload01 i-00000326 output: (No output returned from plugin) [09:25:35] PROBLEM Current Users is now: UNKNOWN on deployment-jobrunner06 i-0000031d output: (No output returned from plugin) [09:45:03] PROBLEM host: deployment-backup is DOWN address: i-000000f8 CRITICAL - Host Unreachable (i-000000f8) [09:51:43] PROBLEM host: dumps-incr is DOWN address: i-000002bb CRITICAL - Host Unreachable (i-000002bb) [09:52:55] 07/16/2012 - 09:52:55 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [09:55:03] PROBLEM host: deployment-cache-upload is DOWN address: i-00000263 CRITICAL - Host Unreachable (i-00000263) [09:55:03] PROBLEM host: nova-daas-1 is DOWN address: i-000000e7 CRITICAL - Host Unreachable (i-000000e7) [09:55:03] PROBLEM host: nova-production1 is DOWN address: i-0000007b CRITICAL - Host Unreachable (i-0000007b) [10:15:03] PROBLEM host: deployment-backup is DOWN address: i-000000f8 CRITICAL - Host Unreachable (i-000000f8) [10:21:43] PROBLEM host: dumps-incr is DOWN address: i-000002bb CRITICAL - Host Unreachable (i-000002bb) [10:25:03] PROBLEM host: deployment-cache-upload is DOWN address: i-00000263 CRITICAL - Host Unreachable (i-00000263) [10:25:03] PROBLEM host: nova-daas-1 is DOWN address: i-000000e7 CRITICAL - Host Unreachable (i-000000e7) [10:25:03] PROBLEM host: nova-production1 is DOWN address: i-0000007b CRITICAL - Host Unreachable (i-0000007b) [10:45:03] PROBLEM host: deployment-backup is DOWN address: i-000000f8 CRITICAL - Host Unreachable (i-000000f8) [10:51:43] PROBLEM host: dumps-incr is DOWN address: i-000002bb CRITICAL - Host Unreachable (i-000002bb) [10:54:57] 07/16/2012 - 10:54:57 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [10:55:03] PROBLEM host: deployment-cache-upload is DOWN address: i-00000263 CRITICAL - Host Unreachable (i-00000263) [10:55:03] PROBLEM host: nova-daas-1 is DOWN address: i-000000e7 CRITICAL - Host Unreachable (i-000000e7) [10:55:03] PROBLEM host: nova-production1 is DOWN address: i-0000007b CRITICAL - Host Unreachable (i-0000007b) [11:15:03] PROBLEM host: deployment-backup is DOWN address: i-000000f8 CRITICAL - Host Unreachable (i-000000f8) [11:21:43] PROBLEM host: dumps-incr is DOWN address: i-000002bb CRITICAL - Host Unreachable (i-000002bb) [11:25:03] PROBLEM host: deployment-cache-upload is DOWN address: i-00000263 CRITICAL - Host Unreachable (i-00000263) [11:25:03] PROBLEM host: nova-daas-1 is DOWN address: i-000000e7 CRITICAL - Host Unreachable (i-000000e7) [11:25:03] PROBLEM host: nova-production1 is DOWN address: i-0000007b CRITICAL - Host Unreachable (i-0000007b) [11:28:43] looks like something crapped out [11:45:03] PROBLEM host: deployment-backup is DOWN address: i-000000f8 CRITICAL - Host Unreachable (i-000000f8) [11:51:43] PROBLEM host: dumps-incr is DOWN address: i-000002bb CRITICAL - Host Unreachable (i-000002bb) [11:55:03] PROBLEM host: deployment-cache-upload is DOWN address: i-00000263 CRITICAL - Host Unreachable (i-00000263) [11:55:03] PROBLEM host: nova-daas-1 is DOWN address: i-000000e7 CRITICAL - Host Unreachable (i-000000e7) [11:55:03] PROBLEM host: nova-production1 is DOWN address: i-0000007b CRITICAL - Host Unreachable (i-0000007b) [11:56:56] 07/16/2012 - 11:56:56 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [12:15:03] PROBLEM host: deployment-backup is DOWN address: i-000000f8 CRITICAL - Host Unreachable (i-000000f8) [12:21:43] PROBLEM host: dumps-incr is DOWN address: i-000002bb CRITICAL - Host Unreachable (i-000002bb) [12:21:53] PROBLEM Puppet freshness is now: CRITICAL on aggregator-test1 i-000002bf output: (Return code of 127 is out of bounds - plugin may be missing) [12:21:53] PROBLEM Puppet freshness is now: CRITICAL on aggregator1 i-0000010c output: (Return code of 127 is out of bounds - plugin may be missing) [12:21:53] PROBLEM Puppet freshness is now: CRITICAL on aggregator2 i-000002c0 output: (Return code of 127 is out of bounds - plugin may be missing) [12:21:53] PROBLEM Puppet freshness is now: CRITICAL on analytics i-000000e2 output: (Return code of 127 is out of bounds - plugin may be missing) [12:21:53] PROBLEM Puppet freshness is now: CRITICAL on asher1 i-0000003a output: (Return code of 127 is out of bounds - plugin may be missing) [12:21:53] PROBLEM Puppet freshness is now: CRITICAL on bastion-restricted1 i-0000019b output: (Return code of 127 is out of bounds - plugin may be missing) [12:57:55] 07/16/2012 - 12:57:55 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [13:59:56] 07/16/2012 - 13:59:55 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [15:01:58] 07/16/2012 - 15:01:58 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [15:55:06] !log deployment-prep Updated ArticleFeedbackv5 778f089..ccbc585 [15:55:09] Logged the message, Master [16:03:55] 07/16/2012 - 16:03:55 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [16:34:47] hello, is it possible to upgrade a labs instance with more cores and more memory? [16:35:16] parsoid.wmflabs.org could use a bit more of both so it doesn't fall over as often [16:35:29] would save us some time on babysitting [16:36:43] gwicke: you can create a new on with a different template and then apply the same classes [16:37:38] we don't have the parsoid installation puppetized yet, so just tweaking the parameters for the existing instance would be much less work [16:38:14] at least if it works similar to XEN or OpenVZ [16:43:40] gwicke: i could probably change your project quota for cores and ram using nova-manage but let's ask Ryan if it makes sense for the instance, these are project quotas after all [16:44:35] ok, thanks [16:45:12] right now we have just a single core and 2G of memory, which is not quite enough if a few people try to test parsoid in parallel on complex pages [16:45:52] there would still be: bring up new instance with large template, rsync your files from one to another [16:53:06] gwicke: ah, so i just learned some more about this and found out "Using the v1.1/{tenant_id}/servers/{server_id}/action API call, you can change the flavor/instance type of the running instance. " [16:53:12] http://ken.pepple.info/openstack/2011/03/03/Configurable-Instance-Types-For-OpenStack-Nova/ [16:53:43] but it is definitely something people have been working on because it did NOT exist [17:05:56] 07/16/2012 - 17:05:56 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [17:16:38] Ryan_Lane: here? [18:07:57] 07/16/2012 - 18:07:57 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [18:19:17] any netadmins around? [18:32:10] JasonDC: well, depends on the project [18:32:13] what's up? [18:34:00] I need a IPv4 address assigned to an instance/project (1 sec ill get you the ID) [18:36:21] i-00000342 [18:36:57] 07/16/2012 - 18:36:57 - User jasonspriggs may have been modified in LDAP or locally, updating key in project(s): bots,bastion,testing,wikibits [18:37:19] 07/16/2012 - 18:37:18 - Updating keys for jasonspriggs at /export/keys/jasonspriggs [18:41:09] JasonDC: you need a public IP? [18:41:11] to which project? [18:41:25] wikibits for a apache server [18:41:26] (and for what purpose?) [18:41:50] we limit IP address allocation, since we're running out, so I always need to ask [18:42:00] understood [18:42:19] what will it be used for? [18:43:26] the apache server? a project for creating a wikibased system for code hosting, colaberation, and 'pastebin' [18:51:58] ah. ok [18:53:46] JasonDC: I've upped the quota [18:53:58] you can now allocate the IP, then associate it with an instance [18:54:37] thank you :) [18:55:57] Anyone happen to know where the openstack page that use to list 'non-incubated projects' has moved to? Totally can't find stuff on their site anymore lol [18:56:18] hm [18:58:19] Damianz: http://wiki.openstack.org/Projects/ [18:58:48] Ah, there they are! Tyvm [19:09:58] 07/16/2012 - 19:09:58 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [19:30:19] 07/16/2012 - 19:30:19 - Updating keys for van-de-bugger at /export/keys/van-de-bugger [19:33:48] !log deployment-prep Updated ArticleFeedback and ArticleFeedbackv5 to latest master. Dropped their tables and ran the updater ( mwscript update.php --quick --wiki=enwiki ). Solves {{bug|38422}} - trash and redo ArticleFeedbackv5 on beta enwiki. See http://en.wikipedia.beta.wmflabs.org/wiki/Special:ArticleFeedbackv5 [19:33:50] Logged the message, Master [19:36:34] !log deployment-prep rebuilding localisation cache [19:36:36] Logged the message, Master [20:11:57] 07/16/2012 - 20:11:57 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [20:25:51] anyone know why when i save security groups; it dosnt save my CIDR settings? [21:11:58] JasonDC: it doesn't? [21:12:03] which cidr are you using? [21:12:31] 0.0.0.0/0 [21:12:45] or 10.4.0.0/24 [21:12:51] depening on what i need blocked [21:12:55] * Ryan_Lane nods [21:12:59] well, it should surely work [21:13:17] its not showing up even tho it says it saved [21:13:55] hm [21:13:58] 07/16/2012 - 21:13:58 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [21:30:40] Change abandoned: J; "does no longer apply" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6109 [22:16:00] 07/16/2012 - 22:16:00 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [22:24:11] Ryan_Lane, I figured it out; if you select a source group no CIDR settings are saved. [22:27:04] JasonDC: in general you don't want to use a source group [22:27:11] I need to turn that into an "advanced option" [23:17:58] 07/16/2012 - 23:17:57 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep