[00:11:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:27:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:41:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:58:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:08:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 175 processes [01:12:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:13:33] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:28:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:42:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:56:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [01:59:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:12:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:29:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:37:52] RECOVERY Free ram is now: OK on sube i-000003d0.pmtpa.wmflabs output: OK: 42% free memory [02:38:52] RECOVERY Free ram is now: OK on wikistream-1 i-0000016e.pmtpa.wmflabs output: OK: 26% free memory [02:42:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:59:04] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:11:53] PROBLEM Free ram is now: WARNING on wikistream-1 i-0000016e.pmtpa.wmflabs output: Warning: 13% free memory [03:13:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:29:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:33:42] PROBLEM Total processes is now: WARNING on bastion1 i-000000ba.pmtpa.wmflabs output: PROCS WARNING: 153 processes [03:44:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:01:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:06:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [04:14:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:32:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:44:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:02:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:14:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:32:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:44:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:03:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:15:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:29:33] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:29:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:33:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:39:33] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 146 processes [06:46:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:49:32] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [06:54:42] PROBLEM Free ram is now: WARNING on swift-be1 i-000001c7.pmtpa.wmflabs output: Warning: 18% free memory [07:03:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:16:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:33:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:47:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:03:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:06:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 11% free memory [08:18:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:33:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:48:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:54:43] RECOVERY Free ram is now: OK on swift-be1 i-000001c7.pmtpa.wmflabs output: OK: 20% free memory [09:03:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:19:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:33:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:49:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:03:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:19:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:33:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:39:32] PROBLEM Current Users is now: CRITICAL on wikidata-dev-8 i-00000532.pmtpa.wmflabs output: Connection refused by host [10:40:13] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-8 i-00000532.pmtpa.wmflabs output: Connection refused by host [10:40:53] PROBLEM Current Load is now: CRITICAL on wikidata-dev-8 i-00000532.pmtpa.wmflabs output: Connection refused by host [10:41:03] PROBLEM Free ram is now: CRITICAL on wikidata-dev-8 i-00000532.pmtpa.wmflabs output: Connection refused by host [10:42:23] PROBLEM Total processes is now: CRITICAL on wikidata-dev-8 i-00000532.pmtpa.wmflabs output: Connection refused by host [10:42:53] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-8 i-00000532.pmtpa.wmflabs output: Connection refused by host [10:47:22] RECOVERY Total processes is now: OK on wikidata-dev-8 i-00000532.pmtpa.wmflabs output: PROCS OK: 105 processes [10:49:32] RECOVERY Current Users is now: OK on wikidata-dev-8 i-00000532.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [10:50:12] RECOVERY Disk Space is now: OK on wikidata-dev-8 i-00000532.pmtpa.wmflabs output: DISK OK [10:50:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:50:52] RECOVERY Current Load is now: OK on wikidata-dev-8 i-00000532.pmtpa.wmflabs output: OK - load average: 0.23, 0.18, 0.08 [10:51:02] RECOVERY Free ram is now: OK on wikidata-dev-8 i-00000532.pmtpa.wmflabs output: OK: 4421% free memory [10:52:52] RECOVERY dpkg-check is now: OK on wikidata-dev-8 i-00000532.pmtpa.wmflabs output: All packages OK [10:53:42] RECOVERY Total processes is now: OK on bastion1 i-000000ba.pmtpa.wmflabs output: PROCS OK: 120 processes [11:00:53] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-8 i-00000532.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [11:03:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:20:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:33:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:50:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:04:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:21:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:34:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:52:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:04:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:22:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:34:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:41:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [13:46:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [13:53:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:04:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:06:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [14:23:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:34:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:51:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [14:53:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:04:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:24:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:34:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:36:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [15:54:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:06:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:24:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:36:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:55:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:07:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:25:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:26:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [17:37:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:56:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:08:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:26:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:38:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:52:22] Damianz: [18:52:24] hi [18:53:13] do you have some time for me? [18:56:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:08:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:26:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:39:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:57:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:09:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:14:06] !ping [20:14:07] pong [20:14:13] :| [20:23:02] !pingall [20:23:04] Ryan_Lane: are you there? [20:23:22] giftpflanze: yep [20:23:25] what's up? [20:23:38] bene would like to be member of bastion [20:23:48] hi Ryan_Lane [20:23:57] that's me ;) [20:23:59] he is waiting like ever and no one is doing it [20:24:02] I gave him shell rights last week, did that not do the trick? [20:24:15] i guess not [20:24:25] benestar_: did you try to connect? [20:24:36] where? [20:24:41] He's in the user list for bastion. [20:24:51] oh, so how to go on? [20:24:55] nscld cache? though I swear I pushed a patch for that [20:25:31] could anybody explain it to me step by step? [20:25:41] hm, ok now i see it [20:26:28] Did you read https://labsconsole.wikimedia.org/wiki/Help:Getting_Started? [20:26:33] It's mostly documented there [20:26:40] * Damianz goes afk for a few as he's only just got home [20:27:30] so i have to create a new project? [20:27:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:27:49] no, you want to join bots ;) [20:28:23] he is already a member of bots. [20:28:41] oh, i understand [20:28:48] and there i can upload my bot, right? [20:29:11] yeah, i did that (don't know how i could do that) but i don't know if that was sufficient: https://labsconsole.wikimedia.org/w/index.php?title=Nova_Resource:Bots&diff=9115&oldid=8796 [20:29:26] 'bots' is a project. A project consists of one or more servers. [20:29:46] So you need to consult with some other bots people for advice about which instance is a good place to run your code. [20:30:02] ('instance' == 'server') [20:30:16] so is here any member of bot project [20:30:18] ? [20:31:06] So basically you probably want to use bots-4 for now to run anything, unless it's pupetized then you can use nr1 [20:32:57] ok [20:39:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:56:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 11% free memory [20:57:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 155 processes [20:57:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:05:37] I really hope it doesn't rain tonight =\ [21:08:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 10.95, 10.25, 7.39 [21:09:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:10:33] PROBLEM Current Users is now: WARNING on kripke i-00000268.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [21:10:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 7.87, 8.15, 6.16 [21:15:28] 12/10/2012 - 21:15:28 - Creating a home directory for bene at /export/keys/bene [21:16:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 8.67, 7.72, 5.73 [21:16:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 8.63, 8.11, 5.90 [21:20:29] 12/10/2012 - 21:20:28 - Updating keys for bene at /export/keys/bene [21:28:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:30:27] 12/10/2012 - 21:30:27 - Updating keys for bene at /export/keys/bene [21:36:33] I have a labs instance which has never successfully run puppet. Here's the sort of error I find in /var/log/messages: puppet-agent[12081]: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class generic::tcptweaks for i-00000459.pmtpa.wmflabs on node i-00000459.pmtpa.wmflabs [21:36:38] jfyi [21:38:53] what does sudo puppetd -tv say [21:39:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:39:09] I can't run that, unfortunately-- the sudo config is not syncing either [21:42:47] surprise! i needed to specify full paths in the sudo config; that part works now [21:42:57] new error: err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Package[apache2] is already defined in file /etc/puppet/manifests/webserver.pp at line 91; cannot redefine at /etc/puppet/manifests/webserver.pp:42 on node i-00000459.pmtpa.wmflabs [21:43:46] awight: if you have Apache on an instance and multiple virtual hosts.. cant repeat the Apache setup twice... [21:44:03] only add additional vhost file [21:44:17] hehe, try explaining this in that case: err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: File[/etc/mysql/my.cnf] is already defined in file /etc/puppet/manifests/mysql.pp at line 431; cannot redefine at /etc/puppet/manifests/mysql.pp:758 on node i-00000459.pmtpa.wmflabs [21:45:07] ... argh... err: Could not retrieve catalog from remote server: Error 400 on SERVER: Exported resource Sshhostkey[abusefilter-global-main.pmtpa.wmflabs] cannot override local resource on node i-00000459.pmtpa.wmflabs [21:45:35] that's kinda normal [21:45:42] sshhostkey issue is ..yea [21:45:56] ;) I am feeling more normal already [21:45:59] and the other one, related to mysql.. also something duplicate [21:46:21] maybe applied 2 mysql classes that conflict? (via labsconsole?) [21:46:24] The deuplicate errors went away when i unchecked the relevant groups in "instance configure" [21:46:41] makes sense.. well.. _one_ of them alone should work [21:46:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.57, 3.64, 4.53 [21:46:57] IMO the ui should handle this stuff [21:47:20] well yeah - but the ui sucks [21:47:35] We don't have proper role classes setup that would allow the ui to be nice [21:47:43] you mean it should understand and run puppet code for every combination of classes you could check in the UI?:o [21:48:00] and test conflicts.. and only then let you changet stuff ... seems hard [21:48:17] nah, i donno quite how I would fix... I mean, I'm impressed there's even a UI [21:48:27] so, how do i fix this sshhostkey thing? [21:48:44] yeah, the real issue is that there are multiple ways to do stuff..like setup an Apache [21:49:00] and we should have just one that is being offered to you in the UI perhaps.. true [21:49:51] also, my sudo policy doesn't work if I only check the "Adamw" user, but it does work if i check "ALL" [21:50:28] even weirder! it was actually the sudo host policy that was blocking me [21:50:38] heh:) [21:51:20] yeah that's known [21:51:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.44, 4.63, 4.99 [21:51:43] thanks to a naming schema change a while back ldap and hostnames argue so ssh policies don't work [21:52:12] cool, thx. I'm more comfortable with the "ALL" hosts box checked than "ALL" users, so fine by me [21:55:24] any clues what I can do about Sshhostkey? [21:56:00] Actually, now that i'm root i can just install the pkgs and forget about puppet :( [21:56:12] done :) [21:56:50] I can help when I get back from a meeting [21:56:59] gimme like 45 mins or so [21:57:02] no worries [21:57:15] awight: You might have figured this out already, but generic:tcptweaks doesn't exist in our repo anymore. [21:57:20] I'll probably find the bug [21:57:31] At least, it looks to me like it's been renamed to base::tcptweaks [21:57:55] good to know! I wish i could blow away the puppet conf and start again... [21:58:04] looks like it got stale in the last few months [21:58:31] That package is still present in the 'configure instance' page :( [21:59:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:59:53] You know the real fucker? [22:00:26] If you use puppetmaster::self and we move stuff around in the repo all the options in the web interface would change :( [22:00:53] PROBLEM dpkg-check is now: CRITICAL on bots-abogott-devel i-00000530.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [22:00:59] * andrewbogott fixed it in labsconsole [22:01:43] can i ask where the "nova" project is hosted? [22:01:51] if that's what the console is based on... [22:02:37] nova is the compute part of openstack [22:02:45] oh k [22:03:15] I don't see console under gerrit projects / labs... [22:03:51] pls say it is written in wikimarkup :P [22:04:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.38, 5.98, 5.41 [22:05:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.06, 6.68, 5.83 [22:05:53] o_O I don't think I'm supposed to have access to this: https://gerrit.wikimedia.org/r/gitweb?p=labs/private.git;a=blob;f=manifests/passwords.pp;h=da973997057c103b9e2a2a63c9f482ebca137992;hb=HEAD [22:05:59] labsconsole is mediawiki [22:06:03] awight: yes you are [22:06:07] with a custom plugin called openstackmanager [22:06:09] kk [22:06:10] or w/e it's called today [22:06:33] awight: random people on the internet are not [22:06:34] Damianz: rad, I see [22:06:40] but you, as a labs user are [22:06:50] Ryan_Lane: On the internet no one knows you're a dog. [22:07:09] Damianz: I know, that's how I get away with it [22:07:48] Ryan_Lane: ok i get it, this is for a test instance... [22:08:01] http://chzmemebase.files.wordpress.com/2012/09/internet-memes-that-dumb-dog.jpg [22:08:07] well, the root password doesn't really get you much [22:08:11] in labs [22:08:33] really I need to make the instances randomly change their root passwords [22:08:59] I hope their pam configs are set to never allow logins anyway [22:09:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:09:23] Totally could change the root pass on every puppet run for lols though [22:09:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 5.16, 5.57, 5.10 [22:10:53] RECOVERY dpkg-check is now: OK on bots-abogott-devel i-00000530.pmtpa.wmflabs output: All packages OK [22:11:05] yep ssh root@ is prevented by default [22:11:11] had to check ;) [22:11:33] Well ssh is simple... pam can be more restrictive than that (local logins etc). [22:18:44] indeed [22:18:49] I turned off root ssh ages ago [22:19:23] before I started allowing access to anyone :) [22:22:35] Change on 12mediawiki a page Developer access was modified, changed by Johannes Kroll (WMDE) link https://www.mediawiki.org/w/index.php?diff=615242 edit summary: [22:29:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:35:32] RECOVERY Current Users is now: OK on kripke i-00000268.pmtpa.wmflabs output: USERS OK - 5 users currently logged in [22:39:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:54:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 3.50, 4.29, 4.99 [22:59:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:00:04] Could someone give me shell access? [23:09:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:29:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:39:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.38, 4.18, 4.85 [23:41:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:47:07] Public IP request. Shall I expound here? [23:47:27] Expound away! [23:47:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.98, 5.73, 5.37 [23:47:45] I'd like to expose fundraising-dev-awight, to offload FR statistics and screenshots away from our secure cluster [23:48:20] ... and cos I'm too lazy to figure out how else to get this stuff somewhere public ;) [23:49:00] what's the project name? [23:49:09] fundraising-awight, i think [23:49:31] yep [23:50:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 2.96, 4.02, 4.76 [23:51:27] awight: If you visit https://labsconsole.wikimedia.org/wiki/Special:NovaAddress you should be able to allocate an IP for your instance. [23:51:27] andrewbogott: thanks! [23:51:47] is there public dns for those? [23:52:25] ok i see "add host name" in the UI [23:52:35] awight: nothing on this instance is going to be private data, right? [23:52:40] nope [23:52:41] ok [23:53:02] yeah, and thank you! It's a huge relief for us to start moving this stuff off our cluster. [23:53:08] yw [23:59:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%