[00:21:43] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [00:32:54] RECOVERY Free ram is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: OK: 87% free memory [00:33:44] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 153 processes [00:33:54] RECOVERY Free ram is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK: 93% free memory [00:38:42] RECOVERY Free ram is now: OK on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: OK: 27% free memory [00:38:52] RECOVERY Free ram is now: OK on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: OK: 26% free memory [00:41:24] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [00:41:46] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 20% free memory [00:44:24] Syntax error at '}'; expected '}' < oreally [00:47:22] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 5.37, 5.42, 5.18 [00:48:42] RECOVERY Total processes is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS OK: 148 processes [00:49:03] Damianz, sounds like a PHP bug [00:49:25] Krenair: A PHP bug in Ruby code? awesome ;P [00:49:57] Even if it's in Ruby code I can imagine PHP doing that. [00:51:42] PROBLEM Free ram is now: WARNING on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: Warning: 18% free memory [00:51:52] PROBLEM Free ram is now: WARNING on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: Warning: 17% free memory [00:52:22] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 2.44, 4.27, 4.82 [00:59:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [01:08:44] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 172 processes [01:09:44] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [01:13:42] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 96 processes [01:15:14] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [02:57:37] where are the dumps located on the bots project again? [02:57:42] * duh forgot [04:39:23] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [04:41:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [04:41:43] RECOVERY Free ram is now: OK on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: OK: 27% free memory [04:41:52] RECOVERY Free ram is now: OK on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: OK: 25% free memory [04:49:52] PROBLEM Free ram is now: WARNING on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: Warning: 17% free memory [04:52:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [04:54:42] PROBLEM Free ram is now: WARNING on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: Warning: 19% free memory [04:59:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [05:53:43] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [05:59:33] PROBLEM Disk Space is now: CRITICAL on syslogcol-ab.pmtpa.wmflabs 10.4.0.223 output: DISK CRITICAL - free space: / 286 MB (2% inode=90%): [05:59:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.87, 4.80, 4.94 [06:29:35] PROBLEM Disk Space is now: WARNING on syslogcol-ab.pmtpa.wmflabs 10.4.0.223 output: DISK WARNING - free space: / 290 MB (3% inode=90%): [06:29:35] PROBLEM Total processes is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS WARNING: 154 processes [06:29:55] RECOVERY Free ram is now: OK on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: OK: 34% free memory [06:32:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 4.72, 5.02, 5.02 [06:37:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.77, 4.91, 4.97 [06:44:34] RECOVERY Total processes is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS OK: 150 processes [06:48:42] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 44% free memory [07:14:32] PROBLEM Disk Space is now: CRITICAL on syslogcol-ab.pmtpa.wmflabs 10.4.0.223 output: DISK CRITICAL - free space: / 286 MB (2% inode=90%): [07:50:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.25, 5.17, 5.08 [07:55:12] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [08:10:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.63, 4.82, 4.97 [08:28:32] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [08:37:23] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [08:38:23] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 6% free memory [08:39:43] RECOVERY Free ram is now: OK on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: OK: 26% free memory [08:45:13] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 20% free memory [08:53:12] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [08:53:32] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [08:55:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [08:57:43] PROBLEM Free ram is now: WARNING on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: Warning: 18% free memory [10:58:53] PROBLEM Current Load is now: CRITICAL on maps-pgsql.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [10:59:33] PROBLEM Disk Space is now: CRITICAL on maps-pgsql.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [11:00:13] PROBLEM Free ram is now: CRITICAL on maps-pgsql.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [11:01:43] PROBLEM Total processes is now: CRITICAL on maps-pgsql.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [11:03:53] RECOVERY Current Load is now: OK on maps-pgsql.pmtpa.wmflabs 10.4.0.136 output: OK - load average: 0.29, 0.67, 0.44 [11:04:33] RECOVERY Disk Space is now: OK on maps-pgsql.pmtpa.wmflabs 10.4.0.136 output: DISK OK [11:05:12] RECOVERY Free ram is now: OK on maps-pgsql.pmtpa.wmflabs 10.4.0.136 output: OK: 94% free memory [11:06:42] RECOVERY Total processes is now: OK on maps-pgsql.pmtpa.wmflabs 10.4.0.136 output: PROCS OK: 90 processes [12:37:42] RECOVERY Free ram is now: OK on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: OK: 26% free memory [12:58:00] I cannot find server key fingerprint information for Bastion https://labsconsole.wikimedia.org/wiki/Bastion . [13:10:44] PROBLEM Free ram is now: WARNING on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: Warning: 18% free memory [15:52:45] What does "token" mean on https://labsconsole.wikimedia.org/w/index.php?title=Special:UserLogin ? Can someone please add to https://labsconsole.wikimedia.org/wiki/Help:Terminology [16:08:50] It's for Two-factor Auth [16:10:16] http://lists.wikimedia.org/pipermail/labs-l/2012-May/000205.html [16:36:13] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 17% free memory [16:38:15] Krenair: is this something for me to know (to learn; where ? [16:39:10] Krenair: I cannot find (official) server key fingerprint information for Bastion [16:39:11] what? [16:39:43] Krenair: overlooked "http://lists.wikimedia.org/pipermail/labs-l/2012-May/000205.html" [16:39:45] ty [16:39:52] sorry [16:40:00] What does that have to do with bastion ssh fingerprints? [16:40:12] nothiung [16:40:12] nothing [16:40:16] It was in my list of questions [16:40:22] open question [16:40:23] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [16:40:43] RECOVERY Free ram is now: OK on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: OK: 26% free memory [16:40:52] What I mean is, the fingerprints should be available somewhere on labsconsole [16:41:13] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 21% free memory [16:48:32] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 100% free memory [16:56:22] PROBLEM Free ram is now: WARNING on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: Warning: 17% free memory [16:56:32] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 2% free memory [17:04:13] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 18% free memory [17:08:12] PROBLEM Free ram is now: CRITICAL on maps-pgsql.pmtpa.wmflabs 10.4.0.149 output: Connection refused by host [17:08:22] PROBLEM dpkg-check is now: CRITICAL on maps-pgsql.pmtpa.wmflabs 10.4.0.149 output: Connection refused by host [17:08:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [17:08:42] PROBLEM Free ram is now: WARNING on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: Warning: 18% free memory [17:08:52] PROBLEM Current Load is now: CRITICAL on maps-pgsql.pmtpa.wmflabs 10.4.0.149 output: Connection refused by host [17:09:32] PROBLEM Disk Space is now: CRITICAL on maps-pgsql.pmtpa.wmflabs 10.4.0.149 output: Connection refused by host [17:09:42] PROBLEM Total processes is now: CRITICAL on maps-pgsql.pmtpa.wmflabs 10.4.0.149 output: Connection refused by host [17:10:20] Wikinaut: We want to put all host key fingerprints on labs console - I think there's a bug for this [17:10:38] Also I wonder if ad has a creation time for instances hmmm [17:13:53] PROBLEM Current Load is now: CRITICAL on maps-pgsql2.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [17:14:33] PROBLEM Disk Space is now: CRITICAL on maps-pgsql2.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [17:15:12] PROBLEM Free ram is now: CRITICAL on maps-pgsql2.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [17:16:42] PROBLEM Total processes is now: CRITICAL on maps-pgsql2.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [17:17:22] PROBLEM dpkg-check is now: CRITICAL on maps-pgsql2.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [17:18:52] RECOVERY Current Load is now: OK on maps-pgsql2.pmtpa.wmflabs 10.4.0.136 output: OK - load average: 0.65, 0.87, 0.49 [17:19:32] RECOVERY Disk Space is now: OK on maps-pgsql2.pmtpa.wmflabs 10.4.0.136 output: DISK OK [17:20:13] RECOVERY Free ram is now: OK on maps-pgsql2.pmtpa.wmflabs 10.4.0.136 output: OK: 96% free memory [17:21:23] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 31% free memory [17:21:43] RECOVERY Total processes is now: OK on maps-pgsql2.pmtpa.wmflabs 10.4.0.136 output: PROCS OK: 100 processes [17:22:23] RECOVERY dpkg-check is now: OK on maps-pgsql2.pmtpa.wmflabs 10.4.0.136 output: All packages OK [17:30:12] what is the best way to request software installs on bots-bnr*? ask in irc, mailing list or bugzilla? [17:37:29] Damianz, ^ maybe you can help giftpflanze? [17:38:04] I can install them [18:19:22] does it need to be puppetized? [18:19:52] Damianz: see https://labsconsole.wikimedia.org/wiki/GiftBot#Dependencies [18:33:23] PROBLEM Free ram is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Critical: 3% free memory [18:38:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [18:42:42] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [20:14:13] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 21% free memory [20:37:42] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 22% free memory [20:38:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [20:38:42] RECOVERY Free ram is now: OK on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: OK: 26% free memory [20:41:24] RECOVERY Free ram is now: OK on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: OK: 25% free memory [20:50:43] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 15% free memory [20:56:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [21:01:43] PROBLEM Free ram is now: WARNING on wordpressbeta-precise.pmtpa.wmflabs 10.4.0.215 output: Warning: 18% free memory [21:09:22] PROBLEM Free ram is now: WARNING on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: Warning: 17% free memory [21:22:13] giftpflanze: Which nr and yes, it should be - I'll just append it to my pending merge stuff that I need to get motivated enough to finish (maybe ams hackup)