[00:04:23] RECOVERY Total processes is now: OK on ee-lwelling2.pmtpa.wmflabs 10.4.1.76 output: PROCS OK: 84 processes [00:05:54] RECOVERY Current Load is now: OK on ee-lwelling2.pmtpa.wmflabs 10.4.1.76 output: OK - load average: 0.26, 0.87, 0.69 [00:05:54] RECOVERY dpkg-check is now: OK on ee-lwelling2.pmtpa.wmflabs 10.4.1.76 output: All packages OK [00:06:34] RECOVERY Current Users is now: OK on ee-lwelling2.pmtpa.wmflabs 10.4.1.76 output: USERS OK - 0 users currently logged in [00:07:06] Ryan_Lane: Work around your mediawiki problem with $ /usr/sbin/dpkg-reconfigure -fnoninteractive mysql-server-5.5 [00:07:14] RECOVERY Disk Space is now: OK on ee-lwelling2.pmtpa.wmflabs 10.4.1.76 output: DISK OK [00:07:27] It should be doing that anyway, but it must be happening at the wrong time [00:07:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 9.46, 7.82, 5.80 [00:07:55] RECOVERY Free ram is now: OK on ee-lwelling2.pmtpa.wmflabs 10.4.1.76 output: OK: 895% free memory [00:07:56] andrewbogott: cool. thanks [00:14:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 7.81, 7.43, 6.00 [00:19:52] PROBLEM host: mwreview-abogott-test2.pmtpa.wmflabs is DOWN address: 10.4.1.77 CRITICAL - Host Unreachable (10.4.1.77) [00:20:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 7.88, 6.72, 5.59 [00:24:53] RECOVERY host: mwreview-abogott-test2.pmtpa.wmflabs is UP address: 10.4.1.77 PING OK - Packet loss = 0%, RTA = 8.70 ms [00:25:53] PROBLEM Current Load is now: CRITICAL on mwreview-abogott-test2.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [00:25:53] PROBLEM dpkg-check is now: CRITICAL on mwreview-abogott-test2.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [00:26:33] PROBLEM Current Users is now: CRITICAL on mwreview-abogott-test2.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [00:27:13] PROBLEM Disk Space is now: CRITICAL on mwreview-abogott-test2.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [00:28:03] PROBLEM Free ram is now: CRITICAL on mwreview-abogott-test2.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [00:29:23] PROBLEM Total processes is now: CRITICAL on mwreview-abogott-test2.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [00:30:52] RECOVERY Free ram is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK: 92% free memory [00:31:52] RECOVERY Free ram is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: OK: 882% free memory [00:33:22] PROBLEM Free ram is now: WARNING on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: Warning: 19% free memory [00:35:53] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 153 processes [00:36:23] PROBLEM Total processes is now: CRITICAL on aggregator1.pmtpa.wmflabs 10.4.0.79 output: PROCS CRITICAL: 257 processes [00:37:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [00:38:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 28% free memory [00:39:23] RECOVERY Total processes is now: OK on mwreview-abogott-test2.pmtpa.wmflabs 10.4.1.77 output: PROCS OK: 92 processes [00:40:53] RECOVERY Current Load is now: OK on mwreview-abogott-test2.pmtpa.wmflabs 10.4.1.77 output: OK - load average: 0.38, 1.03, 0.81 [00:40:54] RECOVERY dpkg-check is now: OK on mwreview-abogott-test2.pmtpa.wmflabs 10.4.1.77 output: All packages OK [00:41:33] RECOVERY Current Users is now: OK on mwreview-abogott-test2.pmtpa.wmflabs 10.4.1.77 output: USERS OK - 0 users currently logged in [00:42:13] RECOVERY Disk Space is now: OK on mwreview-abogott-test2.pmtpa.wmflabs 10.4.1.77 output: DISK OK [00:43:03] RECOVERY Free ram is now: OK on mwreview-abogott-test2.pmtpa.wmflabs 10.4.1.77 output: OK: 749% free memory [00:45:53] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 149 processes [01:06:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [01:07:43] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 176 processes [01:11:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [01:12:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 100 processes [01:35:52] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [01:40:53] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [01:45:48] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 3.01, 4.05, 4.98 [01:57:24] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [02:11:23] PROBLEM Free ram is now: WARNING on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: Warning: 19% free memory [02:12:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 4.28, 4.63, 4.98 [02:14:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 3.33, 3.64, 4.59 [02:16:23] RECOVERY Free ram is now: OK on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: OK: 20% free memory [02:20:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 3.85, 4.78, 5.07 [02:25:53] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [02:25:54] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 4.99, 4.78, 4.99 [02:29:24] PROBLEM Free ram is now: WARNING on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: Warning: 19% free memory [04:02:23] PROBLEM Free ram is now: WARNING on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: Warning: 19% free memory [04:13:23] PROBLEM Disk Space is now: WARNING on mobile-osm.pmtpa.wmflabs 10.4.0.226 output: DISK WARNING - free space: / 484 MB (5% inode=90%): [04:25:53] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [04:37:23] RECOVERY Free ram is now: OK on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: OK: 34% free memory [04:41:52] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 28% free memory [04:43:22] PROBLEM Disk Space is now: CRITICAL on mobile-osm.pmtpa.wmflabs 10.4.0.226 output: DISK CRITICAL - free space: / 215 MB (2% inode=90%): [04:54:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [05:55:54] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 7% free memory [06:00:23] PROBLEM Free ram is now: WARNING on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: Warning: 19% free memory [06:05:22] RECOVERY Free ram is now: OK on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: OK: 20% free memory [06:15:53] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [06:28:35] PROBLEM Free ram is now: WARNING on changefeed-bot.pmtpa.wmflabs 10.4.0.240 output: Warning: 18% free memory [07:47:22] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 15% free memory [08:25:52] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [08:39:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.91, 4.92, 4.96 [08:39:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 28% free memory [08:41:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [08:49:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [08:57:54] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [09:00:53] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [09:22:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.27, 5.10, 5.04 [09:32:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.68, 4.91, 4.99 [10:35:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.57, 5.23, 5.08 [11:14:29] who can add me ('saper') to 'bugtracker'? [11:15:52] PROBLEM Current Load is now: CRITICAL on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: Connection refused by host [11:16:32] PROBLEM Current Users is now: CRITICAL on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: Connection refused by host [11:17:12] PROBLEM Disk Space is now: CRITICAL on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: Connection refused by host [11:17:19] (need to test https://bugzilla.wikimedia.org/show_bug.cgi?id=32504) [11:17:22] PROBLEM Total processes is now: CRITICAL on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: Connection refused by host [11:18:02] PROBLEM Free ram is now: CRITICAL on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: Connection refused by host [11:18:12] PROBLEM dpkg-check is now: CRITICAL on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: Connection refused by host [11:20:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.35, 4.67, 4.92 [11:25:52] RECOVERY Current Load is now: OK on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: OK - load average: 0.73, 0.94, 0.61 [11:25:53] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [11:26:32] RECOVERY Current Users is now: OK on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: USERS OK - 0 users currently logged in [11:27:12] RECOVERY Disk Space is now: OK on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: DISK OK [11:27:22] RECOVERY Total processes is now: OK on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: PROCS OK: 100 processes [11:28:02] RECOVERY Free ram is now: OK on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: OK: 2852% free memory [11:28:12] RECOVERY dpkg-check is now: OK on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: All packages OK [11:35:53] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [12:37:52] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 20% free memory [12:39:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [12:39:42] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 152 processes [12:40:52] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 7% free memory [12:42:02] PROBLEM Free ram is now: WARNING on newchanges-bot.pmtpa.wmflabs 10.4.0.221 output: Warning: 11% free memory [12:48:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.96, 5.58, 5.24 [12:49:43] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 143 processes [12:57:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [13:05:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [13:05:54] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [13:07:03] PROBLEM Free ram is now: CRITICAL on newchanges-bot.pmtpa.wmflabs 10.4.0.221 output: Critical: 4% free memory [13:16:53] PROBLEM dpkg-check is now: CRITICAL on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: DPKG CRITICAL dpkg reports broken packages [13:21:52] RECOVERY dpkg-check is now: OK on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: All packages OK [13:28:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.81, 4.92, 4.99 [13:55:53] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [14:04:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.33, 5.21, 5.06 [14:12:03] PROBLEM Free ram is now: WARNING on newchanges-bot.pmtpa.wmflabs 10.4.0.221 output: Warning: 8% free memory [14:20:52] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [15:19:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.11, 5.12, 5.06 [15:39:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.88, 4.90, 4.99 [16:07:02] PROBLEM Free ram is now: CRITICAL on newchanges-bot.pmtpa.wmflabs 10.4.0.221 output: Critical: 5% free memory [16:09:37] @seenrx addshore [16:09:37] petan: Last time I saw addshore they were quitting the network with reason: Ping timeout: 255 seconds at 1/17/2013 2:10:44 PM (01:58:53.2867790 ago) (multiple results were found: addshore_, addshore__, addshore7, addshore7_) [16:12:02] PROBLEM Free ram is now: WARNING on newchanges-bot.pmtpa.wmflabs 10.4.0.221 output: Warning: 7% free memory [16:25:54] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [16:33:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.39, 5.14, 5.04 [16:40:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 28% free memory [16:58:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.62, 4.78, 4.93 [17:00:53] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [17:02:13] RECOVERY Free ram is now: OK on newchanges-bot.pmtpa.wmflabs 10.4.0.221 output: OK: 32% free memory [17:03:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [18:21:17] Hi andrewbogott, thanks for fixing the strange errors yesterday. I tested my files again and realized that this stupid xml broke when I shortened it so I uploaded it again. [18:21:44] Silke_WMDE: There is one more bug in the sql puppet class which I'm about to fix [18:22:16] …meanwhile you will need to run this after your first puppet run: [18:22:26] /usr/sbin/dpkg-reconfigure -fnoninteractive mysql-server-5.5 [18:22:35] …and then rerun puppet to finish mediawiki install [18:22:36] :) i did [18:22:55] Shall I append more stuff to my "thread" or wait for it to be merged and continue then? [18:23:17] hm... [18:23:17] I think, this is a working setup. [18:23:38] Since it's just you using this at the moment, it's fine for it to go in as one big patch. But I'm happy to merge sooner if that makes your life easier. [18:23:41] up to you. [18:24:17] Yeah, it would make the workflow easier. [18:24:36] Next week, we want to change to a demo system that is generated by puppet [18:24:56] ok. I will look at your patch after I fix this mysql thing [18:25:43] I will submit the variable changes tomorrow. Did too much other stuff today. :D [18:26:48] nevertheless it would be cool to have the other stuff merged already [18:27:08] * Silke_WMDE is heading home. CU! [19:09:33] petan: Do you know if the search project includes a working solr instance? [19:17:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.17, 5.26, 5.20 [19:25:52] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [19:30:53] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 4% free memory [19:57:23] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [20:27:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 5.11, 4.94, 4.98 [20:35:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.37, 5.17, 5.07 [20:37:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [20:37:23] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 6% free memory [20:38:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 20% free memory [20:40:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.59, 4.86, 4.97 [20:40:52] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 7% free memory [20:48:43] PROBLEM dpkg-check is now: CRITICAL on mwreview-abogott-dev3.pmtpa.wmflabs 10.4.0.205 output: DPKG CRITICAL dpkg reports broken packages [20:50:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [20:57:23] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [20:57:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [21:00:53] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 4% free memory [21:14:23] PROBLEM host: mwang-dev1.pmtpa.wmflabs is DOWN address: 10.4.1.67 CRITICAL - Host Unreachable (10.4.1.67) [21:29:53] PROBLEM host: mwang-proxy.pmtpa.wmflabs is DOWN address: 10.4.0.243 CRITICAL - Host Unreachable (10.4.0.243) [21:33:52] RECOVERY host: mwang-proxy.pmtpa.wmflabs is UP address: 10.4.0.243 PING OK - Packet loss = 0%, RTA = 9.15 ms [21:34:22] PROBLEM Total processes is now: CRITICAL on mwang-proxy.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [21:35:53] PROBLEM Current Load is now: CRITICAL on mwang-proxy.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [21:35:53] PROBLEM dpkg-check is now: CRITICAL on mwang-proxy.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [21:36:33] PROBLEM Current Users is now: CRITICAL on mwang-proxy.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [21:37:13] PROBLEM Disk Space is now: CRITICAL on mwang-proxy.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [21:38:03] PROBLEM Free ram is now: CRITICAL on mwang-proxy.pmtpa.wmflabs 10.4.0.243 output: Connection refused by host [21:38:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 4.95, 5.18, 5.14 [21:39:21] mike_wang: Do you mostly know how to proceed with your nxinx module or do you need more guidance? [21:40:52] PROBLEM dpkg-check is now: CRITICAL on mwreview-dev4.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [21:41:32] PROBLEM Current Load is now: CRITICAL on mwreview-dev4.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [21:41:42] PROBLEM Free ram is now: CRITICAL on mwreview-dev4.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [21:42:12] PROBLEM Current Users is now: CRITICAL on mwreview-dev4.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [21:43:05] PROBLEM Disk Space is now: CRITICAL on mwreview-dev4.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [21:43:12] PROBLEM Total processes is now: CRITICAL on mwreview-dev4.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [21:44:22] RECOVERY Total processes is now: OK on mwang-proxy.pmtpa.wmflabs 10.4.0.243 output: PROCS OK: 88 processes [21:45:54] RECOVERY Current Load is now: OK on mwang-proxy.pmtpa.wmflabs 10.4.0.243 output: OK - load average: 0.13, 0.85, 0.70 [21:45:55] RECOVERY dpkg-check is now: OK on mwang-proxy.pmtpa.wmflabs 10.4.0.243 output: All packages OK [21:46:34] RECOVERY Current Users is now: OK on mwang-proxy.pmtpa.wmflabs 10.4.0.243 output: USERS OK - 1 users currently logged in [21:47:14] RECOVERY Disk Space is now: OK on mwang-proxy.pmtpa.wmflabs 10.4.0.243 output: DISK OK [21:48:04] RECOVERY Free ram is now: OK on mwang-proxy.pmtpa.wmflabs 10.4.0.243 output: OK: 854% free memory [21:51:32] RECOVERY Current Load is now: OK on mwreview-dev4.pmtpa.wmflabs 10.4.1.54 output: OK - load average: 0.88, 1.10, 0.69 [21:51:42] RECOVERY Free ram is now: OK on mwreview-dev4.pmtpa.wmflabs 10.4.1.54 output: OK: 568% free memory [21:52:13] RECOVERY Current Users is now: OK on mwreview-dev4.pmtpa.wmflabs 10.4.1.54 output: USERS OK - 1 users currently logged in [21:53:02] RECOVERY Disk Space is now: OK on mwreview-dev4.pmtpa.wmflabs 10.4.1.54 output: DISK OK [21:53:12] RECOVERY Total processes is now: OK on mwreview-dev4.pmtpa.wmflabs 10.4.1.54 output: PROCS OK: 92 processes [21:55:54] RECOVERY dpkg-check is now: OK on mwreview-dev4.pmtpa.wmflabs 10.4.1.54 output: All packages OK [21:55:54] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [22:04:53] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 154 processes [22:12:25] andrewbogott: I am rewriting the puppet code and will upload it tonight. [22:12:54] mike_wang: ok! I'm not sure what to make of paravoid's suggestion… it seems right but also vastly increases the scope of the problem. [22:13:07] Maybe it's simpler than it looks [22:14:53] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 149 processes [22:15:52] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [22:26:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.39, 5.29, 5.14 [22:45:23] Ryan_Lane, did you dig up anything about why we can't create accounts of nova-precise2? [22:45:30] oh. nope [22:45:34] did you enable the ldap log? [22:46:18] does this account exist? uid=novaadmin,ou=people,dc=wikimedia,dc=org [22:46:45] we should probably add an ldif file for adding test entries [22:46:51] my theory is puppet should make this happen like magic [22:47:17] I agree [22:47:59] actually... [22:48:09] it would actually be interesting to have jenkins say spin up an openstack box, unit test osm against the api for real and then just mock the calls normally... dunno how you'd do that in php but in python it's awesome and simple [22:48:29] Hm… lots of 'Failed to start TLS' in the ldap log [22:48:35] oh [22:48:58] use clear for entryption type [22:49:05] encryption [22:49:12] just make sure to use fake passwords [22:49:58] so, you can just populate the local ldap with production's entries [22:50:27] we have a 2000 entry limit and it seems we've over that [22:50:34] but you can search each base at a time [22:50:43] or can use paging [22:50:54] Is it useful to have more than two or three accounts there? [22:51:02] urgh ldap [22:51:04] paging is evil [22:51:07] stupid limit [22:51:17] it's nice to have a full set of test data [22:51:54] or just point it to production *snigger* [22:52:07] ldapsearch -E pr=1000 -LLL -x -D 'cn=proxyagent,ou=profile,dc=wikimedia,dc=org' -W [22:52:23] that'll give you every entry [22:54:02] actually.... [22:54:05] use this: ldapsearch -E pr=1000/noprompt -LLL -x -D 'cn=proxyagent,ou=profile,dc=wikimedia,dc=org' -W + \* [22:54:17] + means show operational attributes [22:54:24] \* means show all other attributes too [22:54:31] -E means use extended operations [22:54:36] pr <— paged results [22:54:55] 1000/noprompt says search for 1000 at a time and don't ask me to continue [22:55:54] I totally just grep the user/pass out the ldap config :) [22:56:05] eh? [22:56:16] oh [22:56:24] yeah. that works too ;) [22:59:33] kinda ironic we require auth to search but leave a set of user details so open... [23:00:42] Lots of things like "Entry ou=netgroup,dc=wikimedia,dc=org cannot be added because it includes attribute subschemaSubentry which is defined as NO-USER-MODIFICATION in the server schema" [23:14:21] https://i.chzbgr.com/maxW500/6952352000/h94A26640/ [23:36:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.62, 4.82, 4.97 [23:44:53] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 151 processes [23:47:59] yep [23:48:05] !access | Wikinaut [23:48:06] Wikinaut: https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [23:48:23] read that [23:48:37] no WinSCP way ? [23:48:42] assuming putty has the ability to use proxy-command, winscp should be able to scp directly to it [23:48:46] no PuTTY way ? [23:48:52] ok [23:48:54] I rely on windows users to write windows docs [23:48:58] I don't have windows [23:49:02] okok [23:49:05] Let's try [23:49:14] together, pls [23:49:14] someone did write some putty docs [23:49:14] I'll document it [23:49:26] https://labsconsole.wikimedia.org/wiki/Help:Putty [23:49:34] Ryan_Lane: install wine? [23:49:40] I use putty [23:49:43] works great [23:49:48] most people don't want to install wine [23:49:53] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 150 processes [23:50:03] Vacation9: how do you handle scp to instances behind bastion? [23:50:16] the scp command? [23:50:19] Vacation9: would you mind updating https://labsconsole.wikimedia.org/wiki/Help:Putty ? [23:50:33] do you set up proxycommand in putty? [23:50:35] using a ssh proxy is bleh [23:50:39] How can I run a command, e.g. tar [23:50:51] do this: ssh (instance) [23:50:57] then you are in that instance [23:51:01] from bastion [23:51:10] uh, I remember [23:51:11] right, but he wants to scp files directly to that instance [23:51:26] Wikinaut: the instance name is openid-wiki.pmtpa.wmflabs [23:51:29] then scp (file name) (instance):(destination path) [23:51:33] I know how you can rsync inline... dunno about scp [23:51:43] that's the command in PuTTy [23:51:50] Damianz: yep, if you use proxycommand it'll scp all the way through [23:52:10] I don't even use proxycommand... just bang the command after ssh and it works (tm) [23:52:25] I need a "How-to-use-labs-instances-with-WinSCP-and-PuTTY-tl;dr" [23:52:28] Damianz: same [23:52:56] a) re-install to ubuntu b) open terminal # damian's how to use puppet guide [23:52:57] Wikinaut: connect to bastion, ssh (instance), you can run commands [23:53:14] Wikinaut or, connect to bastion, and use my scp command above [23:53:15] Vacation9: I remember that I did this 2 years ago [23:53:26] Wikinaut: https://labsconsole.wikimedia.org/wiki/Help:Putty [23:53:32] And, how can I run a program ? [23:53:41] run a program? [23:53:44] what do you mean? [23:53:45] php update.php [23:53:48] for example [23:53:52] after you ssh from bastion to the instance just say the command [23:53:55] tar -xzf ... [23:53:59] ok [23:54:02] ssh [23:54:02] ssh to bastion, with a forwarded agent, then ssh to the instance [23:54:09] yeah - I do remember [23:54:12] after you ssh from bastion to the instance all commands are sent to the instance [23:54:17] yes [23:54:18] yes [23:54:30] it works perfectly [23:54:43] let me ask, why is this so complicated [23:54:45] compared to other VMs [23:54:47] because we don't have many public IP addresses [23:54:50] ok [23:55:02] I NOW understand [23:55:23] will this change with ip6 ? [23:55:40] all instances will have a public ipv6 address by default, yes [23:56:02] ETA ? [23:56:14] after we bring up the zone in eqiad [23:56:18] probably a few months [23:56:19] really ? [23:56:24] ok [23:56:29] nice to read [23:56:41] that's all for today, I think [23:56:42] I don't think ipv6 is going to make things easier ;) [23:57:45] Vacation9: Ryan_Lane just a final question, as far as I understand, for file transfers I only can use scp (via bastion), and no way of the (nice) WinSCP [23:57:52] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 152 processes [23:57:55] you know what would make it easier? GRE tunnel from my router to wikimedia's network :D lololol [23:58:01] Damianz: :D [23:58:23] Wikinaut: I honestly don't know. I don't use windows [23:58:48] WinSCP is a two panel GUI for file transfers [23:59:06] (uses SCP) [23:59:14] winscp uses putty in the background afaik [23:59:19] yes [23:59:32] I use Filezilla [23:59:32] I get that, but I don't know if there's a way to make it transfer directly to the instance [23:59:44] does this work with bastion / instances [23:59:52] it'll work with bastion, yes [23:59:56] Ryan_Lane: I don't think there is, I sftp it to bastion then scp it to my instance