[00:00:05] yw [00:00:18] it's possible the community will complain about this ;) [00:00:27] naturally :) [00:00:28] if that's the case you'll need to defend it [00:00:50] I'll need to do whatever the community decides, if they do complain ;) [00:02:29] thats all I would expect of you ;p [00:09:25] Ryan_Lane, proposals has no admins :) [00:09:32] no? [00:09:38] what's your labsconsole username? [00:09:49] addshore :P [00:10:11] 3 members, no admins [00:10:15] one sec [00:10:37] ooohh [00:10:44] you're looking at the resource page [00:10:57] that doesn't currently display the admins [00:11:03] that's an open bug [00:11:07] Hah! [00:11:26] you can use all of the related openstack interfaces, though [00:11:35] "Manage projects" should show the correct membership [00:12:16] >.< [00:12:50] I see what the problem was, The project filter option had unselected proposals when it was added, so I couldnt see it :) [00:13:54] ah, yes [00:39:22] PROBLEM Total processes is now: CRITICAL on proposals-wikiessentials.pmtpa.wmflabs 10.4.1.75 output: Connection refused by host [00:40:53] PROBLEM Current Load is now: CRITICAL on proposals-wikiessentials.pmtpa.wmflabs 10.4.1.75 output: Connection refused by host [00:40:53] PROBLEM dpkg-check is now: CRITICAL on proposals-wikiessentials.pmtpa.wmflabs 10.4.1.75 output: Connection refused by host [00:41:33] PROBLEM Current Users is now: CRITICAL on proposals-wikiessentials.pmtpa.wmflabs 10.4.1.75 output: Connection refused by host [00:42:13] PROBLEM Disk Space is now: CRITICAL on proposals-wikiessentials.pmtpa.wmflabs 10.4.1.75 output: Connection refused by host [00:43:03] PROBLEM Free ram is now: CRITICAL on proposals-wikiessentials.pmtpa.wmflabs 10.4.1.75 output: Connection refused by host [00:45:53] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 144 processes [01:06:44] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 183 processes [01:10:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.75, 4.91, 4.99 [01:11:42] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 108 processes [01:23:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 4.80, 5.03, 5.03 [01:32:42] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 52% free memory [01:43:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.67, 4.86, 4.99 [01:49:33] Ryan_Lane, please could proposals have 1 IP please? [01:49:52] PROBLEM host: proposals-apache01.pmtpa.wmflabs is DOWN address: 10.4.1.77 CRITICAL - Host Unreachable (10.4.1.77) [01:49:53] PROBLEM host: proposals-mysql1.pmtpa.wmflabs is DOWN address: 10.4.1.75 CRITICAL - Host Unreachable (10.4.1.75) [01:50:11] addshore: yep. one sec [01:50:32] addshore: done [01:50:37] ty [01:53:52] RECOVERY host: proposals-apache01.pmtpa.wmflabs is UP address: 10.4.1.77 PING OK - Packet loss = 0%, RTA = 0.78 ms [01:54:22] PROBLEM Total processes is now: CRITICAL on proposals-apache01.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [01:55:36] Ryan_Lane, is it normal to get Permission denied (publickey). [01:55:36] when first trying to connect to a new instance? [01:55:52] PROBLEM Current Load is now: CRITICAL on proposals-apache01.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [01:55:52] PROBLEM dpkg-check is now: CRITICAL on proposals-apache01.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [01:55:55] it likely isn't finished building [01:56:11] it takes a while for puppet to finish running [01:56:32] PROBLEM Current Users is now: CRITICAL on proposals-apache01.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [01:57:12] PROBLEM Disk Space is now: CRITICAL on proposals-apache01.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [01:58:02] PROBLEM Free ram is now: CRITICAL on proposals-apache01.pmtpa.wmflabs 10.4.1.77 output: Connection refused by host [02:00:13] RECOVERY host: proposals-mysql1.pmtpa.wmflabs is UP address: 10.4.1.75 PING OK - Packet loss = 0%, RTA = 3.20 ms [02:00:53] PROBLEM Current Load is now: CRITICAL on proposals-mysql1.pmtpa.wmflabs 10.4.1.75 output: Connection refused by host [02:01:34] PROBLEM Current Users is now: CRITICAL on proposals-mysql1.pmtpa.wmflabs 10.4.1.75 output: Connection refused by host [02:01:34] PROBLEM dpkg-check is now: CRITICAL on proposals-mysql1.pmtpa.wmflabs 10.4.1.75 output: Connection refused by host [02:02:13] RECOVERY Disk Space is now: OK on proposals-apache01.pmtpa.wmflabs 10.4.1.77 output: DISK OK [02:02:14] PROBLEM Disk Space is now: CRITICAL on proposals-mysql1.pmtpa.wmflabs 10.4.1.75 output: Connection refused by host [02:03:03] PROBLEM Free ram is now: CRITICAL on proposals-mysql1.pmtpa.wmflabs 10.4.1.75 output: Connection refused by host [02:03:03] RECOVERY Free ram is now: OK on proposals-apache01.pmtpa.wmflabs 10.4.1.77 output: OK: 1047% free memory [02:03:53] perfect :) [02:04:05] shame I slightly missnamed one of the instances [02:04:23] RECOVERY Total processes is now: OK on proposals-apache01.pmtpa.wmflabs 10.4.1.77 output: PROCS OK: 88 processes [02:04:24] PROBLEM Total processes is now: CRITICAL on proposals-mysql1.pmtpa.wmflabs 10.4.1.75 output: Connection refused by host [02:05:40] !log proposals addshore: Created and Updated proposals-apache01 and proposals-mysql1 [02:05:42] Logged the message, Master [02:05:52] RECOVERY Current Load is now: OK on proposals-apache01.pmtpa.wmflabs 10.4.1.77 output: OK - load average: 0.79, 0.81, 0.62 [02:06:32] RECOVERY Current Users is now: OK on proposals-apache01.pmtpa.wmflabs 10.4.1.77 output: USERS OK - 1 users currently logged in [02:13:03] RECOVERY Free ram is now: OK on proposals-mysql1.pmtpa.wmflabs 10.4.1.75 output: OK: 686% free memory [02:14:23] RECOVERY Total processes is now: OK on proposals-mysql1.pmtpa.wmflabs 10.4.1.75 output: PROCS OK: 88 processes [02:15:53] RECOVERY dpkg-check is now: OK on proposals-apache01.pmtpa.wmflabs 10.4.1.77 output: All packages OK [02:15:54] RECOVERY Current Load is now: OK on proposals-mysql1.pmtpa.wmflabs 10.4.1.75 output: OK - load average: 0.09, 0.75, 0.68 [02:16:33] RECOVERY Current Users is now: OK on proposals-mysql1.pmtpa.wmflabs 10.4.1.75 output: USERS OK - 1 users currently logged in [02:16:34] RECOVERY dpkg-check is now: OK on proposals-mysql1.pmtpa.wmflabs 10.4.1.75 output: All packages OK [02:17:13] RECOVERY Disk Space is now: OK on proposals-mysql1.pmtpa.wmflabs 10.4.1.75 output: DISK OK [02:24:33] PROBLEM dpkg-check is now: CRITICAL on proposals-mysql1.pmtpa.wmflabs 10.4.1.75 output: DPKG CRITICAL dpkg reports broken packages [02:31:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.72, 5.30, 5.24 [02:34:39] !log proposals apache01 installed apache2 and php5, rebooting [02:39:20] i see [02:48:55] dont appear to be able to log into labsconsole - I am given a lovely error about having cookied disabled O_o [03:19:19] really? [03:19:27] I'll see if I have the same problem [03:21:34] addshore: It's supppper slow for me [03:22:25] addshore: Getting the same error... [03:23:53] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 152 processes [03:33:52] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 144 processes [04:06:53] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 151 processes [04:09:35] hmmm... addshore... [04:09:49] xD [04:40:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 27% free memory [04:41:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 23% free memory [04:58:52] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [05:04:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [05:19:52] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 152 processes [06:28:35] PROBLEM Total processes is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS WARNING: 151 processes [06:29:55] PROBLEM Total processes is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS WARNING: 152 processes [06:33:32] RECOVERY Total processes is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS OK: 147 processes [06:38:54] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 31% free memory [06:54:52] RECOVERY Total processes is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: PROCS OK: 148 processes [07:46:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 4.46, 4.25, 4.92 [08:39:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [08:46:54] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [08:47:24] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 19% free memory [12:05:34] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 10% free memory [12:10:32] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 130% free memory [12:37:22] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 23% free memory [12:38:32] PROBLEM Free ram is now: CRITICAL on bots-4.pmtpa.wmflabs 10.4.0.64 output: Critical: 5% free memory [13:05:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 13% free memory [13:38:52] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 152 processes [13:57:53] RECOVERY Free ram is now: OK on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: OK: 29% free memory [14:23:53] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 147 processes [16:41:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 27% free memory [16:54:01] whats up with the labconsole? [16:54:25] I mean the site. [16:54:33] instance seems to be working fin e [16:57:47] Jyothis, lemme guess - it's slow and logs you out? [16:58:50] Its slow, missing the left toolbox we had before, doesnt show my instances on the project, doesnt let me add or remove any thing. My fellow project memeber is not able to add keys and the argument continues? :) [16:59:31] it's a recurring memcached bug [17:00:02] Ah [17:00:17] any idea when we will resume the normal life? :) [17:00:55] when Ryan restarts it;) [17:01:09] heh [17:01:19] * Jyothis presses the large red button [17:02:42] Jyothis, congratulations! Now that your nuclear missiles have destroyed a few small counries nobody cares about anyway, whom else do you want to destroy? [17:03:52] Jyothis, congratulations! Now that your nuclear missiles have destroyed a few small counries nobody cares about anyway, whom else do you want to destroy? [17:09:52] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [17:09:56] MaxSem: lol [17:16:46] MaxSem: When puppet restarts it ;) [17:18:31] puppet doesn't restart services it doesn't change [18:20:35] Another user and I are unable to see the captcha in the account creation screen. Is that system broken at the moment? [18:20:53] The whole labs site is being slow to load [18:22:56] Pine: Yes, the site is extremely slow for me. Yesterday I couldn't even log in. [18:23:20] robla: ^ is the labs team aware of this? [18:24:12] I think petan is [19:43:45] Labsconsole isn't letting me log in... [19:43:59] Krenair, been like that for nearly 24 hours now :/ [19:45:11] and everything is going rather slowly [19:47:52] PROBLEM Total processes is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS WARNING: 154 processes [20:00:58] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 9% free memory [20:02:52] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 147 processes [20:15:54] RECOVERY Free ram is now: OK on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: OK: 20% free memory [20:23:53] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 18% free memory [20:39:53] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 27% free memory [20:40:24] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 23% free memory [20:52:53] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 8% free memory [21:03:22] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [21:51:53] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 17% free memory