[00:08:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:10:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:14:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:16:33] PROBLEM Total processes is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: PROCS WARNING: 152 processes [00:38:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:41:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:44:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:03:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 19% free memory [01:06:54] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 173 processes [01:08:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:11:16] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:14:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:16:54] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 95 processes [01:38:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:41:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:44:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:03:44] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 25% free memory [02:08:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:11:15] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:14:36] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:38:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:41:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:44:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:51:26] PROBLEM Free ram is now: CRITICAL on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Critical: 5% free memory [02:56:35] RECOVERY Free ram is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK: 24% free memory [03:04:23] PROBLEM Free ram is now: CRITICAL on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Critical: 2% free memory [03:08:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:11:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:14:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:38:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:42:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:46:53] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:02:46] PROBLEM Free ram is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: Warning: 19% free memory [04:07:42] RECOVERY Free ram is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK: 20% free memory [04:09:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:12:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:16:56] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:39:14] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:42:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:45:55] PROBLEM Current Load is now: WARNING on deployment-jobrunner06 i-0000031d.pmtpa.wmflabs output: WARNING - load average: 4.45, 5.13, 5.03 [04:47:05] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:10:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:13:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:17:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:27:23] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Warning: 10% free memory [05:37:32] RECOVERY Free ram is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK: 20% free memory [05:41:37] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:43:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:45:24] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Warning: 15% free memory [05:48:04] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:11:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:13:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:19:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:29:24] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:35:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [06:41:45] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:44:05] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:45:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [06:48:52] PROBLEM dpkg-check is now: CRITICAL on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [06:49:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:49:22] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:50:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [06:53:54] RECOVERY dpkg-check is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: All packages OK [07:11:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:14:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:19:24] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:41:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:44:04] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:49:36] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:52:30] hi, i need an help [07:54:23] * Damianz gives you 1 help [07:57:25] Damianz: thanks for the reply [07:57:33] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Warning: 18% free memory [07:57:34] i thought no one is here :) [07:58:23] i want to test a few tools on bengali wikipedia [07:58:38] there is an instance available at http://bn.wikipedia.beta.wmflabs.org/ [07:59:29] but what i want is a kind of replica of Bengali wikipedia [08:00:02] all setting and gadgets but all the contensts re not required [08:00:12] contents are not required [08:00:36] so how can i get that?? [08:11:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:14:35] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:17:32] RECOVERY Free ram is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK: 37% free memory [08:19:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:41:56] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:43:53] PROBLEM Current Load is now: CRITICAL on eventstream i-000004fa.pmtpa.wmflabs output: Connection refused by host [08:44:33] PROBLEM Current Users is now: CRITICAL on eventstream i-000004fa.pmtpa.wmflabs output: Connection refused by host [08:44:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:45:15] PROBLEM Disk Space is now: CRITICAL on eventstream i-000004fa.pmtpa.wmflabs output: Connection refused by host [08:46:04] PROBLEM Free ram is now: CRITICAL on eventstream i-000004fa.pmtpa.wmflabs output: Connection refused by host [08:47:24] PROBLEM Total processes is now: CRITICAL on eventstream i-000004fa.pmtpa.wmflabs output: Connection refused by host [08:47:54] PROBLEM dpkg-check is now: CRITICAL on eventstream i-000004fa.pmtpa.wmflabs output: Connection refused by host [08:49:34] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:52:23] RECOVERY Total processes is now: OK on eventstream i-000004fa.pmtpa.wmflabs output: PROCS OK: 84 processes [08:52:53] RECOVERY dpkg-check is now: OK on eventstream i-000004fa.pmtpa.wmflabs output: All packages OK [08:53:53] RECOVERY Current Load is now: OK on eventstream i-000004fa.pmtpa.wmflabs output: OK - load average: 0.22, 0.84, 0.68 [08:54:35] RECOVERY Current Users is now: OK on eventstream i-000004fa.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [08:55:12] RECOVERY Disk Space is now: OK on eventstream i-000004fa.pmtpa.wmflabs output: DISK OK [08:56:02] RECOVERY Free ram is now: OK on eventstream i-000004fa.pmtpa.wmflabs output: OK: 1052% free memory [09:12:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:15:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:19:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:42:05] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:45:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:49:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:12:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:15:25] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:19:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:28:20] [bz] (8NEW - created by: 2Ori Livneh, priority: 4Unprioritized - 6normal) [Bug 41622] Unable to create and initialize home directory - https://bugzilla.wikimedia.org/show_bug.cgi?id=41622 [10:43:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:45:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:49:46] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:05:43] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 19% free memory [11:10:45] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 20% free memory [11:13:45] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:15:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:19:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:44:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:45:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:49:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:53:44] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 18% free memory [12:06:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 18% free memory [12:09:13] @infobot-detail docs [12:09:14] Info for docs: this key was created at N/A by N/A, this key was displayed 1 time(s), last time at 10/11/2012 12:26:15 PM (20.23:42:58.0940160 ago) [12:14:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:15:13] !log bots upgrading web server [12:15:15] Logged the message, Master [12:15:44] !log bots expect web server outage [12:15:45] Logged the message, Master [12:16:06] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:17:17] Damianz can you do something about nagios hostnames [12:17:32] I really hate to read the table of hosts which is showing real names instead of cute names [12:17:54] in past it was alias=cute hostname=host [12:17:59] now it's other way round [12:18:14] so it display ugly name in tables [12:19:00] !stats [12:19:19] @search gangli [12:19:19] Results (Found 2): load, load-all, [12:19:25] !load-all [12:19:26] http://ganglia.wikimedia.org/2.2.0/?c=Virtualization%20cluster%20pmtpa&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [12:19:32] PROBLEM dpkg-check is now: CRITICAL on bots-apache1 i-00000450.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [12:21:35] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:24:33] RECOVERY dpkg-check is now: OK on bots-apache1 i-00000450.pmtpa.wmflabs output: All packages OK [12:30:22] PROBLEM Disk Space is now: WARNING on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK WARNING - free space: /export 1027 MB (5% inode=56%): /home/SAVE 1027 MB (5% inode=56%): [12:35:02] PROBLEM host: i-000004fb.pmtpa.wmflabs is DOWN address: i-000004fb.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000004fb.pmtpa.wmflabs) [12:36:53] PROBLEM dpkg-check is now: CRITICAL on glam-gwtoolset i-000004b1.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [12:41:52] RECOVERY dpkg-check is now: OK on glam-gwtoolset i-000004b1.pmtpa.wmflabs output: All packages OK [12:42:36] I am unable to build instances :/ [12:42:44] @seen mutante [12:42:45] petan: Last time I saw mutante they were talking in the channel, they are still in the channel #wikimedia-operations at 11/1/2012 12:12:54 AM (12:29:50.7619960 ago) [12:43:02] paravoid: ping [12:45:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:46:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:48:52] PROBLEM Current Load is now: CRITICAL on bots-apache01 i-000004fc.pmtpa.wmflabs output: Connection refused by host [12:49:32] PROBLEM Current Users is now: CRITICAL on bots-apache01 i-000004fc.pmtpa.wmflabs output: Connection refused by host [12:50:13] PROBLEM Disk Space is now: CRITICAL on bots-apache01 i-000004fc.pmtpa.wmflabs output: Connection refused by host [12:51:05] PROBLEM Free ram is now: CRITICAL on bots-apache01 i-000004fc.pmtpa.wmflabs output: Connection refused by host [12:52:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:52:23] PROBLEM Total processes is now: CRITICAL on bots-apache01 i-000004fc.pmtpa.wmflabs output: Connection refused by host [12:52:53] PROBLEM dpkg-check is now: CRITICAL on bots-apache01 i-000004fc.pmtpa.wmflabs output: Connection refused by host [12:58:55] RECOVERY Current Load is now: OK on bots-apache01 i-000004fc.pmtpa.wmflabs output: OK - load average: 1.04, 1.25, 0.86 [12:59:32] RECOVERY Current Users is now: OK on bots-apache01 i-000004fc.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [13:00:12] RECOVERY Disk Space is now: OK on bots-apache01 i-000004fc.pmtpa.wmflabs output: DISK OK [13:01:05] RECOVERY Free ram is now: OK on bots-apache01 i-000004fc.pmtpa.wmflabs output: OK: 1247% free memory [13:01:42] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 8% free memory [13:02:22] RECOVERY Total processes is now: OK on bots-apache01 i-000004fc.pmtpa.wmflabs output: PROCS OK: 99 processes [13:05:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [13:10:34] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [13:16:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:16:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:23:05] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:30:42] !log wikidata-dev wikidata-dev-2: ULS extension had lost it's branch and was accidentically skipped during yesterday's update. Updated it manually. [13:30:44] Logged the message, Master [13:46:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:46:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:54:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:57:53] RECOVERY dpkg-check is now: OK on bots-apache01 i-000004fc.pmtpa.wmflabs output: All packages OK [14:06:44] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 23% free memory [14:16:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:16:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:23:32] jan_luca, Silke_MWDE_: This is turning out to be a pain, sorry about the delay in my puppet refactor. [14:24:16] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:36:52] PROBLEM dpkg-check is now: UNKNOWN on aggregator-test1 i-000002bf.pmtpa.wmflabs output: Invalid host name i-000002bf.pmtpa.wmflabs [14:40:53] PROBLEM Free ram is now: UNKNOWN on dumps-bot1 i-000003ed.pmtpa.wmflabs output: NRPE: Call to fork() failed [14:42:04] PROBLEM dpkg-check is now: CRITICAL on aggregator-test1 i-000002bf.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [14:43:53] PROBLEM Total processes is now: UNKNOWN on dumps-bot1 i-000003ed.pmtpa.wmflabs output: NRPE: Call to fork() failed [14:45:56] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:46:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:46:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:47:36] PROBLEM Disk Space is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:47:53] PROBLEM SSH is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Server answer: [14:48:33] PROBLEM dpkg-check is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:48:43] PROBLEM Current Users is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:48:53] PROBLEM Total processes is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:51:00] PROBLEM Current Load is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:54:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:55:52] RECOVERY Current Load is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK - load average: 0.59, 1.23, 0.94 [14:55:52] RECOVERY Free ram is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK: 69% free memory [14:57:32] RECOVERY Disk Space is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: DISK OK [14:57:54] RECOVERY SSH is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [14:58:32] RECOVERY dpkg-check is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: All packages OK [14:58:42] RECOVERY Current Users is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [14:58:52] RECOVERY Total processes is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: PROCS OK: 125 processes [15:04:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [15:05:42] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 18% free memory [15:14:14] !log apache server reinstalled now called bots-apache01 [15:14:14] apache is not a valid project. [15:14:34] !log apache server now has 4gb of ram, everything is in puppet except for public_html [15:14:35] apache is not a valid project. [15:14:48] !log bots apache server now has 4gb of ram, everything is in puppet except for public_html [15:14:48] Logged the message, Master [15:16:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:16:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:17:14] !log wikidata-dev wikidata-dev-2/wikidata-dev-3: added bash script /usr/local/bin/update-demo-system.sh that can automate deletion and reimport of test data (chemical elements), and git pull for MW core and all extensions. [15:17:15] Logged the message, Master [15:18:38] Hi andrewbogott! What's the status on a switchable LocalSettings file? Have you had time to look into that? [15:19:19] Silke_WMDE_: It's turning out to be a pain because of puppet's terrible handling of array concatenation. [15:19:38] meh [15:19:43] I'm writing a downscaled version which I should have together in an hour or so… I'm not positive it will get you what you need, but probably... [15:20:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 20% free memory [15:24:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:29:35] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [15:36:46] Silke_WMDE_: This is pretty trivial but maybe it will do what we need for your patch? https://gerrit.wikimedia.org/r/#/c/31252/ [15:43:46] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 19% free memory [15:45:38] andrewbogott: o_O Aha, thx! It will take a moment to figure it out... [15:46:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:46:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:48:20] hi, i am Nasir, i need the Admin right in http://bn.wikipedia.beta.wmflabs.org, can anyone tell me where should i ask [15:49:41] Damianz: ^ can you help nasir8891? [15:50:00] nasir8891: This should be the right place: http://deployment.wikimedia.beta.wmflabs.org/wiki/Global_Requests [15:50:08] oh! [15:50:10] thanks Jan__ [15:50:44] sumanah: No problem [15:50:50] petan: ^ maybe you can help nasir8891? [15:51:09] Jan_Luca: thank [15:51:50] nasir8891, sumanah: Normally Hydriz seems to be the right contract [15:52:01] oh [15:52:18] sj [15:52:31] Jan_Luca: i need an another think, which is the http://bn.wikipedia.beta.wmflabs.org should be the exact replica of the main Bengali Wikipedia [15:53:01] Jan_Luca: should i ask Hydriz for this too? [15:54:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:54:40] nasir8891: it would be good for you to be able to enumerate the differences between the Bengali Wikipedia and the current simulation at http://bn.wikipedia.beta.wmflabs.org [15:54:51] nasir8891: on the main page (http://deployment.wikimedia.beta.wmflabs.org/wiki/Main_Page) there is the note: "Only a few wikis have imported data so far; other wikis will be imported later. If you want to import content to a wiki, ask for the proper permissions (administrator) on the Global requests page (permissions are usually given to anyone who asks for them)." [15:54:51] nasir8891: you should probably file a bug in bugzilla about that [15:55:06] missing extensions and gadgets especially [15:55:54] sumanah: IMHO the beta-cluster should have the same extensions like the production one because there are the changes test. [15:56:08] Or is that wrong? [15:56:40] Jan_Luca: it SHOULD have the same extensions! but sometimes it does not because there has been some kind of error. [15:56:53] Or because things have diverged and need to be brought back into sync. [15:57:01] Jan_Luca: i just need the setting, gadgets and extensions not all the contesnts [15:57:56] the settings and extenions should be the same but the gadgets you have to copy as far as I know [15:59:11] Jan_Luca: ok. let me check then. thank [15:59:17] sumanah: thank you [15:59:59] you're welcome! thank YOU for working on this [16:00:07] Jan_Luca: sumanah oh. i might knock again if i need any other help [16:00:36] :) ok [16:02:14] Silke_WMDE_: Nice SPIEGEL-article about your project :-) [16:02:51] yeah [16:04:38] Jan_Luca: do you use the beta cluster yourself? [16:05:08] not really [16:05:25] I want to test my CentralAuth-changes when they are merged [16:16:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:16:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:24:14] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:25:24] csteipp- Have a few minutes to help me figure something out with the configuration for beta? [16:25:52] anomie: in a meeting atm... in a about 15 mins? [16:25:56] ok [16:47:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:47:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:54:13] hashar- You back, or not really? [16:54:14] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:56:43] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 19% free memory [16:56:54] anomie: not really [16:57:03] anomie: merely checking Jenkins that gone wild again [16:57:12] anomie: if you get some questions please ask :-] [16:58:28] Well, I'm looking at the actual beta config and there's a bunch of uncommitted changes in there. So is the procedure just "make changes in parallel to that and a clean local copy", or is there a less insane way? [16:59:45] that is a bit messy indeed [17:01:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 155 processes [17:02:48] anomie: I can't run git status in the dir :/ [17:03:31] anomie: the most important file is wikiversions.dat [17:03:38] the production and labs versions are indeed different [17:03:51] I think we have a backup in labs, something like wikiversions.labs [17:04:03] the other hack is wmf-config/extension-list [17:04:15] <^demon> If you want to clear the clone so you can work in a clean copy (but not totally trash people's work forever), use git stash. [17:04:18] anything else is probably someone who did a live hack and did not commit [17:04:42] ^demon: that breaks beta since it stashes wikiversions.dat and bring it back to production version [17:04:43] :/ [17:05:13] Hey anomie, I'm back :) [17:05:55] ^demon: jenkins has a huge issue I can't really investigate :/ [17:06:03] ^demon: the update.php just eat all memory :( [17:06:32] csteipp- Good! If only I could stay connected... See the question I asked hashar just above. [17:07:17] Ah, that.... [17:08:22] anomie: Yeah, for wikivoyage, we did make a few live hacks, but everything *except* the .dblist and wikiversions.dat should be merged in git [17:09:03] Until labs has its own dblist files, we can't sync them with the production cluster, so we can't merge them in gerrit [17:10:21] csteipp- I see we already have all-wmflabs.dblist. It's just a matter of the rest of them? [17:11:31] anomie: If labs is actually using thta instaed of all.dblist, then yep, it's just a matter of converting the others over. [17:13:09] I am out again sorry [17:13:20] will take care of gallium later on. [17:16:42] csteipp- It looks like s*.dblist and wik*.dblist are generated by refresh-dblist, and the rest are maintained manually? I also see we have wikiversions.dat in git, and wikiversions.labs on beta; I guess wikiversions.cdb should ideally be generated from wikiversions.labs on labs? How is that generated? [17:17:06] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:17:29] Ah.. there is a script for that... [17:17:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:17:43] <^demon> wikiversions.cdb is generated by refreshWikiversions I believe. [17:18:31] <^demon> And yes, the s*.dblist and wiki*.dblist are done by refresh-dblist based on all.dblist. Other special things (eg: closed|private.dblist) are done by hand. [17:18:47] That sounds about right, and it's not in my history still [17:19:36] Where does refreshWikiversions live? [17:21:14] <^demon> $ which refreshWikiversionsCDB [17:21:14] <^demon> /usr/local/bin/refreshWikiversionsCDB [17:22:20] operations/mediawiki-multiversion, looks like [17:22:59] <^demon> Ah, yes. [17:23:01] <^demon> Sorry :) [17:23:44] No problem. When I asked, I actually meant what you answered. But then I thought "oh, is it in $PATH?" and found it from there [17:24:24] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [17:29:13] Damianz: Jan_Luca - If you'd like to get to watch what used to be a WMF-only meeting, join #wikimedia-metrics-meetings for the monthly meeting - more information: https://meta.wikimedia.org/wiki/Metrics_and_activities_meetings . It'll be a YouTube stream [17:31:33] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Warning: 18% free memory [17:35:13] whoops, I mean #wikimedia-office [17:41:24] RECOVERY Free ram is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK: 20% free memory [17:48:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:48:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:51:48] I am trying to add Tomasz to one of my projects (analytics). He shows up as a member under "Manage Projects" but not under "Members" at the right on the project page: https://labsconsole.wikimedia.org/wiki/Nova_Resource:Analytics [17:51:58] more importantly, he cannot log in [17:52:08] the server says "failed to create home directory" [17:52:11] halp. [17:54:26] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:10:22] PROBLEM Disk Space is now: CRITICAL on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK CRITICAL - free space: /export 516 MB (2% inode=56%): /home/SAVE 516 MB (2% inode=56%): [18:18:57] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:19:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:24:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:35:13] PROBLEM Current Load is now: WARNING on deployment-dbdump i-000000d2.pmtpa.wmflabs output: WARNING - load average: 5.02, 5.11, 5.01 [18:40:16] RECOVERY Current Load is now: OK on deployment-dbdump i-000000d2.pmtpa.wmflabs output: OK - load average: 5.00, 5.05, 5.00 [18:48:56] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:49:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:54:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:03:16] PROBLEM Current Load is now: WARNING on deployment-dbdump i-000000d2.pmtpa.wmflabs output: WARNING - load average: 7.47, 6.57, 5.70 [19:07:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [19:08:15] <^demon> Ryan_Lane: https://gerrit.wikimedia.org/r/#/c/31300/ - we're moving all the replication to the wikimedia github account instead. [19:08:30] ok [19:09:55] <^demon> I'm already on manganese. Let me know when it's on sockpuppet and I can run puppet. [19:11:18] oh [19:11:24] I already did it [19:11:29] <^demon> No worries :) [19:12:06] <^demon> 1031 replication tasks :p [19:16:22] heh [19:18:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:19:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:23:10] Ryan_Lane: I am trying to add Tomasz to one of my projects (analytics). He shows up as a member under "Manage Projects" but not under "Members" at the right on the project page: https://labsconsole.wikimedia.org/wiki/Nova_Resource:Analytics [19:23:27] ah [19:23:27] more importantly, he cannot log in. server says "failed to create home directory". thoughts? [19:23:37] which instance? [19:23:46] we switched to pam_mkhomedir recently [19:24:05] if your mount point hasn't switched to a direct mount pam_mkhomedir will fail [19:24:33] dschoon: which instance? [19:24:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:25:05] I-00000268 aka kripke [19:25:16] i-00000268.pmtpa.wmflabs [19:25:58] ok. gimme a sec [19:26:04] ty [19:26:12] yeah [19:26:21] you may need to reboot [19:26:27] hm. [19:26:36] let me check with people real fast [19:26:46] wait [19:26:47] no [19:26:48] you don't [19:26:55] just needed to restart autofs [19:26:57] it'll work now [19:27:14] aiight. [19:27:29] i'll check with tomasczx [19:27:32] that's likely the fix if this happens on other instances, too [19:30:45] noted. whats the command, exactly? [19:32:52] dschoon: /etc/init.d/autofs restart [19:33:44] ty [19:49:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:50:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:54:47] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:01:25] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 145 processes [20:02:38] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 293 processes [20:07:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [20:19:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:19:25] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 154 processes [20:20:36] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:27:05] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:49:16] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:51:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:58:04] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:01:33] PROBLEM Total processes is now: CRITICAL on ipv6test1 i-00000282.pmtpa.wmflabs output: PROCS CRITICAL: 201 processes [21:19:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:21:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:29:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:35:23] RECOVERY Disk Space is now: OK on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK OK [21:49:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:51:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:59:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [22:00:39] [bz] (8ASSIGNED - created by: 2Daniel Zahn, priority: 4Normal - 6minor) [Bug 36290] replace/rewrite largest_html.php / re-add output in other formats for "largest" all-in-one stats table - https://bugzilla.wikimedia.org/show_bug.cgi?id=36290 [22:02:33] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 293 processes [22:12:34] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [22:19:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:22:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:23:42] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c.pmtpa.wmflabs output: Warning: 19% free memory [22:29:24] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [22:50:22] 11/01/2012 - 22:50:22 - Updating keys for krenair at /export/keys/krenair [22:51:05] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:52:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [22:53:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:59:35] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [23:06:43] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 6% free memory [23:11:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 33% free memory [23:18:42] RECOVERY Free ram is now: OK on bots-2 i-0000009c.pmtpa.wmflabs output: OK: 20% free memory [23:21:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:23:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:29:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [23:51:45] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:54:06]