[00:03:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:06:06] ok chrismcmahon, let's see if anything is broken on beta so far [00:06:23] anomie: looking around... [00:07:17] anomie: no errors, but not loading... [00:07:32] there it is, took a while [00:08:58] anomie: seeing AFTv5 on beta, yay [00:09:25] chrismcmahon- Was it not there before? I didn't change anything that should have made it show up [00:09:48] anomie: no, it was there, but it's been fragile recently, it had some new dependencies this week [00:13:19] anomie: seems much slower than usual. I don't think it's caching, as I'm hitting new pages [00:14:27] chrismcmahon- hitting random pages here seems fast enough? [00:17:15] anomie: does making a new page take a long time? does for me. [00:18:25] chrismcmahon- hmm, let me create an account on beta to test with [00:18:42] anomie: takes a significant amount of time for e.g. the js for the editor to load [00:18:52] PROBLEM Current Load is now: CRITICAL on faidon-test i-00000509.pmtpa.wmflabs output: Connection refused by host [00:19:32] More than before, I guess you mean? [00:19:32] PROBLEM Current Users is now: CRITICAL on faidon-test i-00000509.pmtpa.wmflabs output: Connection refused by host [00:19:53] anomie: all the functions and extensions I'm checking are working, just much slower than I've seen before. (but that might not be connected to what you did just now) [00:20:13] PROBLEM Disk Space is now: CRITICAL on faidon-test i-00000509.pmtpa.wmflabs output: Connection refused by host [00:21:03] PROBLEM Free ram is now: CRITICAL on faidon-test i-00000509.pmtpa.wmflabs output: Connection refused by host [00:22:23] PROBLEM Total processes is now: CRITICAL on faidon-test i-00000509.pmtpa.wmflabs output: Connection refused by host [00:22:53] PROBLEM dpkg-check is now: CRITICAL on faidon-test i-00000509.pmtpa.wmflabs output: Connection refused by host [00:25:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:27:22] RECOVERY Total processes is now: OK on faidon-test i-00000509.pmtpa.wmflabs output: PROCS OK: 88 processes [00:27:52] RECOVERY dpkg-check is now: OK on faidon-test i-00000509.pmtpa.wmflabs output: All packages OK [00:28:54] RECOVERY Current Load is now: OK on faidon-test i-00000509.pmtpa.wmflabs output: OK - load average: 0.40, 0.94, 0.70 [00:29:32] RECOVERY Current Users is now: OK on faidon-test i-00000509.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [00:30:12] RECOVERY Disk Space is now: OK on faidon-test i-00000509.pmtpa.wmflabs output: DISK OK [00:31:02] RECOVERY Free ram is now: OK on faidon-test i-00000509.pmtpa.wmflabs output: OK: 1049% free memory [00:32:48] chrismcmahon- Ok, I'm about to break beta for a minute or two. But then it will be back. [00:34:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:37:25] anomie: OK, I'm around for another 15 minutes or so [00:44:43] chrismcmahon- Ok, should be working again. Also, BTW, my guess is that the network filesystem for beta is being slow right now, because all these git commands are taking longer than usual too. [00:47:06] anomie: sounds reasonable, looking some more [00:53:16] anomie: loading javascript seems particularly slow. I'm about done for today, but I'll poke at beta early tomorrow. [00:53:37] chrismcmahon- ok [00:53:38] anomie: everything I care about seems to be working fine, just not loading in a timely way [00:54:17] chrismcmahon- Hopefully the slowness clears up when whatever other people are doing that's bogging down the whole network filesystem thing gets finished. [00:54:26] yep [00:56:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:04:14] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:07:54] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 175 processes [01:12:55] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:22:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 12.47, 8.97, 6.48 [01:23:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 8.24, 8.14, 6.01 [01:24:13] for x in {0..100}; do head.move(desk); done [01:26:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:26:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [01:34:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:34:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 9.00, 8.19, 5.74 [01:35:34] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 7.24, 7.34, 5.40 [01:38:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 11.77, 9.41, 6.62 [01:40:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 8.15, 7.67, 5.65 [01:56:15] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:04:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:26:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:36:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:39:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 155 processes [02:40:36] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.12, 3.22, 4.74 [02:40:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 5.78, 3.98, 4.73 [02:48:35] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 8.63, 6.89, 5.86 [02:48:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 9.37, 7.15, 5.79 [02:56:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:06:03] Change on 12mediawiki a page Developer access was modified, changed by Pavithrah link https://www.mediawiki.org/w/index.php?diff=602233 edit summary: [03:06:31] Change on 12mediawiki a page Developer access was modified, changed by Pavithrah link https://www.mediawiki.org/w/index.php?diff=602234 edit summary: /* User:Pavithrah */ [03:06:45] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:10:18] Change on 12mediawiki a page Developer access was modified, changed by Omshivaprakash link https://www.mediawiki.org/w/index.php?diff=602235 edit summary: [03:26:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:36:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:56:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:58:32] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 2.53, 2.96, 4.38 [03:58:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 2.75, 3.12, 4.52 [03:59:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.73, 2.94, 4.89 [04:06:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:07:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 8.65, 6.50, 5.69 [04:26:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 5.20, 5.73, 5.61 [04:26:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:26:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 4.36, 4.64, 5.04 [04:31:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 4.92, 4.69, 4.97 [04:36:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:56:34] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 2.28, 3.98, 4.75 [04:57:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:57:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.24, 3.41, 4.57 [04:58:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.18, 3.45, 4.60 [05:06:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:27:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:36:56] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:53:55] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.43, 4.39, 4.94 [05:58:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:07:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:28:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:32] Change on 12mediawiki a page Developer access was modified, changed by Samuelharden link https://www.mediawiki.org/w/index.php?diff=602250 edit summary: [06:32:22] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:32:52] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 4.64, 4.77, 4.96 [06:37:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:41:52] PROBLEM dpkg-check is now: CRITICAL on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [06:43:42] PROBLEM dpkg-check is now: CRITICAL on wordpressbeta-precise i-00000417.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:52] RECOVERY dpkg-check is now: OK on stackfarm-sql2 i-00000508.pmtpa.wmflabs output: All packages OK [06:52:22] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 146 processes [06:53:32] RECOVERY dpkg-check is now: OK on wordpressbeta-precise i-00000417.pmtpa.wmflabs output: All packages OK [06:58:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:05:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 5.03, 5.17, 5.06 [07:07:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:21:57] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 6.10, 5.72, 5.24 [07:29:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:36:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 12.23, 10.49, 6.67 [07:38:45] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:39:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 11.22, 9.89, 6.72 [07:39:56] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 10.88, 9.62, 6.52 [07:40:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 9.75, 9.35, 6.71 [07:50:45] Change on 12mediawiki a page Developer access was modified, changed by Jacobnelson link https://www.mediawiki.org/w/index.php?diff=602295 edit summary: [07:59:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:08:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:29:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:39:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:00:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:04:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [09:04:36] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.88, 4.27, 4.93 [09:04:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 4.34, 3.97, 4.81 [09:09:11] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:10:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.14, 4.37, 4.98 [09:16:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.48, 4.42, 4.99 [09:38:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:38:54] Unable to parse the feed from https://bugzilla.wikimedia.org/buglist.cgi?chfieldfrom=-4h&chfieldto=Now&list_id=151044&product=Wikimedia%20Labs&query_format=advanced&title=Bug%20List&ctype=atom this url is probably not a valid rss, the feed will be disabled, until you re-enable it by typing @rss+ bugzilla [09:42:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:01:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:09:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:14:01] !log test [10:31:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:39:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:48:18] @rss+ bugzilla [10:48:18] This item was enabled now [11:01:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:09:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:22:33] PROBLEM dpkg-check is now: CRITICAL on deployment-integration i-0000034a.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [11:28:16] !log wikidata-dev wikidata-dev-2: (yesterday already) Updated test server. Ran into db problems that might be related to #41112. [11:29:02] hello log bot? [11:31:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:39:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:41:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.45, 4.79, 4.98 [11:49:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 5.78, 5.28, 5.13 [12:01:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:04:54] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.40, 4.67, 4.88 [12:07:52] PROBLEM Free ram is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Critical: 5% free memory [12:09:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:12:34] RECOVERY dpkg-check is now: OK on deployment-integration i-0000034a.pmtpa.wmflabs output: All packages OK [12:31:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:40:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:01:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:05:34] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [13:07:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 6.17, 5.62, 5.33 [13:10:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [13:11:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:32:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:41:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:00:49] !log wikidata-dev Updated the setup and workflow documentation https://labsconsole.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/Documentation [14:02:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:05:46] Change on 12mediawiki a page Developer access was modified, changed by Bhoomit link https://www.mediawiki.org/w/index.php?diff=602391 edit summary: [14:11:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:12:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.17, 4.58, 4.97 [14:32:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:40:52] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 4.39, 4.71, 4.95 [14:41:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:03:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:04:36] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [15:07:34] !log centralauth Updated and rebooted centralauth-frontend, daemons and mysql [15:07:46] !log centralauth Updated and rebooted centralauth-frontend, daemons and mysql [15:09:14] petan: Is there a known problem with !log [15:11:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:16:50] yes [15:17:04] it's broken and I think andrewbogott is working on it? [15:18:13] petan: I looked at the bot but didn't see anything obvious. Do you know more about when it crashes? And/or do you know where its logfile is? [15:18:53] I have no idea I really don't understand python even a bit. I understand Chinese way more :D I only know it crashes almost everytime you want anything from it [15:19:20] Well, that' s my question -- does it crash when there's a labs outage, or does it crash in response to log requests, or...? [15:19:25] last time first !log made it crash [15:19:38] * I made first !log [15:19:41] Heh, that should be easy then. [15:19:58] ok [15:24:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [15:33:59] Change on 12mediawiki a page Developer access was modified, changed by The wub link https://www.mediawiki.org/w/index.php?diff=602414 edit summary: [15:34:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:42:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:49:27] Change on 12mediawiki a page Developer access was modified, changed by Sundar link https://www.mediawiki.org/w/index.php?diff=602421 edit summary: [15:50:50] Hello Damianz, could we get node.js on the bots project [15:51:21] I tried to be sneaky and compile it myself, but there isn't even a c compiler installed :-) [16:04:04] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:12:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:16:01] Damianz. forget node, but a compiler would still be great! [17:04:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:13:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:28:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 156 processes [17:34:49] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:43:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:44:57] drdee: I guess npm2debian is not really mature enough :-] [17:45:21] :( [17:45:23] drdee: it resolves the dependencies tree of a npm module and put everything in the same .deb [17:45:26] that would work [17:46:00] but you end up installing something like ./usr/lib/node_modules/grunt/node_modules/jshint/node_modules/cli/node_modules/glob/node_modules/minimatch/node_modules/lru-cache/lib/lru-cache.js [17:46:34] where grunt is the npm module, then it get jshint which needs cli which in turns needs glob then minimatch then lru-cache [17:46:37] that's ugly [17:46:40] definitely [17:46:56] maybe traverse the DAG after all ;) [17:47:00] I guess npm2debian should simply create another debian package at the specific version needed [17:47:18] which also mean the script need to be hugely enhanced [17:47:59] so I will just install the module on a local machine, push that to some git repository. And in production I would simply git clone that repo [17:48:02] less overhead [17:48:03] ;-] [17:48:12] hashar: look at https://github.com/jordansissel/fpm (from ottomata) [17:48:54] RECOVERY dpkg-check is now: OK on build2 i-000004b7.pmtpa.wmflabs output: All packages OK [17:49:50] drdee: nice [17:49:57] drdee: but I will just use npm install [17:50:00] that is faster [17:50:02] ;-] [17:51:28] but fpm is a more robust npm2debian [18:05:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:08:01] Hmm... Gerrit changeset #29344 is up to patchset 3, but we have patchset 1 actually running (as a live hack) on beta right now. Exciting. [18:13:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:35:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:43:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:05:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:07:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [19:13:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:36:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:36:52] PROBLEM dpkg-check is now: CRITICAL on build2 i-000004b7.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [19:39:12] PROBLEM dpkg-check is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:39:52] PROBLEM Current Load is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:40:02] PROBLEM Disk Space is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:40:02] PROBLEM SSH is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Server answer: [19:41:53] PROBLEM Total processes is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:42:53] RECOVERY Free ram is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: OK: 65% free memory [19:43:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:44:13] RECOVERY dpkg-check is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: All packages OK [19:44:53] RECOVERY Current Load is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: OK - load average: 0.20, 0.79, 0.69 [19:45:12] RECOVERY Disk Space is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: DISK OK [19:45:12] RECOVERY SSH is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:46:52] RECOVERY Total processes is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: PROCS OK: 138 processes [20:02:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 293 processes [20:06:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:07:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [20:11:52] RECOVERY dpkg-check is now: OK on build2 i-000004b7.pmtpa.wmflabs output: All packages OK [20:14:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:15:03] is wm-bot down? its not reporting recent changes in my channel right now... [20:17:12] legoktm: Works for me, but petr changed some stuff recently, so maybe ping him [20:18:11] !ping [20:18:12] pong [20:19:06] hm [20:19:18] well i just made two edits that should have triggered the rc feed [20:19:26] but its responding to stuff like @trusted and what not [20:21:29] petan: any chance you could take a look at wm-bot? [20:36:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:44:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:03:19] wikidata sends raw php code instead of parsed code: http://wikidata-test-repo.wikimedia.de/w/index.php [21:06:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:14:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:17:02] <^demon|busy> Merlissimo: That server is run by wikimedia.de, no labs. Althgouh, works for me? [21:17:07] <^demon|busy> *not [21:17:50] ^demon|busy: it is labs used wmde [21:21:20] <^demon|busy> Orly? I see. [21:21:59] ^demon|busy: sb. enabled the php parser again. Was it you? [21:22:52] <^demon|busy> No, I didn't touch it. [21:22:58] <^demon|busy> I just looked at it and it worked. [21:24:27] ^demon|busy: i got the raw text of localsetting.php. It is not good that this was possible on an server error [21:25:22] <^demon|busy> Well, I don't know how they've got their instance configured. [21:25:22] Any ops alive? [21:25:27] <^demon|busy> That wouldn't happen on prod. [21:25:47] what the heck happen? [21:25:56] Or rather is it fixed? [21:26:06] <^demon|busy> I didn't touch it. [21:26:10] <^demon|busy> I never saw it broken. [21:26:39] No blame game, sorry, but is it fixed? [21:28:01] <^demon|busy> I assume so? I don't even know what was broken. [21:28:35] pm [21:37:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:39:27] ^demon|busy: it did happen once in prod for about 60 secs give or take (somewhere between 10-120 at least) sometime in the last 3-4 weeks [21:41:23] Ordinary swipe and fresh setup or do you people like to live dangerous? ;) [21:46:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:56:48] !ping [21:56:49] pong [21:58:58] !log andrewtestthebot You are hearing me talk. [21:59:26] * andrewbogott loves an easy-to-reproduce crash! [22:00:05] 11/08/2012 - 22:00:04 - Creating a project directory for andrewteststhebot [22:00:49] !log andrewtestthebot Wake up! Time to die. [22:02:34] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 293 processes [22:07:32] PROBLEM Total processes is now: WARNING on bots-2 i-0000009c.pmtpa.wmflabs output: PROCS WARNING: 157 processes [22:08:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:10:19] !log andrewteststhebot so... [22:11:47] !log andrewtestsbot was it really that easy? [22:11:47] Can't contact LDAP for project list. [22:11:48] andrewtestsbot is not a valid project. [22:12:03] !log andrewteststhebot so... [22:12:03] andrewteststhebot is not a valid project. [22:12:20] !log andrewteststhebot it really is though. [22:12:21] andrewteststhebot is not a valid project. [22:12:34] !log bots this is a test message. [22:12:34] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [22:12:35] bots is not a valid project. [22:12:41] Heh. [22:12:50] Hey, at least it's not crashing! [22:12:56] That's true! :) [22:13:23] (that one was on purpose) [22:13:41] *nod* [22:13:57] You don't have to explain to us, andrewbogott, just do your thing :) [22:15:11] !log andrewteststhebot so... [22:15:11] andrewteststhebot is not a valid project. [22:16:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:20:00] !log centralauth Updated and rebooted centralauth-frontend, daemons and mysql [22:20:01] centralauth is not a valid project. [22:20:05] !log centralauth Updated and rebooted centralauth-frontend, daemons and mysql [22:20:05] centralauth is not a valid project. [22:20:12] Jan_Luca: I'm working on it. [22:20:51] andrewbogott: Some hours ago there were no bot so I thought it work maybe now [22:21:06] I figured out why it was crashing but now it doesn't believe in projects. [22:22:08] andrewbogott: Maybe it realized that the existence of beautiful software doesn't necessarily require there to be projects that created it? [22:22:20] andrewbogott: Is the bot going through its rebellious agnostic phase? [22:22:35] If surgery fails I will try talk therapy. [22:23:08] !log andrewteststhebot so... [22:23:08] andrewteststhebot is not a valid project. [22:24:00] !log andrewteststhebot so... [22:24:00] andrewteststhebot is not a valid project. [22:25:30] !log invalidproject invalid! [22:26:39] !log invalidproject invalid! [22:26:39] Can't contact LDAP for project list. [22:27:47] !log invalidproject invalid! [22:27:48] invalidproject is not a valid project. [22:28:22] !log invalidproject invalid! [22:28:22] Can't contact LDAP for project list. [22:28:22] invalidproject is not a valid project. [22:29:09] !log invalidproject invalid! [22:29:10] Can't contact LDAP for project list. [22:29:10] invalidproject is not a valid project. [22:33:45] ping petan [22:37:18] !log invalidproject invalid! [22:37:18] invalidproject is not a valid project. [22:38:09] !log invalidproject invalid! [22:38:09] invalidproject is not a valid project. [22:38:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:39:07] !log invalidproject invalid! [22:39:07] invalidproject is not a valid project. [22:42:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [22:47:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:08:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:17:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:31:06] * andrewbogott is stumped by ldap, will return to the bot tomorrow. [23:38:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:44:11] Damianz: https://labsconsole.wikimedia.org is down - any ideas? [23:44:34] urgh 404 [23:44:41] it's technically in production so bug ops [23:44:58] no reason I know of for it not working [23:45:13] * Damianz jabs paravoid or andrewbogott [23:47:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs)