[00:16:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:21:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:28:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:28:17] Change on 12mediawiki a page Developer access was modified, changed by Doug link https://www.mediawiki.org/w/index.php?diff=600885 edit summary: [00:46:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:51:45] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:59:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:05:57] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 174 processes [01:10:55] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 98 processes [01:11:55] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.17, 4.41, 4.97 [01:16:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:19:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 5.39, 5.14, 5.11 [01:21:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:29:05] Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Dcoetzee link https://www.mediawiki.org/w/index.php?diff=600888 edit summary: +bot credentials [01:29:14] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:34:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.74, 4.83, 4.97 [01:46:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:52:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:59:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:02:11] Are there any bots admins around that can install the python3 package on bots-1? It's called "python3-minimal" [02:17:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:22:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:29:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:37:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 4.28, 5.27, 5.14 [02:48:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:53:25] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:59:36] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:18:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:20:55] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 5.32, 5.22, 5.05 [03:24:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:29:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:36:08] PROBLEM Free ram is now: CRITICAL on spellcheckself-bot i-0000041e.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [03:40:54] RECOVERY Free ram is now: OK on spellcheckself-bot i-0000041e.pmtpa.wmflabs output: OK: 47% free memory [03:49:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:54:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:56:55] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 19% free memory [03:59:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:19:05] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:24:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:29:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:33:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 5.89, 5.58, 5.52 [04:41:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.25, 6.36, 5.36 [04:49:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:51:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 2.51, 4.20, 4.83 [04:54:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:59:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:05:36] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=600896 edit summary: done [05:06:29] danke wm-bot [05:18:54] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.10, 4.52, 4.87 [05:19:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:24:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:29:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:49:16] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:53:04] Any bots admins around that can install python3 on bots-1 please? [05:54:28] legoktm: i think not at this hour... that sounds like the kind of thing that could break stuff [05:54:32] why do you need it? [05:54:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:55:10] to run a python3 script? [05:55:15] I don't see how it could break anything [05:55:25] Its just the "python3-minimal" package [05:56:09] i didn't know there was a -minimal [05:56:24] i could look at it. but i don't want to do it when half asleep [05:56:25] legoktm@bots-1:~$ python3 [05:56:25] The program 'python3' is currently not installed. To run 'python3' please ask your administrator to install the package 'python3-minimal' [05:57:03] I mean, the full version would be nice, but minimal is really all I need. [05:58:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 12.20, 10.04, 7.26 [06:00:37] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:08:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 9.69, 9.10, 6.65 [06:09:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 9.91, 9.03, 6.26 [06:09:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 11.33, 9.98, 7.16 [06:11:34] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 10.81, 9.91, 7.03 [06:19:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:24:47] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:23] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:31:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:33:24] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:50:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:54:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:01:58] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:21:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:24:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:32:06] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:43:56] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.28, 2.92, 4.96 [07:44:55] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 3.98, 3.20, 4.72 [07:51:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:52:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 8.14, 6.48, 5.62 [07:54:48] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:56:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 8.31, 7.35, 6.01 [08:02:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:21:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:25:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:27:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 3.78, 3.86, 4.75 [08:29:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.41, 3.58, 4.76 [08:31:37] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 1.95, 3.35, 4.83 [08:31:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.66, 3.47, 4.61 [08:32:06] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:51:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:56:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:02:05] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:07:58] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.44, 6.29, 5.31 [09:10:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 6.10, 6.52, 5.50 [09:14:34] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 4.81, 5.58, 5.14 [09:19:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 4.29, 5.56, 5.77 [09:21:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:24:33] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.81, 4.64, 4.90 [09:26:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:32:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:32:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.31, 4.25, 4.76 [09:52:06] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:56:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:03:06] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:22:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:27:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:28:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.86, 4.74, 5.00 [10:33:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:46:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 10.15, 7.74, 6.05 [10:52:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:57:35] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 9.08, 8.71, 6.34 [10:57:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:57:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 8.47, 8.13, 6.01 [11:00:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.70, 7.23, 5.88 [11:03:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:03:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 5.21, 6.01, 5.37 [11:23:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:28:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:34:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:47:32] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.34, 3.72, 4.74 [11:48:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 4.69, 4.48, 4.95 [11:53:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:58:15] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:00:57] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.80, 3.92, 4.73 [12:04:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:07:54] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.66, 4.46, 4.97 [12:23:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:28:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:34:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:53:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:59:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:04:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:05:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 199 processes [13:10:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 106 processes [13:11:45] Change on 12mediawiki a page Developer access was modified, changed by Krishnadasji link https://www.mediawiki.org/w/index.php?diff=600989 edit summary: [13:23:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:29:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:34:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:48:10] Change on 12mediawiki a page Developer access was modified, changed by The100rabh link https://www.mediawiki.org/w/index.php?diff=600997 edit summary: /* New user account for the100rabh */ new section [13:51:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.70, 4.63, 4.92 [13:54:27] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:54:27] Change on 12mediawiki a page Developer access was modified, changed by Csgp 1412 link https://www.mediawiki.org/w/index.php?diff=600999 edit summary: [13:59:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:04:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:24:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 5.42, 5.38, 5.07 [14:25:34] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:29:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:34:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:39:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.66, 4.96, 5.00 [14:56:37] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:59:04] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:04:25] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:04:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 199 processes [15:26:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:29:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:29:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 106 processes [15:34:25] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:56:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:59:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:04:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:19:44] moin moin [16:26:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:29:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:34:35] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:56:56] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:59:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:04:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [17:05:53] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 4.59, 4.63, 4.95 [17:26:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:30:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:34:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [17:41:48] Change on 12mediawiki a page Developer access was modified, changed by Bajal link https://www.mediawiki.org/w/index.php?diff=601021 edit summary: [17:57:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:01:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:07:04] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:09:51] Change on 12mediawiki a page Developer access was modified, changed by Viky.nandha link https://www.mediawiki.org/w/index.php?diff=601023 edit summary: [18:27:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:31:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:37:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:57:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:01:47] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:07:35] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 199 processes [19:08:10] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:19:33] RECOVERY Total processes is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: PROCS OK: 108 processes [19:28:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:31:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:39:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:48:39] Change on 12mediawiki a page Developer access was modified, changed by Harsh4101991 link https://www.mediawiki.org/w/index.php?diff=601055 edit summary: [19:58:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:02:33] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 292 processes [20:02:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:07:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 199 processes [20:09:14] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:28:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:32:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:39:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:58:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:02:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:09:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:28:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:32:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:39:15] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:58:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:02:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 292 processes [22:03:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:09:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [22:12:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 199 processes [22:27:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 106 processes [22:29:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:34:06] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:39:14] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [22:59:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:04:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:09:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [23:29:17] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:34:46] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:39:24] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [23:59:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs)