[00:16:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:21:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:24:55] PROBLEM host: i-0000050f.pmtpa.wmflabs is DOWN address: i-0000050f.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000050f.pmtpa.wmflabs) [00:28:53] RECOVERY host: i-0000050f.pmtpa.wmflabs is UP address: i-0000050f.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 6.65 ms [00:29:23] PROBLEM Total processes is now: CRITICAL on toro i-0000050f.pmtpa.wmflabs output: Connection refused by host [00:30:13] PROBLEM dpkg-check is now: CRITICAL on toro i-0000050f.pmtpa.wmflabs output: Connection refused by host [00:30:53] PROBLEM Current Load is now: CRITICAL on toro i-0000050f.pmtpa.wmflabs output: Connection refused by host [00:31:33] PROBLEM Current Users is now: CRITICAL on toro i-0000050f.pmtpa.wmflabs output: Connection refused by host [00:32:14] PROBLEM Disk Space is now: CRITICAL on toro i-0000050f.pmtpa.wmflabs output: Connection refused by host [00:32:47] how long after instance creation does it typically take for SSH keys to sync? [00:33:02] 10 mins? [00:33:03] PROBLEM Free ram is now: CRITICAL on toro i-0000050f.pmtpa.wmflabs output: Connection refused by host [00:33:13] it's basically a full puppet run [00:33:15] maybe a bit more [00:34:27] paravoid: thanks [00:36:34] RECOVERY Current Users is now: OK on toro i-0000050f.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [00:37:14] RECOVERY Disk Space is now: OK on toro i-0000050f.pmtpa.wmflabs output: DISK OK [00:38:04] RECOVERY Free ram is now: OK on toro i-0000050f.pmtpa.wmflabs output: OK: 1022% free memory [00:39:24] RECOVERY Total processes is now: OK on toro i-0000050f.pmtpa.wmflabs output: PROCS OK: 87 processes [00:40:53] RECOVERY Current Load is now: OK on toro i-0000050f.pmtpa.wmflabs output: OK - load average: 0.03, 0.53, 0.53 [00:40:53] RECOVERY dpkg-check is now: OK on toro i-0000050f.pmtpa.wmflabs output: All packages OK [00:46:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:51:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:52:27] andrewbogott / paravoid: last request: 'mattflaschen' needs to be added to bastion [00:52:50] if either of you has a moment [00:53:25] Looks like he has it already. [00:54:33] andrewbogott: [00:54:37] 16:54 I'm not listed at https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bastion [00:54:37] 16:54 And it doesn't work. :) [00:56:13] He is listed, though...? [00:56:19] At the very bottom [00:57:04] oh. huh. oops. i see it now. [00:58:20] superm401: can you try logging in again? [00:59:40] Still no. [00:59:43] "Permission denied (publickey)." [01:00:15] superm401: you changed the config to point to the right key file, right? :) [01:00:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 156 processes [01:00:48] That's start with the simple thing. [01:00:53] *Let's* [01:01:06] I still don't see myself at https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bastion under members. [01:05:53] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 176 processes [01:10:52] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 100 processes [01:11:40] superm401: I don't know what's up with that page, but I can't add you to the project because you're already in it... [01:11:46] I can remove you and re-add you :) [01:12:34] Huh. Did that help? [01:13:33] ^ superm401 [01:13:49] andrewbogott, I'm now on the members list, so progress. [01:13:55] But I still can't SSH in. [01:14:03] I'll try removing then re-adding my key. [01:14:09] nonono [01:14:18] maybe it's a puppet run on bastion that's now required [01:14:35] ori-l? [01:15:22] superm401: if andrewbogott just added you to the project, the keys on bastion still need to be updated. that's done by puppet automatically on a schedule. [01:15:29] superm401: Did you set a key on labsconsole as well as on gerrit? [01:15:35] Yes. [01:15:58] ori-l, okay. I'll check back later. [01:16:35] And I didn't delete it, so I'll just be patient for now. :) [01:16:39] It would only take a few minutes, if that's the issue. And you'll see an alert in this channel. [01:16:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:16:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 7.00, 7.27, 5.94 [01:17:33] andrewbogott, okay, thanks. [01:22:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:33:35] petan, Damianz: anyone know why mysql doesn't work on bots.wmflabs.org? or better yet, how to fix it? [01:46:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:52:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:10:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [02:16:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:22:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:35:41] Hello? [02:35:50] https://www.mediawiki.org/wiki/Developer_access#User:PiRSquared17 <- can someone do this? [02:36:20] PiRQuadratus: someone will go over your request soon. it typically takes less than 24 hours. [02:38:32] RECOVERY Free ram is now: OK on swift-be4 i-000001ca.pmtpa.wmflabs output: OK: 21% free memory [02:46:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:51:08] ori-l: okay [02:52:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:02:54] PROBLEM Free ram is now: WARNING on watchlist-bot i-0000041a.pmtpa.wmflabs output: Warning: 16% free memory [03:06:33] PROBLEM Free ram is now: WARNING on swift-be4 i-000001ca.pmtpa.wmflabs output: Warning: 18% free memory [03:16:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:23:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:24:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 4.01, 5.57, 5.29 [03:31:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 4.79, 5.44, 5.06 [03:33:32] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 151 processes [03:38:33] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 144 processes [03:41:54] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.09, 4.68, 4.87 [03:47:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:53:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:54:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.51, 4.00, 4.65 [04:12:54] RECOVERY Free ram is now: OK on watchlist-bot i-0000041a.pmtpa.wmflabs output: OK: 44% free memory [04:17:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:23:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:47:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:53:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:04:21] I can't log in on deployment.wikimedia.beta.wmflabs.org [05:17:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:23:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:47:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:50:22] 11/14/2012 - 05:50:22 - Updating keys for mattflaschen at /export/keys/mattflaschen [05:54:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:17:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:24:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:26:54] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.22, 4.59, 4.88 [06:28:24] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 154 processes [06:28:34] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:31:22] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:39:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 6.45, 5.65, 5.24 [06:43:34] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [06:48:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:51:23] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:53:23] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [06:54:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:03:33] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 151 processes [07:08:32] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 150 processes [07:18:34] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:24:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:48:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:54:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:56:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 156 processes [08:10:52] PROBLEM Current Load is now: CRITICAL on dumps-2 i-00000510.pmtpa.wmflabs output: Connection refused by host [08:11:02] PROBLEM Free ram is now: CRITICAL on dumps-2 i-00000510.pmtpa.wmflabs output: Connection refused by host [08:11:32] PROBLEM Current Users is now: CRITICAL on dumps-2 i-00000510.pmtpa.wmflabs output: Connection refused by host [08:12:12] PROBLEM Disk Space is now: CRITICAL on dumps-2 i-00000510.pmtpa.wmflabs output: Connection refused by host [08:12:22] PROBLEM Total processes is now: CRITICAL on dumps-2 i-00000510.pmtpa.wmflabs output: Connection refused by host [08:12:52] PROBLEM dpkg-check is now: CRITICAL on dumps-2 i-00000510.pmtpa.wmflabs output: Connection refused by host [08:17:23] RECOVERY Total processes is now: OK on dumps-2 i-00000510.pmtpa.wmflabs output: PROCS OK: 99 processes [08:17:53] RECOVERY dpkg-check is now: OK on dumps-2 i-00000510.pmtpa.wmflabs output: All packages OK [08:18:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:20:52] RECOVERY Current Load is now: OK on dumps-2 i-00000510.pmtpa.wmflabs output: OK - load average: 0.07, 0.54, 0.48 [08:21:02] RECOVERY Free ram is now: OK on dumps-2 i-00000510.pmtpa.wmflabs output: OK: 1873% free memory [08:21:32] RECOVERY Current Users is now: OK on dumps-2 i-00000510.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [08:22:12] RECOVERY Disk Space is now: OK on dumps-2 i-00000510.pmtpa.wmflabs output: DISK OK [08:24:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:33:34] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 151 processes [08:48:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:54:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:56:23] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [09:18:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:25:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:48:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:56:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:19:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:26:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:33:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 4.26, 6.24, 5.63 [10:38:32] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.59, 4.36, 4.96 [10:51:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:56:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:21:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:26:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:29:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 163 processes [11:34:22] PROBLEM Total processes is now: WARNING on preilly i-000004a1.pmtpa.wmflabs output: PROCS WARNING: 153 processes [11:39:22] RECOVERY Total processes is now: OK on preilly i-000004a1.pmtpa.wmflabs output: PROCS OK: 148 processes [11:51:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:56:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:06:34] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 152 processes [12:11:33] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 146 processes [12:14:43] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 19% free memory [12:21:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:26:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:29:45] !ping [12:29:46] pong [12:30:18] 1 sec... progress!!! \o/ [12:30:50] yup [12:34:34] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 157 processes [12:37:18] !ping [12:37:18] pong [12:39:37] !ping [12:39:37] pong [12:47:51] @infobot-shared-trust+ #wikimedia-operations [12:48:09] @infobot-share-trust+ #wikimedia-operations [12:48:09] You inserted channel #wikimedia-operations to shared db [12:51:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:57:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:05:33] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 240 processes [13:10:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 148 processes [13:12:42] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 14% free memory [13:17:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 32% free memory [13:21:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:27:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:44:43] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5.pmtpa.wmflabs output: Critical: 5% free memory [13:51:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:57:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:04:23] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [14:09:33] PROBLEM host: i-000000e5.pmtpa.wmflabs is DOWN address: i-000000e5.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000000e5.pmtpa.wmflabs) [14:21:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:28:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:35:40] !log bots rebooting -3 [14:35:51] andrewbogott_afk your bot is dead again [14:39:53] PROBLEM host: i-000000e5.pmtpa.wmflabs is DOWN address: i-000000e5.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000000e5.pmtpa.wmflabs) [14:45:33] RECOVERY host: i-000000e5.pmtpa.wmflabs is UP address: i-000000e5.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 7.00 ms [14:49:43] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 1439% free memory [14:51:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:59:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:59:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.14, 4.44, 4.90 [15:03:34] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 227 processes [15:21:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:28:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 134 processes [15:29:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:32:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 5.05, 5.16, 5.02 [15:52:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:59:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:07:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 3.90, 4.21, 4.72 [16:22:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:30:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:52:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:55:58] !log bots root: running all beetra's bot's [17:01:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:22:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:31:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:31:49] !log wikidata-dev wikidata-dev-2: Updated test repo and both clients. Load balancer settings required a change of mysql bind address. Is now 0.0.0.0, connecting from outside is not working. [17:52:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:01:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:02:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 152 processes [18:23:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:31:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:53:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:02:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:06:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 227 processes [19:23:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:32:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:53:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:02:32] PROBLEM dpkg-check is now: CRITICAL on integration-apache1 i-000002eb.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:03:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:18:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 208 processes [20:23:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [20:24:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:33:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:54:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:03:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:24:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:25:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 6.90, 6.48, 5.34 [21:33:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:54:14] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:01:58] Is anyone active in here to help? [22:03:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:04:07] You'd be best asking your question... [22:07:29] OK. Is there any way to link a figure from a website into a wikipedia page without going through wikimedia commons? Or can it just be a link to an external image? [22:26:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:34:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:46:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 134 processes [22:56:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:04:04] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:26:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:30:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.42, 5.51, 5.19 [23:34:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:35:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.77, 4.30, 4.78 [23:56:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs)