[00:00:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.77, 4.08, 4.77 [00:02:33] PROBLEM Current Users is now: WARNING on bots-1 i-000000a9.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [00:03:40] hi all [00:03:49] @labs-project-info Wikidata-dev [00:03:49] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 55 seconds ago [00:04:13] @labs-info wikidata-dev-1 [00:04:13] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 19 seconds ago [00:04:24] @labs-info wikidata-dev [00:04:25] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 30 seconds ago [00:04:33] @labs-resolve wikidata-dev [00:04:34] I don't know this instance - aren't you are looking for: I-00000225 (wikidata-dev-3), I-00000259 (wikidata-dev-2), I-000004bb (wikidata-dev-4), I-000004e1 (wikidata-dev-6), I-0000052a (wikidata-dev-9), I-00000532 (wikidata-dev-8), I-0000053a (wikidata-dev-memcached), [00:04:46] @labs-info wikidata-dev-3 [00:04:47] [Name wikidata-dev-3 doesn't exist but resolves to I-00000225] I-00000225 is Nova Instance with name: wikidata-dev-3, host: virt7, IP: 10.4.0.23 of type: s1.medium, with number of CPUs: 1, RAM of this size: 1024M, member of project: wikidata-dev, size of storage: 170 and with image ID: lucid-server-cloudimg-amd64.img [00:05:06] @labs-project-info wikidata-dev [00:05:06] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 12 seconds ago [00:05:10] weird [00:05:58] sumanah, yeah ok [00:06:02] hi petan - how is it going? [00:06:44] hi... [00:06:58] sumanah I just found out that someone deleted all irc logs, that's not really nice [00:07:04] was going to sleep :/ [00:07:13] what?! [00:07:15] how?!! [00:07:16] I have a backup though [00:07:22] who did this? [00:07:27] I don't know [00:07:28] deliberately? [00:07:42] I have no idea if it was accident nor who did it [00:07:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:07:56] I will try to recover some from backups [00:08:10] good, thanks [00:10:39] but it doesn't really make me feel good [00:10:45] I have no idea what happened [00:10:54] it seems that someone already deleted it yesterady [00:11:03] every day .gz of all folders is created [00:11:21] today's .gz file was created from yesterdays log and it's already missing most of them [00:12:18] Ryan_Lane is it possible to enable some kind of auditing on gluster? though it likely wouldn't help [00:12:29] it ran as root anyway :/ [00:12:57] this is so weird, petan [00:12:58] well, it's possible to enable auditing on the os [00:13:02] is this a security glitch? [00:13:05] I'd imagine someone screwed up [00:13:08] not really [00:13:16] everyone has root on bots [00:13:20] so it's not security glitch [00:13:21] we let too many people have root in bots [00:13:41] someone recently -rf on stuff in public_html there is a mail in list [00:13:42] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [00:13:48] it may have been caused by that [00:13:54] D: [00:13:57] but that guy told me it didn't run as root [00:14:09] these folders are restricted to bot's user [00:14:15] ah [00:14:25] so, yeah, we should start planning on removing root [00:14:41] we need a way to deploy bots [00:14:49] ok I restored backups but we may have 1-2 days missing [00:14:52] :( [00:15:16] 3 days [00:15:27] Ryan_Lane: "deploy bots" here meaning "deploy *on the main cluster*"? [00:15:30] petan: should we start a planning document for how to start locking bots down? [00:15:39] sumanah: nah, on bots- instances [00:15:46] oh [00:16:09] hmm, maybe I think it would be best to create some another project for "production" [00:16:11] with no root [00:16:19] and keep current bots for testing [00:16:24] * Ryan_Lane nods [00:16:32] likely a good idea [00:16:39] none of my bots need root for anything [00:16:40] we'll still need a way to deploy bots [00:16:45] yes [00:16:51] we can use my new deployment system [00:16:54] some of them have some installation process [00:17:05] we could just package them [00:17:12] yeah, packaging is hard, though [00:17:17] hm... [00:17:28] we'd also need to have a repo [00:17:40] indeed [00:17:49] my deployment system is generic [00:18:07] it uses git [00:18:18] which means bots would need to be kept in git (which is good) [00:19:06] it could deploy on all instances, but we could choose which bot runs where [00:19:08] in configuration [00:19:13] so that we can move bots around [00:20:00] it's possible to create service accounts now [00:20:08] with self-registration open [00:29:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:33:52] PROBLEM Current Load is now: CRITICAL on abogott-request-tracker i-0000053e.pmtpa.wmflabs output: Connection refused by host [00:34:32] PROBLEM Current Users is now: CRITICAL on abogott-request-tracker i-0000053e.pmtpa.wmflabs output: Connection refused by host [00:35:12] PROBLEM Disk Space is now: CRITICAL on abogott-request-tracker i-0000053e.pmtpa.wmflabs output: Connection refused by host [00:36:02] PROBLEM Free ram is now: CRITICAL on abogott-request-tracker i-0000053e.pmtpa.wmflabs output: Connection refused by host [00:37:16] ok I go sleep it's 1:40 here [00:37:22] PROBLEM Total processes is now: CRITICAL on abogott-request-tracker i-0000053e.pmtpa.wmflabs output: Connection refused by host [00:37:26] will take a look on take tommorow [00:37:34] * that [00:37:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:37:52] PROBLEM dpkg-check is now: CRITICAL on abogott-request-tracker i-0000053e.pmtpa.wmflabs output: Connection refused by host [00:42:23] RECOVERY Total processes is now: OK on abogott-request-tracker i-0000053e.pmtpa.wmflabs output: PROCS OK: 87 processes [00:42:53] RECOVERY dpkg-check is now: OK on abogott-request-tracker i-0000053e.pmtpa.wmflabs output: All packages OK [00:43:43] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [00:43:53] RECOVERY Current Load is now: OK on abogott-request-tracker i-0000053e.pmtpa.wmflabs output: OK - load average: 0.16, 0.85, 0.74 [00:44:33] RECOVERY Current Users is now: OK on abogott-request-tracker i-0000053e.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [00:45:13] RECOVERY Disk Space is now: OK on abogott-request-tracker i-0000053e.pmtpa.wmflabs output: DISK OK [00:46:03] RECOVERY Free ram is now: OK on abogott-request-tracker i-0000053e.pmtpa.wmflabs output: OK: 1130% free memory [00:51:56] Ryan_Lane: when you say repo above you mean a labs brewster? [00:59:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:02:34] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [01:05:52] PROBLEM dpkg-check is now: CRITICAL on abogott-request-tracker i-0000053e.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [01:05:52] RECOVERY Free ram is now: OK on sube i-000003d0.pmtpa.wmflabs output: OK: 25% free memory [01:07:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:08:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 176 processes [01:13:33] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 113 processes [01:13:43] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [01:29:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:36:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [02:02:13] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [02:02:21] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [02:04:17] PROBLEM Disk Space is now: CRITICAL on nova-precise1 i-00000236.pmtpa.wmflabs output: DISK CRITICAL - free space: / 0 MB (0% inode=80%): [02:08:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:08:52] PROBLEM Free ram is now: WARNING on sube i-000003d0.pmtpa.wmflabs output: Warning: 19% free memory [02:11:43] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 147 processes [02:14:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [02:18:49] Krenair: I won't use WMF labs until it has IPv6 xD [02:19:13] Jasper_Deng, um, okay? [02:19:25] Krenair: I like the new registration process [02:19:30] though it might encourage spamming [02:19:33] oh right [02:32:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:36:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [02:38:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:38:53] RECOVERY Free ram is now: OK on wikistream-1 i-0000016e.pmtpa.wmflabs output: OK: 25% free memory [02:38:53] RECOVERY Free ram is now: OK on sube i-000003d0.pmtpa.wmflabs output: OK: 40% free memory [02:39:33] RECOVERY Free ram is now: OK on swift-be4 i-000001ca.pmtpa.wmflabs output: OK: 22% free memory [02:44:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [02:46:52] PROBLEM Free ram is now: WARNING on wikistream-1 i-0000016e.pmtpa.wmflabs output: Warning: 12% free memory [02:47:32] RECOVERY Current Users is now: OK on bots-1 i-000000a9.pmtpa.wmflabs output: USERS OK - 5 users currently logged in [03:02:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:08:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:15:02] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [03:32:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:36:07] ping [03:36:25] * Jasper_Deng pings PiRSquared [03:38:07] pong [03:38:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:45:52] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [03:52:06] Jasper_Deng: do you know how to use a proxy to access labs? [03:52:57] no [04:02:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:09:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:15:52] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [04:32:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:39:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:45:53] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [05:02:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:09:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:13:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.75, 4.74, 4.95 [05:15:53] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [05:21:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [05:32:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:39:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:45:53] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [06:02:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:10:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:15:53] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [06:29:43] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:31:33] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:32:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:32:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 155 processes [06:34:43] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 147 processes [06:40:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:42:32] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [06:45:53] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [06:46:33] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [07:03:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:10:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:13:42] PROBLEM Current Load is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: WARNING - load average: 5.58, 5.54, 5.17 [07:16:32] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [07:23:42] RECOVERY Current Load is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK - load average: 4.34, 4.66, 4.93 [07:33:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:41:42] PROBLEM Current Load is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: WARNING - load average: 5.12, 5.29, 5.09 [07:41:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:46:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [08:03:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:11:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:17:22] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [08:33:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:41:03] @labs-info wikidata-dev-3 [08:41:03] [Name wikidata-dev-3 doesn't exist but resolves to I-00000225] I-00000225 is Nova Instance with name: wikidata-dev-3, host: virt7, IP: 10.4.0.23 of type: s1.medium, with number of CPUs: 1, RAM of this size: 1024M, member of project: wikidata-dev, size of storage: 170 and with image ID: lucid-server-cloudimg-amd64.img [08:41:13] @labs-info bots [08:41:13] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 40 seconds ago [08:41:18] @labs-info bot [08:41:18] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 45 seconds ago [08:41:51] @labs-project-info wikidata-dev [08:41:51] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 18 seconds ago [08:41:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:41:56] @labs-project-info bots [08:41:56] The project Bots has 16 instances and 40 members, description: A project for creating and running bots for use on Wikimedia Foundation sites. [08:42:15] @labs-project-members bots [08:42:20] @labs-user-info bots [08:44:42] !htmllogs [08:44:42] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [08:44:45] !htmllogs [08:44:46] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [08:44:47] @commands [08:44:47] Commands: there is too many commands to display on one line, see http://meta.wikimedia.org/wiki/wm-bot for a list of commands and help [08:47:23] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [09:04:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:11:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:18:02] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [09:22:02] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 12% free memory [09:34:04] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:42:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:48:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [10:04:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:12:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:18:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [10:34:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:43:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:48:52] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [11:04:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:08:16] :-( .. is it my account that has filled up on /home/? [11:08:50] Tried to delete everything, but still no space left .. [11:14:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:19:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [11:34:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:37:08] Grmpf. Got disconnected :-( [11:44:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:50:54] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [12:04:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:14:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:21:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [12:36:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:44:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:50:32] PROBLEM Current Users is now: WARNING on bots-1 i-000000a9.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [12:52:22] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [13:06:44] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:15:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:22:23] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [13:37:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:46:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:52:23] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [14:07:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:16:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:22:24] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [14:37:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:47:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:53:02] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [15:06:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [15:07:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:16:54] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [15:18:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:23:02] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [15:26:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [15:38:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:49:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:53:02] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [16:08:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:19:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:23:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [16:23:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 152 processes [16:28:33] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [16:38:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:49:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:53:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [17:08:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:20:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:23:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [17:37:04] aude: Hi, do you know if there has been any further progress on the tile server? [17:38:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:50:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:53:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [18:09:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:20:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:22:22] Change on 12mediawiki a page Developer access was modified, changed by Bharel link https://www.mediawiki.org/w/index.php?diff=617094 edit summary: [18:23:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [18:24:39] Change on 12mediawiki a page Developer access was modified, changed by Bharel link https://www.mediawiki.org/w/index.php?diff=617096 edit summary: /* User:Bharel */ [18:33:26] Change on 12mediawiki a page Developer access was modified, changed by Fomafix link https://www.mediawiki.org/w/index.php?diff=617101 edit summary: [18:39:04] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:41:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [18:46:54] Can we murder the developer access page now? [18:47:04] Since it's no longer required to get a base account [18:47:24] Damianz, I was considering protecting it [18:47:35] Instead I just replaced the 'Request an account' link with strikethrough [18:47:44] I see it has a note at the top of it... apparently people can't read though =\ [18:47:56] Yeah I forwarded my email to those people [18:48:30] cool [18:49:41] * Damianz goes to find the pizza guy [18:50:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:53:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [18:57:14] Uh... [18:57:22] Got a reply from Bharel [18:57:27] There�s a small problem though, I keep on getting an error: �There was either an authentication database error or you are not allowed to update your external account.�. [19:04:05] hmm [19:04:11] does he have a svn account already? [19:05:18] Well he said no on the form [19:05:42] And I can't find a user in SVN called bharel [19:06:30] yeah I get that error [19:06:44] hmmm weird [19:07:14] IIRC that's usually a conifg setting [19:07:27] * Damianz pokes andrewbogott_afk so he can fix it when he wakes up :P [19:09:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:19:05] Damianz, Fomafix also replied with the same error. [19:20:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:23:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [19:25:50] Damianz, also he confirmed it's when submitting the form. Any chance you could create those accounts? [19:27:28] Don't have access too sorry [19:27:35] I see the same error when trying a test account though [19:41:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:51:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:53:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [20:11:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:21:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:23:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [20:41:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:51:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:53:04] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [21:12:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:22:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:23:53] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [21:41:53] PROBLEM Free ram is now: WARNING on sube i-000003d0.pmtpa.wmflabs output: Warning: 19% free memory [21:42:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:52:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:53:53] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [22:13:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:23:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:24:32] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [22:26:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 11% free memory [22:43:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:54:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:54:32] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [23:05:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 4.76, 5.16, 5.09 [23:13:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:24:32] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [23:24:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:25:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.55, 4.82, 4.95 [23:43:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:43:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 4.93, 5.04, 5.03 [23:54:32] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [23:54:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%