[00:07:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:24:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:38:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:48:32] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 208 processes [00:52:23] PROBLEM Total processes is now: UNKNOWN on dumps-bot3 i-00000503.pmtpa.wmflabs output: NRPE: Call to fork() failed [00:52:53] PROBLEM Free ram is now: UNKNOWN on dumps-bot3 i-00000503.pmtpa.wmflabs output: NRPE: Call to fork() failed [00:53:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [00:54:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:56:43] PROBLEM dpkg-check is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:56:53] PROBLEM Current Load is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:56:53] PROBLEM SSH is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Server answer: [00:57:23] PROBLEM Disk Space is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:57:23] PROBLEM Total processes is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:57:53] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:58:33] PROBLEM Current Users is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:02:53] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 8% free memory [01:03:33] RECOVERY Current Users is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [01:06:42] RECOVERY dpkg-check is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: All packages OK [01:06:52] RECOVERY Current Load is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK - load average: 0.11, 0.78, 0.86 [01:06:52] RECOVERY SSH is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [01:07:22] RECOVERY Disk Space is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: DISK OK [01:08:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:08:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 176 processes [01:13:32] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 98 processes [01:24:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:39:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:44:23] PROBLEM host: i-000000e5.pmtpa.wmflabs is DOWN address: i-000000e5.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:54:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:54:25] someone around who will restart COIBot for me on labs, instance 3 ? [02:09:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:15:23] PROBLEM host: i-000000e5.pmtpa.wmflabs is DOWN address: i-000000e5.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000000e5.pmtpa.wmflabs) [02:24:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:39:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:46:12] PROBLEM host: i-000000e5.pmtpa.wmflabs is DOWN address: i-000000e5.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000000e5.pmtpa.wmflabs) [02:54:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:56:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 10% free memory [03:10:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:16:12] PROBLEM host: i-000000e5.pmtpa.wmflabs is DOWN address: i-000000e5.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000000e5.pmtpa.wmflabs) [03:24:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:27:52] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 4% free memory [03:40:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:46:13] PROBLEM host: i-000000e5.pmtpa.wmflabs is DOWN address: i-000000e5.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000000e5.pmtpa.wmflabs) [03:56:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:57:13] PROBLEM Disk Space is now: WARNING on kubo i-000003dd.pmtpa.wmflabs output: DISK WARNING - free space: / 320 MB (3% inode=66%): [04:02:22] PROBLEM Disk Space is now: CRITICAL on kubo i-000003dd.pmtpa.wmflabs output: DISK CRITICAL - free space: / 273 MB (2% inode=66%): [04:11:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:12:52] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 6% free memory [04:16:32] PROBLEM host: i-000000e5.pmtpa.wmflabs is DOWN address: i-000000e5.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000000e5.pmtpa.wmflabs) [04:17:52] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 4% free memory [04:26:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:35:53] Hello, the running of IRC bots is allowed, right? [04:38:33] uh..................... [04:41:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:46:32] PROBLEM host: i-000000e5.pmtpa.wmflabs is DOWN address: i-000000e5.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000000e5.pmtpa.wmflabs) [04:50:55] someone around who will restart COIBot for me on labs, instance 3 ? [04:56:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:11:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:16:33] PROBLEM host: i-000000e5.pmtpa.wmflabs is DOWN address: i-000000e5.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000000e5.pmtpa.wmflabs) [05:19:35] hi sDrewth [05:19:46] gday jeremyb [05:20:05] who runs it normally? [05:20:06] is bots3 problematic? [05:20:20] Beetstra [05:20:33] it fell down a couple of hours ago [05:20:34] k. i'll look in a min [05:20:34] http://bots.wmflabs.org/~hydriz/minimanual.txt [05:21:07] [12:40] 23* 23UnBlockBot has quit (23Ping timeout: 252 seconds23) [05:21:07] [12:41] 23* 23XLinkBot has quit (23Ping timeout: 245 seconds23) [05:21:07] [12:42] 23* 23COIBot has quit (23Ping timeout: 260 seconds23) [05:21:18] now 16:21 [05:21:28] k [05:21:35] i wonder what TZ you're in [05:21:42] AEDT [05:22:33] are hydriz and beetstra the same person? [05:22:48] nope [05:23:07] I don't know why Hydriz put the notes there beyond make them visible [05:23:28] ok [05:23:29] Beetstra has a note at [[m:User talk:Beetstra]] saying people can restart [05:24:50] $ nc -v bots-3 22; ping -c 1 bots-3 [05:24:50] nc: connect to bots-3 port 22 (tcp) failed: No route to host [05:24:54] From i-000000ba.pmtpa.wmflabs (10.4.0.54) icmp_seq=1 Destination Host Unreachable [05:25:03] so... [05:25:18] the server is done [05:25:40] which will be why the bots disappeared [05:26:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:27:28] http://ganglia.wmflabs.org/latest/?r=day&cs=&ce=&c=bots&h=bots-3&tab=m&vn=&metric_group= [05:30:06] more helpful ... http://ganglia.wmflabs.org/latest/?r=hour&cs=&ce=&s=by+name&c=bots&tab=m&vn= [05:33:42] !log bots bots-3 rebooted from labsconsole. ganglia showed nearly 4 hrs unresponsive, couldn't connect myself, couldn't even view console log on labsconsole. (but console log was working for other instances) [05:33:58] damnit [05:35:41] !log bots bots-3 rebooted from labsconsole. ganglia showed nearly 4 hrs unresponsive, couldn't connect myself, couldn't even view console log on labsconsole. (but console log was working for other instances) [05:35:42] Logged the message, Master [05:36:04] !log bots bots-labs booted labs-morebots (init.d=adminbot) [05:36:05] Logged the message, Master [05:41:22] RECOVERY host: i-000000e5.pmtpa.wmflabs is UP address: i-000000e5.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.49 ms [05:41:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:45:56] grrrrrr, these instructions are horrible [05:46:06] although could be worse i guess [05:46:21] sDrewth: linkwatcher never disappeared? [05:51:52] sDrewth [05:51:52] sDrewth [05:51:53] sDrewth [05:52:01] here [05:52:08] apologies, dealing with LTA [05:52:13] how do i know if they're working? [05:53:45] is on bots-2, and runs other bots [05:54:06] idk what that means [05:54:28] #BeetstraBotChannel [05:55:36] i noticed [05:55:44] linkwatcher has the visible components in IRC LiWa3_(1|2|3) [05:55:53] right [05:56:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:56:44] !log bots bots-3 apparently beetstra's bots don't start on their own (no init script). started them all manually based on http://bots.wmflabs.org/~hydriz/minimanual.txt and root's bash history. they're running as beetstra's user [05:56:46] Logged the message, Master [05:56:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [05:56:54] hi Beetstra [05:57:17] Hey jeremyb [05:57:18] Beetstra! we've been kicking your bot [05:57:19] That was quick [05:57:32] kicking it where? [05:57:37] 25 05:35:40 < jeremyb> !log bots bots-3 rebooted from labsconsole. ganglia showed nearly 4 hrs unresponsive, couldn't connect myself, couldn't even view console log on labsconsole. (but console log was working for other instances) [05:57:40] Beetstra: So you didn't get the pings from eariler? :O [05:57:50] Beetstra: those are the 2 relevant log msgs [05:58:10] nope [05:58:25] strange, why was it suddenly so unresponsive [05:58:29] Seems to be up now, though [05:58:44] there were some OOM kills in the logs but i don't think that was it [05:59:52] Oh wait .. unblockbot [06:00:02] Still has a memory hole that I am unable to find [06:00:04] Bloody perl [18:11:11] Change on 12mediawiki a page Developer access was modified, changed by Ngiamba link https://www.mediawiki.org/w/index.php?diff=609259 edit summary: