[00:03:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:13:02] hi mlitn@i-0000013d :) [00:13:44] mlitn: fyi.. can't cd to /var/www/echo-test/ stuff on that instance.. creates quite a bit of cron mail [00:14:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:17:48] mutante: thanks for heads-up, should be fixed now [00:18:37] mlitn: thanks [00:33:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:34:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 9.22, 7.53, 6.01 [00:38:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.41, 6.14, 5.24 [00:44:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:47:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 7.27, 6.31, 5.40 [00:51:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 11% free memory [01:04:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:05:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 173 processes [01:14:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:15:31] is there any special page to request getting added to a project or do I just have to bug someone :-) [01:15:32] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:22:33] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [01:34:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:35:38] Ryan_Lane: who do I poke or what wiki page do I use to request access to the fundraising project? [01:36:46] !resource fundraising [01:36:46] https://labsconsole.wikimedia.org/wiki/Nova_Resource:fundraising [01:36:57] you should talk to jeff [01:37:01] he can add you [01:37:13] I probably can too, but it's jeff's projet [01:37:31] yeah, he's putting kids to bed. will ask tomorrow [01:37:32] thanks [01:44:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:57:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 4.43, 4.55, 4.93 [02:04:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:14:44] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:24:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 5.76, 4.68, 4.89 [02:34:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:38:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.82, 4.13, 4.72 [02:46:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:59:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.53, 4.63, 4.96 [03:04:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:04:53] PROBLEM Free ram is now: WARNING on newchanges-bot i-00000419.pmtpa.wmflabs output: Warning: 11% free memory [03:16:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:35:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:46:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:54:53] PROBLEM Free ram is now: CRITICAL on newchanges-bot i-00000419.pmtpa.wmflabs output: Critical: 5% free memory [04:06:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:17:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:29:52] PROBLEM Free ram is now: WARNING on newchanges-bot i-00000419.pmtpa.wmflabs output: Warning: 11% free memory [04:36:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:47:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:54:53] PROBLEM Free ram is now: CRITICAL on newchanges-bot i-00000419.pmtpa.wmflabs output: Critical: 5% free memory [05:06:22] PROBLEM SSH is now: CRITICAL on newchanges-bot i-00000419.pmtpa.wmflabs output: CRITICAL - Socket timeout after 10 seconds [05:06:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:16:22] RECOVERY SSH is now: OK on newchanges-bot i-00000419.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [05:18:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:34:34] * jeremyb spies a robla [05:34:42] errr, wrong person [05:34:58] heh [05:35:14] * jeremyb was trigger happy [05:35:16] ;P [05:35:31] * jeremyb forgot to poke sumanah today ;( [05:36:23] [[mediawikiwiki:developer access]] needs some help from notme [05:37:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:39:32] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=612761 edit summary: /* User:Mlpearc */ done [05:48:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:51:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [05:56:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [06:07:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:07:50] hrmmmmmm, is this guy that sent the most recent message to toolserver-l the same guy that's just been announced as a new labs contractor? [06:08:09] no, morten vs. mike [06:08:17] both are mwang though [06:18:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 203 processes [06:31:33] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:31:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 154 processes [06:36:32] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [06:37:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:46:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [06:49:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:51:33] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:53:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 200 processes [07:07:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:19:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:25:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 3.64, 4.16, 4.78 [07:33:14] Can anyone help me with this ? https://labsconsole.wikimedia.org/wiki/Special:NovaKey [07:33:45] :P [07:33:53] "There were no Nova credentials found for your user account. Please ask a Nova administrator to create credentials for you" [07:34:24] https://labsconsole.wikimedia.org/wiki/User:EFDWiki1 [07:37:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:38:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 206 processes [07:49:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:49:15] Mlpearc|Around: hi [07:49:30] I'm assuming this is the first time you've logged in? [07:49:42] there's a bug with initial login. you'll need to log out and back in [07:49:45] then it'll work [08:03:23] Ah, Ryan_Lane ty [08:03:29] yes your right [08:03:36] a newbie [08:03:40] :P [08:04:42] And wha-la [08:08:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:08:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [08:19:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:19:43] @seenrx hashar [08:19:44] petan: hashar is in here, right now (multiple results were found: hashar_, hasharEat, hasharHome, hasharLunch, hasharFood and 3 more results) [08:19:50] hey hashar [08:19:56] did you check the mail regarding beta [08:20:03] why it was broken hm? [08:20:18] good morning [08:20:19] let me wake up first :-] [08:20:21] I haven't looked at beta [08:20:26] * petan shakes with hashar [08:20:41] if you have any details, please share them :-] [08:20:44] yes [08:20:47] check your inbox [08:20:48] :D [08:20:51] apaches were both down [08:20:56] there is like 3k mails in my inbox hehe [08:21:00] / is 100% [08:21:08] ah that is unfortunate [08:21:08] I can't free space [08:21:14] if I remove anything it's still 100% [08:21:15] don't you have sudo there ? [08:21:18] I do [08:21:42] * hashar logs on deployment-apache32 [08:21:45] but it doesn't help if u remove any file it mysteriously get full again like if something was constantly filling it up [08:22:14] hmmm [08:22:18] now it's 51% [08:22:24] but few days ago it was full [08:22:36] apache33 is down :/ [08:22:39] can't ssh there [08:22:58] so when a disk is full [08:23:00] why is /tmp mounted to overflow lol [08:23:17] btw I moved /var/log to gluster [08:23:19] the first thing you want to check is find out the directory that is filling up and on which disk is mounted for it [08:23:20] to free some space [08:23:29] then you want to find out which process write to that file [08:23:35] that [08:23:38] is [08:23:41] not easy :D [08:23:45] damn enter [08:24:00] so the useful commands are df to show disk usage per …. disk [08:24:01] when I du -ks /* [08:24:03] and du [08:24:07] ok you got them :-] [08:24:10] it takes like 2 days to finish [08:24:14] yeah [08:24:21] that it is a bit of a killer [08:24:26] most of the time you can just du /var/log [08:24:27] mm [08:24:37] so that was /var/log filling up ? [08:24:42] I don't know [08:24:48] ohthoo hazehaze [08:24:49] when I removed it disk was still 100% [08:24:58] you can't symlink /var/log to /data/project :-] [08:25:00] really [08:25:03] why :D [08:25:15] if we lost /data/project we have no logs [08:25:22] and loose log on boot up [08:25:24] I suppose we won't loose it [08:25:33] log from boot up is written back from cache [08:25:45] it never writes on the fly [08:25:48] annd [08:25:54] /data/project can't be read :-] [08:25:56] bootlog is stored to RAM and written to disk when they are mounted [08:26:23] why [08:26:30] I reboot apache [08:26:31] 33 [08:26:33] because it's down [08:27:34] so anyway [08:27:46] the issue is that /var/log/glusterfs/data-project.log [08:27:59] so you could have just deleted that file or empty it up with echo -n ;) [08:29:12] ERROR: failed to open logfile /var/log/glusterfs/etc-glusterfs-glusterd.vol.log [08:29:13] hehe [08:29:21] that is why you can't move /var/log on cluster :) [08:30:46] !log deployment-prep on apache32 : removed /var/log symlink, recreated directory, restarted gluster, moving files from /data/project/apache32 [08:30:48] Logged the message, Master [08:32:40] !log deployment-prep rebooting apache32 so all its service knows about /var/log :-] [08:32:42] Logged the message, Master [08:33:30] petan: the bug is https://bugzilla.wikimedia.org/show_bug.cgi?id=41104 [08:35:07] booo [08:35:10] both apaches are dead :/ [08:37:12] automount[1007]: add_host_addrs: hostname lookup failed: Temporary failure in name resolution [08:37:15] we are screwed [08:38:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:38:42] PROBLEM host: i-0000031a.pmtpa.wmflabs is DOWN address: i-0000031a.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031a.pmtpa.wmflabs) [08:38:45] :-( [08:42:14] RECOVERY Disk Space is now: OK on deployment-apache33 i-0000031b.pmtpa.wmflabs output: DISK OK [08:42:15] !log deployment-prep on apache33 : removed /var/log symlink, recreated directory, restarted gluster, moving files form /data/project/apache33 [08:42:18] Logged the message, Master [08:45:28] 12/04/2012 - 08:45:28 - Creating a home directory for efdwiki1 at /export/keys/efdwiki1 [08:46:42] RECOVERY host: i-0000031a.pmtpa.wmflabs is UP address: i-0000031a.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.60 ms [08:47:12] RECOVERY Disk Space is now: OK on deployment-apache32 i-0000031a.pmtpa.wmflabs output: DISK OK [08:49:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:50:25] 12/04/2012 - 08:50:24 - Updating keys for efdwiki1 at /export/keys/efdwiki1 [08:52:23] !log deployment-prep Apache32 is somehow up [08:52:26] Logged the message, Master [08:52:39] petan: I think we will have to reinstall new apaches, the permissions under /var/log are screwed [08:53:11] hashar I don't think they are screwed more than how they are when they are installed by puppet [08:53:24] I mean by default they are screwed on all instances [08:53:34] something like 700 root root [08:53:38] on everything [08:53:45] ? [08:53:58] that is not the case on deployment-bastion [08:53:59] when you create a new instances /var/log is readable by root only [08:54:02] nor on the imagescalers [08:54:03] let me check [08:54:30] at least apaches were always unreadable [08:54:33] PROBLEM host: i-0000031b.pmtpa.wmflabs is DOWN address: i-0000031b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031b.pmtpa.wmflabs) [08:54:40] while on my own ubuntu it's readable [08:55:05] I think that is the default on Ubuntu [08:55:14] you need to be added to the adm group to be able to read them [08:55:21] on apache32 it's ok [08:55:27] iirc the Apche write their error somewhere under /data/project [08:55:32] yeah apache32 is back up [08:55:49] apache33 does not come up, no idea what is happening though [08:55:53] !console [08:55:54] in case you want to see what is happening on terminal of your vm, check console output [09:01:16] will need to ask ops to reboot apache33 for us :-D [09:03:52] RECOVERY host: i-0000031b.pmtpa.wmflabs is UP address: i-0000031b.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.66 ms [09:09:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:19:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:39:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:49:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:51:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 3% free memory [10:10:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:21:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:40:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:51:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:11:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:21:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:41:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:52:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:12:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:22:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:27:18] !log deployment-prep Apache boxes seems to be running again. Had to manually restart apache on apache33. [12:27:20] Logged the message, Master [12:27:24] lunch time [12:42:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:47:23] PROBLEM Total processes is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS WARNING: 166 processes [12:52:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:07:22] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS CRITICAL: 205 processes [13:12:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:16:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 10% free memory [13:22:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:41:37] Change on 12mediawiki a page Developer access was modified, changed by Psubhashish link https://www.mediawiki.org/w/index.php?diff=612873 edit summary: [13:42:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:52:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:12:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:16:03] PROBLEM Free ram is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Warning: 14% free memory [14:22:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:31:02] PROBLEM Free ram is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Critical: 3% free memory [14:36:03] RECOVERY Free ram is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: OK: 517% free memory [14:37:23] RECOVERY Total processes is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS OK: 113 processes [14:42:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:52:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:00:22] PROBLEM Total processes is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS WARNING: 157 processes [15:12:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:18:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 208 processes [15:22:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:23:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [15:25:22] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS CRITICAL: 211 processes [15:42:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:52:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:54:32] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 18% free memory [16:04:32] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 34% free memory [16:12:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:22:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:29:02] PROBLEM Free ram is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Warning: 15% free memory [16:42:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:44:03] PROBLEM Free ram is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Critical: 4% free memory [16:52:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:54:02] RECOVERY Free ram is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: OK: 706% free memory [16:55:22] RECOVERY Total processes is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS OK: 107 processes [17:13:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:22:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:23:22] PROBLEM Total processes is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS WARNING: 163 processes [17:34:32] PROBLEM Current Users is now: CRITICAL on puppetdoc5 i-0000052e.pmtpa.wmflabs output: Connection refused by host [17:35:13] PROBLEM Disk Space is now: CRITICAL on puppetdoc5 i-0000052e.pmtpa.wmflabs output: Connection refused by host [17:35:53] PROBLEM Current Load is now: CRITICAL on puppetdoc5 i-0000052e.pmtpa.wmflabs output: Connection refused by host [17:36:03] PROBLEM Free ram is now: CRITICAL on puppetdoc5 i-0000052e.pmtpa.wmflabs output: Connection refused by host [17:37:23] PROBLEM Total processes is now: CRITICAL on puppetdoc5 i-0000052e.pmtpa.wmflabs output: Connection refused by host [17:37:53] PROBLEM dpkg-check is now: CRITICAL on puppetdoc5 i-0000052e.pmtpa.wmflabs output: Connection refused by host [17:42:24] RECOVERY Total processes is now: OK on puppetdoc5 i-0000052e.pmtpa.wmflabs output: PROCS OK: 86 processes [17:42:54] RECOVERY dpkg-check is now: OK on puppetdoc5 i-0000052e.pmtpa.wmflabs output: All packages OK [17:43:24] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS CRITICAL: 202 processes [17:43:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:44:34] RECOVERY Current Users is now: OK on puppetdoc5 i-0000052e.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [17:45:13] RECOVERY Disk Space is now: OK on puppetdoc5 i-0000052e.pmtpa.wmflabs output: DISK OK [17:45:53] RECOVERY Current Load is now: OK on puppetdoc5 i-0000052e.pmtpa.wmflabs output: OK - load average: 1.05, 0.97, 0.67 [17:46:03] RECOVERY Free ram is now: OK on puppetdoc5 i-0000052e.pmtpa.wmflabs output: OK: 654% free memory [17:52:03] wangatlargo: hi, welcome! [17:52:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:01:36] wangatlargo: how are you settling in? things going well? [18:14:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:22:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:36:43] wangatlargo, do you know much about apache configuration? I could use a hand. [18:44:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:47:02] PROBLEM Free ram is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Warning: 13% free memory [18:52:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:57:03] PROBLEM Free ram is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Critical: 5% free memory [19:03:22] RECOVERY Total processes is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS OK: 98 processes [19:04:52] PROBLEM Current Load is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: WARNING - load average: 6.70, 13.53, 6.94 [19:07:03] RECOVERY Free ram is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: OK: 578% free memory [19:08:42] sumanah: who can accept shell requests for labs like https://labsconsole.wikimedia.org/wiki/Shell_Request/Bene [19:10:20] Merlissimo: as far as I know, wangatlargo is the kind of person who will be taking care of stuff like that. Damianz do you know? [19:10:57] ok, then ping wangatlargo ^^ ;-) [19:14:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:14:52] RECOVERY Current Load is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: OK - load average: 0.06, 1.91, 3.70 [19:16:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [19:21:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 7% free memory [19:22:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:28:32] <^demon> Ryan_Lane: Does the group labsadminbots actually contact gerrit for any read operations? [19:28:44] hm [19:28:49] maybe? [19:28:54] <^demon> s/read/write/ [19:28:56] <^demon> Blah. [19:29:00] <^demon> I don't care about read. [19:29:02] unlikely for writes [19:29:22] it likely writes to mediawiki, not gerrit [19:29:55] <^demon> labsadminbots has Vrfw+1 on a bunch of things. [19:30:13] <^demon> Like refs/meta/config at All-Projects level, which sounds almost certainly wrong. [19:32:32] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 17% free memory [19:36:22] PROBLEM Total processes is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS WARNING: 166 processes [19:44:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:47:33] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 29% free memory [19:52:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:54:18] Andrew: I configured Apache server which running LAMP, nagios, Munin, Cacti etc. I also configured virtual hosts before. So I know some Apache configuration, but not too much. [19:56:22] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS CRITICAL: 206 processes [20:05:27] 12/04/2012 - 20:05:27 - Updating keys for mwang at /export/keys/mwang [20:05:32] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 17% free memory [20:11:37] gah, missed sumanah again [20:12:02] <^demon> jeremyb: She's in other channels. [20:12:12] aha [20:12:24] so she is [20:15:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:16:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [20:17:54] ^demon: that group may have included gerrit2 [20:18:23] <^demon> Bah. [20:18:23] <^demon> FML [20:18:27] <^demon> I hate gerrit2. [20:18:29] <^demon> gerrit2 sucks. [20:18:42] <^demon> But gerrit2 doesn't write anymore, only read, so he's almost dead ;-) [20:19:54] yeah [20:19:59] so you can take away write [20:23:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:24:26] <^demon> Ryan_Lane: I want https://gerrit-review.googlesource.com/Documentation/cmd-ls-user-refs.html [20:24:33] <^demon> It will make auditing crap like this so much easier. [20:25:17] ^demon: heh [20:25:29] lots of stuff to want :) [20:35:29] 12/04/2012 - 20:35:28 - Updating keys for mwang at /export/keys/mwang [20:45:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:53:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:59:00] wangatlargo: were you able to get into bastion-restricted? [21:00:12] PROBLEM Free ram is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Warning: 13% free memory [21:10:53] PROBLEM Free ram is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Critical: 4% free memory [21:15:53] RECOVERY Free ram is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: OK: 634% free memory [21:16:23] RECOVERY Total processes is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS OK: 106 processes [21:16:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:24:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:35:04] wangatlargo: you there? [21:42:53] <^demon> Ryan_Lane: Managed to get a plugin building in jenkins -- good, I was afraid it wouldn't work. [21:43:05] <^demon> (There's been weird SNAPSHOT dependencies in master at various points) [21:44:22] PROBLEM Total processes is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS WARNING: 162 processes [21:46:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:54:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:04:23] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS CRITICAL: 203 processes [22:17:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:24:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:44:38] ^demon|busy: that's odd [22:45:32] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 156 processes [22:47:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:53:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 8.91, 7.17, 5.68 [22:54:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:55:02] <^demon|busy> Ryan_Lane: What's odd? Stupid dependencies? [22:55:11] yeah [22:55:18] <^demon|busy> It's maven, what else do you expect? [22:56:37] <^demon|busy> I think I've fixed all of the non-NPE bugs in the uuid thing. [22:56:45] <^demon|busy> But the NPEs are making me tear my hair out. [23:01:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 4% free memory [23:02:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 9.80, 8.88, 6.60 [23:06:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.24, 7.31, 5.83 [23:07:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.69, 6.85, 5.74 [23:08:52] PROBLEM Free ram is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Warning: 13% free memory [23:17:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:23:52] PROBLEM Free ram is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:25:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:25:33] PROBLEM Current Users is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:26:43] PROBLEM SSH is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Server answer: [23:26:53] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:27:13] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:27:53] PROBLEM Current Load is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:48:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:52:33] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory [23:56:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%