[00:01:05] RECOVERY Disk Space is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: DISK OK [00:01:44] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 58% free memory [00:01:54] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:09:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:32:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:39:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:41:43] RECOVERY Free ram is now: OK on su-fe2 i-000002e6.pmtpa.wmflabs output: OK: 446% free memory [00:47:43] RECOVERY Free ram is now: OK on aggregator-test1 i-000002bf.pmtpa.wmflabs output: OK: 89% free memory [00:48:46] PROBLEM Total processes is now: CRITICAL on su-be1 i-000002e7.pmtpa.wmflabs output: PROCS CRITICAL: 227 processes [00:48:46] PROBLEM Total processes is now: CRITICAL on su-be2 i-000002e8.pmtpa.wmflabs output: PROCS CRITICAL: 235 processes [00:51:42] RECOVERY Free ram is now: OK on pdbhandler-1 i-0000030e.pmtpa.wmflabs output: OK: 796% free memory [00:52:32] PROBLEM Total processes is now: CRITICAL on aggregator-test1 i-000002bf.pmtpa.wmflabs output: PROCS CRITICAL: 241 processes [00:53:42] PROBLEM Total processes is now: CRITICAL on su-be3 i-000002e9.pmtpa.wmflabs output: PROCS CRITICAL: 235 processes [00:53:42] RECOVERY Total processes is now: OK on sultest1 i-0000032d.pmtpa.wmflabs output: PROCS OK: 76 processes [00:54:22] RECOVERY Current Load is now: OK on sultest1 i-0000032d.pmtpa.wmflabs output: OK - load average: 0.20, 0.38, 0.21 [00:55:33] RECOVERY Current Users is now: OK on sultest1 i-0000032d.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [00:56:13] RECOVERY Disk Space is now: OK on sultest1 i-0000032d.pmtpa.wmflabs output: DISK OK [00:56:43] RECOVERY Free ram is now: OK on sultest1 i-0000032d.pmtpa.wmflabs output: OK: 435% free memory [00:56:43] RECOVERY Free ram is now: OK on testing-singer-puppetization i-00000331.pmtpa.wmflabs output: OK: 278% free memory [01:01:42] RECOVERY Free ram is now: OK on tutopuppet i-00000336.pmtpa.wmflabs output: OK: 339% free memory [01:03:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:06:03] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 175 processes [01:06:57] RECOVERY Free ram is now: OK on extrev1 i-00000346.pmtpa.wmflabs output: OK: 653% free memory [01:06:57] RECOVERY Free ram is now: OK on rocsteady-cleanup i-00000349.pmtpa.wmflabs output: OK: 89% free memory [01:09:06] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:11:02] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 99 processes [01:11:52] RECOVERY Free ram is now: OK on wlmpuppet i-0000035c.pmtpa.wmflabs output: OK: 266% free memory [01:21:52] RECOVERY Free ram is now: OK on robh-spl i-00000369.pmtpa.wmflabs output: OK: 254% free memory [01:21:52] RECOVERY Free ram is now: OK on maps-osmrails i-00000373.pmtpa.wmflabs output: OK: 992% free memory [01:21:52] RECOVERY Free ram is now: OK on gerrit-db i-0000038b.pmtpa.wmflabs output: OK: 715% free memory [01:22:03] PROBLEM Free ram is now: CRITICAL on jesusaurus-cleanup i-0000038a.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [01:26:52] RECOVERY Free ram is now: OK on jesusaurus-cleanup i-0000038a.pmtpa.wmflabs output: OK: 66% free memory [01:26:52] RECOVERY Free ram is now: OK on solr-ci i-00000391.pmtpa.wmflabs output: OK: 316% free memory [01:26:52] RECOVERY Free ram is now: OK on maps-osmmapnik i-0000039b.pmtpa.wmflabs output: OK: 1037% free memory [01:31:52] RECOVERY Free ram is now: OK on blamemaps-m1xsmall i-0000039e.pmtpa.wmflabs output: OK: 211% free memory [01:31:52] RECOVERY Free ram is now: OK on mars i-000003a8.pmtpa.wmflabs output: OK: 881% free memory [01:33:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:33:32] RECOVERY Current Users is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [01:39:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:02:42] RECOVERY Disk Space is now: OK on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK OK [02:03:05] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:09:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:10:42] PROBLEM Disk Space is now: WARNING on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK WARNING - free space: / 76 MB (5% inode=51%): [02:21:36] PROBLEM Current Users is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [02:33:04] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:36:55] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 19% free memory [02:39:46] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [02:40:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:40:52] PROBLEM Free ram is now: WARNING on integration-jenkins i-00000363.pmtpa.wmflabs output: Warning: 15% free memory [02:41:55] RECOVERY Free ram is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK: 20% free memory [02:47:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 19% free memory [03:03:05] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:04:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 19% free memory [03:10:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:33:53] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:40:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:49:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 9.30, 8.86, 6.30 [03:51:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 8.69, 7.85, 5.85 [03:55:01] hi Ryan_Lane [03:56:20] howdy [03:56:23] I tried to fix it [03:56:26] didn't work [03:56:36] I can only access it via remote execution [03:56:47] which means I can only do the basics [03:57:30] Ryan_Lane: there's nothing useful left there afais [03:57:37] but labsconsole doesn't work for me now [03:57:42] ah ok [03:57:42] no? [03:57:45] what isue? [03:57:57] ah [03:57:57] it says "No Nova credentials found for your account." but I'm logged in [03:57:57] no creds [03:58:13] log out and back in [03:58:28] I think something is leaking memory and dropping sessions from memcache [03:59:28] are we expecting every use has nova creds? [03:59:39] or maybe I have my timing screwed up between mediawiki and keystone [03:59:44] every nova use requires it [04:00:00] I'm does every wiki user have one? [04:00:04] yeah [04:00:12] but you won't see that error unless you use a nova interface [04:00:12] RECOVERY Disk Space is now: OK on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK OK [04:00:22] if so, maybe we can auto log users out of wiki if we find 'no creds' on him [04:00:38] yeah, probably should [04:01:00] realistically it shouldn't happen [04:01:09] keystone creds shouldn't expire before the mediawiki token [04:01:34] eventually I'll just switch to keystone auth from ldap [04:01:42] but that's going to take some work [04:02:20] Ryan_Lane: I clicked delete on it but it's still listed in Special:NovaInstance? [04:02:31] yeah. takes a sec to go away [04:02:53] it makes a request to nova, nova eventually deletes it [04:03:03] it'll still show it as active until it's actually deleted [04:03:20] ok there's another issue in Special:NovaAddress [04:03:22] oh? [04:03:27] (Remove host name) doesn't work [04:03:28] (that isn't surprising :)) [04:03:35] yeah, the dns code is full of bugs [04:03:37] it says "The requested host does not exist. " [04:03:40] which one? [04:04:05] proxy.wikipedia.wmflabs.org on 208.80.153.147 [04:04:05] which project is this again? [04:04:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:04:17] no need to delete the hostname [04:04:23] you can add it on the new instance [04:04:31] it's a floating IP [04:04:34] the host is added to the IP [04:04:54] I know it but is an unused one wasting resource? [04:05:03] or prevent someone else from using it [04:05:15] do you plan on using it? [04:05:19] on a new instance? [04:05:29] if so, no need to let it go [04:06:29] if you aren't going to use it, I can clean it up manually [04:06:29] not really, or just *maybe* used in some say but allocating one seems easy [04:06:52] * Ryan_Lane nods [04:06:52] ok. lemme clean up the hostname [04:07:30] maybe you'd better just fix the removal code [04:07:41] so, I've been avoiding that [04:07:51] because we're going to completely replace the dns code [04:07:51] with a service [04:07:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: WARNING - load average: 5.50, 5.69, 5.17 [04:08:12] PROBLEM Disk Space is now: WARNING on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [04:08:36] Ryan_Lane: that's ok [04:08:50] lots of other bugs to fix. until then I can just clean manually [04:08:54] it's not too often I need to do so [04:09:46] cleaned it [04:09:53] you should be able to deallocate now [04:11:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:13:34] Ryan_Lane: you cleaned everything on that ip... [04:17:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: OK - load average: 2.83, 3.39, 4.30 [04:18:50] Ryan_Lane: and see the complaint in labs-l [04:26:56] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.05, 3.15, 4.64 [04:27:40] wait, was I not supposed to do that? [04:27:52] liangent: I'm confused [04:28:48] I thought the service wasn't working et [04:28:48] *yet [04:29:54] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.50, 3.38, 4.98 [04:34:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:41:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs CRITICAL - Plugin timed out after 10 seconds [05:02:25] Ryan_Lane: only proxy.wikipedia.wmflabs.org ... the other two should be left [05:02:44] and the proxy is working [05:04:16] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:04:33] Ryan_Lane: btw I can't see its public ip with `ifconfig`? though the ip is working [05:11:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:29:44] PROBLEM Free ram is now: WARNING on aggregator2 i-000002c0.pmtpa.wmflabs output: Warning: 7% free memory [05:30:33] RECOVERY Current Users is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [05:30:47] liangent: ah [05:30:54] liangent: didn't know the others should be there [05:30:57] liangent: I'll put it back [05:31:02] the IP is NAT [05:31:08] so you won't see it [05:31:23] RECOVERY Current Load is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: OK - load average: 0.51, 0.57, 0.59 [05:31:37] Ryan_Lane: I already added them back [05:31:40] oh [05:31:41] cool [05:31:53] RECOVERY SSH is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [05:31:53] RECOVERY Disk Space is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: DISK OK [05:32:53] RECOVERY dpkg-check is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: All packages OK [05:33:33] PROBLEM Total processes is now: UNKNOWN on aggregator2 i-000002c0.pmtpa.wmflabs output: NRPE: Call to fork() failed [05:34:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:34:43] PROBLEM Free ram is now: UNKNOWN on aggregator2 i-000002c0.pmtpa.wmflabs output: NRPE: Call to fork() failed [05:38:32] PROBLEM Current Users is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:38:32] PROBLEM Total processes is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:39:22] PROBLEM Current Load is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:39:42] PROBLEM Free ram is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:39:52] PROBLEM SSH is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: Server answer: [05:39:52] PROBLEM Disk Space is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:40:53] PROBLEM dpkg-check is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:41:56] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:46:12] PROBLEM Disk Space is now: CRITICAL on sube i-000003d0.pmtpa.wmflabs output: DISK CRITICAL - free space: / 11 MB (0% inode=40%): [05:51:16] PROBLEM Disk Space is now: WARNING on sube i-000003d0.pmtpa.wmflabs output: DISK WARNING - free space: / 41 MB (3% inode=40%): [05:52:38] !htmllogs [05:52:41] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [06:04:27] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:12:36] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:34:12] PROBLEM dpkg-check is now: CRITICAL on nova-ldap2 i-00000238.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [06:34:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:38:13] RECOVERY Disk Space is now: OK on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK OK [06:39:13] RECOVERY dpkg-check is now: OK on nova-ldap2 i-00000238.pmtpa.wmflabs output: All packages OK [06:43:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:47:23] PROBLEM dpkg-check is now: CRITICAL on micro-design i-000003e8.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [06:52:45] RECOVERY dpkg-check is now: OK on micro-design i-000003e8.pmtpa.wmflabs output: All packages OK [06:59:58] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Normal - 6normal) [Bug 37080] upload, thumbnails and transcoding on beta (tracking) - https://bugzilla.wikimedia.org/show_bug.cgi?id=37080 [06:59:59] [bz] (8UNCONFIRMED - created by: 2Damian Z, priority: 4Unprioritized - 6minor) [Bug 38792] Thumbnails are broken - https://bugzilla.wikimedia.org/show_bug.cgi?id=38792 [07:02:52] PROBLEM Free ram is now: WARNING on changefeed-bot i-0000041b.pmtpa.wmflabs output: Warning: 15% free memory [07:04:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:07:53] RECOVERY Free ram is now: OK on changefeed-bot i-0000041b.pmtpa.wmflabs output: OK: 20% free memory [07:13:31] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:34:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:44:06] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:51:32] RECOVERY Current Users is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [08:04:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:11:13] PROBLEM Disk Space is now: WARNING on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [08:14:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:32:52] PROBLEM Free ram is now: UNKNOWN on changefeed-bot i-0000041b.pmtpa.wmflabs output: Invalid host name i-0000041b.pmtpa.wmflabs [08:34:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:36:12] RECOVERY Disk Space is now: OK on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK OK [08:45:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:59:15] PROBLEM Disk Space is now: WARNING on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [09:02:19] Change on 12mediawiki a page Developer access was modified, changed by Greeenjohnny link https://www.mediawiki.org/w/index.php?diff=597421 edit summary: [09:03:51] Change on 12mediawiki a page Developer access was modified, changed by Greeenjohnny link https://www.mediawiki.org/w/index.php?diff=597423 edit summary: /* User:Greeenjohnny */ [09:04:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:07:30] Change on 12mediawiki a page Developer access was modified, changed by Greeenjohnny link https://www.mediawiki.org/w/index.php?diff=597425 edit summary: /* User:Greeenjohnny */ [09:15:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:20:52] RECOVERY Free ram is now: OK on integration-jenkins i-00000363.pmtpa.wmflabs output: OK: 118% free memory [09:34:24] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:46:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:04:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:11:06] 10/25/2012 - 10:11:05 - User .docs has been renamed, moving home directory in project(s): wikidata-dev [10:13:35] Maybe I should open a puppet questions channel... [10:14:23] After pulling changes from gerrit I keep getting error messages I didn't see before. [10:15:21] puppet now apparently wants to access the private part that I don't have... [10:15:26] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not parse for environment production: No file(s) found for import of '../private/manifests/passwords.pp' at /etc/puppet/manifests/base.pp:10 on node wikidata-dev-4.pmtpa.wmflabs [10:16:08] Though I do not include base.pp anywhere... [10:16:09] 10/25/2012 - 10:16:08 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:16:46] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:17:18] Any ideas? [10:21:01] 10/25/2012 - 10:21:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:26:02] 10/25/2012 - 10:26:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:30:21] Well, actually the private files *are* there. [10:31:02] 10/25/2012 - 10:30:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:34:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:36:14] 10/25/2012 - 10:36:10 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:41:14] 10/25/2012 - 10:41:10 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:46:12] 10/25/2012 - 10:46:12 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:46:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:51:07] 10/25/2012 - 10:51:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:56:06] 10/25/2012 - 10:56:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [10:56:13] hashar Jan_Luca Hi! Question: What happens to ownership and/or file permissions in a puppetmaster::self when I execute "sudo GIT_SSH=/var/lib/git/ssh git pull --rebase" for the first time? Something seems has changed since yesterday. [10:57:23] Silke_WMDE: files under /var/lib/git/operations/puppet are fetched by puppet and should belong to root:root [10:57:44] so `GIT_SSH=/var/lib/git/ssh git pull --rebase` should probably be run as root too [10:57:52] PROBLEM Free ram is now: WARNING on changefeed-bot i-0000041b.pmtpa.wmflabs output: Warning: 19% free memory [10:57:59] something like sudo 'GIT_SSH=/var/lib/git/ssh git pull --rebase' [10:58:33] ah. I did "sudo" as smeyer. Now files belong to smeyer:root [10:59:15] so maybe use sudo -s first [10:59:24] and then run git [11:01:01] 10/25/2012 - 11:01:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:01:02] Silke_WMDE: When you run "sudo -s" you get a root shell where you can run your commands as root [11:01:43] :) I know. [11:02:13] I'll try to chown the directory [11:03:01] But when using sudo -s new files should be root:root [11:04:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:06:04] 10/25/2012 - 11:06:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:06:38] Jan_Luca: In the documentation for puppetmaster::self, it says "sudo GIT..." so I switched explicitly to smeyer and then ran sudo. [11:06:48] chown root:root doesn't help [11:07:04] It still says "err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not parse for environment production: No file(s) found for import of '../private/manifests/passwords.pp' at /etc/puppet/manifests/base.pp:10 on node wikidata-dev-4.pmtpa.wmflabs" [11:07:08] chown -R root:root should work [11:07:47] it does't [11:08:16] ahhh [11:08:24] Silke_WMDE: that is a different issue I guess [11:08:33] Silke_WMDE: you need to checkout the labs/private repository as well I guess [11:08:38] did you check that there is a symlink /etc/puppet/private -> private-copy [11:08:55] hashar: puppetmaster::sel should do this [11:09:02] ohh [11:09:51] the symlink exists (/etc/puppet/private -> /var/lib/git/labs/private) and there are files in the private repo [11:10:17] also that passwords.pp from the error message is there. [11:11:01] 10/25/2012 - 11:11:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:12:44] passwords.pp is now owned by root:root and has 644 permissions. [11:16:02] 10/25/2012 - 11:15:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:17:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:21:01] 10/25/2012 - 11:21:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:26:06] 10/25/2012 - 11:26:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:31:03] 10/25/2012 - 11:31:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:34:44] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:36:02] 10/25/2012 - 11:36:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:41:00] 10/25/2012 - 11:40:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:46:00] 10/25/2012 - 11:45:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:47:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:48:52] PROBLEM Current Load is now: CRITICAL on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: Connection refused by host [11:49:32] PROBLEM Current Users is now: CRITICAL on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: Connection refused by host [11:50:13] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: Connection refused by host [11:51:04] PROBLEM Free ram is now: CRITICAL on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: Connection refused by host [11:51:08] 10/25/2012 - 11:51:08 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:52:23] PROBLEM Total processes is now: CRITICAL on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: Connection refused by host [11:52:53] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: Connection refused by host [11:56:02] RECOVERY Free ram is now: OK on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: OK: 493% free memory [11:56:09] 10/25/2012 - 11:56:09 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [11:57:22] RECOVERY Total processes is now: OK on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: PROCS OK: 93 processes [11:57:52] RECOVERY dpkg-check is now: OK on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: All packages OK [11:58:52] RECOVERY Current Load is now: OK on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: OK - load average: 1.39, 0.55, 0.20 [11:59:32] RECOVERY Current Users is now: OK on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: USERS OK - 2 users currently logged in [12:00:13] RECOVERY Disk Space is now: OK on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: DISK OK [12:01:09] 10/25/2012 - 12:01:09 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:04:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:06:08] 10/25/2012 - 12:06:08 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:11:03] 10/25/2012 - 12:11:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:16:04] 10/25/2012 - 12:16:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:17:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:21:08] 10/25/2012 - 12:21:08 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:26:10] 10/25/2012 - 12:26:10 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:31:09] 10/25/2012 - 12:31:08 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:34:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:34:52] hashar: The solution was: /var/lib/git/labs should be owned by root:puppet. :) [12:34:53] PROBLEM host: i-000004e2.pmtpa.wmflabs is DOWN address: i-000004e2.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000004e2.pmtpa.wmflabs) [12:36:02] 10/25/2012 - 12:36:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:36:58] Silke_WMDE_: ohhh [12:37:40] Silke_WMDE_: puppet group ;) [12:37:45] Silke_WMDE_: well done [12:37:48] yep [12:38:08] Abraham_WMDE1: I just installed a Wikidata repo via puppet [12:38:30] Silke_WMDE_: awesome, thx [12:38:50] Silke_WMDE_: This seems to be a dangerous thing ;-) [12:40:02] RECOVERY host: i-000004e2.pmtpa.wmflabs is UP address: i-000004e2.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 505.21 ms [12:41:15] 10/25/2012 - 12:41:14 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:43:53] PROBLEM Current Load is now: CRITICAL on integration-jobbuilder i-000004e3.pmtpa.wmflabs output: Connection refused by host [12:44:33] PROBLEM Current Users is now: CRITICAL on integration-jobbuilder i-000004e3.pmtpa.wmflabs output: Connection refused by host [12:45:13] PROBLEM Disk Space is now: CRITICAL on integration-jobbuilder i-000004e3.pmtpa.wmflabs output: Connection refused by host [12:45:53] PROBLEM Free ram is now: CRITICAL on integration-jobbuilder i-000004e3.pmtpa.wmflabs output: Connection refused by host [12:46:04] 10/25/2012 - 12:46:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:47:23] PROBLEM Total processes is now: CRITICAL on integration-jobbuilder i-000004e3.pmtpa.wmflabs output: Connection refused by host [12:47:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:49:33] RECOVERY Current Users is now: OK on integration-jobbuilder i-000004e3.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [12:50:12] RECOVERY Disk Space is now: OK on integration-jobbuilder i-000004e3.pmtpa.wmflabs output: DISK OK [12:50:52] RECOVERY Free ram is now: OK on integration-jobbuilder i-000004e3.pmtpa.wmflabs output: OK: 550% free memory [12:51:03] 10/25/2012 - 12:50:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [12:52:22] RECOVERY Total processes is now: OK on integration-jobbuilder i-000004e3.pmtpa.wmflabs output: PROCS OK: 84 processes [12:53:52] RECOVERY Current Load is now: OK on integration-jobbuilder i-000004e3.pmtpa.wmflabs output: OK - load average: 0.17, 0.50, 0.46 [12:56:12] 10/25/2012 - 12:56:10 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:01:16] 10/25/2012 - 13:01:16 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:04:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:05:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 196 processes [13:05:57] 10/25/2012 - 13:05:57 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:10:36] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 103 processes [13:11:05] 10/25/2012 - 13:11:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:16:07] 10/25/2012 - 13:16:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:18:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs CRITICAL - Plugin timed out after 10 seconds [13:21:07] 10/25/2012 - 13:21:07 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:26:13] 10/25/2012 - 13:26:13 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:31:15] 10/25/2012 - 13:31:15 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:34:47] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:36:05] 10/25/2012 - 13:36:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:41:05] 10/25/2012 - 13:41:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:46:05] 10/25/2012 - 13:46:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:48:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:51:01] 10/25/2012 - 13:51:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [13:56:02] 10/25/2012 - 13:56:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:01:06] 10/25/2012 - 14:01:06 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:04:44] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:06:10] 10/25/2012 - 14:06:10 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:11:06] 10/25/2012 - 14:11:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:12:52] RECOVERY Free ram is now: OK on changefeed-bot i-0000041b.pmtpa.wmflabs output: OK: 20% free memory [14:16:15] 10/25/2012 - 14:16:15 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:18:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:21:09] 10/25/2012 - 14:21:09 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:25:52] PROBLEM Free ram is now: WARNING on changefeed-bot i-0000041b.pmtpa.wmflabs output: Warning: 19% free memory [14:26:00] 10/25/2012 - 14:26:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:26:12] PROBLEM Disk Space is now: CRITICAL on sube i-000003d0.pmtpa.wmflabs output: DISK CRITICAL - free space: / 27 MB (2% inode=40%): [14:31:03] 10/25/2012 - 14:31:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:31:13] PROBLEM Disk Space is now: WARNING on sube i-000003d0.pmtpa.wmflabs output: DISK WARNING - free space: / 41 MB (3% inode=40%): [14:35:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:36:02] 10/25/2012 - 14:36:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:41:07] 10/25/2012 - 14:41:07 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:46:01] 10/25/2012 - 14:46:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:48:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:51:12] 10/25/2012 - 14:51:09 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [14:56:04] 10/25/2012 - 14:56:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:01:05] 10/25/2012 - 15:01:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:04:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 196 processes [15:06:05] 10/25/2012 - 15:06:04 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:06:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:11:09] 10/25/2012 - 15:11:07 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:16:05] 10/25/2012 - 15:16:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:19:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:21:18] 10/25/2012 - 15:21:17 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:24:21] Change on 12mediawiki a page Developer access was modified, changed by Matthewrbowker link https://www.mediawiki.org/w/index.php?diff=597518 edit summary: [15:25:59] 10/25/2012 - 15:25:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:29:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 103 processes [15:31:12] 10/25/2012 - 15:31:12 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:36:15] 10/25/2012 - 15:36:14 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:37:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:41:06] 10/25/2012 - 15:41:05 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:46:11] 10/25/2012 - 15:46:10 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:49:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:51:05] 10/25/2012 - 15:51:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [15:56:10] 10/25/2012 - 15:56:10 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [16:01:11] 10/25/2012 - 16:01:11 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [16:06:01] 10/25/2012 - 16:06:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [16:08:05] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:09:33] PROBLEM Current Users is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [16:11:03] 10/25/2012 - 16:11:03 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [16:15:53] RECOVERY Free ram is now: OK on changefeed-bot i-0000041b.pmtpa.wmflabs output: OK: 20% free memory [16:16:03] 10/25/2012 - 16:16:02 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [16:19:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:21:02] 10/25/2012 - 16:21:01 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [16:23:55] ^demon: Do you know why Gerrit is down? [16:24:06] <^demon> Announced downtime. [16:24:15] <^demon> We're upgrading the server. It's almost done. [16:24:52] Ok, it seems that I overseen this mail... [16:25:57] <^demon> Jan_Luca: Back up now. [16:26:09] 10/25/2012 - 16:26:07 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [16:29:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.04, 6.31, 5.52 [16:31:00] 10/25/2012 - 16:31:00 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [16:31:41] <^demon> Ryan_Lane: manganese & formey are both on precise now. [16:33:53] PROBLEM Free ram is now: WARNING on changefeed-bot i-0000041b.pmtpa.wmflabs output: Warning: 19% free memory [16:36:19] 10/25/2012 - 16:36:18 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [16:37:07] ^demon: \o/ [16:38:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:39:32] RECOVERY Current Users is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [16:41:08] 10/25/2012 - 16:41:07 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [16:46:11] 10/25/2012 - 16:46:10 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [16:46:42] ^demon: Thank you [16:46:54] <^demon> you're welcome. [16:47:32] PROBLEM Current Users is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [16:50:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:51:03] 10/25/2012 - 16:50:59 - User .lib has been renamed, moving home directory in project(s): wikidata-dev [16:53:14] andrewbogott: Could you review my change? [16:53:34] Sure, just a moment... [16:55:18]