[00:10:48] Ryan_Lane, are security groups as listed by 'nova secgroup-list' a different thing from security groups as managed in labsconsole? [00:10:58] no [00:10:59] same [00:11:16] are you passing the project? [00:11:20] when using the nova command? [00:11:34] Trying, but I got the same list each time (possibly because the project isn't getting in right.) [00:11:48] like: OS_TENANT_NAME=visualeditor nova secgroup-list [00:11:56] I'm wondering why the 'security group' column is empty on the manage instances page. Is that something that worked once upon a time? [00:11:57] all on the same line [00:12:05] oh [00:12:07] I know why [00:12:14] because there's no way to get that info from the API [00:12:29] So, never worked then? [00:12:37] it works in the ec2 api [00:12:39] but not in nova's [00:12:48] missing feature... [00:13:28] I'd love for it to be added again [00:13:34] * andrewbogott makes a note [00:13:40] I was going to handle it via the metadata, but that's annoying [00:14:43] It's on the instance page, but… adding it to the API is probably still the right solution for getting it on that main table. [00:16:11] yeah [00:16:26] I think I removed it as a column from "manage instances" right? [00:16:59] I should really make the column list configurable and accessible via callbacks [00:17:14] too much hardcoded crap [00:17:25] that would eliminate a decent amount of code, too [00:18:22] meh. too many things to do :D [00:18:59] The column is still there, caused me brief confustion [00:19:04] I need to fix the hostname/sudo problem [00:19:04] um… confusion [00:19:24] is the hostname/sudo problem the thing I hacked yesterday? [00:19:25] that's actually a fairly easy fix. need to rename all old instance's hostname [00:19:36] eh? how'd you hack it? [00:19:40] that's a fairly icky set of code [00:19:51] https://gerrit.wikimedia.org/r/#/c/34036/ [00:20:13] I was just trying to make the page not crash, didn't think very hard. [00:20:22] oh [00:20:22] no [00:20:24] that's something else [00:20:31] didn't check into why that might be a problem [00:20:44] maybe an instance didn't get updated in ldap for its arecord? [00:22:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:26:33] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 150 processes [00:26:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:52:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:56:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:00:23] 11/20/2012 - 01:00:23 - Creating a home directory for bruceburge at /export/keys/bruceburge [01:05:24] 11/20/2012 - 01:05:23 - Updating keys for bruceburge at /export/keys/bruceburge [01:07:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 174 processes [01:12:32] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:22:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:26:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:53:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:56:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:23:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:26:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:39:43] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory [02:53:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:56:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:05:53] PROBLEM Free ram is now: WARNING on watchlist-bot i-0000041a.pmtpa.wmflabs output: Warning: 13% free memory [03:07:43] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 15% free memory [03:23:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:28:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:45:52] RECOVERY Free ram is now: OK on watchlist-bot i-0000041a.pmtpa.wmflabs output: OK: 42% free memory [03:47:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 155 processes [03:53:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:58:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:59:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 6.69, 6.23, 5.33 [04:23:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:28:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:54:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:58:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:23:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.96, 7.31, 5.71 [05:23:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.44, 6.00, 5.25 [05:24:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:28:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 5.90, 5.61, 5.17 [05:29:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:42:13] PROBLEM Disk Space is now: CRITICAL on deployment-apache33 i-0000031b.pmtpa.wmflabs output: DISK CRITICAL - free space: / 0 MB (0% inode=74%): [05:49:12] PROBLEM Disk Space is now: CRITICAL on deployment-apache32 i-0000031a.pmtpa.wmflabs output: DISK CRITICAL - free space: / 0 MB (0% inode=74%): [05:54:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:58:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 5.06, 4.53, 5.00 [05:59:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:08:33] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.79, 4.47, 4.82 [06:08:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.82, 4.18, 4.91 [06:16:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 4.60, 5.23, 5.13 [06:24:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:29:16] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:30:23] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:30:33] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 154 processes [06:33:25] Change on 12mediawiki a page Developer access was modified, changed by יוסף שמח link https://www.mediawiki.org/w/index.php?diff=607600 edit summary: [06:33:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 201 processes [06:33:53] PROBLEM dpkg-check is now: CRITICAL on abusefilter-global-main i-00000512.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [06:38:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [06:38:53] RECOVERY dpkg-check is now: OK on abusefilter-global-main i-00000512.pmtpa.wmflabs output: All packages OK [06:40:23] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 146 processes [06:40:53] PROBLEM dpkg-check is now: CRITICAL on kubo i-000003dd.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [06:45:33] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 145 processes [06:45:53] RECOVERY dpkg-check is now: OK on kubo i-000003dd.pmtpa.wmflabs output: All packages OK [06:49:57] Change on 12mediawiki a page Developer access was modified, changed by Akoppad link https://www.mediawiki.org/w/index.php?diff=607603 edit summary: [06:54:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:59:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:03:21] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=607604 edit summary: /* User:Akoppad */ done [07:04:00] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=607605 edit summary: /* User:Akoppad */ [07:16:13] PROBLEM Disk Space is now: WARNING on sube i-000003d0.pmtpa.wmflabs output: DISK WARNING - free space: / 42 MB (3% inode=40%): [07:21:12] PROBLEM Disk Space is now: CRITICAL on sube i-000003d0.pmtpa.wmflabs output: DISK CRITICAL - free space: / 30 MB (2% inode=40%): [07:25:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:29:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:32:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [07:56:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:59:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:00:19] Change on 12mediawiki a page Developer access was modified, changed by Mugii link https://www.mediawiki.org/w/index.php?diff=607631 edit summary: [08:26:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:29:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:56:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:59:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:04:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.48, 4.75, 4.95 [09:26:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:29:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:32:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 5.59, 5.28, 5.13 [09:57:27] Change on 12mediawiki a page Developer access was modified, changed by Netha Hussain link https://www.mediawiki.org/w/index.php?diff=607671 edit summary: [09:57:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:01:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:27:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:31:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:58:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:01:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:28:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:31:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:36:47] hey all - I requested developer access at mediawiki.org and got apositive reply (https://www.mediawiki.org/wiki/Developer_access#User:TK-999 ). Now https://labsconsole.wikimedia.org/wiki/Help:Access instructs me to upload my SSH key, but NovaKey says I have no Nova credentials. Is there something I'm missing? [11:52:54] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.15, 4.31, 4.80 [11:59:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:01:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:24:31] Change on 12mediawiki a page Developer access was modified, changed by Adithya link https://www.mediawiki.org/w/index.php?diff=607723 edit summary: [12:29:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:31:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:38:16] nvm, resolved itself [12:59:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:01:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:29:04] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:33:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:35:25] 11/20/2012 - 13:35:25 - Updating keys for thibaultmarin at /export/keys/thibaultmarin [13:59:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:03:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:13:34] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 202 processes [14:18:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [14:29:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:33:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:00:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:03:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:31:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:33:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:43:51] !g I4fcea063cd59860a17bd1058f5998d6871edd10d [15:43:52] https://gerrit.wikimedia.org/r/#q,I4fcea063cd59860a17bd1058f5998d6871edd10d,n,z [15:47:14] !g I9f68819d09bb58b82bd987e8ff00db0d26095fd9 [15:47:14] https://gerrit.wikimedia.org/r/#q,I9f68819d09bb58b82bd987e8ff00db0d26095fd9,n,z [16:01:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:03:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:27:42] RECOVERY Free ram is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: 3167000 [16:31:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:33:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:51:10] Change on 12mediawiki a page Developer access was modified, changed by Odie5533 link https://www.mediawiki.org/w/index.php?diff=607800 edit summary: [17:00:47] Hi! Is any of those present who worked on vagrant and mediawiki? [17:01:16] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:01:43] How do you deal with the problem that in /srv your can't modify file permissions? [17:02:44] This vboxsf doesn't allow it. How does your setup work? [17:03:36] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:31:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:33:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:01:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:03:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:31:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:33:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:01:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:03:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:26:43] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 5% free memory [19:32:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:33:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:03:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:04:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:33:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:34:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:45:34] <^demon> Ryan_Lane: https://gerrit.wikimedia.org/r/#/c/34362/ :) [20:45:38] <^demon> I'm removing crusty code. [20:46:28] do we no longer use the gerrit2 user? [20:46:40] want me to merge this? looks good to me [20:47:06] <^demon> We still use gerrit2, working on that. [20:47:09] <^demon> But less than before. [20:47:11] cool [20:47:20] is this ready for merge? [20:47:26] <^demon> Yeah. [20:47:53] <^demon> I announced that on ops and no one said "hey wait" [20:49:20] cool [20:54:13] ^demon, is it possible to add a feature like that for bugzilla? [21:02:26] <^demon> Krenair: I plan on doing it for BZ and RT. [21:02:31] <^demon> But not in the hacky way I just removed. [21:02:58] ok :) [21:04:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:04:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:34:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:34:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:04:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:04:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:34:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:34:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:04:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:05:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:34:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:35:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 156 processes [23:35:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:42:25] PROBLEM Disk Space is now: CRITICAL on kubo i-000003dd.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [23:42:54] sooo.... [23:42:55] PROBLEM Free ram is now: CRITICAL on bots-sql2 i-000000af.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [23:43:07] may have just caused an issue with bastion-restricted [23:44:33] PROBLEM SSH is now: UNKNOWN on ganglia-test2 i-00000250.pmtpa.wmflabs output: Usage:check_ssh [-46] [-t timeout] [-r remote version] [-p port] host [23:44:38] PROBLEM dpkg-check is now: UNKNOWN on maps-test2 i-00000253.pmtpa.wmflabs output: Invalid host name i-00000253.pmtpa.wmflabs [23:44:38] PROBLEM Current Load is now: UNKNOWN on aggregator2 i-000002c0.pmtpa.wmflabs output: Invalid host name i-000002c0.pmtpa.wmflabs [23:45:15] PROBLEM host: i-0000022c.pmtpa.wmflabs is DOWN address: i-0000022c.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000022c.pmtpa.wmflabs) [23:45:16] PROBLEM host: i-000002b3.pmtpa.wmflabs is DOWN address: i-000002b3.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000002b3.pmtpa.wmflabs) [23:45:16] PROBLEM host: i-000002e7.pmtpa.wmflabs is DOWN address: i-000002e7.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000002e7.pmtpa.wmflabs) [23:45:16] PROBLEM host: i-00000362.pmtpa.wmflabs is DOWN address: i-00000362.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000362.pmtpa.wmflabs) [23:45:16] PROBLEM host: i-000001df.pmtpa.wmflabs is DOWN address: i-000001df.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000001df.pmtpa.wmflabs) [23:45:35] -_- [23:46:15] well shit [23:47:12] this isn't looking good [23:47:15] PROBLEM Disk Space is now: WARNING on kubo i-000003dd.pmtpa.wmflabs output: DISK WARNING - free space: / 291 MB (3% inode=66%): [23:47:45] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 17% free memory [23:47:45] PROBLEM dpkg-check is now: CRITICAL on mobile-varnish i-0000050a.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [23:48:05] PROBLEM SSH is now: CRITICAL on ganglia-test2 i-00000250.pmtpa.wmflabs output: CRITICAL - Socket timeout after 10 seconds [23:48:15] PROBLEM Current Load is now: CRITICAL on bots-salebot i-00000457.pmtpa.wmflabs output: CRITICAL - load average: 104.28, 42.64, 16.25 [23:48:47] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [23:48:47] RECOVERY host: i-000002e7.pmtpa.wmflabs is UP address: i-000002e7.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.49 ms [23:48:47] PROBLEM host: i-0000003a.pmtpa.wmflabs is DOWN address: i-0000003a.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000003a.pmtpa.wmflabs) [23:48:47] PROBLEM host: i-0000013d.pmtpa.wmflabs is DOWN address: i-0000013d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000013d.pmtpa.wmflabs) [23:48:55] PROBLEM host: i-000002ea.pmtpa.wmflabs is DOWN address: i-000002ea.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000002ea.pmtpa.wmflabs) [23:48:55] PROBLEM host: i-00000152.pmtpa.wmflabs is DOWN address: i-00000152.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000152.pmtpa.wmflabs) [23:48:55] PROBLEM host: i-00000103.pmtpa.wmflabs is DOWN address: i-00000103.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000103.pmtpa.wmflabs) [23:48:55] PROBLEM host: i-00000134.pmtpa.wmflabs is DOWN address: i-00000134.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000134.pmtpa.wmflabs) [23:48:55] PROBLEM host: i-000001e6.pmtpa.wmflabs is DOWN address: i-000001e6.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000001e6.pmtpa.wmflabs) [23:48:56] PROBLEM host: i-000002b8.pmtpa.wmflabs is DOWN address: i-000002b8.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000002b8.pmtpa.wmflabs) [23:48:56] PROBLEM host: i-000002e6.pmtpa.wmflabs is DOWN address: i-000002e6.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000002e6.pmtpa.wmflabs) [23:48:57] PROBLEM Current Load is now: WARNING on bots-apache01 i-000004fc.pmtpa.wmflabs output: WARNING - load average: 14.75, 11.80, 5.04 [23:49:05] RECOVERY host: i-0000022c.pmtpa.wmflabs is UP address: i-0000022c.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 9.84 ms [23:49:05] PROBLEM Total processes is now: CRITICAL on aggregator1 i-0000010c.pmtpa.wmflabs output: PROCS CRITICAL: 251 processes [23:49:05] PROBLEM host: i-000001e4.pmtpa.wmflabs is DOWN address: i-000001e4.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000001e4.pmtpa.wmflabs) [23:49:05] PROBLEM host: i-0000023b.pmtpa.wmflabs is DOWN address: i-0000023b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000023b.pmtpa.wmflabs) [23:49:05] PROBLEM host: i-00000238.pmtpa.wmflabs is DOWN address: i-00000238.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000238.pmtpa.wmflabs) [23:49:15] PROBLEM Disk Space is now: CRITICAL on deployment-apache32 i-0000031a.pmtpa.wmflabs output: DISK CRITICAL - free space: / 0 MB (0% inode=74%): [23:49:16] PROBLEM dpkg-check is now: CRITICAL on maps-test2 i-00000253.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:49:16] PROBLEM host: i-0000033b.pmtpa.wmflabs is DOWN address: i-0000033b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000033b.pmtpa.wmflabs) [23:49:16] PROBLEM host: i-00000441.pmtpa.wmflabs is DOWN address: i-00000441.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000441.pmtpa.wmflabs) [23:49:16] PROBLEM host: i-00000346.pmtpa.wmflabs is DOWN address: i-00000346.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000346.pmtpa.wmflabs) [23:49:20] wtf.... [23:49:25] RECOVERY host: i-000002b3.pmtpa.wmflabs is UP address: i-000002b3.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 5.25 ms [23:49:25] PROBLEM Current Load is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:49:35] PROBLEM host: i-0000015c.pmtpa.wmflabs is DOWN address: i-0000015c.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000015c.pmtpa.wmflabs) [23:49:42] all I did was kill dnsmasq and restart nova-network [23:49:45] PROBLEM host: i-000000e5.pmtpa.wmflabs is DOWN address: i-000000e5.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000000e5.pmtpa.wmflabs) [23:50:07] ah. seems like things are working again [23:50:43] RECOVERY host: i-0000033b.pmtpa.wmflabs is UP address: i-0000033b.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 16.57 ms [23:50:43] RECOVERY host: i-0000023b.pmtpa.wmflabs is UP address: i-0000023b.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.68 ms [23:50:43] RECOVERY host: i-00000238.pmtpa.wmflabs is UP address: i-00000238.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 11.01 ms [23:50:43] RECOVERY host: i-000002b8.pmtpa.wmflabs is UP address: i-000002b8.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 3.93 ms [23:50:53] RECOVERY host: i-000002ea.pmtpa.wmflabs is UP address: i-000002ea.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 10.09 ms [23:50:53] RECOVERY host: i-00000362.pmtpa.wmflabs is UP address: i-00000362.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 10.37 ms [23:51:03] RECOVERY host: i-000000e5.pmtpa.wmflabs is UP address: i-000000e5.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 9.68 ms [23:51:43] RECOVERY host: i-0000003a.pmtpa.wmflabs is UP address: i-0000003a.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.98 ms [23:51:53] RECOVERY host: i-00000134.pmtpa.wmflabs is UP address: i-00000134.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.58 ms [23:51:53] RECOVERY host: i-00000152.pmtpa.wmflabs is UP address: i-00000152.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.76 ms [23:51:53] RECOVERY host: i-000001df.pmtpa.wmflabs is UP address: i-000001df.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 1.69 ms [23:51:53] RECOVERY host: i-00000441.pmtpa.wmflabs is UP address: i-00000441.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 5.76 ms [23:51:53] RECOVERY host: i-000001e6.pmtpa.wmflabs is UP address: i-000001e6.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.61 ms [23:51:54] RECOVERY host: i-000001e4.pmtpa.wmflabs is UP address: i-000001e4.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.56 ms [23:52:33] RECOVERY dpkg-check is now: OK on mobile-varnish i-0000050a.pmtpa.wmflabs output: All packages OK [23:52:43] RECOVERY host: i-00000103.pmtpa.wmflabs is UP address: i-00000103.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.53 ms [23:52:53] RECOVERY host: i-00000346.pmtpa.wmflabs is UP address: i-00000346.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.50 ms [23:53:13] RECOVERY host: i-000002e6.pmtpa.wmflabs is UP address: i-000002e6.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 9.84 ms [23:53:13] PROBLEM Current Load is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: WARNING - load average: 1.15, 16.70, 12.40 [23:53:43] RECOVERY host: i-0000015c.pmtpa.wmflabs is UP address: i-0000015c.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 9.83 ms [23:53:53] RECOVERY host: i-0000013d.pmtpa.wmflabs is UP address: i-0000013d.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 11.02 ms [23:53:53] RECOVERY Current Load is now: OK on bots-apache01 i-000004fc.pmtpa.wmflabs output: OK - load average: 0.10, 4.32, 3.64 [23:54:13] PROBLEM dpkg-check is now: CRITICAL on ee-prototype i-0000013d.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages