[00:03:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:23:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:33:14] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:53:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:04:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:07:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 176 processes [01:12:32] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:24:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:30:43] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 16% free memory [01:34:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:54:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:00:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 40% free memory [02:04:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:24:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:33:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 18% free memory [02:34:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:40:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [02:48:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 19% free memory [02:54:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:04:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:16:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [03:24:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:34:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:36:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [03:46:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [03:54:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:06:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:24:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:36:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:42:28] anyone around who would have the opportunity to restart COIBot on instance 3? [04:54:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:07:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:24:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:37:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:53:31] sDrewthedoff: still needed? [05:54:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:56:01] no jeremyb ... Beetstra flounced in and whipped his bot into shape. Thx though [05:56:10] k [05:56:43] * jeremyb just went on a bit of a spree. see end of https://labsconsole.wikimedia.org/w/index.php?title=Special:ListUsers&dir=prev&username=&group=&creationSort=1 [06:07:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:09:19] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=612349 edit summary: /* User:Ladquin */ done [06:09:28] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=612350 edit summary: /* User:William.gathoye */ done [06:09:36] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=612351 edit summary: /* User:Milad Naseri */ done [06:09:44] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=612352 edit summary: /* User:Nick.townsend */ done [06:09:56] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=612353 edit summary: /* User:Satyaakam */ done [06:10:13] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=612354 edit summary: /* User:Mwang */ done [06:11:38] Ryan_Lane: 2 SVN acct reqs pending. both already have gerrit. do you want them or should we wait for sumanah tomorrow? [06:12:01] I don't really want to work on a sunday at 10 pm ;) [06:12:10] I'll gladly take a look tomorrow, though [06:12:27] okey ;) [06:12:50] you have been known to work at weird hours (sometimes) so i thought i'd try ;) [06:12:56] heh [06:13:00] yeah. I often do [06:13:01] although not as weird as some [06:13:07] I'm tired today, though :) [06:17:25] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=612357 edit summary: /* User:Avocato */ not done, i can't do this. will try to find someone else to do it [06:17:28] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=612358 edit summary: /* User:Edinwiki */ half done, i can't do the rest. will try to find someone else to do it [06:17:44] ok, all caught up on what i can do ;) [06:18:25] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=612359 edit summary: /* User:Edinwiki */ fmt [06:24:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 203 processes [06:29:33] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:31:32] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:37:25] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:44:34] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 146 processes [06:47:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 12.75, 9.83, 6.58 [06:53:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 8.13, 7.40, 5.52 [06:54:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:58:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 198 processes [07:01:54] PROBLEM dpkg-check is now: CRITICAL on maps-osmmapnik i-00000527.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [07:03:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.71, 7.15, 5.96 [07:06:53] RECOVERY dpkg-check is now: OK on maps-osmmapnik i-00000527.pmtpa.wmflabs output: All packages OK [07:07:33] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:15:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.80, 6.44, 5.58 [07:25:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:26:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [07:38:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:48:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.30, 3.95, 4.84 [07:53:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.45, 3.75, 4.75 [07:55:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:55:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 2.07, 3.30, 4.63 [07:56:33] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [08:08:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:26:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:38:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:41:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 7.41, 6.85, 5.79 [08:46:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.51, 5.88, 5.21 [08:48:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.69, 6.66, 5.50 [08:51:39] Change on 12mediawiki a page Developer access was modified, changed by Edinwiki link https://www.mediawiki.org/w/index.php?diff=612425 edit summary: /* User:Edinwiki */ [08:56:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:09:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:27:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:30:27] 12/03/2012 - 09:30:26 - Creating a home directory for edin at /export/keys/edin [09:35:28] 12/03/2012 - 09:35:27 - Updating keys for edin at /export/keys/edin [09:39:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:40:27] 12/03/2012 - 09:40:27 - Updating keys for edin at /export/keys/edin [09:43:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.90, 4.46, 4.93 [09:51:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.22, 5.27, 5.19 [09:52:08] Ryan_Lane hi [09:57:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:00:29] 12/03/2012 - 10:00:28 - Updating keys for edin at /export/keys/edin [10:07:52] PROBLEM Free ram is now: WARNING on aggregator1 i-0000010c.pmtpa.wmflabs output: Warning: 19% free memory [10:09:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:16:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.74, 3.82, 4.65 [10:27:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:39:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:51:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.07, 4.25, 4.82 [10:51:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 11% free memory [10:58:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:09:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:28:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:39:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:58:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:09:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:28:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:39:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:58:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:09:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:28:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:39:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:58:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:09:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:28:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:38:32] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 207 processes [14:41:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:43:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 198 processes [14:57:03] What exactly is labs? [14:57:52] <^demon> It's a virtual machine environment built around openstack. [14:58:07] <^demon> So users can spin up VMs on demand. [14:58:23] for the non technical, it's a place to test changes to MediaWiki projects before making the changes live [14:58:31] And what's the connection to gerrit and whatnot? [14:58:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:58:45] <^demon> odie5533: Not strictly true--there's lots of non-mediawiki things in labs. [14:59:14] <^demon> Isarra: Gerrit is our git server. The connection is that most code used in labs lives on Gerrit, and you use the same account for Gerrit & Labs. [15:01:03] Do you have any idea how confusing that is when trying to make a simple mediawiki change? >.< [15:01:34] <^demon> If you're just looking to contribute to mediawiki, you can forget labs exists. [15:02:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 3.52, 3.99, 4.75 [15:11:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:28:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:35:53] PROBLEM Current Load is now: CRITICAL on puppetdoc i-00000528.pmtpa.wmflabs output: Connection refused by host [15:36:33] PROBLEM Current Users is now: CRITICAL on puppetdoc i-00000528.pmtpa.wmflabs output: Connection refused by host [15:37:13] PROBLEM Disk Space is now: CRITICAL on puppetdoc i-00000528.pmtpa.wmflabs output: Connection refused by host [15:37:23] PROBLEM Total processes is now: CRITICAL on puppetdoc i-00000528.pmtpa.wmflabs output: Connection refused by host [15:38:03] PROBLEM Free ram is now: CRITICAL on puppetdoc i-00000528.pmtpa.wmflabs output: Connection refused by host [15:38:53] PROBLEM dpkg-check is now: CRITICAL on puppetdoc i-00000528.pmtpa.wmflabs output: Connection refused by host [15:41:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:45:53] RECOVERY Current Load is now: OK on puppetdoc i-00000528.pmtpa.wmflabs output: OK - load average: 0.35, 0.88, 0.62 [15:46:34] RECOVERY Current Users is now: OK on puppetdoc i-00000528.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [15:47:13] RECOVERY Disk Space is now: OK on puppetdoc i-00000528.pmtpa.wmflabs output: DISK OK [15:47:23] RECOVERY Total processes is now: OK on puppetdoc i-00000528.pmtpa.wmflabs output: PROCS OK: 83 processes [15:48:03] RECOVERY Free ram is now: OK on puppetdoc i-00000528.pmtpa.wmflabs output: OK: 1191% free memory [15:48:54] RECOVERY dpkg-check is now: OK on puppetdoc i-00000528.pmtpa.wmflabs output: All packages OK [15:58:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:11:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:19:32] PROBLEM Current Users is now: CRITICAL on wikidata-dev-8 i-00000529.pmtpa.wmflabs output: Connection refused by host [16:20:13] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-8 i-00000529.pmtpa.wmflabs output: Connection refused by host [16:20:53] PROBLEM Current Load is now: CRITICAL on wikidata-dev-8 i-00000529.pmtpa.wmflabs output: Connection refused by host [16:21:03] PROBLEM Free ram is now: CRITICAL on wikidata-dev-8 i-00000529.pmtpa.wmflabs output: Connection refused by host [16:22:23] PROBLEM Total processes is now: CRITICAL on wikidata-dev-8 i-00000529.pmtpa.wmflabs output: Connection refused by host [16:22:53] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-8 i-00000529.pmtpa.wmflabs output: Connection refused by host [16:28:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:41:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:47:02] Hi! How many instanced can I create for a project in labs? Is there a limitation (of instances? of resources?) [16:53:17] Silke_WMDE_ 10 [16:53:27] it can be increased of cours [16:53:38] ok. [16:54:09] If a newly created one is very red in nagios what can be reasons for this? [16:54:34] the only green part is ssh but it refused my pub key [16:54:45] ok, that means the puppet run failed [16:54:52] !console [16:54:53] in case you want to see what is happening on terminal of your vm, check console output [16:55:06] the conole looks ok [16:55:08] you can either rebuild or wait a little bit more [16:55:16] ok, you see no errors in console output? [16:55:19] no [16:55:35] because if all is red it means puppet didn't update nrpe config and nagios can't connect there [16:55:37] i created an instance half an hour ago [16:55:43] that would mean it didn't even synced the keys [16:55:49] that's why you can't login [16:55:53] yes [16:55:58] but if it's OK it wouldn't happen [16:56:02] that's why I guess it's not OK [16:56:14] I think you should kill it and create again... [16:56:41] good [16:56:47] thank you petan! [16:56:50] np [16:59:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:04:53] PROBLEM host: i-0000052a.pmtpa.wmflabs is DOWN address: i-0000052a.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000052a.pmtpa.wmflabs) [17:08:52] RECOVERY host: i-0000052a.pmtpa.wmflabs is UP address: i-0000052a.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.52 ms [17:09:22] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Connection refused by host [17:10:12] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Connection refused by host [17:10:52] PROBLEM Current Load is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Connection refused by host [17:11:32] PROBLEM Current Users is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Connection refused by host [17:12:12] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Connection refused by host [17:12:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:13:02] PROBLEM Free ram is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Connection refused by host [17:19:22] RECOVERY Total processes is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS OK: 89 processes [17:20:53] RECOVERY Current Load is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: OK - load average: 0.14, 0.76, 0.63 [17:20:53] RECOVERY dpkg-check is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: All packages OK [17:21:33] RECOVERY Current Users is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [17:21:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [17:22:13] RECOVERY Disk Space is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: DISK OK [17:23:03] RECOVERY Free ram is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: OK: 2071% free memory [17:29:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:36:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [17:42:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:51:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [17:53:05] Can someone who understands about apache config help me out? I have this in sites-enabled but still all I get is 'it works!' when I browse to the host. [17:53:06] http://fpaste.org/hFMH/ [17:53:14] (I thought I knew how to do this but… clearly don't) [17:54:27] andrewbogott: Did you restart Apache? [17:54:31] yep [17:54:45] good guess though! [17:56:38] Oh, hm, there was also a 000-default in sites-enabled… removing that changes the behavior. [17:58:22] and… now it works [17:59:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:11:57] oh nice http://semantic-mediawiki.org/wiki/Semantic_MediaWiki_1.8.0_released - we use SMW on Labs https://labsconsole.wikimedia.org/wiki/Special:Version - should we upgrade? [18:12:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:17:50] Ryan_Lane: http://semantic-mediawiki.org/wiki/Semantic_MediaWiki_1.8.0_released - we use SMW on Labs https://labsconsole.wikimedia.org/wiki/Special:Version - should we upgrade? Is that urgent at all, given that it has some new performance improvements? [18:18:16] we're going to upgrade, yes, but the performance isn't really much of an issue for us [18:18:27] got it [18:18:28] labsconsole is pretty small, content wise [18:18:30] nod [18:26:16] petan: ah. missed you [18:26:30] petan: I see you are having issues with gluster [18:26:36] or is it issues with apache? [18:27:01] maybe the apache config doesn't allow following of symlinks? [18:29:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:42:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:59:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:12:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:30:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:40:43] Change on 12mediawiki a page Developer access was modified, changed by Mlpearc link https://www.mediawiki.org/w/index.php?diff=612609 edit summary: [19:43:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:01:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:13:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:32:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:43:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:02:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:09:25] * Damianz pokes wangatlargo [21:11:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [21:13:32] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:16:32] * Damianz sniffs wangatlargo, notes the smell of death and goes back to wondering rather than saying hi [21:16:42] On another note, who's going to FOSDEM? :D [21:17:54] I may be [21:19:14] I'm kinda waiting for the keynotes to be announced, time is running on though :( [21:20:04] well, I submitted for a main room talk [21:25:28] 12/03/2012 - 21:25:28 - Creating a home directory for mwang at /export/keys/mwang [21:30:28] 12/03/2012 - 21:30:27 - Updating keys for mwang at /export/keys/mwang [21:32:13] Damianz: I am [21:32:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:34:04] paravoid: :o [21:34:34] 7th year? [21:34:42] or maybe 6th [21:37:33] wangatlargo: so, you should read up on what labs does [21:37:38] @search start [21:37:38] Results (Found 2): stucked, start, [21:37:43] !start | wangatlargo [21:37:43] wangatlargo: Welcome to Wikimedia Labs! Get yourself started at https://labsconsole.wikimedia.org/wiki/Help:Getting_Started [21:37:54] @search request [21:37:54] Results (Found 2): requests, project-access, [21:37:58] !requests [21:37:59] this is a backlog of all requests needed to be done by ops https://labsconsole.wikimedia.org/wiki/Requests [21:38:02] ugh [21:38:06] that's not correct :D [21:39:06] !requests del [21:39:06] Successfully removed requests [21:39:12] !requests is https://labsconsole.wikimedia.org/wiki/Help:Contents#Requests [21:39:12] Key was added [21:40:45] wangatlargo: Ryan's the labs expert, but I can help you too in case he's otherwise occupied/not available, don't hesitate to ask :-) [21:40:54] Ryan: How do I test if my uploaded pub key works? [21:41:35] wangatlargo: you'll need to ssh into the bastion host for ops members [21:41:39] paravoid totally just made me think of http://www.youtube.com/watch?v=SSUXXzN26zg [21:41:43] !access | wangatlargo [21:41:43] wangatlargo: https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [21:42:48] wangatlargo: you should also add yourself to another project (maybe testlabs) and ensure you can ssh into instances in that project [21:44:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:02:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:09:26] Ryan: I am not in the Bastion project. Can you add me to it? [22:10:54] Ryan: I logged in bastion-restricted. [22:11:33] PROBLEM Current Users is now: CRITICAL on puppetdoc2 i-0000052b.pmtpa.wmflabs output: Connection refused by host [22:12:09] Ryan: Now I know I should login bastion-restricted. [22:12:13] PROBLEM Disk Space is now: CRITICAL on puppetdoc2 i-0000052b.pmtpa.wmflabs output: Connection refused by host [22:13:03] PROBLEM Free ram is now: CRITICAL on puppetdoc2 i-0000052b.pmtpa.wmflabs output: Connection refused by host [22:14:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:14:23] PROBLEM Total processes is now: CRITICAL on puppetdoc2 i-0000052b.pmtpa.wmflabs output: Connection refused by host [22:15:52] PROBLEM Current Load is now: CRITICAL on puppetdoc2 i-0000052b.pmtpa.wmflabs output: Connection refused by host [22:15:53] PROBLEM dpkg-check is now: CRITICAL on puppetdoc2 i-0000052b.pmtpa.wmflabs output: Connection refused by host [22:25:52] RECOVERY Current Load is now: OK on puppetdoc2 i-0000052b.pmtpa.wmflabs output: OK - load average: 1.12, 1.34, 0.88 [22:25:52] RECOVERY dpkg-check is now: OK on puppetdoc2 i-0000052b.pmtpa.wmflabs output: All packages OK [22:26:32] RECOVERY Current Users is now: OK on puppetdoc2 i-0000052b.pmtpa.wmflabs output: USERS OK - 2 users currently logged in [22:27:12] RECOVERY Disk Space is now: OK on puppetdoc2 i-0000052b.pmtpa.wmflabs output: DISK OK [22:28:02] RECOVERY Free ram is now: OK on puppetdoc2 i-0000052b.pmtpa.wmflabs output: OK: 583% free memory [22:29:22] RECOVERY Total processes is now: OK on puppetdoc2 i-0000052b.pmtpa.wmflabs output: PROCS OK: 92 processes [22:32:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:44:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:50:45] wangatlargo: eh? you should be in bastion [22:51:04] it says you are [22:56:59] wangatlargo: are you not able to ssh into bastion-restricted.wmflabs.org? [23:03:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:14:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:29:32] PROBLEM Current Users is now: CRITICAL on puppetdoc4 i-0000052d.pmtpa.wmflabs output: Connection refused by host [23:30:13] PROBLEM Disk Space is now: CRITICAL on puppetdoc4 i-0000052d.pmtpa.wmflabs output: Connection refused by host [23:30:53] PROBLEM Current Load is now: CRITICAL on puppetdoc4 i-0000052d.pmtpa.wmflabs output: Connection refused by host [23:31:03] PROBLEM Free ram is now: CRITICAL on puppetdoc4 i-0000052d.pmtpa.wmflabs output: Connection refused by host [23:32:23] PROBLEM Total processes is now: CRITICAL on puppetdoc4 i-0000052d.pmtpa.wmflabs output: Connection refused by host [23:32:53] PROBLEM dpkg-check is now: CRITICAL on puppetdoc4 i-0000052d.pmtpa.wmflabs output: Connection refused by host [23:33:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:37:22] RECOVERY Total processes is now: OK on puppetdoc4 i-0000052d.pmtpa.wmflabs output: PROCS OK: 89 processes [23:37:32] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 156 processes [23:37:52] RECOVERY dpkg-check is now: OK on puppetdoc4 i-0000052d.pmtpa.wmflabs output: All packages OK [23:39:32] RECOVERY Current Users is now: OK on puppetdoc4 i-0000052d.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [23:40:12] RECOVERY Disk Space is now: OK on puppetdoc4 i-0000052d.pmtpa.wmflabs output: DISK OK [23:40:52] RECOVERY Current Load is now: OK on puppetdoc4 i-0000052d.pmtpa.wmflabs output: OK - load average: 0.61, 0.89, 0.68 [23:41:02] RECOVERY Free ram is now: OK on puppetdoc4 i-0000052d.pmtpa.wmflabs output: OK: 484% free memory [23:44:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:45:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 8.28, 7.69, 5.94 [23:52:32] wangatlargo: ? [23:54:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 8.41, 7.58, 5.75