[11:20:24] @labs-on [11:20:24] Labs utilities were enabled [11:20:37] !ping [11:20:37] pong [11:20:38] o.o [11:20:42] !ping [11:20:42] pong [11:20:52] @labs-info bots-1 [11:20:52] [Name bots-1 doesn't exist but resolves to I-000000a9] I-000000a9 is Nova Instance with name: bots-1, host: virt8, IP: 10.4.0.48 of type: m1.small, with number of CPUs: 1, RAM of this size: 2048M, member of project: bots, size of storage: 30 and with image ID: lucid-server-cloudimg-amd64.img [11:22:02] @labs-resolve I-0000053a [11:22:02] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID [11:22:13] @labs-resolve I-0000053A [11:22:13] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID [11:22:18] @labs-resolve I-0000053 [11:22:19] I don't know this instance - aren't you are looking for: I-00000530 (bots-abogott-devel), I-00000531 (mingle), I-00000532 (wikidata-dev-8), I-00000533 (bots-analytics), I-00000534 (home-migrate-precise), I-00000536 (home-migrate-lucid), I-00000539 (abogott-rt-devel), [11:22:52] @labs-resolve wikidata-dev-memcached [11:22:52] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID [11:23:02] @labs-resolve I-00000532 [11:23:03] The I-00000532 resolves to instance I-00000532 with a fancy name wikidata-dev-8 and IP 10.4.0.67 [11:23:29] @labs-resolve bob [11:23:29] The bob resolves to instance I-0000012d with a fancy name bob and IP 10.4.0.90 [11:23:36] @labs-info bob [11:23:36] [Name bob doesn't exist but resolves to I-0000012d] I-0000012d is Nova Instance with name: bob, host: virt6, IP: 10.4.0.90 of type: m1.small, with number of CPUs: 1, RAM of this size: 2048M, member of project: pediapress, size of storage: 30 and with image ID: lucid-server-cloudimg-amd64.img [11:23:42] pediapress! [11:24:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:26:44] @labs-resolve I-0000051a [11:26:44] The I-0000051a resolves to instance I-0000051a with a fancy name fawikitest and IP 10.4.1.43 [11:27:59] @labs-info I-0000051a [11:27:59] I-0000051a is Nova Instance with name: fawikitest, host: virt6, IP: 10.4.1.43 of type: m1.small, with number of CPUs: 1, RAM of this size: 2048M, member of project: farsi-wikitest, size of storage: 30 and with image ID: ubuntu-12.04-precise [11:28:07] mm [11:29:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:29:53] @labs-info I-000004de [11:29:53] I-000004de is Nova Instance with name: bugzillatesting, host: virt5, IP: 10.4.0.230 of type: m1.small, with number of CPUs: 1, RAM of this size: 2048M, member of project: bugtracker, size of storage: 30 and with image ID: ubuntu-12.04-precise [11:30:18] paravoid are you working on labs? [11:38:42] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [11:49:54] Hi! [11:50:56] How can I customize nagios plugins for my instances in http://nagios.wmflabs.org? Do I have to ask someone to restart the nagios server on the other side? [11:52:51] Silke_WMDE_ customize how? [11:53:05] you need to do both [11:53:48] but tbh I don't know how this new nagios works, Damianz is working on that and I don't think it even support customizations as previous version [11:53:58] customize in a way that I see more stuff than just the defaults. [11:54:31] you need to have these plugins on side of your machine of course [11:54:39] they are there [11:54:41] but changes on nagios server need to be done as well [11:54:46] ok [11:54:56] nagios update its configs using some python script [11:55:01] I don't really understand it [11:55:16] I don't even like it very much :P [11:55:29] hehe [11:55:40] there used to be c++ parser that was pretty weird but at least could resolve creepy names to fancy names [11:56:27] host I-00000045e is down doesn't really tell me much [11:56:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:56:47] like this [11:56:56] @labs-resolve i-00000051a [11:56:56] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID [11:57:06] @labs-resolve i-0000051a [11:57:06] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID [11:57:15] @labs-resolve I-00000051a [11:57:15] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID [11:57:27] @labs-resolve I-0000051a [11:57:27] The I-0000051a resolves to instance I-0000051a with a fancy name fawikitest and IP 10.4.1.43 [11:57:30] here we go [11:57:32] :D [11:59:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:08:43] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [12:09:49] @labs-resolve i-0000053a [12:09:49] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID [12:10:01] @labs-resolve i-0000053a.pmtpa.wmflabs [12:10:01] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID [12:10:23] you need big I [12:10:31] @labs-resolve I-0000053a.pmtpa.wmflabs [12:10:32] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID [12:10:46] though the web interface tells me something [12:11:51] aha, it doesn't support fqdn... [12:11:55] will fix [12:12:03] @labs-resolve I-0000053a [12:12:03] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID [12:12:12] mm [12:12:33] ok, needs some time for it to refresh db [12:17:47] @labs-resolve I-0000053a [12:17:47] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID [12:21:37] petan: in case silke didn't ask already... is there a way to change the firewall settings for a running instance? or does it need a reboot? or even more? [12:21:48] where's firewalling done anyway? at the gateway or at the instance? [12:26:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:29:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:35:17] you can add secu groups/rules and then create instances and choose secu group. In the labs documentation it says you cannot modify this when the instance is already up and running. [12:38:43] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [12:57:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:00:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:08:43] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [13:24:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 13% free memory [13:27:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:30:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:31:48] DanielK_WMDE firewall is done using nova [13:31:54] you don't need to reboot anything [13:32:18] petan: so changing to the firewall are applyied immediatly? [13:32:22] yes [13:32:28] so... if that doesn't work, what could be wrong? [13:32:40] the configuration itself [13:34:15] hm... [13:34:43] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5.pmtpa.wmflabs output: Critical: 5% free memory [13:36:59] petan: thanks [13:37:43] Silke_WMDE_: i can connect to a random port on wikidata-dev-8 from wikidata-dev-9. [13:38:04] how exactly did you try to set up memcached and how did it fail? [13:39:08] I made a security group that allows the test server access to the memcached server and created the memcached server. It talked one-way but of course not back. [13:39:32] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [13:41:43] PROBLEM Total processes is now: WARNING on bastion1 i-000000ba.pmtpa.wmflabs output: PROCS WARNING: 156 processes [13:49:43] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 848% free memory [13:57:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:00:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:09:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [14:27:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:27:31] !log integration deleted -jenkins instance, replaced by -jenkins2 [14:27:33] Logged the message, Master [14:31:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:36:42] RECOVERY Total processes is now: OK on bastion1 i-000000ba.pmtpa.wmflabs output: PROCS OK: 144 processes [14:36:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [14:40:13] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [14:40:53] PROBLEM Current Load is now: CRITICAL on integration-composer i-0000053b.pmtpa.wmflabs output: Connection refused by host [14:41:03] PROBLEM Free ram is now: CRITICAL on integration-composer i-0000053b.pmtpa.wmflabs output: Connection refused by host [14:41:33] PROBLEM Current Users is now: CRITICAL on integration-composer i-0000053b.pmtpa.wmflabs output: Connection refused by host [14:42:13] PROBLEM Disk Space is now: CRITICAL on integration-composer i-0000053b.pmtpa.wmflabs output: Connection refused by host [14:42:23] PROBLEM Total processes is now: CRITICAL on integration-composer i-0000053b.pmtpa.wmflabs output: Connection refused by host [14:42:53] PROBLEM dpkg-check is now: CRITICAL on integration-composer i-0000053b.pmtpa.wmflabs output: Connection refused by host [14:46:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 7% free memory [14:52:22] RECOVERY Total processes is now: OK on integration-composer i-0000053b.pmtpa.wmflabs output: PROCS OK: 87 processes [14:52:52] RECOVERY dpkg-check is now: OK on integration-composer i-0000053b.pmtpa.wmflabs output: All packages OK [14:55:53] RECOVERY Current Load is now: OK on integration-composer i-0000053b.pmtpa.wmflabs output: OK - load average: 0.11, 0.66, 0.62 [14:56:03] RECOVERY Free ram is now: OK on integration-composer i-0000053b.pmtpa.wmflabs output: OK: 889% free memory [14:56:33] RECOVERY Current Users is now: OK on integration-composer i-0000053b.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [14:57:13] RECOVERY Disk Space is now: OK on integration-composer i-0000053b.pmtpa.wmflabs output: DISK OK [14:57:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:01:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:06:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [15:10:53] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [15:27:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:31:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:41:32] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [15:51:35] @labs-resolve test [15:51:36] I don't know this instance - aren't you are looking for: I-000000cb (search-test), I-000000f1 (patchtest), I-000000fd (patchtest2), I-0000013c (test2), I-00000250 (ganglia-test2), I-00000253 (maps-test2), I-00000265 (exim-test), I-00000282 (ipv6test1), I-00000283 (mingledbtest), [15:51:36] !ping [15:51:37] pong [15:51:59] @labs-user DamianZ [15:51:59] That user is not a member of any project [15:52:04] @labs-user DamianZaremba [15:52:05] DamianZaremba is member of these projects: Bastion, Bots, Deployment-prep, Mailman, Openstack, [15:52:12] mwahaha [15:52:22] @labs-project-info bastion [15:52:22] The project Bastion has 2 instances and 241 members, description: The project to allow Labs users to connect to other instances within Wikimedia Labs via tunneling/proxy. [15:52:45] Ryan_Lane I created a simple interface to read nova info using irc [15:57:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:01:08] @labs-resolve i-0000051a [16:01:08] The i-0000051a resolves to instance I-0000051a with a fancy name fawikitest and IP 10.4.1.43 [16:01:19] @labs-user gifti [16:01:19] That user is not a member of any project [16:01:26] err [16:01:26] case sensitive [16:01:34] i'm uppercase?! [16:01:40] … [16:01:42] @labs-user Gifti [16:01:42] Gifti is member of these projects: Bastion, Bots, [16:01:44] everyone is [16:02:00] should remove trailing , [16:02:10] who cares [16:02:14] i do ;) [16:02:17] :D [16:02:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:02:45] @labs-resolve I-0000004de [16:02:45] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 43 seconds ago [16:02:51] @labs-resolve I-000004de [16:02:51] The I-000004de resolves to instance I-000004de with a fancy name bugzillatesting and IP 10.4.0.230 [16:03:03] these 2 instances are spamming whole channels for weeks [16:03:12] we should really get rid of them [16:04:40] @labs-project-instances bots [16:04:40] Following instances are in this project: bots-2, bots-cb, bots-1, bots-sql2, bots-sql3, bots-sql1, bots-3, bots-4, bots-labs, bots-dev, bots-salebot, bots-apache-test, bots-nr1, bots-apache01, bots-abogott-devel, bots-analytics, [16:04:55] bots-salebot o.O [16:04:58] wtf [16:05:06] apache-test? [16:05:26] finally we can clean all that mess [16:06:59] @labs-project-info deployment-prep [16:06:59] The project Deployment-prep has 15 instances and 33 members, description: This project aims to replace the previously buggy en.labs.wikimedia.org and the different prototype wikis with a dedicated area just for testing of new code before it goes live on Wikimedia websites. It is intended to follow the different configuration that Wikimedia uses for its sites, along with the on-wiki MediaWiki configuration so as to allow developers to test code without breaking Wikimedia. [16:07:37] @labs-user legoktm [16:07:37] That user is not a member of any project [16:07:44] er [16:07:48] @labs-user Legoktm [16:07:48] Legoktm is member of these projects: Bastion, Bots, [16:08:06] nice :) [16:08:30] @labs-user Ryan_Lane [16:08:30] That user is not a member of any project [16:08:38] @labs-user Laner [16:08:39] That user is not a member of any project [16:08:44] @labs-user laner [16:08:44] That user is not a member of any project [16:08:47] o.o [16:09:28] @labs-user Ryan Lane [16:09:28] Ryan Lane is member of these projects: Analytics, Bastion, Bots, Category-sorting, Cvresearch, Demo, Deployment-prep, Deployment-prepbackup, Dumps, Editor-engagement, Etherpad, Feeds, Ganglia, Gareth, Gerrit, Gluster, Integration, Juju, Linkcache, Lvs-labs, Mail, Mailman, Maps, Mapstory, Mediahandler, Mediawiki-bugfix, Membase, Mobile, Mobile-sms, Mobile-stats, Mwreview, Nagios, Nginx, Opengrok, Openstack, Otrs, Outreach, Patchtest, Pediapress, Phabricator, Project-proxy, Publicdata, Puppet, Puppet-cleanup, [16:09:31] aha [16:09:33] :D [16:11:33] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [16:28:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:32:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:42:23] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [16:58:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:02:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:09:33] RECOVERY Current Users is now: OK on bots-1 i-000000a9.pmtpa.wmflabs output: USERS OK - 5 users currently logged in [17:12:23] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [17:26:32] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 152 processes [17:28:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:33:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:36:33] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [17:42:23] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [17:44:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 151 processes [17:58:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:04:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:12:23] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [18:28:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:29:32] PROBLEM Current Users is now: CRITICAL on mobile-pg i-0000053c.pmtpa.wmflabs output: Connection refused by host [18:30:13] PROBLEM Disk Space is now: CRITICAL on mobile-pg i-0000053c.pmtpa.wmflabs output: Connection refused by host [18:30:53] PROBLEM Current Load is now: CRITICAL on mobile-pg i-0000053c.pmtpa.wmflabs output: Connection refused by host [18:31:03] PROBLEM Free ram is now: CRITICAL on mobile-pg i-0000053c.pmtpa.wmflabs output: Connection refused by host [18:32:23] PROBLEM Total processes is now: CRITICAL on mobile-pg i-0000053c.pmtpa.wmflabs output: Connection refused by host [18:32:53] PROBLEM dpkg-check is now: CRITICAL on mobile-pg i-0000053c.pmtpa.wmflabs output: Connection refused by host [18:34:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:37:23] RECOVERY Total processes is now: OK on mobile-pg i-0000053c.pmtpa.wmflabs output: PROCS OK: 86 processes [18:37:53] RECOVERY dpkg-check is now: OK on mobile-pg i-0000053c.pmtpa.wmflabs output: All packages OK [18:39:33] RECOVERY Current Users is now: OK on mobile-pg i-0000053c.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [18:40:13] RECOVERY Disk Space is now: OK on mobile-pg i-0000053c.pmtpa.wmflabs output: DISK OK [18:40:53] RECOVERY Current Load is now: OK on mobile-pg i-0000053c.pmtpa.wmflabs output: OK - load average: 0.20, 0.70, 0.59 [18:41:03] RECOVERY Free ram is now: OK on mobile-pg i-0000053c.pmtpa.wmflabs output: OK: 612% free memory [18:42:23] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [18:56:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [18:58:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:04:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:06:52] PROBLEM dpkg-check is now: CRITICAL on abogott-rt-devel i-00000539.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [19:11:50] petan: that's awesome! [19:11:52] thanks! [19:11:52] RECOVERY dpkg-check is now: OK on abogott-rt-devel i-00000539.pmtpa.wmflabs output: All packages OK [19:12:05] petan: there's a webplatform.org user you should meet [19:13:02] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [19:13:32] * Damianz sneezes [19:28:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:34:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:43:02] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [19:58:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:04:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:13:02] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [20:19:53] PROBLEM dpkg-check is now: CRITICAL on abogott-rt-devel i-00000539.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:07] @labs-info bots-3 [20:24:08] [Name bots-3 doesn't exist but resolves to I-000000e5] I-000000e5 is Nova Instance with name: bots-3, host: virt8, IP: 10.4.0.59 of type: m1.small, with number of CPUs: 1, RAM of this size: 2048M, member of project: bots, size of storage: 30 and with image ID: lucid-server-cloudimg-amd64.img [20:24:48] @labs-project-users bots [20:24:48] Following users are in this project: Addshore, Alejrb, Andrew Bogott, Aude, Beetstra, DamianZaremba, DeltaQuad, Dzahn, Fastily, Gifti, Hashar, Hydriz, Hyperon, Jasonspriggs, Jeremyb, Johnduhart, Kaldari, Krinkle, Ryan Lane, Lcarr, Madman, Coren, Novaadmin, Petrb, Reedy, Rich Smith, Sven Manguard, Tanvir Rahman, Thehelpfulone, Dschwen, Legoktm, Gribeco, Cupco, Mardetanha, Quentinv57, DrTrigon, Huji, Hazard-SJ, Odie5533, Bene, [20:24:52] <3 [20:25:14] That is going to get annoying fast.... [20:25:16] this is so incredibly useful [20:25:20] hahaha [20:25:25] Can it reply in PM instead? [20:25:30] ::rolleyes: [20:25:46] heh [20:26:00] @labs-project-info bots [20:26:00] The project Bots has 16 instances and 40 members, description: A project for creating and running bots for use on Wikimedia Foundation sites. [20:26:24] maybe for that specific one it should [20:26:28] since it's a ping machine [20:26:58] doesnt work if you pm the bot :( [20:27:13] well, you could ask in here and it can pm you [20:28:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:34:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:43:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [20:46:54] pingmachine! [20:47:46] pong! :D [20:49:35] @labs-project-users wikidata-dev [20:49:35] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 3 seconds ago [20:49:51] huh [20:50:08] @labs-project-users wikidata-dev [20:50:08] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 36 seconds ago [20:50:13] @labs-project-users maps [20:50:14] Following users are in this project: Aude, Ryan Lane, Apmon, Kolossos, Dschwen, Jeremyb, [20:50:18] that works [20:50:33] it does so in fact [20:55:29] hm. I wonder if it has issues with - [20:55:42] @labs-project-info deployment-prep [20:55:42] The project Deployment-prep has 15 instances and 33 members, description: This project aims to replace the previously buggy en.labs.wikimedia.org and the different prototype wikis with a dedicated area just for testing of new code before it goes live on Wikimedia websites. It is intended to follow the different configuration that Wikimedia uses for its sites, along with the on-wiki MediaWiki configuration so as to allow developers to test code without breaking Wikimedia. [20:55:46] nope [20:57:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 156 processes [20:57:36] @labs-resolve wikistats-01 [20:57:36] The wikistats-01 resolves to instance I-00000042 with a fancy name wikistats-01 and IP 10.4.0.9 [20:57:57] @labs-resolve ip wikistats-01 [20:57:57] I don't know this instance, sorry, try browsing the list by hand, but I can guarantee there is no such instance matching this name, host or Nova ID unless it was created less than 40 seconds ago [20:58:09] ah [20:58:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:58:23] @labs-user dzahn [20:58:23] That user is not a member of any project [20:58:43] @labs-project-users bugtracker [20:58:43] Following users are in this project: Daniel Friesen, Johnduhart, Novaadmin, [20:58:50] @labs-project-users wikistats [20:58:50] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 10 seconds ago [20:59:24] @labs-project-info ceph [20:59:24] The project Ceph has 0 instances and 3 members, description: {unknown} [20:59:37] petan: cool stuff! [20:59:50] just some minor glitches i guess..cant find my project [21:00:24] <^demon> Ryan_Lane: We don't actually use operations/private for anything, right? [21:00:31] right [21:00:40] <^demon> I'm going to go ahead and delete it. It only exists on-disk and has no review history to clean up after. [21:00:44] cool [21:01:16] <^demon> Gone :) [21:01:22] <^demon> Now we only have one private repo :) [21:03:03] @labs-project-info wikidata-dev [21:03:03] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 0 seconds ago [21:03:11] @labs-project-info Wikidata-dev [21:03:11] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 8 seconds ago [21:03:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 6.78, 6.87, 5.95 [21:03:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 8.27, 7.74, 6.07 [21:04:33] @labs-project-members bastion [21:04:54] @labs-project-users bastion [21:04:54] Following users are in this project: Andre Engels, Aaron Schulz, Abartov, Abe.music, Addshore, Adminxor, Akhanna, Chughakshay16, Alejrb, Amgine, Amire80, Andrew Bogott, Anomie, Apmon, ArielGlenn, Arky, Asher, Dash1291, Jack Phoenix, Ataherivand, Aude, Awjrichards, Beetstra, Bhartshorne, Jorm, Brion VIBBER, Bsitu, Cariaso, Catrope, Cmcmahon, CSteipp, Danakim, Mdale, DamianZaremba, Daniel Kinzler, Danny B., Daniel Friesen, DarKnight2012, Darkpsy, Darkyen, Darrell Bishop, DarTar, Declerambaul, DeltaQuad, Demon, Dereckson, Devunt, Dfoy, Diederik, Dmcgrath, Drenfro, Dsc, Dschwen, Dymsza, Dzahn, Edsu, Emufarmers, Emw, Eloquence, Erosen, Erik Zachte, Faidon, FarzanehSarafraz, Fastily, ForrestFuqua, GChriss, GICodeWarrior, Gifti, Guillom, GWicke, Halfak, Happy-melon, TheDJ, Hashar, Henrik, Hoo man, Hydriz, Hyperon, IAlex, IWorld, J, Jalexander, Jan, Jarry1250, Jasonspriggs, Jdlrobson, Jdz, Jeremyb, Jerith, Jeroen De Dauw, Jesusaurus, Jgreen, Jens Ohlig, Johnduhart, Jpostlethwaite, Jyothis, Kaldari, Kangaroopower, Karima Rafes, Kelson, Kai Nissen (WMDE), Kolossos, Koolhead17, Krinkle, Ryan Lane, Lcarr, Leisa, Lemurph, Liangent, Luca de Alfaro, Madman, MarkAHershberger, Coren, Mark Bergsma, MarkTraceur, MaxSem, Mglaser, Domas, Matthias Mullie, Mmovchin, Mschon, Michael Shavlovsky, Multichill, Neilk, Nikerabbit, Nimish Gautam, Nischayn22, NittyG, Novaadmin, Ocaasi, Oren, Ori.livneh, Alpha ori, Ottomata, Brent Garber, Owyn, Petrb, Pgehres, Philinje, Platonides, Preilly, Pyoungmeister, QChris, Raindrift, Reedy, Rical, Rich Smith, RobH, RobiH, SPQRobin, RobLa, R0csteady, Sactage, Sara, Schmir, ShakataGaNai, Shujenchang, Siebrand, Slevinski, Samuel Merritt, Silke Meyer, Smn, Guillaume Beaudoin, Spage, Suhasmonk, Sumanah, Sven Manguard, Tanvir Rahman, Gergő Tisza, Thehelpfulone, Tfinc, Trevor Parscal, Tpt, Tim Starling, Tuxed, Vivekvc, Volker.haas, Denny Vrandecic, Vvuksan, Victor Vasiliev, Vladimir Zapolskiy, Werdna, Whym, Wikinaut, Yaron Koren, Yuvipanda, Zaran, Kozuch, Robmoen, MF-Warburg, Anja Jentzsch, Stefan.petrea, Ebe123, Milimetric, Tim Landscheidt, Legoktm, Merl, Stwalkerster, Dan-nl, Gribeco, Aaron Pramana, Adamw, Ynhockey, Cupco, Hazard-SJ, Huji, Tparis, Quentinv57, Mardetanha, Nemo bis, Louisdang, DrTrigon, Westand, Labslogbot, Leaverou, Paulirish, Swalling, Massaf, Jimmyxu, Mono, Subramanya Sastry, Katie Horn, Mgrover, Umherirrender, Doug, Alolita, Matthewrbowker, Mattflaschen, PiRSquared17, Bruceburge, Odie5533, Tim1357, Nandakv4, Vishal, Rajiff, MZMcBride, Anth1y, Lwelling, Saper, Bene, Mwang, [21:05:06] mutante's so evil [21:05:20] megaping? [21:05:21] heh! [21:05:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:05:32] sorry for highlights :p [21:05:33] is it sorted by shell username? [21:05:52] see e.g. jorm, jack phoenix, etc. [21:05:55] i guess it would need wrapping of long message to multiple lines which sounds a bit complicated [21:06:11] mutante: that would allow eviler [21:06:18] hmmmmtrue [21:13:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [21:28:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:28:48] <^demon> Ryan_Lane: Could you merge https://gerrit.wikimedia.org/r/#/c/38795/? We're finished debugging all the replication stuff. Just ran it manually, so can re-enable in puppet [21:34:57] @labs-user Thehelpfulone [21:34:57] Thehelpfulone is member of these projects: Bastion, Bots, Globaleducation, Mailman, [21:35:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:36:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 10% free memory [21:37:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.53, 6.35, 5.65 [21:43:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [21:47:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.27, 3.95, 4.70 [21:55:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.61, 5.44, 5.14 [21:59:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:00:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 7.59, 6.78, 5.75 [22:05:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:07:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 5.57, 5.99, 5.32 [22:13:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [22:29:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:36:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:43:03] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [22:47:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 4.23, 4.53, 4.97 [22:59:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:05:52] PROBLEM Current Load is now: CRITICAL on abogott-rt i-0000053d.pmtpa.wmflabs output: Connection refused by host [23:06:32] PROBLEM Current Users is now: CRITICAL on abogott-rt i-0000053d.pmtpa.wmflabs output: Connection refused by host [23:06:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:07:12] PROBLEM Disk Space is now: CRITICAL on abogott-rt i-0000053d.pmtpa.wmflabs output: Connection refused by host [23:07:22] PROBLEM Total processes is now: CRITICAL on abogott-rt i-0000053d.pmtpa.wmflabs output: Connection refused by host [23:08:02] PROBLEM Free ram is now: CRITICAL on abogott-rt i-0000053d.pmtpa.wmflabs output: Connection refused by host [23:08:12] PROBLEM dpkg-check is now: CRITICAL on abogott-rt i-0000053d.pmtpa.wmflabs output: Connection refused by host [23:12:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 5.14, 5.12, 5.07 [23:13:42] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [23:15:31] 12/14/2012 - 23:15:31 - Updating keys for mwalker at /export/keys/mwalker [23:15:52] RECOVERY Current Load is now: OK on abogott-rt i-0000053d.pmtpa.wmflabs output: OK - load average: 0.61, 1.02, 0.67 [23:16:32] RECOVERY Current Users is now: OK on abogott-rt i-0000053d.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [23:17:12] RECOVERY Disk Space is now: OK on abogott-rt i-0000053d.pmtpa.wmflabs output: DISK OK [23:17:22] RECOVERY Total processes is now: OK on abogott-rt i-0000053d.pmtpa.wmflabs output: PROCS OK: 87 processes [23:18:02] RECOVERY Free ram is now: OK on abogott-rt i-0000053d.pmtpa.wmflabs output: OK: 1122% free memory [23:18:12] RECOVERY dpkg-check is now: OK on abogott-rt i-0000053d.pmtpa.wmflabs output: All packages OK [23:24:17] Ryan_Lane, so should the pages on mediawiki.org be changed to instruct people to register at labsconsole instead of [[mw:Developer access]]? [23:24:25] yes, please [23:29:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:31:56] hi sumanah [23:32:01] hi there Krenair! [23:32:02] Change on 12mediawiki a page Developer access was modified, changed by Krenair link https://www.mediawiki.org/w/index.php?diff=616919 edit summary: +Historical [23:32:06] It's Christmas early! [23:32:09] self-reg! [23:32:39] * sumanah throws confetti on Ryan_Lane and his associates [23:32:45] :D [23:32:52] I was just about to ask if you've seen that [23:33:20] I wonder if we can set a "create an account" link in gerrit [23:33:31] oh in the skin? [23:33:37] that would be great. [23:33:44] If possible. [23:34:03] auth.registerUrl [23:34:07] it's a config option [23:34:14] I'm going to add that [23:34:45] rockin' [23:34:50] \m/ [23:35:53] PROBLEM Free ram is now: WARNING on sube i-000003d0.pmtpa.wmflabs output: Warning: 19% free memory [23:36:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:42:17] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Krenair link https://www.mediawiki.org/w/index.php?diff=616924 edit summary: no longer beta, self-registration allowed [23:42:46] done. registration link is now https://labsconsole.wikimedia.org/w/index.php?title=Special:UserLogin&returnto=Help%3AGetting+Started&type=signup [23:43:06] when a user creates an account, he'll get sent to Help:Getting_Started [23:43:42] PROBLEM host: i-00000536.pmtpa.wmflabs is DOWN address: i-00000536.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000536.pmtpa.wmflabs) [23:45:03] that sounds great! [23:45:45] Krenair: thanks for the doc updates :) [23:45:53] you're welcome [23:46:04] I don't think I've made everything particularly clear [23:46:27] Maybe we should turn [[mw:Developer access]] into a page about Labs/Git/Gerrit accounts [23:46:51] hm, yeah, may be a good idea [23:49:33] PROBLEM Free ram is now: WARNING on swift-be4 i-000001ca.pmtpa.wmflabs output: Warning: 18% free memory [23:50:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 2.81, 4.17, 4.81 [23:57:09] Krenair: before you do that could you send a quick email to the people with open requests asking them to self-reg? [23:59:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%