[00:09:22] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[00:29:04] <wm-bot>	 Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=593975 edit summary: /* User:Matthias M. */ 
[00:29:48] <wm-bot>	 Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=593976 edit summary: /* User:Thehansmeet */ done
[00:39:23] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[01:06:52] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 172 processes
[01:09:42] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[01:11:53] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 94 processes
[01:39:43] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[02:08:07] <Huji>	 Damianz: You got a minute?
[02:08:24] <Huji>	 petan: or you, maybe?
[02:09:43] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[02:17:37] <Huji>	 or, anyone from the Bots project of the Labs?
[02:38:42] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory
[02:39:52] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[02:51:42] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 14% free memory
[03:10:52] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[03:40:52] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[04:00:12] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Warning: 16% free memory
[04:10:52] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[04:25:13] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Critical: 4% free memory
[04:25:53] <labs-nagios-wm>	 PROBLEM Disk Space is now: WARNING on echo-xmpp i-00000351.pmtpa.wmflabs output: DISK WARNING - free space: / 569 MB (5% inode=91%):
[04:30:12] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f.pmtpa.wmflabs output: OK: 95% free memory
[04:40:52] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[05:10:52] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[05:40:52] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[06:10:53] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[06:42:02] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[07:00:53] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on echo-xmpp i-00000351.pmtpa.wmflabs output: DISK OK
[07:12:03] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[07:20:46] <petan>	 @seen Huji
[07:20:46] <wm-bot>	 petan: Last time I saw Huji they were quiting the network N/A at 10/16/2012 3:52:12 AM (03:28:33.9565890 ago)
[07:42:03] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[08:14:12] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[08:44:12] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[09:05:48] <Silke_WMDE>	 !log wikidata-dev wikidata-dev-2: Updated all extensions on the test client (except for Wikibase, Diff, Data Values) due to problems with APISandbox extension.
[09:05:51] <labs-morebots>	 Logged the message, Master
[09:14:13] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[09:18:32] <labs-nagios-wm>	 PROBLEM Total processes is now: UNKNOWN on ganglia-test2 i-00000250.pmtpa.wmflabs output: Invalid host name i-00000250.pmtpa.wmflabs
[09:23:43] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on ganglia-test2 i-00000250.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds.
[09:37:42] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on aggregator-test1 i-000002bf.pmtpa.wmflabs output: Warning: 19% free memory
[09:44:13] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[10:14:22] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[10:37:26] <wm-bot>	 Change on 12mediawiki a page Wikimedia Labs/Toolserver features wanted in Tool Labs was modified, changed by Nemo bis link https://www.mediawiki.org/w/index.php?diff=594212 edit summary: +All the packages requested by TS users and now taken as a given
[10:44:22] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[11:03:10] <hashar>	 !log integration manually updating operations/puppet on integration-apache1
[11:03:12] <labs-morebots>	 Logged the message, Master
[11:06:37] <Jan_Luca>	 !log centralauth Start test of extension "LabsCentralauth"
[11:06:39] <labs-morebots>	 Logged the message, Master
[11:09:16] <Abraham_WMDE>	 hashar: hi
[11:12:53] <Abraham_WMDE>	 how is it possible to change the security group for a running instance? as a understood labs so far it is only possible to choose the security group(s) when an instance is initial started, right?
[11:13:07] <petan>	 yes
[11:13:11] <petan>	 !security
[11:13:11] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Help:Security_Groups
[11:13:26] <petan>	 that means it's not possible later
[11:14:23] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[11:15:15] <Abraham_WMDE>	 petan: thx
[11:21:16] <labs-home-wm>	 10/16/2012 - 11:21:16 - Creating a home directory for avocato at /export/keys/avocato
[11:26:14] <labs-home-wm>	 10/16/2012 - 11:26:14 - Updating keys for avocato at /export/keys/avocato
[11:41:03] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on integration-apache1 i-000002eb.pmtpa.wmflabs output: DISK OK
[11:41:43] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on integration-apache1 i-000002eb.pmtpa.wmflabs output: OK: 187% free memory
[11:42:26] <hashar>	 !log integration rebooting integration-apache1 in hope the security rule will be applied :(
[11:42:28] <labs-morebots>	 Logged the message, Master
[11:43:23] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on integration-apache1 i-000002eb.pmtpa.wmflabs output: All packages OK
[11:43:33] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on integration-apache1 i-000002eb.pmtpa.wmflabs output: PROCS OK: 97 processes
[11:43:56] <hashar>	 Grrrrmmbmlblblb
[11:44:05] <hashar>	 the security rule does not get applied
[11:44:23] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[11:44:53] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on integration-apache1 i-000002eb.pmtpa.wmflabs output: OK - load average: 0.09, 0.10, 0.05
[11:45:32] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on integration-apache1 i-000002eb.pmtpa.wmflabs output: USERS OK - 1 users currently logged in
[11:46:25] <petan>	 !ping
[11:46:26] <wm-bot>	 pong
[11:46:59] <petan>	 @add ##petan4
[11:55:12] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK OK
[12:02:09] <j^>	 whats the state of deployment-prep these days?
[12:03:06] <j^>	 what puppet class would fix /usr/local/apache right now? setting up deployment-video04 /usr/local/apache was a folder with a symlink common to common-local in it, instead i want /usr/local/apache -> /data/project/apache
[12:03:12] <labs-nagios-wm>	 PROBLEM Disk Space is now: WARNING on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%):
[12:03:18] <hashar>	 j^: hi ):
[12:03:19] <hashar>	 :)
[12:03:32] <j^>	 hi
[12:03:46] <hashar>	 j^: I migrated the files from /home/wikipedia to /data/project which is the shared cluster project
[12:04:15] <hashar>	 is video004 a fresh instance ?
[12:04:34] <petan>	 hashar I moved even /home/log yesterday
[12:04:42] <petan>	 * /home/wikipedia/lo
[12:04:44] <petan>	 g
[12:04:49] <hashar>	 petan: great!
[12:04:55] <petan>	 because disk was full
[12:05:01] <hashar>	 hmm
[12:05:09] <petan>	 these logs had like 300mb
[12:05:09] <hashar>	  /home/wikipedia/logs is still there
[12:05:15] <petan>	 it's a symlink afaik
[12:05:25] <j^>	 hashar: yes, 03 was not responding and since its all in puppet i rebuild it
[12:05:51] <petan>	 eh
[12:05:51] <hashar>	 petan: that is a directory ;-]
[12:05:51] <petan>	 weird
[12:05:54] <petan>	 it was recreated somehow
[12:06:01] <hashar>	 petan: have you made the change in puppet ? 
[12:06:07] <petan>	 no
[12:06:19] <hashar>	 so puppet deleted the symlink and created the directory ;-]
[12:06:23] <petan>	 aha
[12:07:33] <hashar>	 j^: so the symlink from /usr/local/apache/common to /data/project  is provided by a puppet class
[12:07:43] <hashar>	 need to find out which one
[12:08:01] <j^>	 nfs::apache::labs looks alright
[12:08:10] <j^>	 name is a bit dated :)
[12:08:47] <hashar>	 which should be included by one of the role class
[12:09:21] <hashar>	 yeah the role::applicationserver::*  classes should include nfs::apache::labs  apparently
[12:09:54] <hashar>	 maybe there should be a role::applicationserver::videoscaler class
[12:10:25] <j^>	 https://gerrit.wikimedia.org/r/28208 adds it to mediawiki::videoscaler
[12:11:01] <j^>	 role::applicationserver::videoscaler should go (https://gerrit.wikimedia.org/r/#/c/28008/)
[12:11:41] <j^>	 or current mediawiki::videoscaler should become role::applicationserver::videoscaler if thats the up to date naming convention
[12:14:23] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[12:22:48] <Jan_Luca>	 j^: In [[gerrit:28208]] you should remove the tab between "include" and "nfs::apache::labs"
[12:33:52] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on centralauth-puppetmaster i-000004c7.pmtpa.wmflabs output: Connection refused by host
[12:34:32] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on centralauth-puppetmaster i-000004c7.pmtpa.wmflabs output: Connection refused by host
[12:35:12] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on centralauth-puppetmaster i-000004c7.pmtpa.wmflabs output: Connection refused by host
[12:36:02] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on centralauth-puppetmaster i-000004c7.pmtpa.wmflabs output: Connection refused by host
[12:36:30] <j^>	 Jan_Luca: thanks for the review, after reading role::applicationserver::* classes I changed the patch to add role::applicationserver::videoscaler and cleanup
[12:37:25] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on centralauth-puppetmaster i-000004c7.pmtpa.wmflabs output: Connection refused by host
[12:37:53] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on centralauth-puppetmaster i-000004c7.pmtpa.wmflabs output: Connection refused by host
[12:38:44] <Jan_Luca>	 hashar: Sometimes it lasts some time till the new security role works
[12:38:52] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on centralauth-puppetmaster i-000004c7.pmtpa.wmflabs output: OK - load average: 1.25, 1.06, 0.64
[12:39:32] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on centralauth-puppetmaster i-000004c7.pmtpa.wmflabs output: USERS OK - 0 users currently logged in
[12:40:12] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on centralauth-puppetmaster i-000004c7.pmtpa.wmflabs output: DISK OK
[12:41:02] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on centralauth-puppetmaster i-000004c7.pmtpa.wmflabs output: OK: 1465% free memory
[12:42:22] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on centralauth-puppetmaster i-000004c7.pmtpa.wmflabs output: PROCS OK: 91 processes
[12:42:25] <j^>	 1465% free thats a lot
[12:42:45] <Jan_Luca>	 Oh, I have a super instance :-D
[12:42:52] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on centralauth-puppetmaster i-000004c7.pmtpa.wmflabs output: All packages OK
[12:44:11] <Jan_Luca>	 The web interface outputs the same 1465% ...
[12:44:42] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[12:46:26] <hashar>	 Jan_Luca: will have to wait I guess :-(
[12:47:20] <Jan_Luca>	 I had the problem last week, too. It lasts around one hour.
[13:02:02] <Silke_WMDE>	 !log wikidata-dev wikidata-dev-3: installed Solr 3.6.1 for test purposes (in /mnt/apache-solr...)
[13:02:06] <labs-morebots>	 Logged the message, Master
[13:04:33] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes
[13:09:32] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes
[13:14:29] <Jan_Luca>	 !log centralauth Created centralauth-puppetmaster for create the puppet-stuff
[13:14:30] <labs-morebots>	 Logged the message, Master
[13:14:42] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[13:18:52] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on integration-androidsdk i-000004c8.pmtpa.wmflabs output: Connection refused by host
[13:19:32] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on integration-androidsdk i-000004c8.pmtpa.wmflabs output: Connection refused by host
[13:20:12] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on integration-androidsdk i-000004c8.pmtpa.wmflabs output: Connection refused by host
[13:21:05] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on integration-androidsdk i-000004c8.pmtpa.wmflabs output: Connection refused by host
[13:23:52] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on integration-androidsdk i-000004c8.pmtpa.wmflabs output: OK - load average: 0.18, 0.75, 0.50
[13:24:32] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on integration-androidsdk i-000004c8.pmtpa.wmflabs output: USERS OK - 0 users currently logged in
[13:25:13] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on integration-androidsdk i-000004c8.pmtpa.wmflabs output: DISK OK
[13:26:02] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on integration-androidsdk i-000004c8.pmtpa.wmflabs output: OK: 555% free memory
[13:42:05] <j^>	 DB connection error: Can't connect to MySQL server on 'deployment-sql' (4) (deployment-sql)
[13:42:05] <j^>	 anyone got an idea what could cause that?
[13:44:42] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[13:59:42] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on aggregator2 i-000002c0.pmtpa.wmflabs output: Warning: 19% free memory
[14:13:50] <j^>	 are there some access restrictions for deployment-sql somewhere? deplyoment-video04 has 10.4.1.19 previous instances had all 10.4.0.x, could that be a problem?
[14:14:43] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[14:24:18] <wm-bot>	 [bz] (8NEW - created by: 2Jan Gerber, priority: 4Unprioritized - 6normal) [Bug 41083] unable to connect to deployment-sql from deployment-video04 - https://bugzilla.wikimedia.org/show_bug.cgi?id=41083
[14:44:44] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[14:59:13] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on deployment-apache33 i-0000031b.pmtpa.wmflabs output: DISK CRITICAL - free space: / 286 MB (2% inode=78%):
[15:07:37] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 192 processes
[15:14:52] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[15:27:34] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes
[15:45:52] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[16:15:52] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[16:34:03] <^demon>	 https://gerrit-review.googlesource.com/#/c/38453/ \o/
[16:34:06] <^demon>	 About time!
[16:34:20] <^demon>	 robla: gitblit is finally up for review :D https://gerrit-review.googlesource.com/#/c/38453/
[16:35:21] <robla>	 sweet!
[16:35:40] <robla>	 ^demon: is there a demo site anywhere you know of?
[16:35:54] <^demon>	 Not yet. I could try throwing up something on labs perhaps.
[16:35:56] <robla>	 I'm curious how well integrated it is
[16:36:10] <^demon>	 Likewise. I'm going to try building it now
[16:45:52] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[16:50:27] <^demon>	 Building plugins is going to be so much easier once 2.5 has been released and the 2.5 classes are in the maven repos. Building from snapshots is annoying.
[17:13:42] <^demon>	 robla: I'm gonna grab some lunch, but then I'll setup a labs install of this. We'll need to setup some URL rewrites to make it a little less ugly, but otherwise this thing is slick.
[17:13:47] <^demon>	 It's so much faster that gitweb.
[17:14:13] <^demon>	 It replaces all the (gitweb) links we currently have with (gitblit)
[17:15:57] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[17:38:52] <^demon>	 "Labs uses cookies to log in users. You have cookies disabled. Please enable them and try again."
[17:38:54] <^demon>	 I most certainly do not.
[17:39:43] <^demon>	 (Also, labs is painfully slow right now :()
[17:47:02] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[17:50:02] <^demon>	 Hmm, new session solved it.
[18:17:02] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[18:47:05] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[18:55:43] <spagewmf>	 I should use RT for gerrit project permission requests, right?
[18:55:57] <^demon>	 What project?
[18:56:12] <^demon>	 (Generally, no. CT perfers it for ops/* projects though)
[18:57:01] <spagewmf>	 Ahh.  mediawiki/extensions/PostEdit .  It's an E3 extension, so spage, swalling, massaf should have +2 rights.  And ori livneh, but I assume he does already.
[18:57:41] <^demon>	 Ah, yeah that goes through the normal way. <http://www.mediawiki.org/wiki/Git/Gerrit_project_ownership>
[18:58:19] <^demon>	 Just put a request for who needs what, and get an existing user with those rights to say ok.
[18:58:31] <^demon>	 (Or two, if you can :))
[19:02:32] <spagewmf>	 Thx.
[19:03:14] <spagewmf>	 ^demon  BTW, regarding gitweb -> gitblit, I wrote https://www.mediawiki.org/wiki/Template:Git_file , and [[Category:Git]] has other templates to ease the URL change.  But gitweb URLs are so fiddly most people just copy&paste.
[19:04:15] <^demon>	 The gitblit urls are a bit nicer :)
[19:05:32] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes
[19:08:34] <^demon>	 spagewmf: http://{host}/plugins/gitblit-plugin-1.0-SNAPSHOT/commit/All-Projects.git/{hash}
[19:08:45] <^demon>	 (I'll put some sort of rewrites to make the plugins/gitblit part nicer)
[19:09:53] <Damianz>	 http://cdn.publishme.se/cdn/8-1/3190901/images/2011/pony_181258561.jpg < this is how I imagine gerrit doing replication
[19:13:04] <^demon>	 Damianz: It's more like http://www.weyandarun.co.uk/reed-cutting/Bailing_Out_005.jpeg
[19:13:18] <Damianz>	 LOL
[19:17:02] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[19:39:52] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 16% free memory
[19:44:53] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 39% free memory
[19:47:02] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[20:00:33] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 283 processes
[20:17:05] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[20:20:33] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes
[20:49:13] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[21:00:35] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 283 processes
[21:19:13] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[21:30:19] <wm-bot>	 Change on 12mediawiki a page Developer access was modified, changed by Rschmertz link https://www.mediawiki.org/w/index.php?diff=594398 edit summary: 
[21:30:32] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes
[21:49:25] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[21:58:09] <wm-bot>	 Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=594419 edit summary: /* User:Rschmertz */ done
[22:00:32] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 283 processes
[22:19:22] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[22:30:33] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 189 processes
[22:33:52] <bbobb>	 looking to help
[22:34:17] <bbobb>	 know JS pretty well
[22:34:52] <bbobb>	 omitting the first person personal pronoun
[22:49:22] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[22:51:43] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on ve-nodejs i-00000245.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake.
[22:56:42] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on ve-nodejs i-00000245.pmtpa.wmflabs output: OK: 29% free memory
[23:09:43] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on ve-nodejs i-00000245.pmtpa.wmflabs output: Warning: 9% free memory
[23:16:54] <gwicke>	 we are about to start Parsoid round-trip testing on dumps and need some cpu cycles for that
[23:17:40] <gwicke>	 a private vm with as many cores as possible would be great for that
[23:18:02] <gwicke>	 what is the process to request a new vm?
[23:19:10] <gwicke>	 https://labsconsole.wikimedia.org/wiki/Help:Instances says that it should be possible through the web interface with the proper rights
[23:19:22] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)
[23:21:52] <gwicke>	 which we don't currently have
[23:26:20] <Krenair>	 Ryan_Lane should be able to help
[23:26:57] <Ryan_Lane>	 which project?
[23:27:05] <Ryan_Lane>	 parsoid?
[23:27:25] <gwicke>	 yes
[23:27:51] <Ryan_Lane>	 there's no parsoid project
[23:28:01] <gwicke>	 in the meantime we have figured that we can probably use two vms that belong to the visualeditor project
[23:28:02] <Ryan_Lane>	 ah
[23:28:18] <Ryan_Lane>	 it's in visualeditor?
[23:28:23] <gwicke>	 yes
[23:28:46] <gwicke>	 not sure who is admin for the project, I guess Roan
[23:28:50] <Ryan_Lane>	 sec
[23:29:09] <Ryan_Lane>	 you have rights in that project already
[23:29:09] <gwicke>	 seems to be offline right now
[23:29:10] <Ryan_Lane>	 you have sysadmin rights
[23:29:28] <Ryan_Lane>	 so, just create an instance as large as you need
[23:29:29] <gwicke>	 ok, then I missed the 'create instance' link perhaps
[23:29:44] <Ryan_Lane>	 cool
[23:29:45] <Ryan_Lane>	 gotta go
[23:49:23] <labs-nagios-wm>	 PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs)