[00:02:42] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 151 processes
[00:05:33] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 20% free memory
[00:05:33] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 19% free memory
[00:10:32] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 21% free memory
[00:31:20] <milimetric>	 hello my favourite people :)
[00:31:51] <milimetric>	 If anyone has a moment, I'd like cloudadmin or just a new IP so I can assign it to my new instance from https://labsconsole.wikimedia.org/wiki/Special:NovaAddress
[00:32:39] <milimetric>	 I'd like to make redmine.wmflabs.org point to redmine.pmtpa.wmflabs
[00:33:41] <milimetric>	 redmine is a labs instance in the "Analytics" project
[00:35:03] <RoanKattouw>	 milimetric: BTW, you can also use http://redmine.instance-proxy.wmflabs.org
[00:35:13] <milimetric>	 oh that's awesome
[00:35:27] <milimetric>	 that works - scratch my other request
[00:35:32] <milimetric>	 and thanks!
[00:39:22] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 24% free memory
[00:41:03] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-nr2.pmtpa.wmflabs 10.4.1.66 output: Warning: 18% free memory
[00:41:53] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory
[00:53:42] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 143 processes
[00:53:49] <mike_wang>	 andrewbogott: my proxy server mwang-dev is working. Please test. I assigned a public IP(208.80.153.196) to mwang-dev. because I did not change DNS. please put the folloing 3 lines to your /etc/hosts
[00:54:14] <mike_wang>	 208.80.153.196 mwang-dev.wmflabs.org
[00:54:16] <mike_wang>	 208.80.153.196 mwang-dev1.mwang-dev.wmflabs.org
[00:54:17] <mike_wang>	 208.80.153.196 puppetdoc2.mwang-dev.wmflabs.org
[00:54:52] <andrewbogott>	 Hm… seems like just the first should do it…  I'll try that first :)
[00:54:57] <mike_wang>	 then you test http://puppetdoc2.mwang-dev.wmflabs.org/ and 
[00:55:28] <mike_wang>	 http://mwang-dev1.mwang-dev.wmflabs.org/
[00:57:35] <andrewbogott>	 mike_wang, do you mind if I add the dns entry?
[00:57:49] <andrewbogott>	 Or is that broken in labsconsole?
[00:58:21] <mike_wang>	 No. I try to add it. It doesn't work.
[00:58:44] <andrewbogott>	 http://mwang-proxy.wmflabs.org/
[00:59:53] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 17% free memory
[01:00:03] <andrewbogott>	 Hm, so that works but http://puppetdoc2.mwang-proxy.wmflabs.org/ does not.
[01:00:34] <andrewbogott>	 http://puppetdoc2.instance-proxy.wmflabs.org/  works though.  So mwang-dev isn't proxying in the same way
[01:01:54] <andrewbogott>	 mike_wang:  ^^
[01:01:55] <mike_wang>	 It is mwang-dev. not mwang-proxy
[01:02:09] <andrewbogott>	 I added a dns name for mwang-proxy.
[01:03:09] <andrewbogott>	 Oh, hm, maybe I didn't do the DNS right, lemme look
[01:05:20] <andrewbogott>	 Hm, probably I'm just rediscovering something you've already gone over.  Ryan_Lane, any idea how it's possible that *.instance-proxy.wmflabs.org is set in labsconsole but I can't create a new entry with a wildcard like that?
[01:05:57] <andrewbogott>	 mike_wang, OK, I'm convinced that your proxy is working right and I'm just hitting a labsconsole issue.
[01:06:25] <mike_wang>	 I don't know how to do it through labsconsole.
[01:06:29] <andrewbogott>	 so, want to prepare a patch and submit it for review?  You'll want to copy the changes you made in /etc/puppet into your personal repo in ~/puppet.
[01:06:45] <andrewbogott>	 Yeah, maybe it's not possible in labsconsole and requires a comandline hack.
[01:07:26] <mike_wang>	 OK. It is working. Let me check it in tomorrow.
[01:07:42] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 148 processes
[01:08:06] <mike_wang>	 chat with you  tomorrow.
[01:08:10] <Ryan_Lane>	 andrewbogott: because you can only create one inside of a domain
[01:08:11] <andrewbogott>	 ok!
[01:08:24] <Ryan_Lane>	 so, there's likely an instance-proxy domain
[01:09:42] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 179 processes
[01:11:28] <andrewbogott>	 Ryan_Lane:  You're right!  But, how did it get there?
[01:11:37] <Ryan_Lane>	 someone created it
[01:11:41] <Ryan_Lane>	 probably me
[01:12:09] <andrewbogott>	 Via the cli though, not labs console?
[01:12:23] <Ryan_Lane>	 labsconsole….
[01:12:23] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 11% free memory
[01:12:25] <Ryan_Lane>	 "Manage domains"
[01:13:16] <andrewbogott>	 Oh…
[01:13:20] <andrewbogott>	 Ok, mystery solved :)
[01:13:22] <Ryan_Lane>	 heh
[01:14:43] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 100 processes
[01:18:01] <andrewbogott>	 mwang:  OK, now http://pancakes.mwang-proxy.wmflabs.org/  works (it gets us the nginx page on mwang-dev) but http://puppetdoc2.mwang-proxy.wmflabs.org/ does not.  So there's still something not quite right happening.
[01:18:09] <andrewbogott>	 I think.
[01:19:23] <mike_wang>	 My instance name is mwang-dev, not mwang-proxy.
[01:21:02] <andrewbogott>	 Oh, does the proxy do a string compare based on a known address?
[01:21:03] * andrewbogott  changes it

[01:22:14] <mike_wang>	 there is one line in proxy conf
[01:22:18] <mike_wang>	 if ( $host ~ "^([a-z0-9\-_]+)\.mwang-dev\.wmflabs\.org$" )
[01:22:42] <andrewbogott>	 ok, surely that is the problem :)  Let's see...
[01:22:54] <mike_wang>	 so please leave mwang-proxy unchanged. I can change the proxy conf
[01:23:10] <andrewbogott>	 Nope, too late, I changed it already :)
[01:23:17] <mike_wang>	 OK
[01:26:50] <andrewbogott>	 mike_wang, working now!
[01:27:58] <andrewbogott>	 Sorry I doubted you :)
[01:28:46] <mike_wang>	 I will check in the code tomorrow
[01:29:33] <andrewbogott>	 cool.  g'night!
[01:29:52] <mike_wang>	 good night
[02:00:52] <labs-nagios-wm>	 PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed
[02:02:22] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output
[02:05:52] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory
[02:07:22] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: All packages OK
[02:34:52] <labs-nagios-wm>	 PROBLEM host: ee-lwelling.pmtpa.wmflabs is DOWN address: 10.4.1.70 CRITICAL - Host Unreachable (10.4.1.70)
[02:43:53] <labs-nagios-wm>	 RECOVERY host: ee-lwelling.pmtpa.wmflabs is UP address: 10.4.1.70 PING OK - Packet loss = 0%, RTA = 0.81 ms
[02:44:23] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: Connection refused by host
[02:45:53] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: Connection refused by host
[02:45:53] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: Connection refused by host
[02:46:33] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: Connection refused by host
[02:47:13] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: Connection refused by host
[02:48:03] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: Connection refused by host
[03:09:22] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: PROCS OK: 93 processes
[03:10:52] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: OK - load average: 0.28, 0.78, 0.80
[03:10:52] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: All packages OK
[03:11:32] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: USERS OK - 0 users currently logged in
[03:12:12] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: DISK OK
[03:13:02] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: OK: 615% free memory
[03:13:32] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 17% free memory
[03:48:34] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 19% free memory
[04:13:32] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 20% free memory
[04:37:24] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory
[04:38:34] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 22% free memory
[04:39:54] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory
[04:41:48] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 10% free memory
[04:45:24] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory
[04:46:34] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 18% free memory
[04:47:54] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 17% free memory
[05:26:32] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Critical: 3% free memory
[05:31:33] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 20% free memory
[06:09:32] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 19% free memory
[06:48:53] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: DPKG CRITICAL dpkg reports broken packages
[06:53:52] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on ee-lwelling.pmtpa.wmflabs 10.4.1.70 output: All packages OK
[08:40:24] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 26% free memory
[08:41:33] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 17% free memory
[08:48:22] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory
[08:54:52] <labs-nagios-wm>	 PROBLEM host: wikidata-puppet-variables.pmtpa.wmflabs is DOWN address: 10.4.0.224 CRITICAL - Host Unreachable (10.4.0.224)
[09:13:52] <labs-nagios-wm>	 RECOVERY host: wikidata-puppet-variables.pmtpa.wmflabs is UP address: 10.4.0.224 PING OK - Packet loss = 0%, RTA = 0.58 ms
[09:14:22] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: Connection refused by host
[09:15:53] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: Connection refused by host
[09:15:54] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: Connection refused by host
[09:16:33] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: Connection refused by host
[09:17:13] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: Connection refused by host
[09:18:03] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: Connection refused by host
[09:41:33] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Critical: 5% free memory
[09:45:42] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 152 processes
[09:50:43] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 150 processes
[09:57:12] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: DISK OK
[09:58:02] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: OK: 712% free memory
[09:58:42] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 153 processes
[09:59:22] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: PROCS OK: 84 processes
[10:00:53] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: OK - load average: 0.06, 0.64, 0.57
[10:00:53] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: All packages OK
[10:01:33] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: USERS OK - 0 users currently logged in
[10:40:14] <petan>	 !q1
[10:40:14] <wm-bot>	 Damianz where is teh ramcheck in puppet?
[10:53:54] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: DPKG CRITICAL dpkg reports broken packages
[10:58:53] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on wikidata-puppet-variables.pmtpa.wmflabs 10.4.0.224 output: All packages OK
[11:07:32] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 19% free memory
[11:10:23] <hashar>	 !log integration created integration-contintrefactor to test out the refactorization of the contint puppet manifests https://gerrit.wikimedia.org/r/43429
[11:10:25] <labs-morebots>	 Logged the message, Master
[11:16:33] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 29% free memory
[11:20:52] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: Connection refused by host
[11:20:52] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: Connection refused by host
[11:21:37] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: Connection refused by host
[11:22:12] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: Connection refused by host
[11:22:22] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: Connection refused by host
[11:23:12] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: Connection refused by host
[11:27:24] <addshore>	 !tunnel
[11:27:24] <wm-bot>	 ssh -f user@bastion.wmflabs.org -L <local>:server:<remote> -N Example for sftp "ssh chewbacca@bastion.wmflabs.org -L 6000:bots-1:22 -N" will open bots-1:22 as localhost:6000
[11:29:40] <hashar>	 grmblbl
[11:32:08] <addshore>	 hasharar
[11:35:52] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: OK: 1625% free memory
[11:35:52] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: OK - load average: 0.57, 0.96, 0.80
[11:36:32] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: USERS OK - 0 users currently logged in
[11:37:12] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: DISK OK
[11:37:22] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: PROCS OK: 91 processes
[11:38:12] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: All packages OK
[12:00:52] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory
[12:07:34] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 21% free memory
[12:11:22] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory
[12:36:44] <labs-nagios-wm>	 PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 154 processes
[12:38:24] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 22% free memory
[12:40:52] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 23% free memory
[12:41:22] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 20% free memory
[12:49:23] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory
[13:01:22] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 16% free memory
[13:09:12] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: DPKG CRITICAL dpkg reports broken packages
[13:23:53] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory
[13:34:13] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on integration-contintrefactor.pmtpa.wmflabs 10.4.1.52 output: All packages OK
[14:03:53] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 23% free memory
[14:09:07] <petan>	 !q1
[14:09:07] <wm-bot>	 Damianz where is teh ramcheck in puppet?
[14:26:42] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 145 processes
[14:26:52] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory
[14:58:29] <Wiki13>	 I don´t if itś already mentioned here, but http://ganglia.wmflabs.org/latest/ does not work
[14:59:37] <sumanah>	 "There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection refused"
[14:59:58] <Wiki13>	 yup
[15:00:07] <Wiki13>	 same error here
[15:05:06] <sumanah>	 Wiki13: so, andre__ noticed what I didn't which is that of course this is Ganglia *on Labs*
[15:05:14] <sumanah>	 Wiki13: you know about the "real" ganglia?
[15:06:33] <Wiki13>	 I know about the ganglia for the production network, yes...
[15:07:42] <sumanah>	 ok
[15:08:11] <Wiki13>	 ah, maybe I know now
[15:08:12] <Wiki13>	 -.-
[15:08:17] <Wiki13>	 nvm me then
[15:08:48] <sumanah>	 Wiki13: you might leave a message on the talk page for https://labsconsole.wikimedia.org/wiki/Nova_Resource:Ganglia if you want :)
[15:09:03] <Wiki13>	 the one on the production network needs a pass right?
[15:09:20] * sumanah  doesn't know, as she doesn't visit that much

[15:09:37] <^demon>	 Yes, it does right now.
[15:11:10] <Wiki13>	 but afaik the last time I visited it, i could see stats...
[15:11:11] <Wiki13>	 I believe
[15:11:16] <Wiki13>	 on the wmflabs one
[15:26:48] <Wiki13>	 Captcha´s are getting annoying -.-
[15:27:58] <Wiki13>	 sumanah: I posted a message about it on https://labsconsole.wikimedia.org/wiki/Nova_Resource_Talk:Ganglia
[15:31:07] <sumanah>	 thanks Wiki13 
[16:31:54] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 23% free memory
[16:34:41] <MaxSem>	 why could puppetd -tv hang for more than 2 minutes without outputting even the first "Loading facts" message?
[16:41:24] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory
[16:44:55] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory
[17:04:22] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory
[17:06:56] <petan>	 hashar
[17:07:01] <petan>	 @infobot-set-raw beta
[17:07:01] <wm-bot>	 There is no such a key, petan
[17:07:06] <petan>	 meh
[17:07:09] <petan>	 !beta
[17:07:14] <petan>	 wtf
[17:07:21] <petan>	 !ping
[17:07:21] <wm-bot>	 pong
[17:07:26] <petan>	 did you delete it?
[17:07:30] <petan>	 !beta is $*
[17:07:30] <wm-bot>	 Key was added
[17:07:34] <petan>	 !beta del
[17:07:34] <wm-bot>	 Successfully removed beta
[17:07:42] <petan>	 !beta is !log deployment-prep $*
[17:07:43] <wm-bot>	 Key was added
[17:07:49] <petan>	 @infobot-set-raw beta
[17:07:49] <wm-bot>	 This key will be displayed with no extra styling, variables and will ignore all symbols
[17:08:05] <petan>	 !beta this is a creepy log with | and such shitty chars $@#% 6346 w@#%^@# 6bla
[17:08:05] <wm-bot>	 !log deployment-prep this is a creepy log with | and such shitty chars $@#% 6346 w@#%^@# 6bla
[17:08:08] <labs-morebots>	 Logged the message, Master
[17:08:15] <petan>	 hashar: a gift
[17:08:17] <petan>	 for you :D :D
[17:10:32] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 18% free memory
[17:12:31] <hashar>	 petan: :-]
[17:12:55] <hashar>	 petan: jeremyb removed the !beta alias cause the alias does not mention the original username
[17:13:12] <hashar>	 petan: so we end up with wm-bot as a user for the log entry :/
[17:13:19] <hashar>	 nice to see that the pipe is fixed!
[17:25:33] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 20% free memory
[17:27:52] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: Warning: 19% free memory
[17:40:32] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 19% free memory
[17:42:55] <addshore>	 :O
[17:43:50] <addshore>	 alchimis process is using 50% of the ram D:
[18:44:43] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 8.86, 7.79, 5.83
[18:59:53] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory
[19:03:23] <MaxSem>	 I there a way to create fourth-level subdomains for instances?
[19:11:51] <andrewbogott>	 MaxSem:  there is, as I just learned yesterday.  You can add your domain here:  https://labsconsole.wikimedia.org/wiki/Special:NovaDomain
[19:12:04] <andrewbogott>	 And then that will appear in the pulldown on the 'manage addresses' page.
[19:12:18] <MaxSem>	 whee, thanks!
[19:12:18] <andrewbogott>	 If you don't have access to that page or can't alter it then let me know and I'll create the domain for you.
[19:13:46] <MaxSem>	 andrewbogott, indeed I can't - can you create a domain osm.wmflabs.org please?
[19:13:58] <andrewbogott>	 yep
[19:14:21] <andrewbogott>	 hm… I am chronically confused about OpenStackManager vs. OpenStreetMaps
[19:14:28] <andrewbogott>	 But probably I'm the only one who has that problem
[19:16:00] <andrewbogott>	 MaxSem:  OK, done -- probably you'll need to reload the manage addresses page.
[19:16:08] <MaxSem>	 cheers!
[19:17:53] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory
[20:37:54] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 22% free memory
[20:37:54] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: OK: 33% free memory
[20:39:23] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory
[20:45:52] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory
[20:57:59] <^demon>	 Ryan_Lane: I've got a couple of outstanding gerrit changes, all trivial except 1.
[21:02:23] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory
[21:22:25] <hashar>	 Ryan_Lane: and when you have time please abandon https://gerrit.wikimedia.org/r/#/c/12425/  ( operations/debs/ircd-ratbox initial commit ) :-]
[21:22:43] <Ryan_Lane>	 why? do you have something that fixes it?
[21:23:47] <hashar>	 ah maybe you get some specific stuff in it
[21:24:52] <hashar>	 ^demon: easter egg in Gerrit.  Just had a user "Anonymous Coward 75" that voted V-1 on a change :-]
[21:30:53] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 23% free memory
[21:35:52] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on techvandalism-bot.pmtpa.wmflabs 10.4.0.194 output: Warning: 19% free memory
[22:08:53] <labs-nagios-wm>	 PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory
[22:53:42] <preilly>	 Ryan_Lane: sudo: unknown uid 2026: who are you?
[23:08:55] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on performance.pmtpa.wmflabs 10.4.1.69 output: Connection refused by host
[23:09:35] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on performance.pmtpa.wmflabs 10.4.1.69 output: Connection refused by host
[23:10:23] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on performance.pmtpa.wmflabs 10.4.1.69 output: Connection refused by host
[23:10:53] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on performance.pmtpa.wmflabs 10.4.1.69 output: Connection refused by host
[23:12:26] <labs-nagios-wm>	 PROBLEM Total processes is now: CRITICAL on performance.pmtpa.wmflabs 10.4.1.69 output: Connection refused by host
[23:13:13] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on performance.pmtpa.wmflabs 10.4.1.69 output: Connection refused by host
[23:15:14] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on performance.pmtpa.wmflabs 10.4.1.69 output: DISK OK
[23:15:54] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on performance.pmtpa.wmflabs 10.4.1.69 output: OK: 1639% free memory
[23:17:24] <labs-nagios-wm>	 RECOVERY Total processes is now: OK on performance.pmtpa.wmflabs 10.4.1.69 output: PROCS OK: 90 processes
[23:18:14] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on performance.pmtpa.wmflabs 10.4.1.69 output: All packages OK
[23:18:54] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on performance.pmtpa.wmflabs 10.4.1.69 output: OK - load average: 0.05, 0.58, 0.53
[23:19:34] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on performance.pmtpa.wmflabs 10.4.1.69 output: USERS OK - 0 users currently logged in
[23:26:42] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 6.07, 6.02, 5.27
[23:33:33] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 3.88, 5.14, 5.10
[23:36:12] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on performance.pmtpa.wmflabs 10.4.1.69 output: DPKG CRITICAL dpkg reports broken packages
[23:43:33] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 5.35, 4.85, 4.90