[06:11:16] <shinken-wm>	 PROBLEM - SSH on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:19:57] <shinken-wm>	 RECOVERY - SSH on tools-webgrid-lighttpd-1409 is OK: SSH OK - OpenSSH_6.9p1 Ubuntu-2~trusty1 (protocol 2.0)
[06:20:37] <shinken-wm>	 PROBLEM - Puppet failure on tools-bastion-02 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0]
[06:55:24] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[06:55:42] <shinken-wm>	 RECOVERY - Puppet failure on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[07:30:27] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0]
[14:47:48] <Luke081515>	 !log rcm deleted rcm-3 (unused)
[14:47:59] <Luke081515>	 !log Rcm deleted rcm-3 (unused)
[14:47:59] <labs-morebots>	 Rcm is not a valid project.
[14:48:01] <Luke081515>	 hm
[14:48:49] <Luke081515>	 !log RCM deleted rcm-3 (unused)
[14:48:49] <labs-morebots>	 RCM is not a valid project.
[14:48:59] <valhallasw`cloud>	 tools.rcm?
[14:49:12] <Luke081515>	 no, Nova Resource:Rcm
[14:49:37] <Luke081515>	 what was wrong?
[14:51:33] <valhallasw`cloud>	 oh, right. The first one worked, exceeept
[14:51:49] <valhallasw`cloud>	 !log rcm deleted rcm-3 (unused)
[14:51:53] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL, Master
[14:52:01] <valhallasw`cloud>	 there's a bug in the bot, where it needs to have at least one header to work
[14:52:18] <Luke081515>	 ah, ok, so it will work the next time?
[14:52:26] <valhallasw`cloud>	 yeah
[14:52:35] <Luke081515>	 Ok, thank you
[15:04:14] <wikibugs>	 6Labs: Enable NFS at own project - https://phabricator.wikimedia.org/T112465#1805645 (10Luke081515) 5Open>3Resolved a:3Luke081515
[15:07:04] <wikibugs>	 6Labs, 6Phabricator: Labs' Phabricator uses Wikimedia logo - https://phabricator.wikimedia.org/T117235#1805648 (10Luke081515) Any ideas how we can fix this? At the moment I can just try to don't confuse user, when I'm writing a note at the hompage like: IMPORTANT: This in not a wikimedia project  and I set thi...
[15:29:19] <valhallasw`cloud>	 Nemo_bis: why are you scraping the entire mailman archive on tools-login?
[16:23:58] <wikibugs>	 10Tool-Labs-tools-Other, 6Wikisource: OCR scripts need updating at tools labs - https://phabricator.wikimedia.org/T117711#1805661 (10Aklapper)
[16:34:27] <Luke081515|Busy>	 Question: the section "Instances for this project", is this section created automaticaly? There are only two of my five instances
[16:37:05] <valhallasw`cloud>	 Luke081515|Busy: it's automatically updated, but it's not instantaneous
[16:37:13] <Luke081515|Busy>	 hm, ok
[16:37:15] <Luke081515|Busy>	 tahnks
[16:37:18] <Luke081515|Busy>	 *thanks
[16:37:20] <valhallasw`cloud>	 wikitech.wikimedia.org/wiki/Special:NovaInstance should be up to date
[16:37:25] <valhallasw`cloud>	 if not, try logging out and in again
[16:37:53] <Luke081515|Busy>	 yeah, it's up to date, but don't has the same information, like instance type
[16:43:28] <wikibugs>	 10Tool-Labs-tools-Other, 6Wikisource: OCR scripts need updating at tools labs - https://phabricator.wikimedia.org/T117711#1805670 (10valhallasw) Tool Labs has these files installed via apt-get: http://packages.ubuntu.com/trusty/tesseract-ocr-ben , but the last release was in 2012 (!). More recent debian releas...
[16:45:20] <wikibugs>	 6Labs, 10Tool-Labs: Backport newer tesseract-ocr-* packages from sid - https://phabricator.wikimedia.org/T118643#1805671 (10valhallasw) 3NEW
[17:09:48] <Luke081515>	 !log rcm recreated rcm-3 to do some specific tests
[17:09:52] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL, Master
[17:09:56] <Luke081515>	 ah :)
[18:17:27] <Nemo_bis>	 valhallasw`cloud: so that I can watch what it does, but can stop if it's a problem (memory went slightly over expected)
[18:18:24] <wikibugs>	 10Quarry, 5Patch-For-Review: Add chat or forum (Quarry) - https://phabricator.wikimedia.org/T117647#1805719 (10XXN) 5Open>3Resolved a:3XXN It's not exactly what I personally expected and prefered, but is a sollution and a good alternative. Thanks. I think this task can be closed now.
[18:19:10] <wikibugs>	 10Quarry: Add chat or forum (Quarry) - https://phabricator.wikimedia.org/T117647#1805723 (10XXN) a:5XXN>3yuvipanda
[18:24:25] <valhallasw`cloud>	 Nemo_bis: yeah, I was mainly wondering if there wasn't a more direct way to do what you want, by asking someone with access to the mailman server
[18:25:14] <valhallasw`cloud>	 my shell is lagging, but that might also be my internet connection
[18:28:57] <wikibugs>	 10Quarry: Show last run time on query page in Quarry - https://phabricator.wikimedia.org/T118646#1805743 (10XXN) 3NEW
[18:52:56] <wikibugs>	 6Labs, 10wikitech.wikimedia.org, 7Wikimedia-log-errors: Hook SMWParseData::onTitleMoveComplete has invalid call signature; Parameter 3 to SMWParseData::onTitleMoveComplete() expected to be a reference, value given - https://phabricator.wikimedia.org/T118649#1805783 (10Legoktm) 3NEW
[18:59:46] <Nemo_bis>	 valhallasw`cloud: nope, wget is the only proper tool to use for the purpose; almost done anyway
[19:13:42] <phe>	 looking for duplicated task starting, crontab -l give me the same, non empty, cron jobs on both @tools-submit and @tools-bastion-01 , I don't remember to setup cron on bastion, and is it normal to allow crontab from bastion?
[19:15:42] <valhallasw`cloud>	 phe: crontabs are always stored and executed on tools-submit
[19:15:48] <valhallasw`cloud>	 so it's the same crontab
[19:16:08] <phe>	 but visible from bastion too?
[19:16:13] <valhallasw`cloud>	 yes
[19:16:30] <phe>	 ok, so my duped task comes from elsewhere
[19:17:28] <valhallasw`cloud>	 unless something has gone wrong, yes.
[19:46:38] <shinken-wm>	 PROBLEM - Host tools-andrew-puppettest is DOWN: CRITICAL - Host Unreachable (10.68.21.109)
[20:09:03] <wikibugs>	 6Labs, 10Tool-Labs: Define policy for tools-local packages - https://phabricator.wikimedia.org/T114645#1805864 (10valhallasw) Working from two main premises:  1) we need to prevent attack vectors on the infrastructure, and  2) we need to reduce the load on admins as much as possible,  I suggest the following....
[20:09:04] <valhallasw`cloud>	 yuvipanda: ^ wheee
[20:10:10] <wikibugs>	 6Labs, 10Tool-Labs: tools-webgrid-lighttpd-1412 refuses ssh connections - https://phabricator.wikimedia.org/T118654#1805866 (10scfc) 3NEW
[20:22:44] <wikibugs>	 6Labs, 10Tool-Labs: tools-webgrid-lighttpd-1412 refuses ssh connections - https://phabricator.wikimedia.org/T118654#1805893 (10valhallasw) Odd. Graphite doesn't seem to have /any/ recent information, and shinken is not complaining about anything. Has this host ever been online to begin with...?  example: {F296...
[20:27:59] <wikibugs>	 6Labs, 10Tool-Labs: tools-webgrid-lighttpd-1412 refuses ssh connections - https://phabricator.wikimedia.org/T118654#1805897 (10valhallasw) Nothing obvious in the console log either:  ``` Cloud-init v. 0.7.5 finished at Mon, 02 Nov 2015 22:07:58 +0000. Datasource DataSourceOpenStack [net,ver=2].  Up 183.07 seco...
[21:04:51] <wikibugs>	 6Labs, 10Tool-Labs: tools-webgrid-lighttpd-1412 refuses ssh connections - https://phabricator.wikimedia.org/T118654#1805941 (10yuvipanda) I think this host was stillborn and I kept it alive to see if @andrewbogott wanted to investigate it, but I think it's ok to just kill it and rebirth it now. This was due to...
[22:29:48] <wikibugs>	 10Tool-Labs-tools-Global-user-contributions: Tool no longer shows number of projects with edits - https://phabricator.wikimedia.org/T118662#1806012 (10MC8) 3NEW