[00:49:24] <wikibugs>	 6Labs, 10LabsDB-Auditor, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs, and 8 others: Labs' Phabricator tags overhaul - https://phabricator.wikimedia.org/T89270#1253943 (10Dzahn)
[00:50:08] <wikibugs>	 6Labs, 10LabsDB-Auditor, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs, and 8 others: Labs' Phabricator tags overhaul - https://phabricator.wikimedia.org/T89270#1031944 (10Dzahn) removed tag "wikistats" because that has already been renamed from the former labs-related name and wasn't relevant here...
[00:51:30] <yuvipanda>	 !log deployment-prep created deployment-boomboom to test
[00:51:51] <labs-morebots>	 Logged the message, Master
[01:57:51] <yuvipanda>	 !log tools increased tools instance quota
[01:58:06] <labs-morebots>	 Logged the message, Master
[01:59:23] <yuvipanda>	 !log tools created tools-webgrid-lighttpd-14{01-10}
[01:59:27] <labs-morebots>	 Logged the message, Master
[02:12:17] <yuvipanda>	 !log tools created tools-webgrid-lighttpd-12{01-05} and tools-webgrid-generic-14{01-04}
[02:13:22] <labs-morebots>	 Logged the message, Master
[02:31:00] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 85.71% of data above the critical threshold [0.0]  
[02:31:14] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[02:35:33] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[02:37:04] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[02:41:00] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:41:10] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:44:05] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[02:45:05] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[02:45:29] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1405 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0]  
[02:46:59] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:52:15] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[02:53:03] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[02:55:20] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[02:55:30] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1405 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:04:39] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-07 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[03:05:31] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:09:03] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:10:01] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:12:12] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:18:04] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:20:17] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:29:39] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-07 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:41:49] <shinken-wm>	 PROBLEM - Host tools-webgrid-06 is DOWN: CRITICAL - Host Unreachable (10.68.17.163)  
[03:42:00] <legoktm>	 umm
[03:42:01] <shinken-wm>	 PROBLEM - Host tools-webgrid-05 is DOWN: CRITICAL - Host Unreachable (10.68.16.239)  
[03:43:30] <wikibugs>	 6Labs, 10Tool-Labs: Rebuild a bunch of tools instances - https://phabricator.wikimedia.org/T97437#1254083 (10yuvipanda) Trusty webgrid has been expanded and replaced with new hosts.  I built new precise and generic webgrid hosts too but they seem to be dead on arrival and not running puppet at all for some str...
[03:43:57] <yuvipanda>	 !log tools drained and deleted old trusty webgrid tools-webgrid-{05-07}
[03:44:00] <yuvipanda>	 legoktm: that’s all me
[03:44:04] <labs-morebots>	 Logged the message, Master
[03:44:08] <legoktm>	 okay
[03:44:12] <yuvipanda>	 legoktm: well, not the nfs load
[03:44:13] <yuvipanda>	 that’s not me
[03:44:17] <legoktm>	 :P
[03:44:28] <shinken-wm>	 PROBLEM - Host tools-webgrid-07 is DOWN: CRITICAL - Host Unreachable (10.68.16.97)  
[03:44:33] <yuvipanda>	 that’s me too ^
[03:51:18] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0]  
[03:56:19] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:26:33] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[06:31:57] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[06:34:25] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 71.43% of data above the critical threshold [0.0]  
[06:35:41] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1408 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[06:39:30] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[06:39:32] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[06:44:26] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:45:10] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[06:49:36] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [0.0]  
[06:49:38] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [0.0]  
[06:50:14] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:51:34] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:54:04] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[06:56:50] <yuvipanda>	 !log tools pooled tools-webgrid-generic-{01-04}
[06:56:55] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:56:58] <labs-morebots>	 Logged the message, Master
[06:59:35] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:00:45] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:02:11] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[07:04:30] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:06:10] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[07:19:40] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[07:31:03] <yuvipanda>	 !log tools disabled catmonitor task / cron, was heavily using an sqlite db on NFS
[07:31:11] <labs-morebots>	 Logged the message, Master
[07:31:23] <yuvipanda>	 !log depooled and deleted tools-webgrid-{01,02}
[07:31:24] <labs-morebots>	 depooled is not a valid project.
[07:31:28] <yuvipanda>	 !log tools depooled and deleted tools-webgrid-{01,02}
[07:31:33] <labs-morebots>	 Logged the message, Master
[07:32:43] <shinken-wm>	 PROBLEM - Host tools-webgrid-generic-02 is DOWN: CRITICAL - Host Unreachable (10.68.16.215)  
[07:33:10] <shinken-wm>	 PROBLEM - Host tools-webgrid-generic-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.152)  
[07:36:05] <wikibugs>	 6Labs, 10Tool-Labs: Rebuild a bunch of tools instances - https://phabricator.wikimedia.org/T97437#1254180 (10yuvipanda) Generic hosts migrated now. Just precise ones left :)
[08:37:36] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[08:52:09] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0]  
[08:52:34] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0]  
[08:54:01] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0]  
[08:54:27] <shinken-wm>	 PROBLEM - Puppet staleness on tools-mailrelay-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]  
[08:54:39] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0]  
[08:56:06] <yuvipanda>	 !log tools drained and deleted tools-webgrid-01
[08:56:07] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0]  
[08:56:15] <labs-morebots>	 Logged the message, Master
[08:59:58] <shinken-wm>	 PROBLEM - Host tools-webgrid-01 is DOWN: CRITICAL - Host Unreachable (10.68.16.28)  
[09:07:54] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[09:17:01] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[09:17:53] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0]  
[09:19:22] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[09:19:48] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[09:24:34] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[09:59:22] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0]  
[09:59:44] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [0.0]  
[10:02:05] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0]  
[10:09:20] <valhallasw`cloud>	 !log toolsbeta created [[Nova_Resource:I-00000c01.eqiad.wmflabs|toolsbeta-logstash]] to play around with logstash and figure out what we need for tools ([[phab:T97861]])
[10:09:50] <valhallasw`cloud>	 labslogbot dead?
[10:09:51] <labs-morebots>	 Logged the message, Master
[10:10:11] <valhallasw`cloud>	 just slow
[10:12:24] <yuvipanda>	 !log tools pooled tools-webgrid-lighttpd-{06-10}
[10:12:28] <labs-morebots>	 Logged the message, Master
[10:12:50] <valhallasw`cloud>	 yuvipanda: why is openstack so incomprehensible? :P
[10:13:07] <valhallasw`cloud>	 yuvipanda: how do toolsbeta instances know to use the local puppetmaster? it's not configured anywhereh
[10:13:18] <valhallasw`cloud>	 yuvipanda: and why can't I choose role::logstash?
[10:13:41] <yuvipanda>	 !log tools cleaned out wegrid  jobs from tools-webgrid-03
[10:13:44] <yuvipanda>	 hi valhallasw`cloud :)
[10:13:45] <labs-morebots>	 Logged the message, Master
[10:14:26] <yuvipanda>	 valhallasw`cloud: http://wikitech.wikimedia.org/wiki/Hiera:toolsbeta is where that’s configured. it’s documented wherever per project puppetmasters are, IIRC
[10:14:30] <yuvipanda>	 I should also sleep
[10:14:33] <valhallasw`cloud>	 (also, how are you still awake? it's π time over there)
[10:14:34] <yuvipanda>	 valhallasw`cloud: if you want to apply new roles
[10:14:34] <valhallasw`cloud>	 haha
[10:14:43] <yuvipanda>	 valhallasw`cloud: try ‘manage puppet groups'
[10:14:45] <yuvipanda>	 on left firewall
[10:14:55] <valhallasw`cloud>	 ah
[10:14:58] <yuvipanda>	 err
[10:15:00] <yuvipanda>	 left sidebar
[10:15:04] <yuvipanda>	 clearly I should sleep :|
[10:15:40] <valhallasw`cloud>	 I'll add a 'toolsbeta' group for testing
[10:16:03] <yuvipanda>	 cool
[10:16:34] <valhallasw`cloud>	 also, those variables; are they basically hiera stuff again?
[10:16:40] <valhallasw`cloud>	 or are they different altogether
[10:16:54] <yuvipanda>	 totally different :P
[10:17:02] <yuvipanda>	 (and are being phased out, etc)
[10:17:08] <valhallasw`cloud>	 ok, I'll just ignore them then :P
[10:17:22] <yuvipanda>	 :)
[10:17:26] <valhallasw`cloud>	 !log toolsbeta added toolsbeta group to make role::logstash available there
[10:21:34] <yuvipanda>	 !log tools drained all the old webgrid nodes, pooled in all the new webgrid nodes! POTATO!
[10:21:38] <labs-morebots>	 Logged the message, Master
[10:21:55] <yuvipanda>	 did it answer to me or to valhallasw`cloud?
[10:21:57] * yuvipanda isn’t sure
[10:23:01] <yuvipanda>	 valhallasw`cloud: am off to bed. night :)
[10:23:05] <valhallasw`cloud>	 night
[10:23:18] <wikibugs>	 6Labs, 10Tool-Labs: Rebuild a bunch of tools instances - https://phabricator.wikimedia.org/T97437#1254314 (10yuvipanda) Done for all webgrid instances too \o/  Now we just figure out when to kill the rest of the running non-restartable task jobs.
[10:23:55] <valhallasw`cloud>	 wikitech is continuously losing session data
[10:23:57] <valhallasw`cloud>	 :/
[10:24:26] * valhallasw`cloud hits 'save' a few more times
[10:24:35] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0]  
[10:25:04] <valhallasw`cloud>	 !log toolsbeta set Hiera variable "elasticsearch::cluster_name": toolsbeta-logstash-eqiad
[10:25:12] <labs-morebots>	 Logged the message, Master
[10:30:47] <valhallasw`cloud>	 !log toolsbeta pulled new changes into puppetmaster to get https://github.com/wikimedia/operations-puppet/commit/4afd23d8e2905a84ef211ad92e8314173eb743ba in
[10:30:51] <labs-morebots>	 Logged the message, Master
[10:37:04] <valhallasw`cloud>	 !log toolsbeta that doesn't seem to be applied... setting has_ganglia: false manually in wikitech hiera
[10:37:08] <labs-morebots>	 Logged the message, Master
[10:39:09] <valhallasw`cloud>	 also, why on earth doesn't puppet give a backtrace 'hey, this was required from here, then here, then here'
[10:39:13] <valhallasw`cloud>	 *confused*
[11:11:40] <valhallasw`cloud>	 !log toolsbeta commenting out include ::elasticsearch::ganglia in role::logstash seems to work. I think we have to write our own tools logstash roles anyway in the end, as the role::logstash code contains e.g. mediawiki specific code
[11:11:46] <labs-morebots>	 Logged the message, Master
[11:29:59] <wikibugs>	 10Tool-Labs: Provide centralized logging (logstash) - https://phabricator.wikimedia.org/T97861#1254380 (10valhallasw) 10:09 valhallasw`cloud: created toolsbeta-logstash to play around with logstash and figure out what we need for tools (phab:T97861) 10:25 valhallasw`cloud: set Hiera variable "elasticsearch::clus...
[11:41:30] <wikibugs>	 10Tool-Labs: Provide centralized logging (logstash) - https://phabricator.wikimedia.org/T97861#1254382 (10valhallasw) Which was because elasticsearch wasn't started. OK, that starts logstash, but that doesn't actually give us an interface yet...
[12:21:39] <grrrit-wm>	 (03PS1) 10Alex Monk: Send Graph-VE MW extension project to VE channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/208384 
[12:28:40] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] Send Graph-VE MW extension project to VE channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/208384 (owner: 10Alex Monk)
[12:28:52] <grrrit-wm>	 (03Merged) 10jenkins-bot: Send Graph-VE MW extension project to VE channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/208384 (owner: 10Alex Monk)
[12:37:27] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: f9b9d5bda60b9f1f6aac196254f8b6cfff6d58a2 Send Graph-VE MW extension project to VE channel
[12:37:31] <labs-morebots>	 Logged the message, Master
[13:11:07] <Base>	 is Error occurred during initialization of VM
[13:11:07] <Base>	 Could not reserve enough space for object heap because I want too much memory or because I want too little?
[13:13:16] <Base>	 "Error occurred during initialization of VM
[13:13:16] <Base>	 Could not reserve enough space for object heap" (wasn't clear where quotation end and my text begins)
[13:32:14] <Base>	 the answer was too little
[14:50:28] <wikibugs>	 6Labs, 10Tool-Labs: Rebuild a bunch of tools instances - https://phabricator.wikimedia.org/T97437#1254591 (10valhallasw) tools-exec 02,07,08,13,14 and 15 are still in the @general host group; is that intended?  On tools-webgrid-03 and -08, I have killed a few php-cgi processes that were stuck in some gettimeof...
[14:51:00] <shinken-wm>	 PROBLEM - Host tools-webgrid-08 is DOWN: CRITICAL - Host Unreachable (10.68.18.35)  
[14:51:30] <shinken-wm>	 PROBLEM - Host tools-webgrid-03 is DOWN: CRITICAL - Host Unreachable (10.68.17.123)  
[14:52:04] <shinken-wm>	 PROBLEM - Host tools-webgrid-02 is DOWN: CRITICAL - Host Unreachable (10.68.17.9)  
[16:12:03] <valhallasw`cloud>	 yes, that was the idea, shinken-wm.
[16:20:32] <valhallasw`cloud>	 yuvipanda: we should find a way to unregister hosts in the various tools
[16:21:07] <valhallasw`cloud>	 or just do it manually, I guess
[16:41:07] <wikibugs>	 10Tool-Labs: Make a decommissioning checklist - https://phabricator.wikimedia.org/T97904#1254671 (10valhallasw) 3NEW
[16:54:18] <yuvipanda>	 valhallasw`cloud: it does get unlisted in shinken - after 5 minutes
[16:54:25] <yuvipanda>	 valhallasw`cloud: you won’t find these in the web interface
[16:54:59] <wikibugs>	 6Labs, 10Tool-Labs: Rebuild a bunch of tools instances - https://phabricator.wikimedia.org/T97437#1254683 (10yuvipanda) Yes, the remaining tools-exec hosts are still in the hostgroup but they have been disabled, so is ok.
[17:27:09] <valhallasw`cloud>	 yuvipanda: I see. How does that work?
[17:28:24] <valhallasw`cloud>	 oh, shinkengen just reads from ldap, which is sourced in wikitech
[17:28:38] <valhallasw`cloud>	 fair enough
[17:31:03] <wikibugs>	 10Tool-Labs: Make a decommissioning checklist - https://phabricator.wikimedia.org/T97904#1254698 (10valhallasw)
[17:55:50] <wikibugs>	 10Tool-Labs: Make a decommissioning checklist - https://phabricator.wikimedia.org/T97904#1254715 (10scfc) Cf. T95537 what I did for `tools-webgrid-04`.  <dream>There should be a Phabricator template with all the check boxes, etc. in place.  Then, in the "master" task that requires to decommission a host, you cli...
[18:07:01] <Magog_the_Ogre>	 is there a way to find out which tools have which license? 
[18:09:03] <yuvipanda>	 Magog_the_Ogre: not quite. I guess ‘ask the maintainer'
[18:09:07] <yuvipanda>	 Magog_the_Ogre: or look at the source
[18:14:42] <wikibugs>	 10Tool-Labs: Make a decommissioning checklist - https://phabricator.wikimedia.org/T97904#1254743 (10valhallasw)
[18:16:01] <wikibugs>	 10Tool-Labs: Make a decommissioning checklist - https://phabricator.wikimedia.org/T97904#1254750 (10yuvipanda)
[18:16:11] <wikibugs>	 10Tool-Labs: Make a decommissioning checklist - https://phabricator.wikimedia.org/T97904#1254671 (10yuvipanda) Should go on https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin
[18:16:58] <wikibugs>	 10Tool-Labs: Make a decommissioning checklist - https://phabricator.wikimedia.org/T97904#1254752 (10valhallasw) A simple userscript should be able to do that, I think. See @Aklapper s work at https://github.com/wikimedia/wikimedia-bugzilla-triagescripts/blob/master/wikimedia-maniphest-task.user.js
[18:27:02] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-1201 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[18:30:09] * valhallasw`cloud blinks
[18:33:42] <valhallasw`cloud>	 yuvipanda: ^ /var/log/puppet.log:1121:Error: /bin/cat /data/project/.system/store/hostkey-* >/etc/ssh/ssh_known_hosts~ returned 1 instead of one of [0]
[18:33:54] <valhallasw`cloud>	 could just be nfs, I'm rerunning puppet agent
[18:34:02] <valhallasw`cloud>	 yeah, works fine now
[18:34:08] <yuvipanda>	 if it could be just nfs it’s probably just nfs...
[18:38:26] <wikibugs>	 10Tool-Labs: Remove redundant ssh host keys from users' known_hosts - https://phabricator.wikimedia.org/T92497#1254755 (10scfc)
[18:47:02] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-1201 is OK: OK: Less than 1.00% above the threshold [0.0]  
[18:58:08] <wikibugs>	 10Tool-Labs: Make a decommissioning checklist - https://phabricator.wikimedia.org/T97904#1254763 (10yuvipanda)
[19:35:32] <valhallasw`cloud>	 yuvipanda: qdel causes bigbrother spam :p
[19:35:52] <yuvipanda>	 valhallasw`cloud: until we get rid of bigbrother :P
[22:38:18] <wikibugs>	 10Tool-Labs: Tool Labs: Install php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#1254841 (10Danmichaelo) Strange. I got `Fatal error:  Call to undefined function mcrypt_module_open()`. And on the command line, `php -i | grep mcrypt` gives no results. Both indicating it's not installed.
[22:46:24] <wikibugs>	 10Tool-Labs: Tool Labs: Install php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#1254842 (10Danmichaelo) Perhaps it's installed, but not activated? I vaguely remember that some ubuntu php packages do not create the necessary symlinks in `/etc/php5/`
[22:57:30] <CristianCantoro>	 Hi, I know that it ssssis possible to request a (virtual) server to host a Wikimedia-related service. In particular I would like to move the hosting of http://wikidataldf.com/ (which is now on DigitalOcean)... I have been asking myself "I'll look for this tomorrow" for too much time. Can somebody point out a page where it is explained how to make a request. Thanks.
[23:00:51] <yuvipanda>	 hi CristianCantoro 
[23:00:54] <yuvipanda>	 https://wikitech.wikimedia.org/wiki/Main_Page is the main page
[23:01:06] <yuvipanda>	 CristianCantoro: there’s a ‘request a project’ link on the left sidebar with info
[23:22:35] <CristianCantoro>	 yuvipanda: I am still lost. I remember there was a page that explained which type of instances you could request... I think that before asking "can you give me a machine?" one should be able to see which are the possibilities available based on the needs
[23:23:40] <yuvipanda>	 CristianCantoro: heh, all the documentation sucks, but here’s a screenshot of machine types: https://www.dropbox.com/s/ek68q3ekv60prgh/Screenshot%202015-05-02%2016.23.26.png?dl=0
[23:23:46] <yuvipanda>	 you can get jessie, precise or trusty
[23:24:03] <yuvipanda>	 https://lists.wikimedia.org/mailman/listinfo/labs-l is also a good source of help
[23:26:04] <CristianCantoro>	 yuvipanda: that helps, the current machine ha 2GB of RAM and 40GB of storage/root partition
[23:26:20] <yuvipanda>	 :) should fit in easily
[23:26:33] <yuvipanda>	 you can also use our flexible shared-ish hosting service (tools.wmflabs.org)
[23:27:46] <CristianCantoro>	 I am little lost between the difference among "create a new tool" in tools.wmflabs.org and "you can request a machine hosted on Wikimedia Labs"
[23:27:52] <CristianCantoro>	 the procedure is the same?
[23:30:15] <CristianCantoro>	 yuvipanda: https://lists.wikimedia.org/mailman/listinfo/labs-l is also a good source of help ... oh noes! Another Wikimedia mailing list to subscribe to... sooner or later my inbox will explode like a supernova
[23:30:54] <Krenair>	 They are very different things
[23:31:06] <yuvipanda>	 CristianCantoro: toollabs is like shared hosting. you don’t get root, but you don’t have to worry about sysadmin-y stuff. ‘Request a machine’ is you get a VM and you get full responsibility / root on it
[23:31:26] <Krenair>	 Which reminds me, I need to run updates on that one instance I'm responsible for
[23:33:24] <CristianCantoro>	 I understand the difference, my question is if the procedure to request access to labs, create a new tool or get access to an instance is the same... i.e. you have to make a request in the same place, i.e. "Request project"
[23:33:57] <Krenair>	 You need labs access to do either a tool or a whole instance
[23:34:30] <Krenair>	 Requesting a project is if you want to run your own group of instances, don't do it if you just want a tool
[23:34:57] <CristianCantoro>	 Krenair: ok, thanks
[23:35:04] <Krenair>	 Your own instance can get a public IP, I think?
[23:36:02] <Krenair>	 if you want to host your wikidata-related site from labs, you should probably request a project for it
[23:36:53] <CristianCantoro>	 I want to move the hosting of http://wikidataldf.com and so I was thinking of requesting a machine with similar characteristics on Labs
[23:38:10] <Krenair>	 any particular requirements?
[23:38:10] <CristianCantoro>	 so, in Labs language I should request a project if I get this correctly
[23:38:49] <Krenair>	 I believe so. yuvipanda?
[23:39:14] <yuvipanda>	 the answer, of course, is ‘it depends’ :)
[23:39:20] <yuvipanda>	 if it fits the usage patterns of toollabs
[23:39:23] <yuvipanda>	 it should be in toollabs
[23:39:36] <yuvipanda>	 what exactly does the VM run, CristianCantoro?
[23:40:07] <yuvipanda>	 is it a querying webservice?
[23:40:20] <yuvipanda>	 what is the webservice being run as? 
[23:40:36] <CristianCantoro>	 yuvipanda: this https://github.com/LinkedDataFragments/Server.js
[23:40:50] <CristianCantoro>	 (more background at: http://linkeddatafragments.org/)
[23:41:27] <CristianCantoro>	 is a Node.js application
[23:41:30] <CristianCantoro>	 *it is
[23:41:53] <yuvipanda>	 ah, toollabs does support nodejs apps :)
[23:42:11] <yuvipanda>	 CristianCantoro: so is your ‘backing data’ in the file system? 
[23:42:34] <CristianCantoro>	 yes, around 30 GB of data
[23:43:11] <CristianCantoro>	 basically they Wikidata RDF dumps converted to another format 
[23:43:19] <CristianCantoro>	 *they are
[23:43:47] <yuvipanda>	 ah, I see.
[23:43:59] <yuvipanda>	 so toollabs the only space you can keep 30G of data consistently is (sadly still) on NFS
[23:44:06] <yuvipanda>	 and I guess that’ll not be fast enough for you
[23:44:17] <yuvipanda>	 in which case you should request a labs project and I’ll happily approve it
[23:44:27] <yuvipanda>	 if you think NFS is fast enough you should just request a toollabs project
[23:44:40] <Krenair>	 toollabs tool*
[23:44:43] <yuvipanda>	 the advantage of toollabs is you’ve to do far less maintenance over time
[23:44:53] <yuvipanda>	 since it has dedicated sysadmins.
[23:44:59] <yuvipanda>	 while for your own VM you’ve to do things yourself
[23:45:15] <Krenair>	 the terminology is confusing enough for newbies already, let's not add more ambiguity :p
[23:45:21] <yuvipanda>	 we need to write so much documentation
[23:45:58] <CristianCantoro>	 I haven't developed the application (i.e. the server) so I really have no clue about this kind of requirements. What I have done is that I took this application, crunched the data from the Wikidata RDF dump and setup the whole thing. 
[23:46:44] <Krenair>	 It should be fine to experiment with a toollabs tool IMO
[23:46:46] <yuvipanda>	 I suggest askin for a labs project
[23:47:02] <yuvipanda>	 Krenair: disagree. NFS is going to kill it.
[23:47:09] <Krenair>	 ok
[23:47:11] <CristianCantoro>	 Krenair, yuvipanda... so you two are suggesting two different things, right?
[23:47:14] <CristianCantoro>	 ;D
[23:47:15] <yuvipanda>	 Krenair: and it’s going to kill NFS :P
[23:47:28] <Krenair>	 CristianCantoro, yuvipanda knows what he's talking about, I do not
[23:47:37] <yuvipanda>	 haha
[23:48:04] <CristianCantoro>	 yuvipanda: Krenair: I am in touch with the developer of the app/server so I could ask him what he think about using NFS
[23:48:10] <yuvipanda>	 CristianCantoro: sure!
[23:54:05] <CristianCantoro>	 ok, I'll do that... 
[23:54:13] <CristianCantoro>	 just a couple more questions:
[23:54:14] <CristianCantoro>	 1) I saw that sometimes the app crashes (not too often, I am running the server since 6 months and it happened two times), seemingly at random (I haven't dug in the logs yet). Furthermore, I saw that I had simply to restart the server. So I was thinking to setup a service supervisor (I was thinking about runit: https://en.wikipedia.org/wiki/Runit). With a machine/project I would be able to do that, with a tool on toollabs is it 
[23:54:51] <CristianCantoro>	 2?
[23:55:00] <CristianCantoro>	 *?
[23:55:49] <yuvipanda>	 CristianCantoro: if you’re doing that with toollabs, it’s automatically taken care of for you
[23:56:08] <yuvipanda>	 CristianCantoro: if you’re doing that on a labs project, I’d suggest using upstart since that comes with ubuntu by default
[23:56:16] <yuvipanda>	 should be a simple 5-10 line config for it and it’ll work
[23:58:04] <CristianCantoro>	 yuvipanda: good to know... I have been reading about the various init systems, I was picking the one that seemed the nicest from my almost-zero previous knowledge 
[23:59:03] <CristianCantoro>	 second question: if a request a tool or a project and I want to give access to other people they have to register an account and get labs access first, right?