[03:43:50] <YuviPanda|nap>	 andrewbogott: heh, yesterday’s puppet storm was also from a real issue.
[03:44:06] <andrewbogott>	 That's good!  Unless I caused the issue :)
[03:44:43] <YuviPanda|nap>	 andrewbogott: heh, not this time either :)
[03:45:40] <YuviPanda|nap>	 andrewbogott: wish there was an easier way to check puppet logs for failure messages than sshing into each machine tho
[05:01:05] <Coren>	 This time, it was a tweak by Faidon for Jessie that didn't sit well with debian vm images.  Quickly fixed.
[05:01:17] <paravoid>	 which one?
[05:03:13] <Coren>	 Something about grub; it tried to make a patch for serial to /etc/defaults/grub which doesn't exist in Labs.
[05:03:37] <paravoid>	 oh the one I fixed?
[05:03:48] <paravoid>	 I thought you were talking about another one
[05:03:51] * Coren nods.
[05:04:27] <Coren>	 I was refering to andrew wondering if the last puppet storm was his.
[05:05:48] <YuviPanda>	 yeah, our recent puppet storms have been caused by actual breakages.
[05:05:50] <YuviPanda>	 this is exciting
[05:09:57] <YuviPanda>	 andrewbogott: any news on the new virt hosts?
[05:41:00] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[05:41:04] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[05:42:20] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[05:42:30] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[05:42:52] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[05:44:38] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[05:44:50] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[05:45:48] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[05:46:16] <shinken-wm>	 PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[05:47:05] <shinken-wm>	 PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[05:47:07] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[05:48:49] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[05:50:57] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[05:51:20] <YuviPanda>	 andrewbogott: ^ apt-get issue again, but co-incides with apt-get issue on prod
[05:51:50] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[05:52:11] <shinken-wm>	 PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[05:53:17] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[05:54:12] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0]  
[05:54:19] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[05:55:28] <shinken-wm>	 PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[05:56:04] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]  
[05:56:32] <shinken-wm>	 PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[05:56:42] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[05:58:04] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[05:58:06] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[05:58:22] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[05:58:26] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[05:59:08] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0]  
[05:59:33] <shinken-wm>	 PROBLEM - Puppet failure on tools-trusty is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[05:59:40] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[06:02:44] <shinken-wm>	 PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[06:07:06] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:08:26] <shinken-wm>	 PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0]  
[06:09:48] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:12:06] <shinken-wm>	 RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:12:10] <shinken-wm>	 RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:13:20] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:16:32] <shinken-wm>	 RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:16:52] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:19:13] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:20:27] <shinken-wm>	 RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:21:41] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:22:19] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:22:51] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:23:21] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:23:27] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:24:07] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:24:33] <shinken-wm>	 RECOVERY - Puppet failure on tools-trusty is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:24:41] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:25:42] <YuviPanda>	 GerardM-: hey
[06:26:01] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:26:04] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:27:30] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:27:41] <shinken-wm>	 RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:28:27] <shinken-wm>	 RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:29:39] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:30:50] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:31:16] <shinken-wm>	 RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:33:50] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:34:34] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[06:38:44] <shinken-wm>	 PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[06:38:52] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[06:39:18] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:40:46] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[06:40:58] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:41:52] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]  
[06:43:02] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:43:10] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:43:29] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[06:44:27] <shinken-wm>	 PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[06:45:43] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[06:45:59] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0]  
[06:47:51] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[06:48:28] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0]  
[06:49:49] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0]  
[06:51:58] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0]  
[06:52:32] <shinken-wm>	 PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0]  
[06:54:15] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 87.50% of data above the critical threshold [0.0]  
[06:56:29] <shinken-wm>	 PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[06:57:42] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[06:57:55] <shinken-wm>	 PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]  
[06:58:19] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[06:59:08] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[06:59:26] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:00:42] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[07:03:08] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[07:08:29] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:08:41] <shinken-wm>	 RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:09:27] <shinken-wm>	 RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:10:40] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:13:28] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:14:02] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]  
[07:14:52] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:17:25] <GerardM->	 hey Yuvi
[07:17:39] <GerardM->	 I think oAuth in Widar is up the fritz
[07:17:56] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:18:19] <GerardM->	 YuviPanda: ^^
[07:19:11] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:19:29] <YuviPanda>	 GerardM-: don’t think I can do anything about OAuth :)
[07:19:41] <shinken-wm>	 PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[07:20:12] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0]  
[07:20:42] <GerardM->	 right so what to do ?
[07:21:28] <shinken-wm>	 RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:22:33] <shinken-wm>	 RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:22:43] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:23:01] <wikibugs>	 3Tool-Labs: oAuth on Labs does not work - https://phabricator.wikimedia.org/T78247#840650 (10GerardM) 3NEW
[07:23:23] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:23:29] <GerardM->	 YuviPanda: https://phabricator.wikimedia.org/T78247
[07:23:55] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:24:24] <YuviPanda>	 GerardM-: now someone who knows about OAuth and the tool author would need to figure out what is wrong and fix it, I suppose.
[07:25:20] <GerardM->	 This is EXACTLY why I hate Tools... Widar is an official tool and it is not supported. There are no recovery plans
[07:25:37] <YuviPanda>	 GerardM-: ‘official tool’?
[07:25:41] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:25:43] <GerardM->	 nothing to do with you.. but everything to do with frustration
[07:25:45] <GerardM->	 Widar is
[07:25:49] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:26:00] <YuviPanda>	 GerardM-: well, it’s not running on wikimedia.org, so it’s up to the tool author to fix things
[07:26:13] <YuviPanda>	 GerardM-: I checked if OAuth itself is working fine - quarry.wmflabs.org is able to use OAuth just fine
[07:26:50] <GerardM->	 YuviPanda: Labs has production tools that people rely on
[07:27:12] <YuviPanda>	 GerardM-: ops doesn’t consider anything not on wikimedia.org production :) 
[07:27:20] <GerardM->	 given the lack of support it fails miserably to people who are not the author of the tool
[07:27:28] <GerardM->	 They are full of shit
[07:27:37] <GerardM->	 production is what the customer says it is
[07:27:54] <shinken-wm>	 RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:27:57] <YuviPanda>	 GerardM-: good luck getting much help with that attitude.
[07:27:59] <GerardM->	 PS it is healthy to be full of shit
[07:28:12] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:28:48] <GerardM->	 FYI I want the chapters to take responsibility in this... they are the organisational customer in this
[07:31:49] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:34:17] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:36:56] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:37:03] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[07:37:29] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: T1316: Also report {icon umbrella} projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/179088 
[07:39:04] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0]  
[07:39:30] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[07:41:42] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]  
[07:43:08] <shinken-wm>	 PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[07:44:08] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[07:44:28] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[07:45:07] <YuviPanda>	 shut up shinken-wm
[07:45:10] <YuviPanda>	 heh, that was effective :)
[07:45:12] <YuviPanda>	 (or not)
[07:45:20] <YuviPanda>	 !log shinken shut up shinken-wm
[07:45:23] <labs-morebots>	 Logged the message, Master
[07:46:03] <JamesR>	 Hmm... I've just been kicked off shell for labs. When reconnecting I'm getting an error. Any ideas?
[07:47:14] <YuviPanda>	 JamesR: what error?
[07:47:58] <JamesR>	 jamesmac:~ james$ ssh tools-login.wmflabs.org
[07:47:58] <JamesR>	 Connection closed by 208.80.155.130
[07:48:03] <JamesR>	 @YuviPanda
[07:48:10] <YuviPanda>	 hmm
[07:48:36] <JamesR>	 I've been having some connection issues here, maybe maxed logins?
[07:48:49] <YuviPanda>	 JamesR: possibly. am investigating, hold on
[07:49:01] <JamesR>	 Thank you :)
[07:52:41] <YuviPanda>	 JamesR: in the meantime, you can ssh via tools-dev.wmflabs.org
[07:52:57] <JamesR>	 YuviPanda: will do, thanks
[07:58:42] <YuviPanda>	 !log tools rebooted tools-login, wasn’t responsive.
[07:58:44] <labs-morebots>	 Logged the message, Master
[08:02:27] <legoktm>	 or trusty.tools.wmflabs.org :)
[08:03:47] <YuviPanda>	 JamesR: -login should be back up
[08:04:01] <JamesR>	 YuviPanda: will try again, thank you
[08:04:42] <YuviPanda>	 JamesR: yw
[08:10:14] <GerardM->	 YuviPanda: could that have affected Widar ?
[08:11:35] <GerardM->	 it seems not to have made a difference
[08:58:07] <Kelson42>	 YuviPanda: Hi
[08:58:13] <YuviPanda>	 Kelson42: heya
[08:58:15] <YuviPanda>	 ‘sup
[08:58:26] <Kelson42>	 YuviPanda: VM is in "SHUTOFF" state https://wikitech.wikimedia.org/wiki/Special:NovaInstance
[08:58:33] <YuviPanda>	 andrewbogott: ^
[08:58:41] <YuviPanda>	 hmm
[08:58:44] <Kelson42>	 YuviPanda: and I think this is the reason why I can not connect to it anymore
[08:58:57] <YuviPanda>	 Kelson42: right.
[08:59:00] <YuviPanda>	 Kelson42: let me take a look
[08:59:07] <Kelson42>	 YuviPanda: "reboot" link fails too
[08:59:12] <andrewbogott>	 Kelson42: that link doesn't help a lot, can you tell us what VM and project specifically?
[08:59:44] <Kelson42>	 andrewbogott: i-0000075f.eqiad.wmflabs -  mwoffliner1 - project offliner
[08:59:52] <Kelson42>	 andrewbogott: project "mwoffliner" sorry
[09:00:31] <YuviPanda>	 andrewbogott: I’m doing a nova start now.
[09:00:51] <andrewbogott>	 YuviPanda: ok.  Watch out for instances that were mothballed during the eqiad migration -- they generally need more work than just a startup.
[09:01:08] <YuviPanda>	 andrewbogott: yeah, no, this was started a few weeks ago, I think
[09:01:35] <YuviPanda>	 andrewbogott: hmm, ‘Invalid nova credentials’, even after sourcing novaenv.
[09:01:50] <andrewbogott>	 you need to specify the tenant, maybe...
[09:02:19] <YuviPanda>	 andrewbogott: nova --os-tenant-name offliner list
[09:02:27] <andrewbogott>	 mwoffliner
[09:02:35] <YuviPanda>	 oh
[09:02:36] <YuviPanda>	 right
[09:03:05] <YuviPanda>	 Kelson42: see if it is still in SHUTOFF state?
[09:03:48] <Kelson42>	 YuviPanda: no, it's active now
[09:03:58] <Kelson42>	 YuviPanda: but I still have a pb to connect
[09:04:12] <YuviPanda>	 Kelson42: give it a minute to finish booting up
[09:08:31] <Kelson42>	 YuviPanda: works now :)
[09:08:36] <Kelson42>	 YuviPanda: andrewbogott thx
[09:09:37] <YuviPanda>	 Kelson42: yw
[10:11:59] <wikibugs>	 3Tool-Labs: oAuth on Labs does not work - https://phabricator.wikimedia.org/T78247#841119 (10Aklapper) Please provide steps to reproduce.  See https://www.mediawiki.org/wiki/How_to_report_a_bug
[10:12:39] <wikibugs>	 3Tool-Labs-tools-Other: Move catbot from toolserver to toollabs - https://phabricator.wikimedia.org/T63825#841124 (10Nosy79)
[10:12:40] <wikibugs>	 3Tool-Labs-tools-Other: [tracking] toolserver.org tools that have not been migrated - https://phabricator.wikimedia.org/T60865#841123 (10Nosy79)
[10:12:41] <wikibugs>	 3Tool-Labs-tools-Other: Migrate https://toolserver.org/~daniel/WikiSense/* to Tool Labs and provide redirect - https://phabricator.wikimedia.org/T60869#841121 (10Nosy79) 5Open>3Resolved
[10:12:55] <wikibugs>	 3Tool-Labs: Toolserver migration to Tools (tracking) - https://phabricator.wikimedia.org/T60788#841128 (10Nosy79)
[10:12:57] <wikibugs>	 3Tool-Labs, Labs-Team: Toolserver final migration steps - https://phabricator.wikimedia.org/T75996#841127 (10Nosy79) 5Open>3Resolved
[10:13:25] <wikibugs>	 3Tool-Labs: oAuth on Labs does not work - https://phabricator.wikimedia.org/T78247#841134 (10Aklapper) Wondering if this is related to / a dup of T78223 and https://gerrit.wikimedia.org/r/#/c/153983/
[10:31:00] <wikibugs>	 3Tool-Labs-tools-Other: Migrate https://toolserver.org/~daniel/WikiSense/* to Tool Labs and provide redirect - https://phabricator.wikimedia.org/T60869#841182 (10Nemo_bis) I'm confused: what was redirected to what?
[10:34:18] <wikibugs>	 3Tool-Labs-tools-Other: Migrate https://toolserver.org/~daniel/WikiSense/* to Tool Labs and provide redirect - https://phabricator.wikimedia.org/T60869#841197 (10Nosy79) Redirect 301 /~daniel/WikiSense/CatScan.php http://tools.wmflabs.org/catscan2/catscan2.php  Redirect 301 /~daniel/WikiSense/CategoryIntersect.p...
[12:15:45] <wikibugs>	 3Tool-Labs: oAuth on Labs does not work - https://phabricator.wikimedia.org/T78247#841384 (10Sjoerddebruin) Probaly related to T78209.
[12:40:57] <Kelson42>	 YuviPanda: it's me again, I have created an instance with 150GB of storage space... and I remark now I don't see that space free.
[12:41:35] <YuviPanda>	 Kelson42: indeed, go to ‘configure’ page in special:novainstance, and select the role with ‘srv’ in it, save, and then force a puppet run (‘sudo puppet agent -tv'
[12:41:36] <YuviPanda>	 )
[12:41:40] <YuviPanda>	 and viola, you shall have a /srv that big
[12:43:39] <Kelson42>	 YuviPanda: ok, let me try this
[12:55:06] <Kelson42>	 YuviPanda: it works, thx
[12:55:13] <YuviPanda>	 yw
[13:19:40] <Kelson42>	 YuviPanda: first zims were generated, I'm currently setting up a rsync server to be able to migrate them migrated to the download master
[13:19:56] <YuviPanda>	 Kelson42: \o/ cool
[13:20:05] <Kelson42>	 YuviPanda: have achieved to configure the security groups to allow access on rsync tcp port
[13:20:11] <Kelson42>	 YuviPanda: at least I guess it works
[13:20:11] <YuviPanda>	 nice!
[13:20:20] <Kelson42>	 YuviPanda: but I believe I still don't have a public IP
[13:20:44] <Kelson42>	 YuviPanda: I have nothing in "manage adresses" and if I ask a new public ip address, I get an error
[13:20:46] <YuviPanda>	 Kelson42: indeed, we don’t have public IPs assigned by default.
[13:20:59] <YuviPanda>	 Kelson42: do you need rsync, or a web server be enough for you?
[13:21:08] <YuviPanda>	 Kelson42: if latter, you can set up a web proxy from ‘manage web proxies'
[13:21:15] <Kelson42>	 YuviPanda: I need rsync
[13:21:39] <Kelson42>	 YuviPanda: I don't need a web server
[13:21:42] <YuviPanda>	 Kelson42: alright, give me 5 minutes and I’ll allocate one for you
[13:21:46] <Kelson42>	 YuviPanda: merci
[13:56:35] <YuviPanda>	 !log mwoffliner granted quota of 1 public ip
[13:56:37] <labs-morebots>	 Logged the message, Master
[13:56:38] <YuviPanda>	 Kelson42: ^
[13:56:51] <Kelson42>	 YuviPanda: great, let me check this
[13:56:52] <YuviPanda>	 Kelson42: now you should be able to use ‘allocate ip’ in the addresses special page, and then associate it with your instance
[13:58:06] <Kelson42>	 YuviPanda: new address configured
[13:58:21] <YuviPanda>	 \o/ cool
[14:03:05] <Kelson42>	 YuviPanda: ok, I still have problem with rsyncd.. but it looks like I achieve to reach the server from the kiwix download master server
[14:03:07] <Kelson42>	 YuviPanda: thx
[14:04:00] <YuviPanda>	 Kelson42: cool! :0
[14:13:57] <Kelson42>	 YuviPanda: hm... my instance wen again spontanously in SHUTOFF mode
[14:14:10] <YuviPanda>	 hmm, bah.
[14:21:51] <YuviPanda>	 Kelson42: I’ve started it again.
[14:22:07] <YuviPanda>	 Kelson42: labs is a little short on CPU these days. we’ve 3 more machines coming in, so should get better this week or next.
[14:22:16] <YuviPanda>	 I don’t know for sure if that is the reason, but it could be
[14:34:09] <Coren>	 YuviPanda: Actually, strictly speaking, it's ram we're overcommited on.
[14:34:16] <YuviPanda>	 oh
[14:34:20] <YuviPanda>	 right.
[14:58:49] <Kelson42>	 YuviPanda: ok, thx, this is maybe related, we use a lot of cpuy
[15:32:18] <wikibugs>	 3Labs-Team, Tool-Labs: Labs proxy + tools proxy should use ssl_ciphersuite - https://phabricator.wikimedia.org/T77987#841861 (10scfc) 5Open>3Resolved a:3scfc AFAICS, this should have been resolved by Gerrit change #178493.
[15:32:46] <wikibugs>	 3Labs-Team, Tool-Labs: Labs proxy + tools proxy should use ssl_ciphersuite - https://phabricator.wikimedia.org/T77987#841864 (10scfc) a:5scfc>3Dzahn
[15:54:13] * werdna waves
[15:54:57] <werdna>	 I want to set up a MediaWiki instance to demonstrate an extension. Should I have a new project or should I use tool labs etc?
[15:55:15] <werdna>	 It's for the new Living Style Guide (from the UX team)
[16:36:06] <YuviPanda>	 werdna: the design team actually already has a project.
[16:36:11] <YuviPanda>	 werdna: poke prtksxna about it :)
[16:36:23] <YuviPanda>	 werdna: he’s also maintaining something on toollabs, I think/thought
[16:39:54] <prtksxna>	 werdna: Added you to https://wikitech.wikimedia.org/wiki/Nova_Resource:Design
[16:40:17] <prtksxna>	 werdna: spagewmf is maintaining the toollabs stuff.
[16:45:26] <wikibugs>	 3Tool-Labs: oAuth on Labs does not work - https://phabricator.wikimedia.org/T78247#842009 (10greg) @GerardM: T78223 is now fixed, which is probably (hopefully!) the same issue as this, can you confirm?
[17:43:09] <wikibugs>	 3Tool-Labs: oAuth on Labs does not work - https://phabricator.wikimedia.org/T78247#842154 (10Stryn) At least authorizing doesn't work for me. If I go to http://tools.wmflabs.org/widar/ and click "Authorize WiDaR now" it redirects to http://tools.wmflabs.org/widar/index.php?action=authorize which gives a message:...
[17:49:46] <werdna>	 YuviPanda: cheers
[17:49:51] <werdna>	 also prtk, when he gets back :p
[17:49:56] <YuviPanda>	 :)
[17:50:00] <YuviPanda>	 he sleeps early
[17:50:30] <werdna>	 yeah, he gets up early too
[17:50:37] <werdna>	 the man is awake at 0430 or something insane
[17:51:10] <YuviPanda>	 yeah
[17:51:29] <YuviPanda>	 when I was staying with him I never managed to go to sleep before he woke up
[17:55:27] <werdna>	 hahaha
[17:55:35] <werdna>	 impressive, a little scary
[17:55:44] <werdna>	 hey YuviPanda, can I get an address for the design project?
[17:55:55] <werdna>	 or, is there something else I should do to have a public webserver?
[17:55:59] <YuviPanda>	 werdna: no need to get IP addresses for web services anymore :)
[17:56:01] <werdna>	 I just want a plain MW install that I can fuck with :D
[17:56:02] <werdna>	 oh sweet
[17:56:04] <werdna>	 what do I do
[17:56:04] <YuviPanda>	 werdna: there hsould be a ‘manage web proxies'
[17:56:13] <werdna>	 got it
[17:56:16] <werdna>	 <3
[17:56:20] <YuviPanda>	 werdna: also use a small instance? we’re kind of low on RAM and CPU until next week
[17:56:50] <werdna>	 I used medium because the labs vagrant docs say to use medium
[17:56:52] <werdna>	 but I can use small
[17:57:12] <werdna>	 can I reconfigure that myself or do I need to destroy/re-add?
[17:58:17] <YuviPanda>	 werdna: this is just going to be doing the styles stuff, right? destroy/re-add :)
[17:58:45] <werdna>	 YuviPanda: it's an MW instance for testing an extension I wrote for doing styles stuff
[17:59:20] <YuviPanda>	 werdna: yeah, as long as you’re not going to be doing heavy stuff a small should be ok
[17:59:30] <werdna>	 buuut I run MW with 2 GB of RAM on another site anyway
[17:59:35] <YuviPanda>	 yeah
[17:59:41] <YuviPanda>	 dunno why labs-vagrant reccomends medium
[18:01:41] <werdna>	 Permission denied (publickey).
[18:01:42] <werdna>	 hmph
[18:01:48] * werdna checks the help pages
[18:02:11] <YuviPanda>	 werdna: you should wait about 10m after instance creation for puppet to run, system to boot up, etc
[18:02:19] <werdna>	 okay, I'll let it do its thing
[18:02:24] <werdna>	 make a cup of tea or whatever :p
[18:02:30] <YuviPanda>	 +1
[18:03:13] <wikibugs>	 3Labs-Team, Tool-Labs: Labs proxy + tools proxy should use ssl_ciphersuite - https://phabricator.wikimedia.org/T77987#842256 (10Dzahn) https://gerrit.wikimedia.org/r/#/c/178493/  thanks scfc, yea, it's resolved
[18:08:00] * werdna checked the console output, still puppeting away
[18:09:58] <mutante>	 <wikibugs> Tool-Labs: add package kteatime on exec nodes
[18:11:16] <YuviPanda>	 mutante: link?
[18:11:42] <YuviPanda>	 oh
[18:11:42] <YuviPanda>	 damn
[18:11:44] <YuviPanda>	 joke
[18:11:45] <YuviPanda>	 nevermind
[18:11:59] * YuviPanda declines mutante’s request citing ‘GUI application, does not make sense on toollabs’ :P
[18:13:01] <mutante>	 YuviPanda: heheh, i was hoping for a cmdline one but couldnt find :)
[18:18:09] <werdna>	 :p
[18:18:20] <werdna>	 lol
[18:18:36] <werdna>	 is this the right way to make it refresh puppet? # puppet agent --no-daemonize --onetime
[18:21:18] <Coren>	 werdna: It is, although most people are lazier and do the equivalent 'puppet agent -t'  :-)
[18:21:29] <Coren>	 (Or -tv when you want to see what it's doing)
[18:28:04] <werdna>	 Coren: well, the latter worked but the former didn't
[18:28:49] <YuviPanda>	 werdna: the former might be from a time ago, I haven’t seen it before
[18:28:51] <YuviPanda>	 puppet agent -tv
[18:28:53] <YuviPanda>	 is what I use
[18:45:07] <halfak>	 hey folks.  It looks like I can't initialize a new instance on Wikitech at the moment.  Is that intentional?
[18:51:56] <YuviPanda>	 halfak: probably not, but perhaps for the best.
[18:52:04] <YuviPanda>	 overloaded on RAM, instances are randomly going to shutoff state
[18:52:41] <halfak>	 That's fine.  Anywhere else that I can see a list of instance types that are generally available. 
[18:52:45] <halfak>	 Search is failing me :( 
[18:54:34] <YuviPanda>	 halfak: interesting, wikitech seems down?
[18:54:40] <YuviPanda>	 oh
[18:54:40] <YuviPanda>	 no
[18:54:58] <YuviPanda>	 halfak: https://www.dropbox.com/s/il4uy87b4cm6jrr/Screenshot%202014-12-12%2000.24.50.png?dl=0
[18:55:04] <halfak>	 Thanks
[18:56:09] <halfak>	 What's the ETA on a solution for "overloaded RAM"?
[18:57:09] <YuviPanda>	 halfak: next week, hopefully.
[18:57:26] <YuviPanda>	 halfak: all the hardware is here and racked and powered, just some kinks to work out
[18:57:31] <YuviPanda>	 Coren: ^
[18:57:49] <halfak>	 Gotcha.  Thanks 
[18:58:18] <Coren>	 Have no more news than that, I expect andrewbogott_afk is sleeping.  Not sure what his timzeone is atm but I expect he may have news upon his return.
[18:59:49] <grrrit-wm>	 (03CR) 10Jforrester: [C: 031] T1316: Also report {icon umbrella} projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/179088 (owner: 10Merlijn van Deen)
[19:03:33] <FaFlo>	 hi, one question: if I would request e.g. a m1.xlarge instance and it would come to pass that 160GB doesn't suffice (for now it surely will), is that disk space extendable? 
[19:07:16] <YuviPanda>	 Coren: ^
[19:07:21] <YuviPanda>	 FaFlo: I… doubt it.
[19:07:59] <Coren>	 FaFlo: No; but you shouldn't be storing large amounts of data on local storage anyways; you should be using the shared network storage instead
[19:08:07] <FaFlo>	 ok
[19:08:15] <FaFlo>	 thanks
[19:09:55] <psi_>	 so you have access to a larger NAS?
[19:10:27] <FaFlo>	 and just to be clear (bc I'm not sure about the infrastructure), does that mean I could use shared net. storage on top, instead or how does that work?
[19:11:14] <YuviPanda>	 FaFlo: so shared storage is just NFS
[19:11:19] <YuviPanda>	 FaFlo: and is quite a few terabytes.
[19:11:26] <YuviPanda>	 FaFlo: and you can use that however you want.
[19:11:31] <YuviPanda>	 just remember that it is NFS and hence would be slower
[19:11:46] <FaFlo>	 YuviPanda:  alright
[19:15:19] <wikibugs>	 3Tool-Labs: oAuth on Labs does not work - https://phabricator.wikimedia.org/T78247#842398 (10greg) @mmodell / @chasemp / @legoktm thoughts here?
[19:15:33] <legoktm>	 :o
[19:31:02] <FaFlo>	 one other question: when on a Labs VM, what would be the best way to access the current XML (complete)history dump of a specific article? 
[19:31:30] <YuviPanda>	 FaFlo: they are all available automatically via NFS
[19:31:34] <YuviPanda>	 FaFlo: in /public/dumps
[19:32:01] <FaFlo>	 I see
[19:32:14] <mutante>	 oh heh, i was about to say stuff like curl https://www.mediawiki.org/wiki/Special:Export/MediaWiki :p
[19:32:25] <FaFlo>	 :) 
[19:32:39] <FaFlo>	 and the dumps are generated/updated how often? 
[19:33:41] <YuviPanda>	 FaFlo: depends on each wiki, usually it is available as soon as the dump is available on dumps.wikimedia.org
[19:36:43] <FaFlo>	 hm ok. I was looking for something that can give me the most up-to-date revision history in xml
[19:41:35] <YuviPanda>	 FaFlo: what exactly are you planning on doing?
[19:42:24] <FaFlo>	 well, we are working on an API that extracts certain information about a requested article (and it's complete revision history)
[19:42:56] <FaFlo>	 and somebody should also be able to query including the most up-to-date revision
[19:44:02] <mutante>	 "If $wgExportAllowHistory is set to false in LocalSettings.php, only the current version can be exported, not the full history."
[19:44:50] <mutante>	 FaFlo: https://www.mediawiki.org/wiki/Manual:Parameters_to_Special:Export#Retrieving_earliest_5_revisions
[19:45:37] <mutante>	 FaFlo: https://meta.wikimedia.org/wiki/Help:Export#Exporting_the_full_history
[19:45:45] <FaFlo>	 mutante: thanks, I'm aware of Special:Export 
[19:45:57] <FaFlo>	 thought there might be another, maybe better way
[19:47:07] <mutante>	 i see, yea
[19:47:18] <mutante>	 maybe directly out of the (sanitized) databases
[19:49:05] <YuviPanda>	 mutante: our databases don’t offer content.
[19:49:28] <YuviPanda>	 FaFlo: if you’re looking for up to date as in ‘as it is right now’, I am not sure. halfak might be able to think up a way, maybe.
[19:49:35] <YuviPanda>	 FaFlo: but yeah, dumps are usually a week or so old
[19:51:14] <FaFlo>	 ok, thanks, now I'm at least a bit wiser than before :)
[19:55:17] <YuviPanda>	 FaFlo: :)
[20:52:48] <Mpaa-irc>	 Hi. I am trying to get up_property from enwiki_p.user_properties but I do not get what I expect to find (I am trying to get up_property WHERE (up_property LIKE 'gadget-%')
[20:53:23] <Mpaa-irc>	 Is there a permission issue in trying o get that type of info?
[20:54:58] <Mpaa-irc>	 I expected to find something starting with "gadget-***" as in here: https://en.wikipedia.org/wiki/Wikipedia:Database_reports/User_preferences#Gadgets
[20:58:12] <YuviPanda>	 Mpaa-irc: gagdet preferences and a lot of preferences aren’t made available
[20:59:46] <YuviPanda>	 Mpaa-irc: https://phabricator.wikimedia.org/T60196
[21:15:05] <Mpaa-irc>	 YuviPanda, thanks. Long story short, this is not likely to happen shortly, am I correct?
[21:15:19] <YuviPanda>	 Mpaa-irc: yeah.
[22:47:33] <cscott>	 !log deployment-prep updated OCG to version bfc3812ef346c9f767135b339cedd123a1bcac98
[22:47:38] <labs-morebots>	 Logged the message, Master
[23:36:24] * Fiona looks at T60196.
[23:37:17] <Fiona>	 Yeah, wtf.
[23:44:32] <wikibugs>	 3Tool-Labs: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#843096 (10MZMcBride) >>! In T60196#630652, @LuisV_WMF wrote: > If the users are OK with the month-filtered view, and exclusion of small wikis, that seems like a workable compromise to me.  Okay,...
[23:46:47] <wikibugs>	 3Tool-Labs: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#843100 (10MZMcBride) a:5LuisV_WMF>3None