[00:08:36] <Cyberpower678>	 Coren, big brother is having a nervous breakdown.
[00:08:53] <Cyberpower678>	 andrewbogott, YuviPanda ^
[00:09:03] <Reedy>	 That's nice
[00:09:15] <Cyberpower678>	 It's unable to restart lighttpd
[00:09:45] <Reedy>	 Most issues are known
[00:10:02] <Cyberpower678>	 It's running in circles on xtools and spamming my inbox
[00:11:59] <Cyberpower678>	 Reedy, are we still in the reboot phase?
[00:12:05] <Reedy>	 yeah,
[00:12:09] <Reedy>	 or at least, fixing up the fallout
[00:12:18] <Cyberpower678>	 Wow.  It's been several hours
[01:39:23] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:39:30] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:39:40] <shinken-wm>	 PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:39:48] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:39:50] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:40:14] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:40:14] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:40:16] <shinken-wm>	 PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:40:26] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-generic-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:40:28] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[01:40:30] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:40:42] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:40:50] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:40:55] <shinken-wm>	 PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0]  
[01:40:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:41:05] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:41:07] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0]  
[01:41:15] <shinken-wm>	 PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:41:38] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]  
[01:41:48] <shinken-wm>	 PROBLEM - Puppet failure on tools-static is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:41:50] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:42:01] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[01:42:02] <shinken-wm>	 PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[01:57:15] <Nemo_bis>	 https://tools.wmflabs.org/magnustools/ seems to need a restart
[01:57:20] <Nemo_bis>	 Well, perhaps something more
[02:01:29] <YuviPanda>	 Nemo_bis: everything is down
[02:02:39] <GEOFBOT>	 :o
[02:08:40] <andrewbogott>	 https://wikitech.wikimedia.org/wiki/Virt1009_casualties
[02:08:52] <andrewbogott>	 (I have hope for them yet)
[02:16:45] <Mjbmr>	 do you need any help?
[02:16:47] <YuviPanda>	 Nemo_bis: coming back up. magnustools is back up, so is autolist
[02:16:56] <YuviPanda>	 Mjbmr: pointing out which tools are down would help, yeah :)
[02:17:14] <Mjbmr>	 that's a hard work
[02:17:42] <Mjbmr>	 quentinv57-tools
[02:19:17] <Mjbmr>	 my tool is down
[02:19:58] <YuviPanda>	 Mjbmr: quentin’s should be back up now
[02:20:47] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:20:54] <shinken-wm>	 RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:21:00] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:21:00] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:21:06] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:21:14] <shinken-wm>	 RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:21:36] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:22:42] <shinken-wm>	 RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:40:11] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:44:44] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:45:16] <shinken-wm>	 RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:47:02] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:47:03] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:47:09] <wikibugs>	 3Tool-Labs, WMF-Legal: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#997462 (10yuvipanda) 3NEW
[02:49:39] <shinken-wm>	 RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:51:47] <shinken-wm>	 RECOVERY - Puppet failure on tools-static is OK: OK: Less than 1.00% above the threshold [0.0]  
[02:56:51] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:00:39] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:04:58] <T13|detached>	 What is state Rr in qstat?
[03:05:14] <T13|detached>	 YuviPanda: ^
[03:05:31] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:06:42] <YuviPanda>	 T13|detached: running continuous maybe? Is it running?
[03:07:58] <T13|detached>	 Not sure.  Don't think it was restarted after server restarts today
[03:08:14] <T13|detached>	 This is for fengtools/reflinks
[03:09:00] <T13|detached>	 There are two fengtools, and I'm not sure how to restart them.
[03:09:47] <T13|detached>	 Webservice restart says use webservice2 but that just lists some things and doesn't restart.
[03:09:51] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:10:24] <T13|detached>	 Do you have time to take a peek and educate me?
[03:10:47] <T13|detached>	 I can add you to look if needed.
[03:11:20] <CP678>	 T13|detached, he has root.  He can peak in without access.
[03:11:55] <T13|detached>	 If he's busy, think you can restart them Cyberpower678 ?
[03:12:11] <Cyberpower678>	 Restart what?
[03:12:19] <T13|detached>	 Fengtools
[03:12:27] <T13|detached>	 Reflinks
[03:12:36] <Cyberpower678>	 I don't have access.
[03:12:45] <T13|detached>	 I can add you
[03:12:46] <Cyberpower678>	 I'm no root. ;P
[03:13:15] <T13|detached>	 You're a maintainer
[03:13:32] <Cyberpower678>	 I'm busy myself.  I just spent the last few hours finding loose ends I left and trying to tie it into the new OAuth module I plan to deploy.
[03:13:45] <Cyberpower678>	 Now I'm doing EE 350 HW.
[03:13:48] <T13|detached>	 Kk
[03:14:13] <T13|detached>	 Will have to wait for zhao then
[03:14:24] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:14:46] <Cyberpower678>	 I'm hungry for cookies though.
[03:15:24] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-generic-01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:15:34] <Cyberpower678>	 shinken-wm, shut up
[03:16:45] <YuviPanda>	 Cyberpower678: T13|detached hi
[03:16:46] <YuviPanda>	 I’m here
[03:17:16] <Cyberpower678>	 YuviPanda, an you bake me some cookies?
[03:18:00] <Cyberpower678>	 T13|detached, ^
[03:18:17] <T13|detached>	 Can you hack the power grid and reroute power for me?  No seriously though, can you teach me how to restart fengtools /reflinks?
[03:18:39] <YuviPanda>	 T13|detached: https://tools.wmflabs.org/fengtools/ back up
[03:18:45] <YuviPanda>	 T13|detached: so if webservice2 stop doesn’t work
[03:18:46] <YuviPanda>	 do qstat
[03:18:47] <YuviPanda>	 get the jobid
[03:18:52] <YuviPanda>	 do ‘qdel <jobid>'
[03:18:58] <YuviPanda>	 and then just do webservice2 —release trusty start
[03:19:29] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:19:37] <Cyberpower678>	 T13|detached, I rerouted the power grid for you, now how about those cookies?
[03:19:42] <T13|detached>	  Stop worked
[03:20:13] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:20:23] <T13|detached>	 Invalid choice trusty yuv
[03:20:41] <T13|detached>	 YuviPanda: ^
[03:21:18] <YuviPanda>	 T13|detached: ‘webservice2 —release trusty start’ worked for me
[03:21:21] <YuviPanda>	 and fengtools is up
[03:21:38] <T13|detached>	 Was that 1 or 2 -?
[03:21:53] <YuviPanda>	 webservice2
[03:21:54] <T13|detached>	 Looks like one on my client, but I'm not sure
[03:22:02] <YuviPanda>	 https://tools.wmflabs.org/fengtools/reflinks/ is back up
[03:22:02] <YuviPanda>	 oh
[03:22:05] <YuviPanda>	 two --
[03:22:18] <T13|detached>	 -release or --release
[03:22:28] <T13|detached>	 Ahh that's probably why.  Thanks
[03:22:55] <T13|detached>	 Gets notepad and jots stuff down
[03:23:26] <YuviPanda>	 :)
[03:50:30] <comets>	 most tools still dead :/
[03:51:14] <YuviPanda>	 comets: which ones?
[03:51:18] <Cyberpower678>	 I might have to do some damage control.
[03:51:30] <CoE>	 geohack is down, or is that by design?
[03:51:30] <YuviPanda>	 Cyberpower678: where?
[03:51:42] <Cyberpower678>	 xTools
[03:51:47] <comets>	 GUC too, a few commons tools
[03:51:52] <Cyberpower678>	 NVM
[03:52:05] <Cyberpower678>	 I just remembered that the error messages are my doing
[03:52:17] <Cyberpower678>	 I'm rewriting a few things on xTools
[03:52:36] <Cyberpower678>	 I'll just blame it on the labs restart. :D
[03:52:41] <comets>	 Krinkle, GUC is down..
[03:52:55] <comets>	 we can blame them for everything :DD
[03:53:07] <Krinkle>	 all tools are down afaik. some came back up, I don't know
[04:11:42] <YuviPanda>	 Krinkle: hi. Most tools are back and the infra itself seems ok
[04:12:12] <YuviPanda>	 Cyberpower678: Krinkle tell me which are down? I'll restart 
[04:12:29] <Cyberpower678>	 supercount xtools-articlinfo
[04:12:35] <Cyberpower678>	 No trusty
[04:33:41] <Krinkle>	 YuviPanda: Can you start webservice for 'commons-delinquent' ?
[04:33:47] <Krinkle>	 I'm not a member of that tool service group
[04:35:35] <YuviPanda>	 Krinkle: moment 
[04:35:59] <Krinkle>	 I tried starting 'static', but it seems to be 504 erorring. Not sure if it's an issue with the tool or not
[04:36:07] <Krinkle>	 https://tools.wmflabs.org/static/  https://tools.wmflabs.org/delinker/  https://tools.wmflabs.org/commons-delinquent/
[04:37:16] <Krinkle>	 YuviPanda: Also, message from multichill: Can you start https://tools.wmflabs.org/heritage/ for us?
[04:39:42] <YuviPanda>	 Krinkle: doing ‘em all
[04:53:06] <CoE>	 https://tools.wmflabs.org/geohack/
[04:53:20] <YuviPanda>	 is that down?
[04:53:20] <CoE>	 geohack is linked to by millions of pages!
[04:53:25] <CoE>	 yes
[04:54:05] <CoE>	 timeouted earlier for me, now it just gives a blank page
[04:55:54] <YuviPanda>	 CoE: ok, bought that back
[04:56:35] <CoE>	 thanks! :)
[04:57:09] <Earwig>	 hi, so I have a job (earwigbot in tools.earwigbot) "running" on execnode 14 stuck in the deleting state that I can't kill (qdel is not working, and I can't ssh into the node because of 'no route to host')
[04:57:10] <CoE>	 can you also bring back https://tools.wmflabs.org/wiwosm/osm-on-ol/embed-labs.html? it's in a frame on the geohack page
[04:58:14] <YuviPanda>	 Earwig: can you give me the id from qstat? I can force kill it
[04:58:22] <Earwig>	 4204835
[04:58:37] <YuviPanda>	 Earwig: deleted that manually
[04:58:40] <YuviPanda>	 CoE: looking
[04:58:43] <Earwig>	 okay, thanks
[05:07:12] <Krinkle>	 tool static (static-browser) is now back up. delinker (commons-delinquent), an heritage still pending 
[05:08:19] <YuviPanda>	 andrewbogott: https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000005f5.eqiad.wmflabs
[05:10:18] <YuviPanda>	 Krinkle: I restarted them, but something else seems to be messed up now
[05:10:21] * YuviPanda looks
[05:13:29] <CoE>	 I gotta run, thanks for fixing things!
[05:20:56] <shinken-wm>	 RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0]  
[05:22:13] <Krenair>	 YuviPanda, hey
[05:22:29] <YuviPanda>	 Krenair: heya
[05:23:49] <Krenair>	 YuviPanda, is it possible to make me an admin of deployment-prep?
[05:23:55] <Krenair>	 right now I'm just a member
[05:25:59] <YuviPanda>	 Krenair: I should leave that to greg-g’s discretion, I think
[05:47:41] <Krenair>	 YuviPanda, can you please bring up deployment-parsoidcache02 for us?
[05:47:49] <Krenair>	 It's down and causing VE to be broken
[05:52:35] <YuviPanda>	 Krenair: alex is looking at it
[06:02:54] <KevinPayravi>	 Hey all - I know there were those patches and server restarts today, though my tool "traffic-grapher" (http://tools.wmflabs.org/traffic-grapher/) has been down for at least 6 hours. Is this normal during this time? Just wanted to make sure. Thanks!
[06:05:55] <YuviPanda>	 KevinPayravi: heya!
[06:05:58] <YuviPanda>	 KevinPayravi: let me restart it
[06:08:33] <KevinPayravi>	 YuviPanda: Looks like it's working now. Thank you!
[06:09:31] <YuviPanda>	 KevinPayravi: :)
[06:43:59] <shinken-wm>	 PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[06:45:31] <shinken-wm>	 RECOVERY - Host tools-exec-catscan is UP: PING OK - Packet loss = 0%, RTA = 0.85 ms  
[06:45:57] <shinken-wm>	 RECOVERY - Host tools-exec-14 is UP: PING OK - Packet loss = 0%, RTA = 0.87 ms  
[06:51:53] <shinken-wm>	 RECOVERY - Host tools-exec-15 is UP: PING OK - Packet loss = 0%, RTA = 1.14 ms  
[07:09:03] <shinken-wm>	 RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0]  
[08:35:46] <wikibugs>	 3Tool-Labs-tools-Erwin's-tools: Add Gini coefficient to xcontribs - https://phabricator.wikimedia.org/T87740#997910 (10Nemo_bis) 3NEW a:3Nemo_bis
[08:54:52] <NeoMahler>	 I don't know how to run a bot form the labs
[08:55:07] <NeoMahler>	 and I don't understand (or I can't find) the lbas help
[09:20:18] <shinken-wm>	 PROBLEM - Puppet staleness on tools-exec-07 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [43200.0]  
[10:14:07] <marcmiquel>	 hi guys. i am using the tool labs media wiki ddbb for processing articles and i have a problem with a query. anyone could help me?
[11:28:27] <petan>	 andrewbogott_afk, can I add you to all stars team of wm-bot?
[11:28:29] <petan>	 https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation/wm-bot
[11:28:45] <petan>	 I suppose you know how to stop or start the bot :)
[11:28:49] <petan>	 which is the only requirement :D
[11:36:40] <Linedwell>	 hi
[12:14:23] <devunt>	 Can I change wikitech wiki username without changing ldap/shell name?
[12:44:26] <GerardM->	 Warning: fopen(http://wdq.wmflabs.org/api?q=Claim%5B31%3A5%5D+and+noclaim%5B166%3A7626301%5D): failed to open stream: HTTP request failed! HTTP/1.1 502 Bad Gateway in /data/project/catscan2/public_html/omniscan.inc on line 132
[12:44:41] <GerardM->	 Maybe there is a problem ?
[12:45:32] <valhallasw`cloud>	 GerardM-: unless that just changed from 'working' to 'non-working', it's probably related to the mass reboots last night
[12:45:39] <Linedwell>	 according to the chan topic, there are some problems with webservices
[12:56:08] <GerardM->	 yes ... right ... so I ask for attention, service? 
[12:56:23] <GerardM->	 Is that not what this channel is there for as much as anything ?
[13:38:12] <GerardM->	 can someone restart WDQ https://wikitech.wikimedia.org/wiki/User:Yuvipanda/Restarting_magnus_wdq
[13:41:49] <shinken-wm>	 PROBLEM - Puppet staleness on tools-submit is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0]  
[13:57:21] <valhallasw`cloud>	 GerardM-: file a bug
[13:58:07] <valhallasw`cloud>	 GerardM-: many devs are currently in SF, so it might take a few hours before it's picked up.
[14:07:38] <GerardM->	 valhallasw`cloud: Really
[14:07:48] <valhallasw`cloud>	 GerardM-: yes, really.
[14:07:51] <GerardM->	 so why is support like this
[14:07:58] <GerardM->	 it is not only today
[14:08:34] <valhallasw`cloud>	 GerardM-: because, last time I checked, you aren't paying for support, so you'll have to wait until a volunteer restores service for you.
[14:09:23] <GerardM->	 <grin> so we have to pay to get support
[14:09:38] <valhallasw`cloud>	 GerardM-: or you can, say, wait until someone has time to look at it
[14:09:50] <Mjbmr>	 this is not good
[14:10:07] <devunt>	 well can I change wikitech wiki username without changing ldap/shell name?
[14:10:27] <devunt>	 I want to change my wikitech.wikimedia.org username only
[14:10:41] <valhallasw`cloud>	 devunt: iirc wikitech usernames shouldn't be changed at all
[14:12:35] <devunt>	 oh, that's shame.
[14:12:46] <devunt>	 thank you for responding
[14:13:02] <magnus__>	 http://wdq.wmflabs.org/ gives "502 bad gateway" since today. Anyone here who could fix it?
[14:14:15] <devunt>	 and valhallasw`cloud:
[14:14:35] <devunt>	 did you find out what caused phab's url encoding problem?
[14:14:38] <valhallasw`cloud>	 magnus__: is the web server running?
[14:15:01] <valhallasw`cloud>	 magnus__: I don't have access to the WDQ servers, but iirc you do?
[14:15:11] <grrrit-wm>	 (03PS1) 10Merlijn van Deen: Labs-Team was renamed to Labs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187114 
[14:15:36] <grrrit-wm>	 (03CR) 10Merlijn van Deen: [C: 032] Labs-Team was renamed to Labs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187114 (owner: 10Merlijn van Deen)
[14:15:49] <grrrit-wm>	 (03Merged) 10jenkins-bot: Labs-Team was renamed to Labs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/187114 (owner: 10Merlijn van Deen)
[14:16:42] <magnus__>	 valhallasw`cloud: Yuvi set up a load balanced, I don't think I can access it
[14:17:13] <valhallasw`cloud>	 magnus__: oh :( one quick thing to check is whether the web server instance is actually running; it might not have come up after last night
[14:20:12] <magnus__>	 valhallasw`cloud: wdq-mm-01 and wdq-mm-02 show as "active", but I can't ssh into wdq-mm-01 ("no route to host"). That might be it. No idea how to fix it though
[14:22:52] <valhallasw`cloud>	 magnus__: hm, yeah, and wdq-mm-01 was booted 2014-12-09T23:40:40, so before last nights outage. Maybe try restarting it.
[14:23:14] <valhallasw`cloud>	 https://graphite.wmflabs.org/render/?title=wdq-mm-01+Network+bytes+last+day&width=800&height=250&from=-24h&hideLegend=false&uniqueLegend=true&target=alias%28wdq-mm.wdq-mm-01.network.eth0.rx_byte.value%2C%22Bytes+received%22%29&target=alias%28wdq-mm.wdq-mm-01.network.eth0.tx_byte.value%2C%22Bytes+sent%22%29
[14:25:04] <valhallasw`cloud>	 ^ this suggests it's indeed related to last nights' outage. 
[14:27:29] <magnus__>	 I tried rebooting, but all I get is "The requested host does not exist.". I've mailed Yuvi, I guess he'll restart it once he's awake :-)
[14:29:20] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: 9f4845ee4937cc9bf890bc7ea2251ca6613080e0 Merge "Labs-Team was renamed to Labs"
[14:29:20] <wikibugs>	 3Wikidata, wikidata-query-service, Labs: WDQ down (502 Bad Gateway) - https://phabricator.wikimedia.org/T87765#998267 (10valhallasw) * <magnus__> valhallasw`cloud: wdq-mm-01 and wdq-mm-02 show as "active", but I can't ssh into wdq-mm-01 ("no route to host"). That might be it. No idea how to fix it though * <valh...
[14:29:25] <labs-morebots>	 Logged the message, Master
[14:29:59] <magnus__>	 thanks!
[14:59:23] <T13|detached>	 CommonsHelper tool was a mangus tool right? I'm assuming it was migrated to labs, anyone know?
[15:00:17] <T13|detached>	 Found it
[15:00:24] <T13|detached>	 Was an easy one
[16:20:56] <wikibugs>	 3Tool-Labs, WMF-Legal: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#998371 (10Cyberpower678)
[16:21:56] <wikibugs>	 3Tool-Labs, WMF-Legal: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#997462 (10Cyberpower678) Some of Hedonil's tools are integrated into xTools.  I've been getting buggered into having this fixed, but I can't do anything without access.
[16:36:48] <MusikAnimal>	 Question: does anyone use Ruby on tool labs? I notice at least on the login server only Ruby 1.8.7 is installed, which I'm guessing comes with Ubuntu
[16:37:10] <MusikAnimal>	 Ruby 2.1.x would be far more desirable, and faster
[16:54:49] <valhallasw`cloud>	 MusikAnimal: trusty has 2.x, I think.
[16:55:24] <valhallasw`cloud>	 MusikAnimal: and is available on the trusty nodes (so trusty.tools.wmflabs.org and the trusty exec nodes)
[16:58:10] <MusikAnimal>	 valhallasw`cloud: cool, so when I go to set up a new tool I can request a trusty instance?
[16:58:42] <valhallasw`cloud>	 MusikAnimal: when you submit a task to the grid, you can request a trusty host, yes. Check the docs for how, I'm not sure about the exact syntax.
[17:00:00] <MusikAnimal>	 ok I'll do more reading. Just wanted to know if Ruby was even a thing on tool labs. Sounds like everyone uses python or PHP
[17:01:22] <valhallasw`cloud>	 MusikAnimal: also, I think most ruby people actually compile their own interpreter?
[17:01:51] <valhallasw`cloud>	 MusikAnimal: that's what I noticed (and was put off by...) when trying to run ruby-based software packages
[17:03:28] <MusikAnimal>	 Interesting. Well, I wasn't going to bother with Rails as I don't need a whole framework for what I'm trying to do, and I think I can safely assume there's faster interpreters, maybe JRuby
[17:03:57] <valhallasw`cloud>	 MusikAnimal: I don't think speed of the interpreter is typically limiting for web apps
[17:04:18] <Cyberpower678>	 Is there anyone here that can approve an OAuth consumer
[17:05:13] <MusikAnimal>	 valhallasw`cloud: You don't happen to know any existing tools offhand using Ruby?
[17:05:21] <valhallasw`cloud>	 MusikAnimal: not directly, no.
[17:05:29] <valhallasw`cloud>	 MusikAnimal: maybe hay's tools, but I think those were Scheme
[17:06:24] <valhallasw`cloud>	 hm, actually a python/php mix. Not sure who used Scheme then....
[19:11:29] <wikibugs>	 3Wikidata, wikidata-query-service, Labs: WDQ down (502 Bad Gateway) - https://phabricator.wikimedia.org/T87765#998880 (10yuvipanda) 5Open>3Resolved a:3yuvipanda wdq-mm-01 (which hosts the load balancer as well) went down due to firefighting yesterday. Is back now.
[19:59:03] <Krenair>	 YuviPanda, upload.beta.wmflabs.org is down?
[20:07:20] <YuviPanda>	 Krenair: probably. I asked greg-g to get someone not me look at it :)
[20:09:10] <Krenair>	 deployment-upload seems to be returning what I would expect, guess the problem is with varnish on deployment-cache-upload02...
[20:09:25] <ragesoss>	 YuviPanda: (from yesterday) Is there anything except add a new labs user to a project that needs to be done to allow them to log in to an instance? They uploaded their key yesterday and set up the same ssh config that I use (with the proxycommand stuff for bastion). But they were getting a pubkey error.
[20:09:52] <YuviPanda>	 ragesoss: heya! Did they copy paste their key without newlines onto wikitech?
[20:09:57] <YuviPanda>	 ragesoss: that has been a common problem in the past
[20:10:19] <Krenair>	 can't connect to host when I curl deployment-cache-upload02
[20:10:30] <ragesoss>	 so, the answer is that, otherwise, it should just work?
[20:12:49] <greg-g>	 Krenair: YuviPanda beta cluster stuff -> #wikimedia-releng :)
[20:40:12] <fhocutt>	 who do I ask about updating a python library on labs?
[20:41:00] <fhocutt>	 there's a new version of mwclient I'm using for the co-op bot, and I'd like to avoid having to use a virtualenv once it's up and running
[20:41:30] <YuviPanda>	 fhocutt: hey!
[20:41:52] <YuviPanda>	 fhocutt: file a bug? I could package it and update. But I usually reccomend people use virtualenvs
[20:42:25] <fhocutt>	 YuviPanda, done already: https://phabricator.wikimedia.org/T87483
[20:42:39] <fhocutt>	 virtualenv doesn't play nicely with sqlalchemy/mysqldb
[20:42:47] <YuviPanda>	 ragesoss: it should just work yeah
[20:43:01] <fhocutt>	 I have to go in and modify the path and I suspect that's a fragile solution
[20:43:22] <YuviPanda>	 fhocutt: oh? It should in labs because we have the underlying libs installed 
[20:43:40] <YuviPanda>	 fhocutt: what issue are you running into?
[20:43:49] <fhocutt>	 hm, let me see
[20:43:51] <YuviPanda>	 Also are you in the office 
[20:43:56] <fhocutt>	 yep, I'm upstairs
[20:44:32] <Cyberpower678>	 YuviPanda, who here has the power to approve an OAuth consumer?
[20:45:21] <YuviPanda>	 Cyberpower678: anomie csteipp Reedy maybe
[20:45:45] <Reedy>	 I don't on my non staff account IIRC
[20:45:46] <Cyberpower678>	 Reedy are you here?
[20:45:47] <Reedy>	 I should fix that
[20:46:09] <Cyberpower678>	 Reedy, do you think you can take a quick minute to approve my consumer?
[20:47:00] <ragesoss>	 YuviPanda: now instead of pubkey, he's getting 'channel 0: open failed: administratively prohibited: open failed'
[20:47:23] <ragesoss>	 Cyberpower678: also Deskana
[20:48:07] <valhallasw`cloud>	 ragesoss: try without port forwarding?
[20:48:38] <Cyberpower678>	 Reedy, Deskana: https://en.wikipedia.org/w/index.php?title=Special:OAuthListConsumers/view/57865053fcbe6580e346b973a3a687af&name=&publisher=Cyberpower678&stage=0
[20:48:38] <fhocutt>	 YuviPanda, here's the hack I'm using right now: https://github.com/fhocutt/matchbot/blob/dev/mblog.py#L23
[20:48:57] <valhallasw`cloud>	 ragesoss: I don't think tunnels should be disallowed though, so that's a bit weird
[20:49:16] <ragesoss>	 his ssh config is the same one I use successfully.
[20:50:38] <valhallasw`cloud>	 fhocutt: there's three alternative methods I can think of -- In order of preference: 1) install the libraries (again) in the virtualenv, so pip install mysqldb sqlalchemy, 2) if that doesn't work, use a virtualenv with --system-site-packages, or 3) in the environment, set PYTHONPATH=/usr/lib/python2.7/dist-packages/
[20:51:05] <fhocutt>	 mysqldb is weird to install.
[20:51:23] <valhallasw`cloud>	 yeah, it needs to compile stuff. The libs needed for that /should/ be on tools, though, I think.
[20:51:38] <YuviPanda>	 Yeah they should be
[20:52:16] <Cyberpower678>	 *sigh
[20:52:19] <Reedy>	 Cyberpower678: done
[20:52:20] <fhocutt>	 checked--if I comment out that line, I get ImportError: No module named MySQLdb
[20:52:44] <valhallasw`cloud>	 fhocutt: pip install  mysql-python         should work
[20:52:51] <Cyberpower678>	 Reedy, it's still marked as proposed
[20:53:00] <Reedy>	 lol
[20:53:05] <Reedy>	 I didn't give a message
[20:53:05] <valhallasw`cloud>	 fhocutt: right; virtualenvs don't check the global system packages directory by default
[20:53:13] <Reedy>	 now it's done
[20:53:20] <YuviPanda>	 ok, am back on my thing now.
[20:53:28] <YuviPanda>	 thing -> laptop
[20:53:36] <Cyberpower678>	 There we go.  Thanks!!
[20:53:44] <YuviPanda>	 fhocutt: which tool is this?
[20:53:59] <Cyberpower678>	 Reedy, ^
[20:54:01] <fhocutt>	 this is the bot I'm writing for the Co-op
[20:54:25] <fhocutt>	 valhallasw`cloud, that seemed to install fine, let's see how this works now
[20:55:54] <fhocutt>	 valhallasw`cloud, YuviPanda, thanks--that works now without the path hack
[20:56:04] <valhallasw`cloud>	 great!
[20:56:09] <YuviPanda>	 fhocutt: \o/
[20:56:27] <fhocutt>	 thanks! May be worth updating mwclient anyway, the new version allows you to set custom user-agents
[20:56:27] <YuviPanda>	 fhocutt: so I can still install a newer mwlib if you want, but that’s probably going to take until next week. And for python stuff I really do reccomend virtualenvs
[20:56:57] <fhocutt>	 YuviPanda, totally fine, and thanks for the advice.
[20:57:29] <YuviPanda>	 fhocutt: yw! Tool Labs definitely needs way more communicty documentation and just a better developer story. Hopefully I’ll get some time to work on it in a bit
[20:58:20] <fhocutt>	 that would be great, it's been kind of frustrating getting going and I had J-Mo's existing hostbot set-up to work off of
[20:59:00] <YuviPanda>	 fhocutt: yup, yup. Needs lots of love.
[20:59:07] <YuviPanda>	 fhocutt: am currently focused on betalabs, though.
[20:59:11] <fhocutt>	 totally
[21:08:45] <wikibugs>	 3Labs: Update mwclient on labs - https://phabricator.wikimedia.org/T87483#999312 (10Fhocutt) p:5High>3Normal
[21:15:26] <ragesoss>	 thanks for the suggestions, valhallasw`cloud and YuviPanda. We got it fixed. 
[21:15:35] <YuviPanda>	 ragesoss: yay. what was the issue?
[21:18:03] <ragesoss>	 mainly, some slight miscommunication. In the ssh config I gave him, I was calling the server "dashboardlabs2", but the actual name is educationdashboard2. From the help materials, he was doing 'ssh dashboardlabs2.eqiad.wmflabs', when it should have been either 'ssh dashboardlabs2' or 'ssh educationdashboard2.eqiad.wmflabs'
[21:18:23] <ragesoss>	 I had him send me the -vv out put and then I caught it.
[21:18:59] <ragesoss>	 (that's what you had me do the first time I was unsuccessfully trying to log in on labs, YuviPanda)
[21:19:05] <ragesoss>	 :D
[21:31:28] <valhallasw`cloud>	 fhocutt: okay, I now registered the 'mysql' package, so pip install mysql now /also/ works :-D
[21:32:03] <fhocutt>	 valhallasw`cloud, thanks!
[21:36:46] <matanya>	 YuviPanda: hi there ?
[22:05:28] <YuviPanda>	 matanya: hey
[22:05:40] <matanya>	 forgot what i wanted to ask
[22:06:08] <YuviPanda>	 andrewbogott: do you want to build new ubuntu images with the patched partitions? or are you still tired off the firefighting?
[22:09:13] <andrewbogott>	 YuviPanda: there's still breakage with the jessie image so I'm not ready to rebuild everything yet
[22:09:17] <andrewbogott>	 And, still firefighting a bit
[22:09:30] <YuviPanda>	 andrewbogott: alright!
[22:14:02] <wikibugs>	 3Tool-Labs, Analytics-Engineering: Copy paid MaxMind geolocation dbs to tool labs - https://phabricator.wikimedia.org/T87151#999591 (10yuvipanda) 5Open>3stalled
[22:23:29] <valhallasw`cloud>	 YuviPanda: ^ a non-free database? *frowny face*
[22:23:42] <YuviPanda>	 valhallasw`cloud: yeah, I don’t expect it to happen
[22:25:31] <valhallasw`cloud>	 YuviPanda: *nod* would be awesome to have, but it would be pretty hypocritical
[23:18:35] <Krinkle>	 YuviPanda: ping
[23:22:06] <YuviPanda>	 Krinkle: pong
[23:22:36] <Krinkle>	 YuviPanda: I was gonan ask you to run a command using salt-master on all integration-slave* instances but did it manually for now
[23:22:46] <YuviPanda>	 Krinkle: ah
[23:32:23] <wikibugs>	 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#999884 (10yuvipanda) I'll note that this thing is finally going to be fixed shortly by T87003