[00:52:58] <wikibugs>	 06Labs, 10Tool-Labs: Missing replicy.my.conf on Tools Labs - https://phabricator.wikimedia.org/T138037#2387776 (10Fabian24d97) 05Open>03Resolved a:03Fabian24d97
[01:22:05] <wikibugs>	 06Labs, 10Tool-Labs: Missing replicy.my.conf on Tools Labs - https://phabricator.wikimedia.org/T138037#2387374 (10Krenair) @Fabian24d97: I noticed that you marked this as resolved by yourself. How did you fix this issue? Other users have been experiencing it too, so it'd be useful to know... Semi-relevant XKCD...
[06:09:04] <shubinator>	 one of the jobs for my bot is stuck in the "deleted" state, and calling qdel again on the job does nothing
[06:09:15] <shubinator>	 could one of the powers that be force-kill the job...?
[06:10:49] <shubinator>	 it's in the "dr" state to be precise
[06:12:29] <yuvipanda>	 hey shubinator 
[06:12:33] <yuvipanda>	 which tool / job?
[06:12:37] <yuvipanda>	 a job id even perhaps?
[06:14:23] <shubinator>	 job id 7033590
[06:15:36] <shubinator>	 runDYKUpdateBot
[06:16:31] <shubinator>	 running under tools.dykbot
[06:17:25] <yuvipanda>	 !tools forced deletion of 7033590 for dykbot for shubinator 
[06:17:25] <wm-bot>	 There are multiple keys, refine your input: tools-admin, toolsbeta, tools-bug, toolscors, tools-equiad, toolsmigration, tools-request, tools-status, toolsvslabs, tools-web, 
[06:17:33] <yuvipanda>	 !log tools forced deletion of 7033590 for dykbot for shubinator 
[06:17:34] <yuvipanda>	 there
[06:17:38] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[06:18:10] <shubinator>	 excellent, thank you!!
[06:18:36] <yuvipanda>	 yw!
[10:27:24] <wikibugs>	 06Labs, 10Tool-Labs: Missing replicy.my.conf on Tools Labs - https://phabricator.wikimedia.org/T138037#2388333 (10Fabian24d97) @Krenair  I have missed that only tools have a `replica.my.conf`, so I created a tool which then had one.
[11:24:46] <SoniWP>	 Hello
[11:25:03] <SoniWP>	 I have been trying to login to Tool Labs to run a few SQL Queries.
[11:25:16] <SoniWP>	 But I am not quite sure what exactly I am supposed to do.
[11:25:33] <SoniWP>	 I have been granted Tool and Shell access and SSHed into my tool account
[11:25:42] <SoniWP>	 But what am I supposed to do next?
[11:27:30] <jynus>	 well, you have quarry
[11:28:09] <jynus>	 and documentation on https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database
[11:28:36] <SoniWP>	 I am aware of Quarry but it times out on the types of Query I am trying. Hence I have been advised to try Tool Labs
[11:28:55] <jynus>	 then see may second suggestion
[11:29:21] <SoniWP>	 ERROR 1045 (28000): Access denied for user 'soni'@'10.68.23.58' (using password: YES)
[11:29:38] <SoniWP>	 This is the error I get when trying to run "sql enwiki"
[11:29:55] <jynus>	 that is not possible, you are not using or have a replica.my.cnf
[11:29:59] <SoniWP>	 Which password am I supposed to enter? WikiTech?
[11:30:14] <SoniWP>	 Oh okay.
[11:30:16] <jynus>	 SoniWP, please read the docs
[11:31:38] <SoniWP>	 Alright thank you.
[11:32:06] <SoniWP>	 I'll get back to you if I figure out what is wrong
[16:13:04] <wikibugs>	 06Labs, 10Tool-Labs: Disable xdebug on Tool Labs - https://phabricator.wikimedia.org/T137146#2388850 (10Matthewrbowker) a:05Matthewrbowker>03None Tom did some testing in my stead, I'm going to unassign as there's no more testing I can do.  @tom29739 would you like to give your observations?
[16:26:55] <wikibugs>	 06Labs, 10Tool-Labs: Disable xdebug on Tool Labs - https://phabricator.wikimedia.org/T137146#2388908 (10tom29739) a:03tom29739 I discovered that NFS was a huge slowdown of all webservices, and that running a webservice in /tmp, I copied the files to the bastion /tmp and made a symlink, and it worked (strange...
[16:37:07] <tom29739>	 How can I get access to toolsbeta? I'd like to do some testing on there to see whether xdebug makes a difference to the speed of tools webservices, and I need a same or similar environment to tools to do so. :)
[16:58:15] <wikibugs>	 06Labs, 10Tool-Labs: Missing replicy.my.conf on Tools Labs - https://phabricator.wikimedia.org/T138037#2388986 (10scfc) 05Resolved>03Open a:05Fabian24d97>03None That's not true: //All// user and tool accounts (in the #Tool-Labs project) should have a `~/replica.my.cnf` (cf. https://wikitech.wikimedia.o...
[17:08:19] <tom29739>	 On second thoughts, it'd probably easier to have a labs project to do the testing on.
[17:12:08] <wikibugs>	 06Labs: Create new labs project tools-xdebug-testing - https://phabricator.wikimedia.org/T138097#2389036 (10tom29739)
[17:17:28] <wikibugs>	 06Labs, 10Tool-Labs: Missing replicy.my.conf on Tools Labs - https://phabricator.wikimedia.org/T138037#2389059 (10scfc) I tested `modules/labstore/files/create-dbusers`'s logic to find all users and servicegroups (`print(all_users)`), and a) it does not seem to include the user account `fdamken` or `Fabian24d9...
[17:24:04] <wikibugs>	 06Labs, 06Design-Research-Backlog: Public IP and Wildcard DNS for REFLEX project - https://phabricator.wikimedia.org/T92273#1104460 (10Capt_Swing) Reflex project is no longer active.
[17:25:02] <wikibugs>	 06Labs, 06Design-Research-Backlog: Public IP and Wildcard DNS for REFLEX project - https://phabricator.wikimedia.org/T92273#2389098 (10ggellerman) 05Open>03Invalid
[18:10:51] <SoniWP>	 Hello
[18:10:58] <SoniWP>	 I think I am SSHed into Tool labs
[18:11:09] <SoniWP>	 But I do not see any UserID-Password
[18:11:26] <SoniWP>	 Where should I be searching for them?
[18:12:44] <Matthew_>	 SoniWP: For database work?  
[18:12:51] <Matthew_>	 Should be in ~/replica.my.cnf
[18:13:33] <tom29739>	 Phabricator is erroring out every frequently with: Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #2003: Can't connect to MySQL server on 'm3-master.eqiad.wmnet' (99).
[18:13:49] <SoniWP>	 soni@tools-bastion-03:~$ vi ~/replica.my.cnf
[18:14:05] <SoniWP>	 Matthew_, I tried the above. Dont see anything
[18:14:09] <Matthew_>	 But there was a bug reported: https://phabricator.wikimedia.org/T138037#2388986
[18:14:15] <Matthew_>	 Sorry had to find it.
[18:14:16] <Matthew_>	 Hm.
[18:14:29] <SoniWP>	 Am I doing something wrong or is the file missing for me?
[18:14:40] <Matthew_>	 Is it in there if you do "ls ~"
[18:16:07] <yuvipanda>	 hey SoniWP 
[18:16:09] <yuvipanda>	 am taking a look now
[18:16:23] <wikibugs>	 06Labs: Create new labs project tools-xdebug-testing - https://phabricator.wikimedia.org/T138097#2390406 (10tom29739)
[18:16:27] <wikibugs>	 06Labs, 10Tool-Labs: Disable xdebug on Tool Labs - https://phabricator.wikimedia.org/T137146#2390405 (10tom29739)
[18:16:36] <SoniWP>	 Matthew_, No.
[18:16:39] <SoniWP>	 yuvipanda, Thanks
[18:16:41] <paladox>	 tom29739 Hi i think that is because we now track all branches in operation/puppet repo
[18:16:55] <tom29739>	 wikibugs is slow.
[18:16:56] <paladox>	 refs/changes/ are being processed and this repo is big.
[18:17:06] <tom29739>	 OK.
[18:17:09] <Matthew_>	 SoniWP: There's your answer.  yuvipanda is far smarter than I am, and will get you hooked up :)
[18:17:16] <yuvipanda>	 SoniWP in general we reccomend creating a tool and doing your work there - you can do that just now to unblock yourself
[18:17:20] <tom29739>	 paladox, will it finish soon?
[18:17:52] <yuvipanda>	 tools also have other advantages, such as allowing you to easily share your work with others, etc
[18:18:20] <paladox>	 tom29739 hopefully. But i carn't say.
[18:18:49] <SoniWP>	 yuvipanda, I do not know how to. Can you give me a link where I can figure out how to create a tool?
[18:19:06] <paladox>	 tom29739 we had this problem before when we did it a few months back and had this problem. But as it is one repo hopefully it will be done in an hour or so.
[18:19:27] <paladox>	 But you should still be able to use phabricator. Just it may time out more often then usual.
[18:19:35] <yuvipanda>	 SoniWP sure! Our documentation kinda sucks unfortunately, so let me guide you through it
[18:19:58] <yuvipanda>	 SoniWP if you go to http://tools.wmflabs.org/ you'll find a 'create a new tool' link under 'getting started'
[18:19:59] <paladox>	 yuvipanda hi there's gerrit-dev but it seems apache has been stoped on there
[18:20:03] <paladox>	 gerrit-dev.wmflabs.org
[18:20:04] <yuvipanda>	 err, under 'developing your own tool'
[18:20:29] <paladox>	 https://gerrit-dev.wmflabs.org/
[18:20:55] <yuvipanda>	 paladox hmm, I've no idea who maintains that though, so you should contact them. do you know who it is? if not I can try to find out for you
[18:21:05] <tom29739>	 SoniWP, go to this link: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaServiceGroup&action=addservicegroup&projectname=tools
[18:21:26] <paladox>	 yuvipanda Hi, nope i doint know who owns it but it is linked from the gerrit page on wikitech
[18:21:41] <paladox>	 Saying that is what they used before pushing it out to production gerrit.
[18:21:49] <paladox>	 But old references such as gerrit 2.4.
[18:21:57] <tom29739>	 SoniWP, think of a tool name, restart your shell session, type 'become <whatever you typed for a tool name>' and you're in.
[18:21:58] <paladox>	 Could you find me the owner please.
[18:22:00] <paladox>	 :)
[18:22:07] <yuvipanda>	 yup, moment
[18:22:43] <paladox>	 Thankyou very much :)
[18:27:12] <yuvipanda>	 paladox so it points to gerrit-dev.eqiad.wmflabs, but no such instance exists. so whatever it was, has been deleted a long time ago
[18:27:42] <paladox>	 yuvipanda Oh, ok thanks for finding that out. Should we delete the website since it is no longer in use.
[18:27:59] <yuvipanda>	 paladox yes I think so
[18:28:06] <yuvipanda>	 paladox can you file a bug and assign it to me?
[18:28:18] <paladox>	 yuvipanda Ok, thanks :_
[18:28:20] <paladox>	 :)
[18:29:02] <yuvipanda>	 tom29739 I can add you to the toolsbeta project, but I rarely use it so I've no idea what state it is in
[18:29:15] <Matthew_>	 Real quick question: What is the memory usage limit on tool labs?  
[18:29:36] <yuvipanda>	 Matthew_ for jsub, you can set it with the -mem parameter
[18:29:48] <Matthew_>	 What about for webservice?
[18:29:55] <yuvipanda>	 for webservices, it defaults to '4G' but that's not a 'real' number, it's just vague heuristics that gridengine uses
[18:29:56] <SoniWP>	 yuvipanda, thanks. It worked
[18:30:23] <Matthew_>	 Okay thanks.
[18:32:23] <wikibugs>	 06Labs: Delete gerrit-dev.wmflabs.org website - https://phabricator.wikimedia.org/T138101#2390444 (10Paladox)
[18:32:32] <paladox>	 yuvipanda https://phabricator.wikimedia.org/T138101 :)
[18:33:22] <tom29739>	 yuvipanda, can you create the labs project tools-xdebug-testing instead? https://phabricator.wikimedia.org/T138097
[18:34:06] <yuvipanda>	 you know you can just disable xdebug for your own tool on tool labs, right?
[18:35:05] <tom29739>	 Completely? Like, unloading the entire module?
[18:36:13] <tom29739>	 Unless the entire module is unloaded, you won't get the speedup. :/
[18:36:28] <tom29739>	 Sorry for being a pain :D
[18:37:18] <yuvipanda>	 I think you can do that too
[18:37:28] <tom29739>	 You can?
[18:37:32] <yuvipanda>	 yeah
[18:37:37] <yuvipanda>	 there's a 'lighttpd-plain' webservice type
[18:37:41] <yuvipanda>	 which doesn't load anything
[18:38:17] <tom29739>	 I'd have to load PHP myself, right?
[18:38:28] <yuvipanda>	 yeah, I think so
[18:38:41] <yuvipanda>	 it just sets up mimetype defaults 
[18:38:56] <yuvipanda>	 you can find the exact configs in https://github.com/wikimedia/operations-software-tools-webservice
[18:50:39] <paladox>	 yuvipanda Hi, do you know how we do the dns for git.wikimedia.org for https://phabricator.wikimedia.org/T137224 please
[18:51:32] <yuvipanda>	 paladox unfortunately no, I haven't really done anything with prod DNS
[18:51:46] <paladox>	 yuvipanda Oh, ok. thanks.
[18:52:12] <yuvipanda>	 yw
[19:07:10] <wikibugs>	 06Labs, 10Tool-Labs: Some Tools users do not show up in create-dbusers query - https://phabricator.wikimedia.org/T138102#2390481 (10scfc)
[19:08:36] <wikibugs>	 06Labs, 10Tool-Labs: Calculation of home directories is broken for users - https://phabricator.wikimedia.org/T138103#2390499 (10scfc)
[19:09:10] <wikibugs>	 06Labs, 10Tool-Labs: Calculation of home directories is broken for users - https://phabricator.wikimedia.org/T138103#2390499 (10scfc) a:03scfc
[19:31:13] <wikibugs>	 06Labs: Nodepool has trouble taking snapshots on OpenStack labs - https://phabricator.wikimedia.org/T138106#2390567 (10Krenair)
[19:31:41] <paladox>	 yuvipanda Hi, im trying to setup gerrit-test.wmflabs.org 
[19:31:45] <paladox>	 but getting error
[19:31:46] <paladox>	 : [client 10.68.21.68:55101] AH00035: access to /review_site/tmp/index.htm denied (filesystem path '/var/www/html/review_site/tmp/index.htm') because search permissions are missing on a component of the path
[19:32:06] <yuvipanda>	 I suppose that's a gerrit related issue? I've no experience setting up gerrit, sorry
[19:32:17] <paladox>	 Oh ok
[19:47:16] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure: Nodepool has trouble taking snapshots on OpenStack labs - https://phabricator.wikimedia.org/T138106#2390592 (10hashar)
[19:51:05] <wikibugs>	 06Labs, 10Horizon, 13Patch-For-Review: Create puppet backend with REST api for labs instance configuration - https://phabricator.wikimedia.org/T133412#2390599 (10Andrew) a:05Andrew>03yuvipanda The backend frequently errors out, like this:  Traceback (most recent call last):   File "/usr/lib/python3/dist-...
[20:13:31] <SoniWP>	 Hello. I am trying to run a query via Tool labs
[20:14:58] <SoniWP>	 BUt it got killed in about 30 mins.
[20:15:33] <SoniWP>	 So i wanted to know if the Quarry limit that kills queries within 30 mins also extends to Tool Labs.
[20:15:37] <SoniWP>	 Would anyone know?
[20:15:53] <SoniWP>	 This is the query for reference - https://quarry.wmflabs.org/query/10293
[20:16:05] <SoniWP>	 It was being run by the tool "sonitool", if it helps\
[20:17:29] <tom29739>	 I'm pretty sure the db kills queries after 30 mins.
[20:18:03] <tom29739>	 I think you can add something to the query that bypasses the limit (or was that toolserver?)
[20:18:46] <Matthew_>	 That was toolserver I think.  "/* SLOW OK */"
[20:19:03] <SoniWP>	 tom29739, Would you know what I can do to bypass the limit?\
[20:19:36] <tom29739>	 The usual answer is 'Optimise your SQL'
[20:19:49] <tom29739>	 What are you trying to do with that query?
[20:22:33] <SoniWP>	 I dont think I can do much to Optimise this particular query, actually.
[20:23:09] <SoniWP>	 I'm trying to generate a randomised sample of Pages made, based on their first revision.
[20:24:15] <SoniWP>	 There will be a limit on the first revision added in the query later but other than that, the query will remain pretty identical
[20:24:35] <tom29739>	 Could you do it in batches?
[20:25:09] <tom29739>	 Maybe limit it more, and then keep running it.
[20:25:26] <tom29739>	 Then join it up at the end.
[20:25:27] <SoniWP>	 Like a limit 20 or something, you mean?
[20:25:49] <tom29739>	 As big as you can get.
[20:26:00] <tom29739>	 (without it being killed)
[20:26:39] <SoniWP>	 Hm, okay. Is there any other way to try bypassing the query time limit altogether?
[20:26:57] <SoniWP>	 My final data size is expected to be about 20,000 rows
[20:27:23] <SoniWP>	 So there's only so much I can get by making it into batches
[20:38:04] <wikibugs>	 06Labs, 06Operations, 13Patch-For-Review: Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#2390675 (10hashar)
[20:52:31] <wikibugs>	 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Database-Queries: Run a Tool Labs query without Timing out - https://phabricator.wikimedia.org/T138111#2390701 (10Soni)