[00:27:13] <Magog_the_Ogre>	 hello, everyone
[00:30:37] <Betacommand>	 hi
[01:30:52] <Coren>	 Krenair: I'm here.
[01:31:22] <Coren>	 Krenair: Sorry, had to run out for a bit.  Yes, it'll support your use case.
[01:33:17] <Krenair>	 cool
[01:34:48] <Krenair>	 are you planning for this to be available soon?
[01:56:45] <wikibugs>	 3Wikimedia Labs / 3tools: deletion queries joined with tokudb replication tables are really slow - 10https://bugzilla.wikimedia.org/68918#c7 (10Sean Pringle) So we have index differences too? Is the toolserver schema dumped anywhere I can see it?
[02:08:14] <Betacommand>	 jackmcbarn: you still around?
[02:08:19] <jackmcbarn>	 yeah
[02:08:23] <Betacommand>	 jackmcbarn: http://tools.wmflabs.org/betacommand-dev/reports/single_red_links.txt
[02:08:31] <Betacommand>	 go ahead and post a link to it
[02:08:37] <Betacommand>	 warning it is a large file
[02:08:49] <Betacommand>	 almost 100Mb
[02:09:02] <jackmcbarn>	 it crashed firefox...
[02:09:34] <Betacommand>	 jackmcbarn: Ill get a compressed version of it in a few minutes
[02:09:38] <Betacommand>	 http://tools.wmflabs.org/betacommand-dev/reports/
[02:09:47] <Betacommand>	 and download it instead of opening
[02:11:10] <Betacommand>	 jackmcbarn: in the process of creating a 7z
[02:16:40] <Betacommand>	 jackmcbarn: the compressed version is about 30Mb http://tools.wmflabs.org/betacommand-dev/reports/single_red_links.7z
[02:16:59] <jackmcbarn>	 ok
[07:30:00] <katpatuka>	 hi - is there a way to additional enable searching template names in the searching suggestions dropdown-lists of the search field ?
[07:34:22] <katpatuka>	 or is there another channel I can go and ask?
[09:02:07] <wikibugs>	 3Wikimedia Labs / 3deployment-prep (beta): deployment-mediawiki02 and mediawiki03 have puppet disabled - 10https://bugzilla.wikimedia.org/71410#c1 (10Antoine "hashar" Musso) 5NEW>3RESO/FIX Puppet has been reenabled somehow.
[14:18:28] <mbh>	 Hi. I'm new on labs, how I can run my service, written on C#? I load project folder into ~/public_html, launch webservice, what's next? My tool is http://tools.wmflabs.org/mbh/ , shellname mbh, account name Maxbiohazard
[14:23:06] <YuviPanda>	 hello mbh
[14:23:25] <YuviPanda>	 mbh: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Other_web_servers might be able to help
[14:28:14] <mbh>	 "First of all, create a file called 'httpserver.sh', with the following contents:" - create where?
[14:28:34] <YuviPanda>	 mbh: anywhere in your tool account :)
[14:29:12] <hashar>	 YuviPanda: !!!! :-]
[14:29:19] <YuviPanda>	 hashar: ?
[14:29:25] <hashar>	 YuviPanda: I was wondering whether the beta icinga monitoring emails could be tweaked
[14:29:36] <hashar>	 YuviPanda: i could use the instance hostname in the mail Subject hehe
[14:30:27] <YuviPanda>	 hashar: will have to wait for shinken, I'm afraid
[14:30:57] <mbh>	 "exec portgrabber <tool name> <command to run tool>" - what is "command to run tool"? View locate in "/trans_ns_moves/trans_ns_moves/Views/records/Index.cshtml"
[14:31:48] <valhallasw`cloud>	 mbh: how would you normally run it?
[14:32:12] <hashar>	 YuviPanda: works for me :]
[14:32:17] <YuviPanda>	 hashar: :) cool
[14:33:12] <mbh>	 until I run it only within Visual Studio
[14:33:26] <hashar>	 and another ops question:  I have a production machine in 10.0.0.0/8 and can't reach out a labs instance despite having set the security rule
[14:33:39] <hashar>	 I guess there is either no route or some firewall in between
[14:34:22] <valhallasw`cloud>	 mbh: you probably want to follow http://www.mono-project.com/docs/web/fastcgi/
[14:34:22] <YuviPanda>	 mbh: hmm, so this is running Mono on Linux
[14:34:49] <YuviPanda>	 hashar: you can't ping prod machines from labs at all, unless they've a public IP, or  you poke mark into creating a firewall exception
[14:35:00] <YuviPanda>	 hashar: there's a rather strict firewall between prod and labs
[14:35:12] <valhallasw`cloud>	 mbh: alternatively, use xsp with portgrabber
[14:35:26] <valhallasw`cloud>	 mbh: but I'd suggest the fcgi route
[14:35:54] <hashar>	 YuviPanda: ah thanks for confirming :]
[14:36:15] <YuviPanda>	 hashar: so file an rt ticket in the network queue, but usually it requires a damn good reason :)
[14:36:23] <hashar>	 YuviPanda: na there is no reason for it
[14:36:25] <hashar>	 I can adjust :]
[14:36:31] <YuviPanda>	 hashar: :) cool
[14:39:10] <hashar>	 YuviPanda: I should just run all tests in labs :]
[14:39:17] <YuviPanda>	 hehe
[14:39:19] <YuviPanda>	 yaaaa
[14:46:44] <wikibugs>	 3Wikimedia Labs / 3wikidata: make instance for wbdoc.wmflabs.org work with normal puppet master - 10https://bugzilla.wikimedia.org/71511 (10Lydia Pintscher) p:5Unprio>3Low s:5normal>3minor
[14:51:15] <wikibugs>	 3Wikimedia Labs / 3tools: Mail/LDAP query is broken - 10https://bugzilla.wikimedia.org/71392#c1 (10Marc A. Pelletier) 5NEW>3RESO/FIX This was accidentally broken when the LDAP change was done; unbroken.
[15:15:53] <andrewbogott>	 !log social-tools re-enabled puppet on instance 'gadgets'
[15:15:57] <labs-morebots>	 Logged the message, dummy
[15:39:22] <Nemo_bis>	 Barras: can you redirect #wikimedia-rcom to #wikimedia-research? halfak wants so
[15:39:40] <Nemo_bis>	 (he's not formally op or foudner of -rcom but he's on RCOM etc. etc.)
[15:39:50] <halfak>	 I confirm. 
[15:40:07] <halfak>	 DarTar is running the channel.  I work with him at the WMF.  He's too busy to deal with this right now. 
[15:42:04] <Barras>	 Nemo_bis / halfak : I will take care of that...
[15:42:23] <halfak>	 Thanks Barras
[16:45:45] <wikibugs>	 3Wikimedia Labs / 3tools: Mail/LDAP query is broken - 10https://bugzilla.wikimedia.org/71392#c2 (10Tim Landscheidt) (Cf. Gerrit change #164337.)
[17:07:10] <quiddity>	 labs is down?
[17:07:11] <quiddity>	 no response in ~5 minutes from http://en.wikipedia.beta.wmflabs.org/  and   http://mwui.wmflabs.org/
[17:07:12] <JohnFLewis>	 Accounts.wmflabs.org as well
[17:07:13] <quiddity>	 i've mentioned it in -operations
[17:07:15] <JohnFLewis>	 quiddity: I saw after I posted the above :)
[17:07:17] <Coren>	 labs outage atm, should return shortly.
[17:08:02] <andrewbogott>	 Coren:  What happened?
[17:08:11] <andrewbogott>	 Related to your network changes?
[17:08:30] <Coren>	 andrewbogott: Yes, but in a completely insane way.
[17:32:13] <andrewbogott>	 !log tools removing ryan lane from tools admins, because his email in ldap is defunct and I get bounces every time something goes wrong in tools
[17:32:24] <labs-morebots>	 Logged the message, dummy
[17:33:17] <jeremyb>	 hah
[17:33:25] <jeremyb>	 that's why i showed up :)
[17:33:33] <jeremyb>	 andrewbogott: springle not ohere
[17:33:36] <jeremyb>	 not here*
[17:34:02] <jeremyb>	 hrmmmm, tools.labsdb Host '10.68.17.1' is blocked because of many connection errors
[17:34:08] <andrewbogott>	 jeremyb: Sean lives in Australia, generally opposite hours of NA
[17:34:21] <jeremyb>	 and DBD::mysql::db do failed: MySQL server has gone away at
[17:34:44] <jeremyb>	 andrewbogott: so hard to remember he's in OC... i'm sure i knew that.
[17:34:59] <jeremyb>	 ok, bbl
[17:41:28] <Krinkle>	 Labs going out of memory?
[17:41:35] <Krinkle>	 Or did the way of measuring change?
[17:41:49] <andrewbogott>	 Krinkle: context?
[17:41:52] <Krinkle>	 Past 1-2 hours various unrelates tool labs bots died after a high uptime
[17:41:59] <Krinkle>	 and the restart immediately got killed as well
[17:42:07] <Krinkle>	 e-mails with "Out of memory!" are being sent
[17:42:16] <Krinkle>	 I've had those before, not for OOM anyway.
[17:42:30] <Krinkle>	 Cron <tools.krinklebot@tools-submit> /usr/bin/jsub -N fileprotectionsync -once -quiet -mem 500m ~/liveprotect-run.sh
[17:42:34] <Krinkle>	 > Out of memory!
[17:42:51] <Krinkle>	 Cron <tools.wmfdbbot@tools-submit> /usr/bin/jsub -N dbbot-wm -once -continuous -quiet -mem 1280M -o /dev/null php ~/apps/ts-krinkle-Kribo/Init.php
[17:42:54] <Krinkle>	 > Out of memory!
[17:43:22] <andrewbogott>	 Krinkle: I don't know -- Coren and Mark were adjusting network connections.  I wouldn't think that would cause an OOM but it's possible.
[17:43:56] <Coren>	 It may have caused OOM as a side effect if there were a LOT of cron jobs piling up waiting on the disk.
[17:57:24] <Krinkle>	 Cron <tools.delinker@tools-submit> /usr/bin/jsub -once -mem 700m -j y -o $HOME/job-cdh.out -N cdh $HOME/bin/start-cdh >>$HOME/job-submit.out
[17:57:25] <Krinkle>	 Use of uninitialized value $prog in scalar chomp at /usr/bin/jsub line 112.
[17:57:27] <Krinkle>	 Use of uninitialized value $prog in -l at /usr/bin/jsub line 113.
[17:57:30] <Krinkle>	 Use of uninitialized value $prog in pattern match (m//) at /usr/bin/jsub line 127.
[17:57:31] <Krinkle>	 /data/project/delinker//data/project/delinker/bin/start-cdh: No such file or directory
[17:57:34] <Krinkle>	 Testing in prod:P ?
[17:58:44] <YuviPanda>	 Coren: ^ awjr (and others) also got similar errors
[17:58:51] <YuviPanda>	 Krinkle: I guess that's mark :)
[17:59:18] <Coren>	 Those are probably all side effect of the mess a bit earlier.  If this recurs, please notify me.
[17:59:27] <andrewbogott>	 !log tools added Ryan back to tools admins because that turned out to not have anything to do with the bounce messages
[17:59:29] <labs-morebots>	 Logged the message, dummy
[20:22:39] <andrewbogott>	 Coren: emails telling me I should "unblock with 'mysqladmin flush-hosts'" -- is there any reason for me not to do so?
[20:23:04] <Coren>	 andrewbogott: Not that I know of; but I'd double check with Sean.
[21:19:33] <Betacommand>	 Coren: can you take a look at job 4490576 ? Im not seeing the script come up
[23:58:34] <mutante>	  56             'labs' => $::site ? {
[23:58:34] <mutante>	  57                 'pmtpa' => [ "i-00000051" ],
[23:58:35] <mutante>	  58                 default => undef,
[23:58:56] <mutante>	 from prod lvs config
[23:59:07] <mutante>	 would you want something not undef?