[00:03:19] SMalyshev: could it be this.. just best guess here: [00:03:28] in updater.pp , line 23 [00:03:58] eh, while i said it.. nevermind [00:04:08] i thought maybe $path is not set [00:04:22] because there is that conditional for the initsystem [00:04:41] nah, I thought on that one too but unfortunately it's not it [00:04:57] unless it somehow misinterprets it [00:06:43] maybe you can split it into smaller chunks? [00:26:57] (03PS1) 10Ricordisamoa: Make it work on Python 3 [labs/tools/translatemplate] - 10https://gerrit.wikimedia.org/r/225267 [00:27:09] bd808: I tried to enable role::deployment::server and I get tons of errors about mysql-client-5.1 which should be 5.5 and it goes downhill from there [00:28:52] Hooray for Python 3 [00:32:49] Unrelatedly [00:32:54] bd808: does it require some specific os (jessie/ubuntu)? [00:33:00] Why aren't we allowed to run "explain" sql queries? [00:33:44] (03CR) 10Ricordisamoa: "Tool Labs doesn't support py3k + uWSGI yet (T104374), but it'd be nice to get ready :)" [labs/tools/translatemplate] - 10https://gerrit.wikimedia.org/r/225267 (owner: 10Ricordisamoa) [00:36:56] SMalyshev: fwiw, on the production deployment server using that role. it is 5.5 [00:37:40] mutante: that's what I was getting: https://gist.github.com/smalyshev/b711923a4337dd276eab [00:38:51] SMalyshev: i see this: modules/mysql_wmf/manifests/client.pp: package { 'mysql-client-5.1': [00:38:59] but that is "mysql_wmf" [00:39:14] for some reason I imaging deployment role pulls it in [00:39:20] imagine [00:39:29] i just checked, it says "include mysql" [00:39:41] so i just recently applied that role on a new server, mira [00:39:45] and i got 5.5 [00:40:08] i think mysql =! mysql_wmf [00:40:10] != [00:40:45] the one other place this shows up is in the "snapshot" module, but i doubt you use that [00:41:02] yeah probably but maybe something like 'include mediawiki' pulls it in? [00:41:55] dunno. I'll re-init the instance and try again [00:42:34] try just including that one role without any others first [00:42:40] the deployment server tole [00:42:43] role [00:47:53] SMalyshev: what distro version are you on now? [00:49:08] trusty [00:55:12] SMalyshev: i can confirm that this role works on physical server "mira" which is also running trusty and includes role::deployment::server and has mysql-client-5.5 without these errors [00:55:25] i guess it must be the combo with another role or puppetmaster::self [00:55:47] the mediawiki part then ? [00:56:00] did you say you also include a mediawiki role? [01:00:46] mutante: I think deploy server role does that: include mediawiki in its role file [01:01:33] mutante: I tried on jessie before, but now trying on trusty in case something is different [01:01:35] SMalyshev: .. it does. it's weird how i got away without this error [01:01:42] ah [01:01:53] well if it's jessie that makes sense [01:02:02] we did not get to that yet on any deployment server [01:02:05] though it would be nice [01:02:13] and is something that should be fixed then [01:02:46] ok, let's see how far it gets on trusty... I already see some red stuff but may be just timeouts [01:02:47] so i assume the case is that on trusty the 5.1 package is still existing as a dummy package that pulls in 5.5 [01:02:51] but on jessie it is gone [01:03:12] they usually keep dummy packages like that for one distro version but not 2 [01:08:31] Error: Cannot create /home/l10nupdate/.ssh; parent directory /home/l10nupdate does not exist [01:09:01] Error: /Stage[main]/Role::Deployment::Server/Keyholder::Private_key[mwdeploy_rsa]/File[/etc/keyholder.d/mwdeploy_rsa]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///private/ssh/tin/mwdeploy_rsa [01:10:11] SMalyshev: the work around for the second part is to put a fake key into the labs/private repo [01:10:25] in general, if it is failing about stuff in puppet://private/ [01:10:38] then you put some placeholder in labs/private [01:11:15] how I do that? would that require patch against puppet repo? [01:12:03] no mention of any of it in https://wikitech.wikimedia.org/wiki/Trebuchet#Using_Trebuchet_in_Labs [01:12:06] no, it's a separate repo [01:12:22] https://gerrit.wikimedia.org/r/#/q/project:labs/private,n,z [01:12:32] see how other people do similar stuff there [01:12:35] to fix puppet runs in labs [01:12:57] "empty", "dummy" etc [01:12:59] ah... ok, I'll look into it, thanks... looks like it's not that easy to install it [01:13:17] will probably take a break for today and try again tomorrow [01:13:34] i hear you, yea, it's always tedious [01:13:39] ttyl [01:17:31] (03PS1) 10Ricordisamoa: Allow to serialize Element objects as JSON [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/225269 [02:05:48] SMalyshev: The prod and beta cluster deploy hosts are 12.04. tgr is trying to build a 14.04 one and hit at least one problem so far that he was working around (apache 2.2 -> 2.4 config changes) [02:06:13] I'm pretty sure it will fail badly if you try to use jessie [03:00:30] I have an unmerged patch which is dependent on two another unmerged patches. How can I add dependency? [03:02:08] bd808: The tin-esque hosts? I got it mostly working iirc... [03:02:19] When I was working on staging [03:02:29] On jessie? [03:03:08] I thought there was something in trebuchet that didn't work on jessie [03:03:26] at least on the salt-master/deploy server side [03:04:03] tgr hit a problem in the apache config for the trebuchet apache vhost that it doesn't work on apache 2.4 [03:04:21] easy enough to fix, just a problem with the auth syntax change [03:47:51] hello, sorry to bother... still having issues with the cronjob running my bash script as apparently the wrong user (?), it says "ruby: Permission denied -- core.rb (LoadError)" [03:48:27] the cronjob is calling a bash script under trusty, setting the ruby version and running "ruby core.rb" [03:49:01] all was well for quite some time, but this is the 3rd time this has happened, which just as mysteriously fixes itself [03:49:32] no code changes... I think it's the NFS mount changing the permissions, everything is 0777 on my end [03:51:30] any ideas? my bot is responsible for clerking WP:PERM on enwiki, but is a no go right now :( [03:55:17] I guess I should open a phab task? [03:55:46] I feel like maybe this is a Ruby-specific issue... seems like other bots run continuously with no issues [04:06:44] MusikAnimal: that's strange, the cronjob runs as the same user. Can you try it with "ruby -d core.rb" ? [04:07:36] sitic: thank you! I'll try putting that in the bash script that the cronjob calls [04:08:08] the only part missing would be the `-d` [04:08:42] cron will run again in about 1 minute [04:09:07] for the record, I can run the bash script directly and all is fine [04:09:14] the same way the cron runs it [04:16:50] sitic: just checked the logs... looks like the Ruby script successfully ran!! however there is a wealth of new errors still reported at dash.err which is the file the cronjob writes to [04:17:57] looks like arbitrary Ruby exceptions... namely one that reads "No such file or directory @ rb_file_s_stat" [04:20:12] not sure if that is connected, -d is just a debug switch, I would expect it to add all sorts of messages [04:21:53] if it ran successfully the -d change was maybe just coincidental, no good idea what is going on [04:22:06] sitic: dah.. that makes sense as to why there's so much output. Helpful nonetheless [04:22:08] yeah [04:22:21] at UTC 4:20 it did not run [04:22:35] same error... Permission denied -- core.rb (LoadError) [04:22:54] okay [04:23:03] it's not loading Ruby at all [04:23:20] "cannot load such file -- rubygems/defaults/operating_system" [04:23:34] followed by "cannot load such file -- rubygems/defaults/ruby" [04:23:47] along with the other gems I'm requiring [04:24:18] YuviPanda: Bigbrother still isnt restarting things the way it should be [04:24:28] this time I'm sure it's a process that hasn't been dead since hte NFS outage [04:24:32] it was running a few days ago [04:26:49] MusikAnimal: the debug switch is a bit tricky, it prints all raised errors even if caught [04:27:21] sitic: this is still very helpful... thank you, I'm lead to believe this is purely environmental [04:27:31] why would it load Ruby just fine before and not now [04:28:11] the bash script exports the bin directories to PATH [04:28:20] maybe it's a timing issue? [04:28:34] here's the bash script: [04:28:50] export PATH="$HOME/.rbenv/bin:$PATH" [04:28:51] export PATH=$PATH:/data/project/musikbot/.rbenv/versions/2.1.0-dev/bin [04:28:51] rbenv local 2.1.0-dev [04:28:52] cd ~/MusikBot [04:28:54] ruby -d core.rb [04:29:25] ~~~~~ (end of output, sorry for the clutter) [04:30:49] my Ruby script outputs to its own log file, and I see that it ran at 4:10 UTC, which as you say must have been coincidence [04:31:59] hmmm I honestly have no clue what's wrong here [04:32:43] it runs on trusty, if that helps [04:33:05] last time this lasted for about a day before it started working again [04:33:18] that was a few weeks ago [04:34:38] anyway I think I've narrowed down it's not the command ran in the cronjob but something within trusty [04:34:57] because I can run ~/exec.sh and it works fine [04:35:20] whereas in the cronjob it is `jsub -l release=trusty sh ~/exec.sh` [04:38:12] MusikAnimal: can you give it more mem? Like jsub -l release=trusty -mem 2G ~/exec.sh [04:38:53] 2G = 2 gigs? this is a tiny script! I shall try that [04:39:30] miraculously it worked again at 4:30 UTC, still have the same output from the cron though [04:39:33] this is so strange! [04:41:03] just in case, more memory is always better ;-) I think it has to be a small g though, not sure if it accepts G [04:42:15] okay trying that [04:42:22] the cron runs every 10 minutes btw [04:42:33] did not run at 4:40 [04:53:20] hmm the error output moved from dash.err (cronjob output) to exec.err (bash script output) [04:53:40] it's showing similar errors, `cannot load such file -- rubygems/defaults/operating_system` [04:54:35] I think those are still from the -d flag [04:54:43] but did it run succesfully? [04:54:53] no not this time [04:54:54] (03PS1) 10Sitic: Improve translation setup [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/225282 (https://phabricator.wikimedia.org/T101438) [04:55:14] hmm then I'm out of ideas [04:55:57] I keep checking the timestamps with `ls -al` and I see that this time dash.err was last modified at 4:40 and exec.err at 4:50 [04:56:12] anyway... I think I have to sign off for the night. Thank you so much for the help!!! [04:56:36] just going to hope it will run sufficiently over the weekend :) [04:56:38] sure, evene if it remains a mystery [04:56:47] :-) [04:57:32] from New York, g'night! [05:36:42] PROBLEM - Free space - all mounts on tools-webgrid-lighttpd-1404 is CRITICAL tools.tools-webgrid-lighttpd-1404.diskspace.root.byte_percentfree (<30.00%) [06:37:52] 10Tool-Labs-tools-DrTrigonBot---General: DRTRIGON-18 F56 (copied from wiki) - https://phabricator.wikimedia.org/T61585#1459719 (10Xqt) p:5Triage>3Lowest "private" script [06:41:40] RECOVERY - Free space - all mounts on tools-webgrid-lighttpd-1404 is OK All targets OK [06:42:53] 10Tool-Labs-tools-Other: Edits by user results - https://phabricator.wikimedia.org/T106040#1459728 (10Sigma) I fixed it. [07:41:24] 6Labs, 10wikitech.wikimedia.org, 3Labs-Sprint-105: remove nutcracker from wikitech - https://phabricator.wikimedia.org/T102993#1459801 (10Nemo_bis) [08:09:16] 6Labs, 6operations: upgrade salt to 2015.5 - https://phabricator.wikimedia.org/T106074#1459870 (10Joe) I Strongly object to upgrading without a thorough evaluation, we upgraded to 2014.7 for similar reasons and look where it got us. I'm pretty sure 2015.5 has its own bunch of problems. We should first pin dow... [08:09:23] 6Labs, 6operations: upgrade salt to 2015.5 - https://phabricator.wikimedia.org/T106074#1459871 (10Joe) 5Open>3declined [08:11:41] 6Labs, 6operations: upgrade salt to 2015.5 - https://phabricator.wikimedia.org/T106074#1459882 (10MoritzMuehlenhoff) > We should first pin down what problems do we have, maybe work in order to even the zeromq versions across the cluster, I created https://phabricator.wikimedia.org/T106093 for that yesterday. [08:51:29] (03PS2) 10Sitic: Improve translation setup, add qqq.json, rtl support [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/225282 (https://phabricator.wikimedia.org/T101438) [08:58:30] (03PS3) 10Sitic: Improve translation setup, add qqq.json, rtl support [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/225282 (https://phabricator.wikimedia.org/T101438) [08:58:55] (03CR) 10Sitic: [C: 032 V: 032] Improve translation setup, add qqq.json, rtl support [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/225282 (https://phabricator.wikimedia.org/T101438) (owner: 10Sitic) [09:50:34] 10Gerrit-Patch-Uploader, 7Easy: Serve static resources from //tools-static.wmflabs.org or /static/ project - https://phabricator.wikimedia.org/T86354#1459972 (10valhallasw) I'm not sure. I tried changing from local resources to /static/ and I think I couldn't find the right address to use for the chosen sprite... [09:52:40] 6Labs, 10Tool-Labs: missing database on replica server - https://phabricator.wikimedia.org/T105713#1459973 (10jcrespo) Hello, back from being sick. Your table was creating **huge** corruption issues on the replica host, so it was "soft-removed". It should be available on `/srv/tmp` @springle, but I would like... [10:01:28] 6Labs, 10Tool-Labs: missing database on replica server - https://phabricator.wikimedia.org/T105713#1459974 (10Superyetkin) >>! In T105713#1459973, @jcrespo wrote: > @Superyetkin If I create the database and table structure for you, is that ok with you? Yes, restoring the database structure would be of help.... [10:26:28] is there a service that can be used to monitor (time and error logging) cron jobs on a labs instance? [11:39:18] jakob_WMDE: you can get error logs from a cronned job with 2> [11:40:03] or if you need to just base this on return code you can create a simple wrapped such as your_job || echo "`date` failed" > error_log [11:43:11] petan: yes, I was looking for something a bit more powerful that could also send out notifications in case it takes too long or throws errors [11:43:52] you can always script a wrapper for this kind of stuff, there might be something for this but TBH to me it would be easier to write this myself than googling it [11:43:59] it's literally like 3 lines of shell script [11:46:38] petan: alright, I might do that if I can't find anything better. thanks [11:48:14] jakob_WMDE: send message if run longer than 10 secs [11:48:35] timeout 10 "command here" || mail ... [11:48:42] 1 liner [12:15:49] petan: you're probably right. i'll try that out and then look for something more fancy when it's really needed. thanks! [12:22:41] 6Labs, 10Tool-Labs: missing database on replica server - https://phabricator.wikimedia.org/T105713#1460118 (10jcrespo) I've recreated the structure as it was: ``` MariaDB LABS localhost s51698__yetkin > use s51698__yetkin Database changed MariaDB LABS localhost s51698__yetkin > SHOW TABLES; +-----------------... [13:01:36] 10Tool-Labs-tools-Database-Queries: HELP! Database is getting Slow: A test which took less than 10 min, now it takes 3 hours. I cannot do my experiments. - https://phabricator.wikimedia.org/T105964#1460174 (10jcrespo) a:5jcrespo>3None [13:14:46] Hi all [13:14:51] Is it normal that from here: https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-ldf.wikidata-ldf.eqiad.wmflabs [13:14:55] if I click confugre [13:15:15] I get: "The specified resource does not exist." [13:45:01] CristianCantoro: yes, SNAFU; logout and login again [14:15:24] 6Labs, 10wikitech.wikimedia.org: Provide web API for instance creation, deletion and configuration - https://phabricator.wikimedia.org/T106139#1460247 (10scfc) 3NEW [14:44:54] Coren|MX: YuviPanda: any last things you’d like me to do/look at today before I vanish for the week? [14:50:09] andrewbogott: nothing I can think of :) [14:50:19] andrewbogott: actually, would be great if you can fill up https://phabricator.wikimedia.org/T105720 and subtasks with things that are missing? [14:50:27] ‘k [14:51:54] do we have automatic backups yet? If not, what’s blocking? [14:58:46] 6Labs, 10Labs-Infrastructure: labnet1001 is a spof - https://phabricator.wikimedia.org/T106141#1460313 (10Andrew) 3NEW [14:59:35] 6Labs, 10Labs-Infrastructure: labnet1001 is a spof - https://phabricator.wikimedia.org/T106141#1460326 (10Andrew) [14:59:37] 6Labs: Labs team reliability goal for Q1 2015/16 - https://phabricator.wikimedia.org/T105720#1460325 (10Andrew) [15:04:35] 6Labs, 10Labs-Infrastructure: holmium is a spof - https://phabricator.wikimedia.org/T106142#1460338 (10Andrew) 3NEW [15:05:05] 6Labs, 10Labs-Infrastructure: holmium is a spof - https://phabricator.wikimedia.org/T106142#1460346 (10Andrew) [15:05:07] 6Labs: Labs team reliability goal for Q1 2015/16 - https://phabricator.wikimedia.org/T105720#1460345 (10Andrew) [15:06:01] 6Labs, 10Labs-Infrastructure: holmium is a spof - https://phabricator.wikimedia.org/T106142#1460338 (10Andrew) [15:06:02] 6Labs, 10Labs-Infrastructure: labnet1001 is a spof - https://phabricator.wikimedia.org/T106141#1460349 (10Andrew) [15:06:04] 6Labs: Eliminate SPOFs in Labs infrastructure (Tracking) - https://phabricator.wikimedia.org/T105723#1460347 (10Andrew) [15:06:15] 6Labs: Labs team reliability goal for Q1 2015/16 - https://phabricator.wikimedia.org/T105720#1450103 (10Andrew) [15:06:15] 6Labs, 10Labs-Infrastructure: holmium is a spof - https://phabricator.wikimedia.org/T106142#1460338 (10Andrew) [15:06:17] 6Labs, 10Labs-Infrastructure: labnet1001 is a spof - https://phabricator.wikimedia.org/T106141#1460313 (10Andrew) [15:07:56] 6Labs: Make an evacuation plan for labs instances - https://phabricator.wikimedia.org/T106144#1460363 (10Andrew) 3NEW [15:08:31] 6Labs: Labs team reliability goal for Q1 2015/16 - https://phabricator.wikimedia.org/T105720#1460369 (10Andrew) [15:08:32] 6Labs: Make an evacuation plan for labs instances - https://phabricator.wikimedia.org/T106144#1460363 (10Andrew) [15:08:47] 6Labs: Make an evacuation plan for labs instances - https://phabricator.wikimedia.org/T106144#1460363 (10Andrew) [15:08:48] 6Labs: Eliminate SPOFs in Labs infrastructure (Tracking) - https://phabricator.wikimedia.org/T105723#1460371 (10Andrew) [15:10:20] 6Labs: Fix up cold instance migration scheme - https://phabricator.wikimedia.org/T106145#1460373 (10Andrew) 3NEW a:3Andrew [15:12:05] 6Labs: Get instance block-migration working reliably; script and document - https://phabricator.wikimedia.org/T106146#1460380 (10Andrew) 3NEW [15:13:54] 6Labs: Upgrade labs cluster to Trusty - https://phabricator.wikimedia.org/T90821#1460387 (10Andrew) [15:13:54] 6Labs: Get instance block-migration working reliably; script and document - https://phabricator.wikimedia.org/T106146#1460380 (10Andrew) [15:14:44] 6Labs, 6operations, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1460389 (10Andrew) [15:14:46] 6Labs: Get instance block-migration working reliably; script and document - https://phabricator.wikimedia.org/T106146#1460388 (10Andrew) [15:18:51] 6Labs, 10Labs-Infrastructure, 10hardware-requests, 6operations: New server: labdns1001 - https://phabricator.wikimedia.org/T106147#1460393 (10Andrew) 3NEW [15:19:53] YuviPanda: ok, I added some blockers. I’ll work on https://phabricator.wikimedia.org/T106144 a bit today. [15:23:21] andrewbogott: : do you know why high replag is still not monitored here? s3 has currently a lag of more than one day [15:39:22] Merlissimo: I don’t know why it isn’t monitored. I suggest you open a phab task — if you’re able to include simple tests for detecting lag and reasonable bounds it will make it quicker for us to implement. [15:41:00] 6Labs, 10Tool-Labs: missing database on replica server - https://phabricator.wikimedia.org/T105713#1460465 (10Superyetkin) 5Open>3Resolved [15:41:01] Merlissimo: or, rather, open /two/ phab tasks :) One that is ‘fix the replag right now’ and one about monitoring [15:41:41] jynus: Merlissimo is reporting replag, can you follow up? [15:42:30] 6Labs, 10wikitech.wikimedia.org: Provide web API for instance creation, deletion and configuration - https://phabricator.wikimedia.org/T106139#1460475 (10Krenair) It sounds like this should be an #OpenStackManager task, rather than a #wikitech task. Which would be replaced by functionality in Horizon anyway. [15:43:37] well, monitoring is broken in labsdb1002, so I will assume that is where it is happening [15:44:23] jynus: high replag is currently on s3 not s2 [15:44:38] s3 on s3.labsdb [15:45:04] all servers have all shards, that terminilogy is not helpful to me, sorry [15:45:17] investigating [15:50:54] jynus: $ mysql -hs3.labsdb -e "select * from frrwiki_p.recentchanges order by rc_timestamp desc limit 1;" [15:50:58] 20150716125910 [15:52:19] last edit on frrwiki was today on 20150717112648 [16:01:10] 6Labs, 7Database: write irc bot to report high replag of s{1,2,3}.labsdb on #wikimedia-labsdb - https://phabricator.wikimedia.org/T106151#1460526 (10Krenair) [16:03:14] I see it now [16:03:32] it is not a problem with the labs hosts [16:03:49] but with the filtering of sensitive data [16:19:34] 10Quarry: Allow filtering of the final report - https://phabricator.wikimedia.org/T61764#1460572 (10yuvipanda) Wait you can already do this there is a search option in data tables... [16:22:19] Merlissimo, I've just fixed some corruption on a table on s3 [16:22:34] that should improve now [16:28:16] 10Quarry: REPORTS-52 Add option to output report results to a wiki page - https://phabricator.wikimedia.org/T60833#1460588 (10yuvipanda) Hmm, interesting. But this would require bot approval for all wikis? [16:28:29] 10Quarry: REPORTS-52 Add option to output report results to a wiki page - https://phabricator.wikimedia.org/T60833#1460590 (10yuvipanda) p:5Triage>3Low [16:45:41] jynus: thanks! [16:50:15] 6Labs: Make an evacuation plan for labs instances - https://phabricator.wikimedia.org/T106144#1460687 (10Andrew) This blueprint, fixed in Kilo, claims to provide us with a complete evacuation solution: https://blueprints.launchpad.net/horizon/+spec/migrate-all-instances-from-hosts-in-maintenance-mode [17:09:03] hello, I am back again seeking help about why my cronjob that runs a bash script for my bot is having unpredictable environmental issues [17:09:33] the workflow is as follows: cron runs a bash script under trusty, with `jsub -l release=trusty -mem 2g ~/exec.sh` [17:09:46] (the `-mem 2g` was in hopes it would resolve the issue) [17:10:12] next the bash script exports the Ruby bins to PATH, sets the local ruby, then runs the ruby script [17:10:37] I can run the bash script directly and it works every time, but when the cron runs it works sometimes, other times it does not [17:11:01] the error log suggests Ruby isn't getting loaded, e.g. `cannot load such file -- rubygems/defaults/ruby` [17:11:32] then the final error it always ends on is `ruby: Permission denied -- core.rb (LoadError)` but I'm not convinced it's actually a permission issue [17:11:36] core.rb is my ruby script [17:13:11] this is for MusikBot, which is responsible for clerking WP:PERM on enwiki. It's supposed to run every 10 minutes but only successfully runs about every 60 to 90 minutes [17:13:55] Coren|MX: you helped me set up the bash script. Any idea what's going on? Something up with trusty, perhaps? cc YuviPanda [17:14:49] 6Labs, 10wikitech.wikimedia.org: Provide web API for instance creation, deletion and configuration - https://phabricator.wikimedia.org/T106139#1460804 (10scfc) I wanted to aim for the general concept. So … I'll remove #wikitech, leaving #Labs. [17:14:58] 6Labs: Provide web API for instance creation, deletion and configuration - https://phabricator.wikimedia.org/T106139#1460805 (10scfc) [17:34:54] 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1460822 (10MusikAnimal) 3NEW [17:35:41] 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1460830 (10MusikAnimal) [17:36:10] 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1460822 (10MusikAnimal) @Coren, CC'ing you only because you helped me set this up when I first got the bot going. Thank you!! [17:38:06] 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1460837 (10MusikAnimal) [17:39:29] 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1460822 (10MusikAnimal) [17:50:49] [13gerrit-patch-uploader] 15ricordisamoa opened pull request #32: Load jQuery, Bootstrap and Chosen from cdnjs (06master...06cdnjs) 02http://git.io/vm6kX [17:54:04] 10Gerrit-Patch-Uploader, 7Easy: Serve static resources from //tools-static.wmflabs.org or /static/ project - https://phabricator.wikimedia.org/T86354#1460868 (10Ricordisamoa) @valhallasw: https://github.com/valhallasw/gerrit-patch-uploader/pull/32 should work, see https://tools-static.wmflabs.org/cdnjs/ajax/li... [17:55:29] [13gerrit-patch-uploader] 15paladox closed pull request #23: Update and rename jquery.1.6.4.js to jquery.1.11.1.js (06master...06patch-2) 02http://git.io/vm6LO [18:33:22] Hi all, I have a question... how can I request a public IP for my instance/project? [18:51:21] 10Quarry: Add list of query executions to the query page side-bar - https://phabricator.wikimedia.org/T100982#1460951 (10Capt_Swing) Might want to put this in a "history" tab, rather than crowding the query window with a persistent sidebar. [18:51:52] 10Quarry: Add list of query executions to the query page side-bar - https://phabricator.wikimedia.org/T100982#1460953 (10Capt_Swing) p:5Triage>3Normal [18:53:00] 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1460955 (10Ricordisamoa) [18:54:00] 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1403141 (10Ricordisamoa) @Krinkle thanks for adding your own tools! Will you take care of them? [18:57:10] 10Quarry, 6Analytics-Backlog: it would be useful to run the same Quarry query conveniently in several database - https://phabricator.wikimedia.org/T95582#1460961 (10Capt_Swing) p:5Triage>3High [18:58:39] 10Quarry: Allow published query titles to be searched or filtered by tag - https://phabricator.wikimedia.org/T90509#1460975 (10Capt_Swing) Let's start with simple search within 'title' and description field? [18:58:48] 10Quarry: Allow published query titles to be searched or filtered by tag - https://phabricator.wikimedia.org/T90509#1460976 (10Capt_Swing) p:5Triage>3High [19:01:57] [13intuition] 15Krinkle pushed 1 new commit to 06master: 02https://github.com/Krinkle/intuition/commit/4bfeb32121232af71136ec5bb960328d28480370 [19:01:58] 13intuition/06master 144bfeb32 15lokal-profil: Add messages for dcatap... [19:02:20] [13intuition] 15Krinkle closed pull request #50: Add support for dcatap (06master...06dcatap) 02https://github.com/Krinkle/intuition/pull/50 [19:03:25] 10Quarry, 10Tool-Labs-tools-tsreports: Quarry-TSreports feature parity - https://phabricator.wikimedia.org/T78549#1460992 (10Capt_Swing) possibly use Jinja? [19:04:06] 10Quarry, 10Tool-Labs-tools-tsreports: Quarry-TSreports feature parity - https://phabricator.wikimedia.org/T78549#1460995 (10Capt_Swing) p:5Triage>3Normal [19:06:34] 10Quarry: Recurring queries - https://phabricator.wikimedia.org/T101835#1460999 (10Capt_Swing) p:5Triage>3Normal [19:07:25] 10Quarry: Recurring queries - https://phabricator.wikimedia.org/T101835#1348993 (10Capt_Swing) Have recurring queries: to start, specify that a query can be run weekly or monthly, etc. [19:08:53] 10Quarry: Allow published query titles to be searched or filtered by tag - https://phabricator.wikimedia.org/T90509#1461012 (10Capt_Swing) p:5High>3Normal [19:09:49] 10Quarry: Make "Home" navlink go to profile for logged-in users. - https://phabricator.wikimedia.org/T85175#1461013 (10Capt_Swing) p:5Triage>3Normal [19:10:59] 10Quarry: Allow comments on queries - https://phabricator.wikimedia.org/T71543#1461016 (10Capt_Swing) [19:11:19] 10Quarry: Add 'download as SQL' option - https://phabricator.wikimedia.org/T71191#1461021 (10Capt_Swing) [19:11:46] 10Quarry: Build a schema explorer - https://phabricator.wikimedia.org/T71192#1461023 (10Capt_Swing) [19:12:08] 10Quarry: Show the execution time in the table of queries - https://phabricator.wikimedia.org/T71264#1461025 (10Capt_Swing) [19:13:07] 10Quarry: Provide a way to hyperlink texts - https://phabricator.wikimedia.org/T74874#1461043 (10Capt_Swing) related to T78549 [19:14:27] 10Quarry: Add a list/table of popular queries - https://phabricator.wikimedia.org/T71266#1461053 (10Capt_Swing) p:5Triage>3Normal [19:17:04] 10Quarry: Replace spaces with underscores for wiki usernames in URL - https://phabricator.wikimedia.org/T72166#1461064 (10Capt_Swing) p:5Triage>3Low [19:18:33] 10Quarry: Make the table sortable - https://phabricator.wikimedia.org/T71265#1461067 (10Capt_Swing) p:5Triage>3Lowest [19:19:30] 10Quarry: Make the table sortable - https://phabricator.wikimedia.org/T71265#1461072 (10Capt_Swing) @yuvipanda and I think that T90509 will cover most of these use cases, plus some others. So we'll keep this as a lower priority for now? [19:21:12] 10Quarry: Categorizing the queries - https://phabricator.wikimedia.org/T76127#1461077 (10Capt_Swing) p:5Triage>3Lowest [19:21:49] 10Quarry: Categorizing the queries - https://phabricator.wikimedia.org/T76127#1461087 (10Capt_Swing) @yuvipanda and I think T90509 will solve this (user can put #tags in query description, and search for them in the search interface). [19:24:28] 10Quarry: Database dump for analysis - https://phabricator.wikimedia.org/T93907#1461092 (10Capt_Swing) p:5Triage>3Normal [19:24:51] MusikAnimal: try running with -cwd /shot in the dark [19:25:31] or not. That probably wouldn't change anything [19:25:47] MusikAnimal: do you have the script in a repo somewhere? [19:25:47] 10Quarry: Raise query limits - https://phabricator.wikimedia.org/T74342#1461098 (10Capt_Swing) @halfak does the 20 minute limit solve this? [19:26:14] for MusikBot? it's at https://github.com/MusikAnimal/MusikBot [19:26:15] 10Quarry: Raise query limits - https://phabricator.wikimedia.org/T74342#1461103 (10Capt_Swing) p:5Triage>3Normal [19:26:24] the issues come and go without changes to the script [19:26:35] and everything was stable for weeks with the same codebase [19:28:26] MusikAnimal: side note: CC probably isn't the best for code [19:28:41] CC? [19:29:04] Creative Commons [19:29:10] CC-BY-SA [19:29:32] could you pastebin or gist the script? [19:31:13] 10Quarry, 6Discovery, 10Labs-Other-Projects, 10Wikidata, and 2 others: Setup sparqly service at https://sparqly.wmflabs.org/ (like Quarry) - https://phabricator.wikimedia.org/T104762#1461107 (10Capt_Swing) [19:32:02] MusikAnimal: ^ [19:32:56] how about GNU general public v2.0? [19:33:02] I'll get you a pastebin [19:33:40] I'm ok with the code being in the open domain or whatever [19:34:08] anyway, is GitHub not sufficient? here's `core.rb` which is the file that gets run by the bash script: https://github.com/MusikAnimal/MusikBot/blob/master/core.rb [19:34:31] it runs this file `https://github.com/MusikAnimal/MusikBot/blob/master/perm_clerk.rb` [19:35:16] so the bash script just runs core.rb with ruby? [19:35:49] Negative24: see https://phabricator.wikimedia.org/T106170 [19:35:59] that first code block at the top is the bash script [19:36:37] Coren|MX helped me get this up, since it's on trusty we needed to export the bin directories to PATH, and set what version of Ruby to use [19:39:38] Negative24: thank for looking into this, by the way! [19:39:42] np [19:40:12] MusikAnimal: at first look it seems that the script is running as another user that can't execute the binaries in your home [19:40:31] that was my guess initially [19:40:40] but sometimes it works [19:40:57] oh yes [19:41:04] that makes it interested [19:41:07] *-ing [19:41:23] it last ran at 18:50 UTC [19:41:40] supposed to be every 10 minutes [19:42:12] 10Quarry: Search or filter queries by title or summary - https://phabricator.wikimedia.org/T90509#1461148 (10Capt_Swing) [19:42:36] I'm not sure if this is the right place to ask this question, running a self join on revision even on very small wikis is taking a very long time, any idea what I'm missing? [19:42:48] Negative24: I was told the cron should run the bash script as the same user, musikbot [19:44:00] MusikAnimal: gah! sourceforge is still down which is where the grid engine docs are [19:44:40] bummer [19:44:44] 10Quarry, 10Tool-Labs-tools-tsreports: Quarry-TSreports feature parity - https://phabricator.wikimedia.org/T78549#1461156 (10Capt_Swing) Making this the 'Epic' ticket for all report-related tickets, which we @valhallasw can either close as duplicates, or they can just sit in the backlog for now. [19:47:21] 10Quarry, 10Tool-Labs-tools-tsreports: Quarry-TSreports feature parity - https://phabricator.wikimedia.org/T78549#1461158 (10Capt_Swing) [19:47:24] 10Quarry: Provide a way to hyperlink texts - https://phabricator.wikimedia.org/T74874#1461157 (10Capt_Swing) [19:48:26] 10Quarry: Add a stop button to halt the query - https://phabricator.wikimedia.org/T71037#1461161 (10Capt_Swing) @yuvipanda does reloading the page stop the query? [19:48:51] 10Quarry: Add a stop button to halt the query - https://phabricator.wikimedia.org/T71037#1461164 (10yuvipanda) Nope, does not. Submitting a new query stops the old one though. Am not sure if this is strictly necessary... [19:50:16] hey Jeph_paul :) [19:50:23] have you tried using revision_userindex? [19:51:15] No I'm not using revision_userindex, http://quarry.wmflabs.org/query/4457 [19:51:51] do I need to? I'm not refering to the user in anyway [19:52:26] MusikAnimal: maybe the previous job isn't exiting in time for the next job so some files are still locked [19:53:14] hmm I don't think so, the script finishes in under a minute [19:53:58] 10Quarry: Only load 'head' of result set - https://phabricator.wikimedia.org/T71076#1461173 (10Capt_Swing) @halfak now that Quarry paginates results, with 100 per page, is this resolved? [19:55:13] Jeph_paul: hmm, not sure [19:55:18] * YuviPanda doesn't know much SQL [19:55:31] Jeph_paul: try it anyway? :) [19:55:39] Trying it now [19:55:40] halfak: ^ someone with SQL questions from quarry, if you can help... [19:56:11] o/. [19:56:15] * halfak reads [19:56:52] Jeph_paul, checking it out [19:57:43] thanks :-) [19:58:31] Jeph_paul, your where clause. [19:58:40] It's going to scan the entire table slowly. [19:59:00] I was under the impression that rev_deleted > 0 wasn't available in the public DBs. [19:59:24] I tried it without the where clause too, http://quarry.wmflabs.org/query/4462 [19:59:55] Jeph_paul, cool. Checking that out too. [20:00:50] MusikAnimal: is there a reason you're using rubygems 2.1.0-dev? [20:01:13] some bug reports are hinting at a SSL bug that's fixed in 2.2.2 [20:01:25] https://phabricator.wikimedia.org/T96261 [20:01:48] wait that doesn't say anything about -dev [20:02:07] Jeph_paul, looks like you found something interesting. [20:02:11] no there it is, in the comments [20:02:31] When I run it on the internal DBs, the query takes 1.2 seconds for kgwiki [20:02:34] Negative24: it complained about "mkmf" missing when I tried to install the non-dev version [20:02:41] I can try upgrading to 2.2.2 though [20:03:21] 10Quarry: Raise query limits - https://phabricator.wikimedia.org/T74342#1461189 (10Halfak) Yes. This was a badly specified bug. The query limit has been raised as requested, but I still can't really run this query. Let's mark this resolved and I'll look into why the query takes so long before I re-file. [20:03:29] 10Quarry: Raise query limits - https://phabricator.wikimedia.org/T74342#1461190 (10Halfak) 5Open>3Resolved [20:04:30] is it got to do with quarry ? or the public replica? [20:04:45] Jeph_paul, I suspect the replica. [20:04:58] Here's the query plan I get on the analytics team DBs: https://gist.github.com/halfak/f2488cd87431f32e9427 [20:05:10] It looks like it's simply doing a primary key lookup. [20:05:17] Which is *very* fast for 39k revisions. [20:05:19] 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1461193 (10Ricordisamoa) I'd like to hear from #WMF-Legal about images like https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png [20:05:20] So... [20:05:32] YuviPanda, ^ thoughts? [20:06:08] revision_userindex is no help [20:07:35] halfak: not sure at all [20:07:36] ... [20:07:48] Woah. Got something weird. [20:07:49] Jeph_paul: can you try on a different wiki that's not kgwiki? [20:09:19] I tried it on a smaller wiki hz, same thing happens, http://quarry.wmflabs.org/query/4462 [20:09:30] MusikAnimal: could you change the crontab to run jsub with the -once option as well? [20:09:49] only 4421 rows in revision [20:10:04] YuviPanda: does tools' grid engine restart jobs if they fail? [20:10:13] ^ with jsub [20:11:38] Jeph_paul: halfak can you file a bug? I can ask the DBAs to take a look [20:11:44] Negative24: if they are set to be continuous jobs, yes [20:12:18] YuviPanda: but without -continuous? [20:12:23] no it does not [20:12:56] I just ran the querry logining into toollabs & it came fine [20:13:20] Jeph_paul: oh, I see. [20:13:23] YuviPanda, working on translation stuff. [20:13:26] halfak: ok! [20:13:26] Maybe Jeph_paul can file :) [20:13:39] Jeph_paul: can you try: [20:13:45] run 'sql enwiki' [20:13:48] then run your query? [20:13:52] Negative24: so `jsub -l release=trusty -once ~/exec.sh` ? [20:13:53] (including the use kgwiki) [20:14:29] ok, doing it now [20:15:00] MusikAnimal: yeah [20:16:06] done, 4 minutes until next cron run [20:17:07] I can't run 'sql enwiki' and then use kgwiki, it say user not ...... [20:17:30] if I directly run kgwiki, the result comes fine [20:17:47] MusikAnimal: sounds good [20:19:58] So shall I file a bug saying self joins on revision take forever? [20:20:14] on quarry [20:21:49] Negative24: no dice, same errors [20:22:03] MusikAnimal: hmm [20:22:04]