[01:15:42] <andrewbogott>	 Coren: I'm trying to figure out how to edit the sidebar on wikitech… I feel like I should be editing something like MediaWiki:Sidebar/Group_projectadmin but in fact wikitech happily serves any possible MediaWiki:Sidebar/Group_<anything here> as the exact same content.
[01:16:00] <legoktm>	 isn't it just mediawiki:sidebar?
[01:16:26] <Coren>	 andrewbogott: IIRC there is some black magic coming from the Openstack extension itself.
[01:16:26] <legoktm>	 https://en.wikipedia.org/wiki/MediaWiki:Sidebar
[01:16:38] <Coren>	 andrewbogott: I think we'll have to lookit da sourcez.
[01:16:46] <andrewbogott>	 Coren: I am, and it doesn't help… 
[01:16:57] <andrewbogott>	 I mean, there's a callback that determines which sidebars to display
[01:17:08] <andrewbogott>	 but not the content of those sidebars.  that, I think, has to be on a wiki page someplace.
[01:17:21] <Coren>	 You're sure it's just /which/ sidebars that the callback handles?
[01:17:33] <andrewbogott>	 legoktm: look for yourself, all the interesting stuff is just called 'GROUP-SIDEBAR'
[01:17:45] <legoktm>	 ah
[01:17:45] <legoktm>	 i see
[01:17:53] * legoktm  has no clue :)

[01:18:02] <andrewbogott>	 Coren, pretty sure… $roles = $user->getRoles(); $groups = array_merge( $groups, $roles );
[01:18:05] <andrewbogott>	 can that be adding content?
[01:18:34] <Coren>	 Not in a way that'd make sense no.
[01:18:58] <andrewbogott>	 I think what I need is someone who has used dynamic sidebar and GROUP-SIDEBAR before.
[01:19:05] <andrewbogott>	 For which, I am probably in the wrong channel.
[01:19:13] <Coren>	 Probably.  :-)
[01:19:52] <andrewbogott>	 doesn't really seem like a -dev question...
[01:26:44] <andrewbogott>	 Ah!  there is something at https://wikitech.wikimedia.org/wiki/MediaWiki:Sidebar/Group:projectadmin
[01:26:53] <andrewbogott>	 weird that when I make up URLs it returns something rather than nothing
[01:33:44] <Coren>	 Mediawiki space is odd that way.
[01:52:59] <Atethnekos>	 I was wondering if anyone who knows about hosting php scripts on Labs could give me some advice?
[02:34:17] <gry>	 Atethnekos: hi.
[02:34:53] <gry>	 Atethnekos: 'php' is an available command, and public_html supports php; if the former, use the grid engine to submit the script as a job.
[02:44:03] <Atethnekos>	 I mean, far more general advice
[02:44:27] <Atethnekos>	 There was a tool that was hosted offsite which is used by a few templates on en.wiki which is now done
[02:44:40] <Atethnekos>	 down rather
[02:44:54] <Atethnekos>	 I'm not a programmer.
[02:46:36] <legoktm>	 which tool?
[02:52:50] <Atethnekos>	 Sorry. It is described here https://en.wikipedia.org/wiki/Template_talk:Bibleverse#Bibleverse_template_broken
[02:53:12] <Atethnekos>	 The code seems to be all here: http://code.google.com/p/codetestmap/source/detail?r=240
[02:53:29] <Atethnekos>	 It used to be hosted at http://bibref.hebtools.com/
[02:54:02] <Atethnekos>	 This Bibleverse template and other related ones (bibleref) are used on around 10,000 pages I believe, but now they don't work.
[03:00:18] <Atethnekos>	 Based on my limited understanding it should be just a matter of moving the bibleversefinder.php and the two .txt files into an accessible directory.  Then the address for the templates can be changed. But I don't know if this is something that Labs could do or not?
[03:01:12] <mutante>	 Atethnekos: there is "Tool Labs" and "Labs", Tool Labs is a subset of it, and yes, should be able to do that
[03:03:13] <Atethnekos>	 I see the en.wiki page should be updated which conflates the two together.
[03:03:49] <mutante>	 Atethnekos: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help  
[03:04:49] <mutante>	 since you say you're not a programmer, maybe you can find a "sponsor" to help you get your script up or so ?
[03:05:49] <mutante>	 but if it's true it's just a .php and 2 text files and it's in use on projects it should be fine
[03:06:16] <Atethnekos>	 Well, I don't mind a challenge.  Maybe I'll try applying for tools access and give it a go that way?
[03:06:37] <Atethnekos>	 Yeah, it should just be moving the three files over.
[03:07:02] <Atethnekos>	 Unless you advise I don't get involved since I don't really know what I'm doing.
[03:07:28] <mutante>	 if you dont mind a challenge, go ahead :)
[03:07:41] <mutante>	 and apply, make a wiki page describing your tool
[03:08:05] <mutante>	 then get a php dev to look over the code
[03:09:01] <Atethnekos>	 Great, thanks for the encouragement.  I'll start with getting an SSH key!  I've done that before when I hosted some webpages at my University.
[03:09:19] <mutante>	 :) welcome
[03:29:22] <Coren>	 Atethnekos: Welcome!  Also beware:  you'll find making tools to be painfully addictive.  :-)
[03:55:27] <Atethnekos>	 Thanks, Coren!  I'm just waiting for approval; hint hint wink wink https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Atethnekos
[04:01:23] <mutante>	 Coren: tried to help him out. didnt do this for a while  Failed to add Atethnekos to tools. This needs user Atethnekos to have the "loginviashell" right. 
[04:01:50] <Coren>	 mutante: You need to give him the 'shell' userright first.
[04:02:09] <Coren>	 Sorry, I'm having a movie watching night with my SO.  :-)
[04:02:45] <mutante>	 Coren: thanks, run for movie! 
[04:06:36] <mutante>	 Atethnekos: https://wikitech.wikimedia.org/wiki/Special:Log/rights
[04:06:59] <mutante>	 and Successfully added Atethnekos to tools. 
[04:07:27] <Atethnekos>	 Thanks so much mutante!
[04:08:00] <mutante>	 yw, i'll also head off for now though, hope it works well with the next steps
[04:08:10] <mutante>	 otherwise...ping the channel. and cya 
[04:09:48] <Atethnekos>	 thanks again.
[05:52:25] <Atethnekos>	 Works perfectly, thanks mutante and Coren.
[05:52:57] <Coren>	 Atethnekos: I'm glad the documentation was clear enough.
[10:00:31] <giftpflanze>	 Coren: remember the dead welink checking i didn't want to do in the grid. turns out to be not doable the way i wanted and i have to use the grid now. to have an acceptable speed we probably will have to increase the grid capacity
[10:00:48] <giftpflanze>	 /grid./grid?/
[12:06:07] <giftpflanze>	 aw, mass inserting into redis doesn't work for me :(
[12:06:15] <giftpflanze>	 fuck it
[12:33:49] <giftpflanze>	 Coren: can we install redis-cli?
[12:50:08] <giftpflanze>	 ok, works now, forget redis-cli
[13:32:11] <alkamid>	 in my tool's folder (alkamidbot/scripts/output) there is one file with odd permissions: -rw-r--r-- 1 alkamid          wikidev            14957 Nov 21 18:42 visits.txt
[13:32:15] <alkamid>	 I can't "take" it as alkamidbot
[13:32:25] <alkamid>	 it says "visits.txt: You need to share a group with the file"
[13:33:32] <alkamid>	 I can't chown it as alkamid: chown: changing ownership of `/data/project/alkamidbot/scripts/output/visits.txt': Operation not permitted
[14:56:17] <giftpflanze>	 Coren: do we have a redis connection timeout set? if so, why??
[16:24:09] <Coren>	 giftpflanze: Not that I know of.
[16:38:23] <giftpflanze>	 Coren: can we add more nodes to the grid?
[16:38:55] <Coren>	 giftpflanze: Yeah, no reason we couldn't.  Are you hitting job delays?
[16:39:25] <giftpflanze>	 i want to start an array job, only 26 are executed in parallel
[16:47:17] <giftpflanze>	 Coren: do i have to ping you when i answer you?
[16:49:09] <Coren>	 Ideally.  It's saturday and I'm doing other things in the house and not always at my computer.  You'll hit into the per-user limit if you do this on the normal grid; what I can do instead is give you a dedicated node with the adequate resources for that task.
[16:49:51] <giftpflanze>	 Coren: that would be great
[16:51:18] <Coren>	 giftpflanze: open a bz with details (number of expected parallel jobs, resource requirements).  Will you need the actual grid engine parallel job support or are all the jobs independent?
[16:51:43] <giftpflanze>	 um
[16:53:09] <giftpflanze>	 Coren: what does actual grid engine parallel job support give me?
[16:55:17] <Coren>	 If you don't know about it you don't need it.  It's for running jobs that need message passing for synchronization (such as with http://www.open-mpi.org/).  Unless you have very complex parallelization and coded it that way, you don't need it.
[16:55:39] <giftpflanze>	 ok, don't need it
[17:02:09] <giftpflanze>	 Coren: the more resources the better, is it ok then if i request a m1.xlarge instance for the node?
[17:02:45] <Coren>	 giftpflanze: I'm probably not going to okay an xlarge for a single tool unless you make a damn good case that you need all that ram!  :-)
[17:03:19] <giftpflanze>	 coren: when i was testing it in my own project it was all needed
[17:03:47] <Coren>	 I'm sure it was sufficient, but was it necessary?  :-)
[17:03:53] <giftpflanze>	 hm, well, if it isn't stored in memory but written to disk it isn't needed
[17:04:20] <giftpflanze>	 but i can make more web requests at the same time
[17:04:48] <Coren>	 Sure, but you realize than an xlarge is 16G of ram?  How many requests do you want to /do/?
[17:05:04] <giftpflanze>	 some thousand
[17:05:34] <Coren>	 ... wait woah.  What *are* you doing exactly?
[17:05:57] <giftpflanze>	 i check ~5m urls for brokenness
[17:06:54] <Coren>	 Fair enough, but how often do you expect to be doing that check?
[17:07:05] <giftpflanze>	 every two weeks
[17:07:44] <Coren>	 That's less than 300/min
[17:09:19] <giftpflanze>	 ok, then i guess everything as it is is enough
[17:09:39] <Coren>	 make it 500/min if you want your run to have some elbow room, though.
[17:09:55] <Coren>	 and then double that for safety.  :-)
[17:10:01] <giftpflanze>	 heh :)
[17:10:13] <giftpflanze>	 timeout is 200 sec
[17:10:29] <Coren>	 So realistically, you need to be able to handle ~1000 web requests by minute.
[17:11:08] <Coren>	 Sure, but statistically timeouts are going to be rare compared to connection refused, 404s and such.
[17:11:18] <giftpflanze>	 right
[17:11:59] <Coren>	 So I'd plan for 10s per connection as a reasonable median.
[17:12:54] <Coren>	 So if you want to make sure you can do 1000/min on average, and you can expect 6 per minute per thread or so, then you only need 166 parallel connections.
[17:13:07] <Coren>	 Make it 200 and you've added another safety margin.
[17:13:25] <Coren>	 And 200 won't need that much resources.  :-)
[17:16:00] * Coren  ponders.

[17:16:19] <Coren>	 Wait, why do you even need to parallelize this?
[17:16:34] <Coren>	 That sounds perfectly doable with a single job.
[17:16:55] <giftpflanze>	 except that it runs months
[17:17:42] <giftpflanze>	 with our calculation 2 years …
[17:17:43] <Coren>	 It shouldn't.  We just calculated that 200 parallel web connections suffices to do 5 million links in ~ 5 days.
[17:21:34] <giftpflanze>	 Coren: should i do that with threading in my own project or in a single job on the grid, for the second scenario i think the memory isn't enough
[17:22:17] <giftpflanze>	 also threading is so prone to errors
[17:22:29] <Coren>	 giftpflanze: That's probably best.  Make a pool of worker threads handling exactly one connection asynchronously.
[17:23:06] <Coren>	 giftpflanze: It's not so bad if what threads to is really really self contained.  The only thing they should do is "pick an URL from the list, fetch the contents, store the result on a list"
[17:23:31] <giftpflanze>	 about a third of results get dropped in my tests
[17:23:44] <giftpflanze>	 if stored in lists memory usage gets huge
[17:24:04] <Coren>	 giftpflanze: You don't need to have both lists populated completely.
[17:24:28] <Coren>	 giftpflanze: Have a thread refill the "pending" list when it gets below a certain level.
[17:24:56] <Coren>	 giftpflanze: What's your normal proportion of 'good' links that won't need any action?
[17:25:22] <giftpflanze>	 i have a thread pool with a maximum of threads, that gets filled by a loop from file
[17:25:41] <giftpflanze>	 let me look …
[17:27:21] <giftpflanze>	 ~19% of urls are broken
[17:27:52] <giftpflanze>	 but i need to know if a bad url turns good in a later run
[18:59:17] <anomie>	 Coren: What is the per-user limit on the grid?
[18:59:26] <Coren>	 anomie: 12 jobs, iirc
[19:00:09] <anomie>	 Coren: And if it's exceeded, the extra jobs wait for one of the others to exit?
[19:00:37] <Coren>	 They should.
[19:03:45] <giftpflanze>	 they don't for me
[19:03:51] <anomie>	 I suppose they do. With the change I made to get rid of AnomieBOT's "updater" job, I have 9 continuous jobs and occasionally submit up to 8 task jobs, and all the task ones seem to get run.
[20:56:31] <alkamid>	 Hi. I'm experiencing different behaviour of my script on login and exec servers: http://pastebin.com/48VrD1Cj
[20:56:45] <alkamid>	 This is basically looking for the latest dump in the dumps folder
[20:57:32] <alkamid>	 on login, it finds it properly (20131115). On exec, it raises DumpNotFound
[20:59:34] <alkamid>	 oh, and config.path['dumps'] is /public/datasets/public/plwiktionary
[21:06:54] <alkamid>	 surely dumps are mounted on exec servers?
[21:26:29] <alkamid>	 Coren, could you check if dumps folder is properly mounted on all exec servers?
[21:27:18] <alkamid>	 because my script (that check if dump filename exists) sometimes runs properly, and sometimes not
[21:27:55] <alkamid>	 most of the times it doesn't
[21:35:04] <alkamid>	 it seems to be ok on exec-08
[22:04:34] <giftpflanze>	 Coren: can you increase my job limit?
[22:05:53] <giftpflanze>	 they pile up in waiting state