[01:22:56] 3Wikimedia Labs / 3wikistats: Add enciclopedia.us.es and ateneodecordoba.org URLs to stats - 10https://bugzilla.wikimedia.org/68398#c3 (10Daniel Zahn) Alejandro, so the first wiki is already added, would be duplicate: Duplicate entry 'http://enciclopedia.us.es/api.php' for key .. and the second wiki doesn... [07:19:45] the current puppet provisioning command is sudo puppet apply right ? [09:51:27] 3Wikimedia Labs / 3wikistats: Add enciclopedia.us.es and ateneodecordoba.org URLs to stats - 10https://bugzilla.wikimedia.org/68398#c4 (10Alejandro Sánchez Marín) API enabled on ateneodecordoba.org [12:11:52] tonythomas: sudo puppet agent -tv [12:13:11] YuviPanda|zzz: thanks ! [12:14:47] Yw [14:15:22] Dispenser: For God's sake! pls. run your stuff on the grid. You're blocking tools-login again. https://tools.wmflabs.org/paste/view/593e799c [14:16:03] that's still running?\ [14:16:18] Dispenser: yes [14:17:08] Guess we're gonna have to find a better way of indexing these thumbnails [14:22:16] Anyone interested in developing an image mosaic tool? [14:34:19] hedonil: How many IP address are allocated to the GRID? [14:34:44] public IPv4 addresses* [14:35:07] Dispenser: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Overview#Sun_Grid_Engine_.28SGE.29 [14:35:48] Any way I can use all 12? [14:36:28] Dispenser: if you have a job on each node - probably yes [14:39:31] Last time I tried the GRID on the Toolserver it was impossible to make it useful, documentation was shit, and long-term debugging was impossible. Have thing notably improved since then? [14:41:54] Dispenser: RTFM and find out? [14:42:34] also 'impossible to make it useful' is awfully unspecific [14:43:11] long-term debugging is marginally more difficult, but not much (after all, even in an interactive session, you would find something typically crashed a while ago) [14:43:16] valhallasw`cloud: I did RTFM of writing on the wiki the most useful configurations -- That how I know its shit [14:43:57] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Submitting.2C_managing_and_scheduling_jobs_on_the_grid [14:45:27] Does not email stderr, instead writing to some fucking log in ~. Impossible to see the output of what's running (e.g. is it actually doing what its suppose to). [14:46:26] The command _appeared_ to work, but were actually silently failing [14:46:52] last time I checked, bash also did not mail stderr :-p [14:47:16] cron does, but they're disabled that :-( [14:47:39] err, no? [14:47:47] did you forget to confirm your wikitech mail address? [14:48:21] Its only for become tool part [14:49:13] There _is_ option to email, but it email stderr + stdout and a fucking header every time. And (IIRC) clogs up the home dir [14:53:01] * valhallasw`cloud feels Dispenser should RTFM. [14:53:09] seriously, clogging op the home dir was a /toolserver/ issue [14:53:21] which hasn't been an issue on tool labs since, well, since tool labs /exists/ [14:54:00] and it's /literally/ in the docs: By default, jsub will (...) print the eventual output to files (‘jobname.out’ and ‘jobname.err’) in your home directory. [14:54:12] so no, no jobname.o1234567's on tool labs [14:55:45] I'm not sure about email; iirc you have to pipe email yourself [15:25:35] Your job ### ("X") has been submitted, but is silent if there's a problem (e.g. no executable) [15:25:52] not executable* [15:27:00] does anyone know why none of my tools seems to be responding? I can't find any error logs for it. [15:27:05] that goes to the err file [15:27:53] in theory [15:27:56] and yet no error file exists [15:28:30] actually I don't think I'm even getting that far. I get the same blank page if I try to access a non-existent URL [15:29:29] gifi, that's were I found it. Its inconsistent. It know already that it wont work and decides to tell me nothing. [15:30:25] Magog_the_Ogre: lighttpd error logs are in error.log [15:30:40] URI? [15:30:53] Magog_the_Ogre ~/error.log [15:31:18] cat: /data/project/magog/error.log: No such file or directory [15:31:22] and you can add debug.log-request-handling = "enable" to your .lighttpd.conf to get more info [15:31:57] * Dispenser hates lighttpd [15:32:13] Magog_the_Ogre: oh, it's throwing a 502 bad gateway, but the message doesn't show up. Did you start your webservice? [15:32:37] Dispenser: if you use jsub, try -stderr? [15:32:37] valhallasw`cloud, it might have terminated for some reason [15:33:19] gifti: I'm just point out something, so I remember it better [15:33:36] if there's no error.log, which suggests that it never started in the first place [15:34:02] and how do I restart the service [15:34:11] webservice restart [15:34:25] what's the tool name, btw? [15:35:20] there we go, that did it valhallasw`cloud [15:37:59] Magog_the_Ogre: ah, and the error.log has appeared. Good :-) [15:38:34] (it's not very informative for php stuff, though, as that will show up in access.log anyway) [16:01:08] tools.irc-cloudbot@tools-login:~/Cloudbot$ screen -U gives: Cannot open your terminal '/dev/pts/183' - please check. [16:01:28] any idea? :p [16:01:41] They disabled screen for tool accounts [16:01:54] you suppose to SGE or some shit [16:02:06] your suppose to use SGE IIRC* [16:02:16] huh [16:02:35] so I have to browse Help... [16:03:22] Good Luck. Haven't been able to understand it for years. And judging by mailing list posts, most of the community hasn't as well. [16:03:43] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Submitting.2C_managing_and_scheduling_jobs_on_the_grid [16:04:23] Thanks for link, Dispenser [16:04:43] I might have to use my private server, since my tool won't work well with it (with quick browse) [16:04:52] in fact, irc bot :p [16:04:56] Want to join up? [16:05:56] join where? :9 [16:07:21] Already running my stuff on my own private server. However, it would make sense to build a community server where we could run tools together, like a toolser... [16:08:37] hmm, I don't think I have a idea to use SGE, so meh :p [16:13:07] compilation terminated: mysql++.h: No such file [16:13:20] Dammit, I really wanted rmytop [16:17:46] Dispenser: If it may help you a bit, run: $ watch sql enwiki -e "show processlist;" - or whatever server you're on [16:18:30] Atleast it doesn't flash like mytop [16:19:32] Dispenser: And if this my help you, too: Check your queries with: http://tools.wmflabs.org/tools-info/optimizer.py [16:19:57] That doesn't work for temporary tables [16:20:09] Dispenser: right [16:20:43] Dispenser: wait... really? [16:21:25] I found out I'm mostly getting killed by latency between my server and WMF databases [16:21:58] They exist per connection and loaded in a strange way (LOAD DATA INFILE) [16:22:24] Dispenser: It's online execution, so it /should/ work - but I nerver tried out before [16:23:32] Dispenser: of course, per connection - but IIRC: LOAD DATA INFILE doesn't work at all [16:23:57] it requires a special right, which "normal" users don't have [16:24:33] oursql.Connection(local_infile=True) [16:26:40] Dispenser: could you stop spewing nonsense? [16:26:46] screen is not blocked for tool accounts [16:27:02] Then why can't I use it? [16:27:57] because it /never/ works through sudo, unless you use the ugly toolserver hack which logged (!) all i/o to your terminal [16:28:16] it has nothing to do with SGE [16:29:49] So how would Revi use it? [16:30:31] !screenhack [16:30:34] hmm. [16:30:38] it's indeed not in the manual :-p [16:30:45] iirc script /dev/null works [16:31:30] http://bots.wmflabs.org/~wm-bot/dump/%23wikimedia-labs.htm [16:31:30] @info [16:32:04] https://wikitech.wikimedia.org/wiki/Screen [16:35:13] Thanks valhallasw`cloud [16:35:36] Though I turned off my laptop few minutes ago and almost bed :p [16:41:47] Dispenser: in case you're no confusing read_default_file with local_infile, just discard "local_infile=True" from connection [16:43:33] I found out I'm mostly getting killed by latency between my server and WMF databases [16:43:40] Dispenser: err, 'my server'? [16:43:46] you mean tools-dev or tools-db? [16:43:53] Its a security risk enabling local-infile=1, plus that .my.cnf could be nuked by an update script [16:44:00] dispenser.homenet.org [16:44:34] Dispenser: what do you mean by 'WMF databases' and how are you accessing them? [16:44:43] ssh tunnel [16:45:08] hmm, I'm unsure if that is ok with Privacy Policy / ToU. I think it should, but let me check anyway. [16:45:26] Coren: confidential ping ;) What's your opinon on json.cookie assisted directory for Tools (Labs) ? [16:46:34] Coren: If you issue the blessing on this, we could set up some lines of POC code [16:46:37] hedonil: confidential pong? :) what do you mean by json.cookie assited directory [16:46:52] Are we adding yet another rule because of me? [16:47:09] YuviPanda: the labs-l thing from Hay [16:47:25] hedonil: aaaah, right. I was thinking of directories as a directory on the FS and was confused :) [16:48:00] YuviPanda: yeah, kind of pimp'd LDAP for labs [16:48:15] Dispenser: *shrug* I think ssh tunnel to db access is a good thing, but I would like to get that clarified. [17:06:39] YuviPanda: can you do me a favour and paste me some sample output lines (~50) from $ ldapsearch -x 'ou=servicegroups' ? [17:07:02] * hedonil is an ldap paria [17:47:15] hedonil: heh, me too. [17:47:26] I think, at least? [17:47:27] let me try [17:48:14] hedonil: 'text: You do not have sufficient privileges to perform an unindexed search' [17:48:42] YuviPanda: :/ - single queries work fine ldapsearch -x 'cn=tools.xtools' [17:49:05] hedonil: yeah, I guess you need to be 'real' ops for that to work [17:49:12] hedonil: although I think that should be accessible to most [17:49:15] YuviPanda: grrrit-wm [17:49:23] hedonil: hmm? [17:49:33] YuviPanda: ??? typo !! [17:49:39] hehehe [17:49:43] ;-) [17:49:46] hedonil: "ldaplist -l servicegroups" AFAIR. [17:50:00] * hedonil checks [17:50:28] scfc_de: ahhh! thanks [17:52:55] hedonil: now we know whom to poke for ldap queries ;) [17:54:16] YuviPanda: yeah [17:55:14] scfc_de: this is perfect, afaics there are all servicegroups, including tools + projects [17:56:19] that's ecatly was I was looking for ;-) [17:56:28] x [17:57:19] *xac [17:57:24] heh [17:59:15] * YuviPanda writes JavaScript [18:01:32] How do I list SGE's -l requirement resources thingies [18:02:46] scfc_de: YuviPanda: best thing: ldaplist is no binary like ldapsearch, but a python script [18:03:23] Dispenser: qconf -sc [18:03:23] Dispenser: I'm not sure whether that's available to all users though [18:03:36] Dispenser: Ah, it is. [18:03:46] hedonil: does that call into ldapsearch in the end? [18:04:16] YuviPanda: import ldap [18:04:21] aaah cool [18:06:47] YuviPanda: this should make a /comprehensive/ script/web supported directory, pimp'd with custom json cookies, a cakewalk [18:06:53] niice [18:07:13] Coren: ^ [18:09:31] hedonil: If you want/need to stick to ldapsearch, you can also query à la modules/toollabs/files/maintainers: "/usr/bin/ldapsearch -LLL -x -b "ou=servicegroups,dc=wikimedia,dc=org" member" [18:10:41] scfc_de: great, even moar possibilities! [18:11:34] am trying out Atom to replace Vim [18:15:15] ..and adding a handful of lines will even provide I18-support [18:17:34] is a wmf labs staff member currently here? [18:19:14] se4598: what for? Coren was speaking a while ago, and I will be 'wmf labs staff member' in a couple months... [18:20:08] YuviPanda: privacy concerns [18:20:59] se4598: ah, hmm. I suggest you email mpelletier@wikimedia.org and also cc me? (yuvipanda@wikimedia.org) [18:53:59] se4598: thanks! [19:36:00] Thanks Coren. You wouldn't happen to know the way of forcing 5 jobs to be 5 different servers (i.e. IP addresses)? [19:37:00] Dispenser: There's no way of doing that specifically that I know of (it's rare enough a scenario, I think, that there is no general mechanism). You /could/ request specific hosts but then you'd be at the mercy of scheduling. [19:37:22] Dispenser: why do you want 5 different hosts? [19:37:40] Plus server availability changes [19:38:31] because upload.wikimedia.org idiotically restricts requests to 2.4 per second (for each IP) [19:39:19] I see [19:40:11] Dispenser: I'm pretty sure that anything that needs to hit upload.wm.o that hard should probably be discussed further to find a better mechanism. [19:40:48] (idiotic because 20 req/sec wasn't even noticed and single browser users with standard pipelining of 6 these days hits it [19:41:52] Dispenser: that reasoning makes no sense. It makes batch downloading large amounts of files harder, not impossible [19:42:37] *of course* 20 req/sec is minimal compared to the normal load, that's not the point [19:43:04] I'm rather weary of discussions, often they take longer than the task (3-4 months) and I'm not very good in them [19:44:20] If I load up a category page, my browser is hitting 6+ requests/second [19:44:55] Dispenser: then I don't get why you would hit a 2.4/s limit from tool labs [19:45:37] Well now postfix half the images aren't going to display [19:46:26] (anyway, trying to subvert policies by using more IPs is not cool) [23:57:36] Where is there documentation on how to get a "what links here" from the database