[06:12:23] addsleep: thanks :) [07:34:22] !labsdoc [07:34:24] er [07:34:26] !toolsdoc [07:34:31] !tooldocs [07:34:31] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [07:34:34] ty [07:43:18] :o [07:52:33] i can never remember the names for the db servers and stuff [07:52:39] so i wrote a quick library for it [07:52:42] https://github.com/legoktm/wmflabs-lib [07:52:49] mainly https://github.com/legoktm/wmflabs-lib/blob/master/wmflabs/db.py [07:52:58] legoktm you know there is sql command [07:53:10] yes, but not if i'm using python :) [07:53:19] hmm python [07:53:29] anyways [07:53:35] yes [07:53:35] this took care of it :) [07:53:49] and ill add a function to submit jobs to jsub since that's weird too [07:54:48] legoktm: does charset=None mean it returns str instead of unicode? [07:55:04] valhallasw: I'm not sure, that was what was recommended on tswiki. [07:55:15] * legoktm looks [07:55:51] > Extra flags are required for oursql to force binary mode since VARCHAR fields on sql-s2 are mislabeled with latin-1. Manual decoding is required even after upgrading since the fields will be VARBINARY instead. [07:56:20] https://wiki.toolserver.org/view/Mysql#Python [08:04:43] how can I change ownership of files from oren to local-orwell01 ? [08:05:01] OrenBochman use take [08:05:06] become orwell01 [08:05:15] take blablafvd [08:10:19] legoktm: ok, that makes sense [09:11:53] petan is there take for groups ? [09:32:55] Exists a page with sources of the Tools (toolserver, wmflabs)? [09:35:43] OrenBochman no but I will make it soon [09:35:54] Steinsplitter? [09:36:10] sources of what [09:36:42] example: http://toolserver.org/~luxo/contributions/contributions.php?blocks=true&user= [09:36:42] who i can find the source of toolserver/wmflabs tools? [09:41:14] Steinsplitter: by asking luxo [09:41:19] in this case [09:41:23] there is no central repository [09:41:31] mh, okay. thx [09:55:44] valhallasw: asking DaB could work, I guess [09:59:08] https://github.com/Pathoschild/Wikimedia-contrib.toolserver aaaha :) [09:59:13] som tools [10:05:43] fale: yes, but it's not really clear whether source may be shared [10:06:44] * Steinsplitter is testing som tools :9 [10:06:45] :) [10:09:01] valhallasw: is possible that DaB (or any other root) looking at the source sees a license file, and act conseguently ;) [10:13:00] fale: yes, true [10:57:41] petan, Coren, maybe it's an idea to set the default umask for projects to 0002? Then group also can write things. [11:08:47] I set the username and password (mysql) on my pc equal to the toollabs one. As soon as I try logging in it returns me a password mismatch error even if I use copy and paste two times (first to set, second to use) -.- [11:09:30] fale: you're trying to login to the labs SQL servers from home? or...? [11:51:36] is there an api to look things up like name of french article english name of which i already know? [11:51:49] yeah [11:52:05] gry: what article are you starting from? [11:54:23] i translate things a lot and often i end up with input of 'illinois university' kind, which i currently handle by opening an english wikipedia article on that and clicking a french interwiki, copy-pasting article name of that - this means a lot of waiting on page load and manual work to figure out where french is located in the interwiki list [11:55:19] it could be so much easier if there was an api to query for things like this [11:55:43] Betacommand: --^^ [11:56:18] hm, langlinks is broken since wikidata, it seems :/ [11:56:22] gry: can I get a link to an example page> [11:57:05] oh wait, that's because it was a redirect [11:57:07] gry: https://en.wikipedia.org/w/api.php?action=query&prop=langlinks&titles=University%20of%20Illinois%20at%20Urbana%E2%80%93Champaign [11:57:22] https://en.wikipedia.org/w/api.php?action=query&prop=langlinks&titles=University%20of%20Illinois%20at%20Urbana%E2%80%93Champaign&lllang=fr if you only want fr [11:57:56] https://en.wikipedia.org/w/api.php?action=query&prop=langlinks&titles=Illinois_University&redirects=&lllang=fr if you want to follow redirects [11:57:59] valhallasw: you beat me to it :P [11:59:23] Betacommand: http://en.wikipedia.org/wiki/Illinois_University [11:59:35] gry: see valhallasw's link [11:59:41] valhallasw: ok [12:21:49] http://jsfiddle.net/gryllida/M9Qzd/ why doesn't this work then? [12:23:17] https://en.wikipedia.org/wiki/Same_origin_policy [12:23:19] gry: ^ [12:23:50] other api queries work fine from js ... [12:25:36] gry: do you have an example/ [12:46:55] gry: http://jsfiddle.net/M9Qzd/2/ [12:47:32] thanks, what is 'jetzer'? [12:47:48] jetzer? [12:48:19] yes [12:48:45] gry: oh, that was a debug variable [12:48:49] which didn't work [12:49:50] :D [12:52:13] if you were wondering about the word - it's used by students as 'thing that's somewhat big' [12:52:24] is it possible to get it working without window.setResult function? I don't like defining new window.* things [12:53:12] gry: due to the same origin policy, the included javascript has to call back to your code [13:04:13] valhallasw: ok, wth did i break here? http://jsfiddle.net/gryllida/AqMV3/ [13:04:46] valhallasw: http://jsfiddle.net/gryllida/AqMV3/1/ here even [13:05:03] gry: you removed the callback in the url [13:05:33] ah, I see, thanks, I didn't even see it [13:59:27] Coren ping [13:59:52] Coren is there any point in requiring group to be correct for take [14:00:10] my c++ version doesn't require it by default but it can be changed in config file [14:13:27] petan: just a cookie? :-( [14:18:01] is ther a possibelety to change a fiel nema acroll all wikis (a quickly methode?) [14:18:03] *ß [14:18:06] *? [14:18:26] *name acoss (sorry... :( ) [14:46:50] valhallasw did you find some :P [14:51:09] :'( [15:12:28] valhallasw: nope, to local server... just mysql not liking the password :( [15:47:15] " Coren is there any point in requiring group to be correct for take" [15:47:23] ...! It's the *point* of it! [15:47:31] huh? [15:47:51] why you shouldn't be able to overtake a file just because you own the parent directory? [15:47:58] why should you need to be in a group as well [15:48:28] if you are owner you can change the group anyway [15:49:12] Coren ^ [15:50:57] petan: If you are the owner you can change the group, yes, but to become the owner you must already be part of the group. That's how take works. Besides, your code is full of race conditions and unsafe handling of paths; I just send an email. [15:51:06] sent* [15:51:38] hmm [15:52:03] petan: It's not an accident that take.cc doesn't handle pathnames, keeps directory inodes open, and uses fchown() and not chown(); those are all guards against very specific scenarios you were apparently unaware of. :-) [15:52:39] I disagree with the philosophy: imagine you create a folder with permissions 777 in your home, user bob create a folder there, with different group which you are in and with bunch of subfiles [15:52:46] how would you ever overtake it? [15:52:50] take has one purpose exactly: allow tools to take files owned by its maintainers and vice-versa. [15:53:39] petan: If you create a folder with permissions 777 in your home, you deserve to be flogged. [15:53:56] mhm... [15:56:30] Incidentally, /what/ issues? First I hear of it. [15:57:12] And, btw, take doesn't follow symlinks already (and for good cause) [16:04:50] !seen hashar [16:04:50] you probably wanted to use @seen [16:04:58] @seen hashar [16:04:58] Steinsplitter: Last time I saw hashar they were quitting the network with reason: Quit: I am a manual virus, please copy me to your quit message. N/A at 6/21/2013 8:56:02 PM (19:08:56.5052280 ago) [16:08:00] Coren: for example today Oren pointed that it's hard to overtake the group as well (you want files to be local-blah:local-blah) [16:08:05] --group does it [16:08:42] also recursive by default is evil [16:08:59] petan: You shouldn't be /able/ to do so. Either a file belongs to your project, which you can then take, or it doesn't in which case you shouldn't be able to in the first place. [16:09:51] ok, now it works like this: if you aren't in a group which file is owned by, you can't overtake it, but IF YOU ARE, you can overtake owner and even reset the group (which is useful, and not less secure) [16:10:21] Once you own the file, yes. [16:10:53] what race conditions you mean? [16:11:26] you mean when something happen with files while I am overtaking them? [16:12:10] like I would check if owner and groups is correct and between the check and actual change of permission, the permission could change again? [16:12:15] There are many. You're not holding on to the inode between checks, which means that another program can change one file for another (or even a symlink!) between them; and you're not holding the directories open while yo recurse down them (which is even worse) [16:12:18] I don't really think this is a problem... [16:12:39] petan: It /is/ a problem. It allows anyone to change ownership of any file on the filesystem. [16:12:47] nonsense [16:12:54] give me example how could it happen [16:12:58] Hi there, I'm looking to get a new project created on Labs? [16:13:04] Bleep hi [16:13:09] petan: shouting 'nonsense' is not really a good attitude to security :P [16:13:15] :P [16:13:31] valhallasw I just don't believe it, I was rethinking it million times [16:13:34] petan: Heh. A long history of security exploits behind you say otherwise. [16:13:43] petan: How old are you? [16:13:54] Coren: on a related subject - could the default umask for project be 0002 (i.e. allow group +w) instead of 0022? [16:13:54] Coren 24 [16:13:59] can you give me that example :P [16:14:14] because my tool /is not/ doing anything with symlinks /ever/ [16:14:33] it is using ftw function to grab a tree of files, and then it works with them [16:14:33] petan: not at the moment you check, but also not at the moment you *apply*? [16:14:37] it is pretty secure [16:14:42] petan: even *that* is false. You are checking if something is a symlink indeed. At some point. Then not anymore while you do other checks. [16:15:09] petan: Your checks are worthless unless you do them on an open file descriptor. [16:15:13] Coren but changing owner of symlink is not causing any harm [16:15:39] petan: No, but recursing down one who points at, say, /etc would be! [16:15:51] that function build a tree list in memory, and then it walk through the files, changin anything after, doesn't change the recursion [16:16:21] petan: It means that someone can change everything undeneath the program once it builds its list. [16:16:27] it is already in memory, it can't be changed later, it 1) build a tree 2) check the permissions and type of file 3) change the ownership [16:16:36] it is already in memory, it can't be changed later, it 1) build a tree 2) check the permissions and type of file 3) change the ownership [16:16:41] petan: I've been doing security programming since you were 5. That code is completely insecure. [16:16:48] meh [16:16:49] my bouncer :/ [16:16:50] lagging [16:17:22] (See, that's why I wanted to know your age so I knew if I could say "before you were born" or not) :-) [16:17:34] heh [16:18:15] I mean you are right - there are security flaws... worst possible flaw is that it change the owner of symlink in a subdirectory... which is IMHO pathetic issue [16:19:00] the file checks are done after the file list is built in memory, so swapping files with links can't change the recursion [16:19:02] that is what I meant [16:19:05] petan: No, you're wrong. Allow me to illustrate. In my home, I have a directory "booboo" with a file "passwd" in it. [16:19:14] ok [16:19:29] petan: You build your list. It now has "$home/booboo" and "$home/booboo/passwd" Right? [16:19:36] yes [16:19:57] petan: At that point, I substitute booboo with a symlink to "/etc". I now own "/etc/passwd" [16:20:24] Because your code will gleefully chown "$home/booboo/passwd" [16:20:28] let me try it on my own laptop [16:20:56] petan: You'll need a slow filesystem and make the box trash to get it on the first try; a race condition is hard to engineer on purpose. [16:21:08] petan: But always presume that attackers are more patient than you are. :-) [16:21:15] on I don't need to, I can insert some sleep(20) [16:21:18] * no [16:21:50] petan: Yeah, if you force sleeps you can make it easier to try them -- provided you guess right all the places where there is a race conditions. [16:22:10] petan: In your case, since you use pathnames and not inodes, it's between *each* operation on files. [16:22:10] I know everything right :3 [16:23:09] Verbosity: 1 [16:23:10] DEBUG: your uid: 1000 [16:23:11] DEBUG: your euid: 0 [16:23:12] DEBUG: Taking etc [16:23:13] DEBUG: etc resolved to /etc [16:23:14] DEBUG: Invalid path (not allowed for security reasons) /etc [16:23:15] WARNING: Not taking the ownership of /etc because you don't meet the requirements [16:24:14] but maybe in some other hypotetical situations it could be flaw... I just need to invent some [16:24:42] btw that etc thing was actually a symlink in my own directory to /etc [16:24:47] I didn't do take /etc :P [16:25:46] anyway, if I open a FD to each file I am to overtake, that is enough to protect them? Coren? [16:26:10] is there a reason to not just add a --help to the existing take? [16:26:54] valhallasw yes :3 my code is easier to read and that makes it superior XD [16:28:02] valhallasw check both sources, you will find out it's not just --help what I added to mine [16:28:22] it's like 20 more various features, including these debug logs and crap like that [16:31:29] petan: Your code isn't easier to read, it's much *harder* to read because the flow of control is not clear. And it is, by design, completely insecure. [16:32:08] petan: Perhaps I had not been clear; unless it is rewritten to recurse securely (i.e.: not use pathnames, keep the chain of directories open) it doesn't get suid on tests ever. :-) [16:33:33] petan: You are welcome to suggest cosmetic improvements to the current implementation of take which is, admitedly, rather stoic. :-) [16:34:12] Oh, I never put it in git have it? Sorry. [16:34:32] no you never did I had to use sudo to read it [16:34:36] anyway I am fixing all that now [16:35:15] so what you suggest is to open a FD before I work with file and once I am finished with all of them close them all? [16:35:36] BTW, we need a deployment path for (Coren's) take. At the moment, it's only on tools-login. If we add it to the labs/toollabs repo with some Makefile, I'd like to piggyback that to add a testing framework (autoconf's autotest) for the jsub family. [16:35:52] You don't need to keep them open once you're done with the fchown(), but you want to keep all the directories along your path though. [16:36:06] scfc_de: Yes. [16:37:11] New patchset: coren; "Tool Labs Tools: add take.cc" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/69981 [16:37:28] Oh. Amend w/ license. [16:38:21] New patchset: coren; "Tool Labs Tools: add take.cc" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/69981 [16:39:41] Coren: Okay, I'll research a bit how to best integrate the Debian packaging with a build system in one repo. [16:42:03] New review: coren; "Known working." [labs/toollabs] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/69981 [16:42:04] Change merged: coren; [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/69981 [16:43:30] petan: Also, on readability, I'd sincerely love to understand why you feel that a single straightforward function that's strictly linear is harder to read than your complex use of STL to build a datastructure which you then walk. :-) [16:47:04] scfc_de, Coren my version already has Makefile and also package-build.sh script which makes a .deb from it [16:47:16] that's how I installed it on toolsbeta [16:47:44] Coren because that hardcore structure I made is more extendable... [16:48:19] if you attempted to implement some of these features I have your 1-function code would become horrid [16:59:45] * ConfusedPanda reads backscroll of very interesting conversation between petan, Coren|Away and others [17:09:23] Any user arrount with a Interwiki Bot? [17:16:58] petan: ping [17:16:59] petan: can you help me find where the puppet stuff for tools-mc is? [17:17:13] yes, it's nowhere because it never was merged... [17:17:22] link?? [17:17:30] lemme search my box [17:18:45] oh [17:18:47] it was merged... [17:18:50] weird [17:18:59] memcachesasl.pp [17:19:14] it's empty though [17:19:18] hmm? [17:19:20] link? [17:19:28] https://gerrit.wikimedia.org/r/#/c/66328/1/modules/toollabs/manifests/memcachesasl.pp [17:19:47] petan: lol [17:55:02] Coren|Away: ok try to find a flaw now :P [17:55:21] Coren|Away I implemented all what you required + far more and it's even faster and more memory efficient [17:55:35] it works as a charm <3 [18:30:12] petan: i'm adding a puppet class for redis on labs. [18:30:27] ConfusedPanda pls work on toolsbeta [18:30:27] petan: can we have tools-redis dedicated to redis? And maybe decommission tools-mc? [18:30:38] or just keep it as memcached? [18:30:45] ConfusedPanda why decommission something what we need? :< [18:30:54] mc is being used as hell [18:30:56] petan: well, then the later one [18:31:02] which is 'just keep it as memcached' [18:31:11] and use tools-redis for redis :) [18:31:12] tbh I don't know why we should use 2 servers for 2 services which are barely used [18:31:22] it's IMHO wasting of resources... but idk [18:31:27] * idc [18:31:42] meh. fine [18:31:49] * ConfusedPanda does git reset --hard, goes to find something else to do [18:31:51] ConfusedPanda first set it up on toolsbeta and then we can consider pushing these changed to production ;) [18:32:00] * changes [19:07:20] zz_ConfusedPanda: Ideally, redis (and memcached and ...) should be set up in Puppet in a way that the server is run on any host that includes the module. [19:08:53] Coren|Away I must admit your version has significantly better performace: [19:09:13] petrb@toolsbeta-login:~$ /usr/bin/time take -r linux-3.9.7 [19:09:14] ERROR: Not overtaking linux-3.9.7/arch/microblaze/boot/dts/system.dts because it is a symlink [19:09:15] 0.88user 11.58system 2:40.79elapsed 7%CPU (0avgtext+0avgdata 5216maxresident)k [19:09:16] 0inputs+0outputs (0major+381minor)pagefaults 0swaps [19:09:17] petrb@toolsbeta-login:~$ [19:09:18] petrb@toolsbeta-login:~$ /usr/bin/time ./take -r linux-3.9.7 [19:09:19] -r: No such file or directory [19:09:20] system.dts: will not follow or touch symlinks [19:09:21] Command exited with non-zero status 1 [19:09:22] 0.31user 7.81system 2:09.86elapsed 6%CPU (0avgtext+0avgdata 4704maxresident)k [19:09:23] 0inputs+8outputs (0major+361minor)pagefaults 0swaps [19:09:27] (I hate pastebin :P) [19:09:28] FLOOOOOOOOOOOOOOOOOOD [19:09:37] * Cyberpower678 stabs petan [19:09:55] * Hazard-SJ sighs [19:10:03] when I use pastebin nobody ever clicks the link :D [19:10:21] and it involves launch of browser which is ineffective [19:10:34] also it requires more network traffic [19:10:37] :D [19:11:02] * Cyberpower678 greases his network daily. [19:11:40] petan: While you're at it, please tell Coren|Away that I received no email [19:11:56] Coren|Away he received no email [19:12:36] * Hazard-SJ thanks petan [19:12:42] :> [19:12:46] @notify Coren|Away [19:12:46] This user is now online in #wikimedia-labs. I'll let you know when they show some activity (talk, etc.) [19:13:41] * Hazard-SJ wonders how many pings Coren|Away got so far [19:14:46] you and your lame irc clients... crying over spam and pings :) [19:16:47] Hazard-SJ, to answer your question, it is uncertain how many pings Coren|Away has so far. [19:17:02] Cyberpower678: Thanks :) [19:17:04] For all we know, Coren|Away is busy breaking labs. [19:17:27] But there is one thing we do know about Coren|Away, Hazard-SJ [19:17:44] petan: BTW, could you help me out with SGE, please? :P [19:17:53] yup [19:18:10] And that is Coren|Away now has 4 more pings now after answering your question. :p [19:18:10] * Hazard-SJ wonders what Cyberpower678 is going to say that we know about Coren|Away [19:18:20] 5 more [19:18:22] (5) :P [19:18:37] Hazard-SJ we don't run SGE :P [19:18:42] we run OGE [19:18:46] open grid engine [19:18:56] sge is proprietary [19:18:57] * Hazard-SJ facepalms [19:19:00] GE :p [19:19:04] it's same :P [19:19:18] ok so what you need? :P [19:19:38] Cyberpower678 if you are bored you can go proove that my take is totally hacker resistant [19:19:38] petan: Firstly, which is better? qsub or jsub? [19:19:51] depends [19:19:58] jsub is a wrapper for qsub :P [19:20:07] :/ [19:20:09] it just does some things automagically for you [19:20:15] it's better for newbies [19:20:27] qsub can do same things as jsub, but it may be more complicated [19:20:44] petan: So can jsub do everything qsub does? [19:21:08] I can't answer that because I don't know, but I think it can do everything you need :P [19:21:29] Okay :) [19:21:33] jsub is written by Coren|Away and he is indeed a clever guy :D [19:21:47] who got pinged 6 times now [19:22:29] * Hazard-SJ facepalms at petan saying Coren|Away got 6 pings, who knows Coren|Away actually got more than that :P [19:23:27] So, petan, should I submit jobs through cron or directly as a command (if you understand what I'm asking :/ ) [19:24:07] why not, just remember that jsub may need to be defined with absolute path in cron [19:24:22] at least it didn't work to me without that [19:24:41] ah, I mean you can use cron if you want [19:25:00] it depends if it's a task that needs to be started repeatedly or one time task [19:25:07] petan: And I should set those in cron on the login server, right? [19:25:07] if it's 1 time you shouldn't cron it :P [19:25:13] yes [19:25:58] petan: The last time I tried using _GE it didn't work [19:26:12] there is .err log [19:26:17] which has the reason why [19:30:14] * Hazard-SJ goes to try again [19:39:36] * Hazard-SJ will brb [19:40:22] petan: The first test I just made seemed to have worked [19:40:35] o/ [21:36:51] is the idea that each tool has its own VM, rather than being on one shared server? [22:12:40] On a related vein, could somebody create Nova credentials for me? [22:13:50] lfaraone: Are you looking at creating a tool (bot/web stuff) in the Tools project, or do you have a more complicated setup? [22:14:12] scfc_de: oh, there's a Tools project? That's probably what I want. [22:14:23] petan, ^ [22:14:41] scfc_de: I'm looking to create such a tool, yes. [22:16:03] lfaraone: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help