[00:43:53] Hello [00:44:10] Any chance of handling this request soon? :-) [00:51:01] InezK: which request? [00:51:30] ah. shell request? [00:51:53] https://wikitech.wikimedia.org/wiki/Shell_Request/Inez [00:52:01] looks like someone already did it [00:52:05] but didn't mark it as done [00:52:39] ok, so I still have problem with access bastion [00:52:40] ah. you had a very old account that didn't get bastion access [00:52:51] ssh -A inez@bastion.wmflabs.org [00:52:51] Permission denied (publickey). [00:52:59] one sec [00:53:05] I put my public key https://wikitech.wikimedia.org [00:53:40] try now [00:53:49] same problem still [00:53:59] let me check auth.og [00:54:00] *log [00:54:17] failed public key [00:54:20] * Ryan_Lane checks something else [00:55:01] I can't see a reason for it to be failing [00:55:09] I got in [00:55:10] ah. seems it worked now [00:55:17] nice nice :) [00:55:29] ok, let me check if I can break something [00:55:53] crap. I wonder if I removed you from any other projects [00:55:58] were you previously in VE? [00:56:23] yes, you did [00:56:27] now I can't see my instance in here https://wikitech.wikimedia.org/wiki/Special:NovaInstance [00:56:36] which projects were you in? [00:56:42] visualeditor only [00:56:44] ok [00:56:52] projectadmin as well? [00:57:14] RoanKattouw: Was I projectadmin? should I be? [00:57:41] InezK: were you able to create and/or delete instances? [00:57:54] and reboot and such? [00:58:04] yes [00:58:05] I can probably check backups of ldap [00:58:06] and yes [00:58:09] ah. you were project admin then [00:58:25] ok, your rights are back [00:58:32] I need to be more careful when managing shell [00:59:29] Ryan_Lane: thank you so much! [00:59:32] btw.. http://visualeditor-parsoid-tests.instance-proxy.wmflabs.org/ [00:59:34] yw [00:59:37] I guess I don't have apache [00:59:44] heh [01:00:08] should I just install via shell.. or you have some nice clickable interface for it? [01:00:15] I also need PHP + MySQL [01:01:09] Ryan_Lane: I don't suppose we have a nice Puppet class for just plain Apache + PHP + MySQL? [01:01:19] Coren: can you add role::labs::tools::redis to tools-mc? [01:01:20] we should under configure [01:01:26] hm [01:01:30] lemme look [01:01:56] webserver::php5-mysql ? [01:02:00] * RoanKattouw looks at what that does [01:02:05] role::lamp::labs ? [01:02:08] Successfully modified instance i-000007e9 (visualeditor-parsoid-tests). [01:02:23] so.. it will take some time for it to apply? [01:02:40] InezK: On the instance, run sudo puppetd -tv [01:02:56] Oh I see [01:02:58] Nice [01:03:08] inez@visualeditor-parsoid-tests:~$ sudo puppetd -tv [01:03:09] info: Loading facts in /var/lib/puppet/lib/facter/projectgid.rb [01:03:10] info: Loading facts in /var/lib/puppet/lib/facter/default_gateway.rb [01:03:11] InezK: Install role::lamp::labs instead, that looks like a nice one [01:03:12] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class db::core for i-000007e9.pmtpa.wmflabs on node i-000007e9.pmtpa.wmflabs [01:03:14] warning: Not using cache on failed catalog [01:03:16] err: Could not retrieve catalog; skipping run [01:03:23] ugh. damn virtualbox [01:03:25] * Ryan_Lane reboots [01:03:43] indeed it does [01:04:13] sudo puppetd -tv works now apparently [01:04:25] I should have sweet brand new lamp there in matter of seconds [01:06:10] yay: http://visualeditor-parsoid-tests.instance-proxy.wmflabs.org/ [01:06:50] \o/ [02:30:59] Coren: Can you add account 'catrope' to tools, so that I can add him to tools/local-visualeditor? [02:31:39] I got it [02:32:24] done [03:24:33] InezK: Do you have a labs account? [03:24:39] InezK: What's your user name [03:24:49] inez [03:25:13] Ryan_Lane: Looks like I can't add him to a tools project either, can you add him to the tools project itself? [03:25:27] yep [03:26:08] done [06:09:37] Coren: on ToolServer we had toolserver.namespace table for wikis NS. Is there something similar on ToolLabs? [06:15:04] Coren: it's a lookup table to expand the NS name having the NS id [06:26:48] fale: there's a bug open for it… Krenair had set something up [06:46:19] hi henna_: how you're doing? [07:11:09] legoktm: Thanks. Would not be enough to make a table dump from TS and upload it back to ToLabs [07:11:12] *? [07:12:08] fale: maybe. https://bugzilla.wikimedia.org/show_bug.cgi?id=48625 [07:13:52] legoktm: Thanks. So I'll wait :D [08:03:50] !log toolsbeta petrb: updating logsplitter [08:03:52] Logged the message, Master [08:05:07] It seems that I don't have the push right on gerrit... maybe I'm doing something wrong :( [08:06:01] you don't push into gerrit, you do git review [08:06:19] MaxSem: even for my own repo? [08:07:19] yes [08:07:22] fale: gerrit is ugly and evil [08:07:29] petan: :D [08:07:37] MaxSem: thanks :) [08:07:44] www.mediawiki.org/wiki/User:Petrb/Git_for_idiots [08:07:58] what sucks are UI and git-review, the underlying idea is great [08:08:05] idea is [08:08:09] implementation is not [08:08:21] it's Java, sir [08:10:24] thanks :) [08:10:38] according to awstats visitors are growing for tools :o [08:11:01] addsleep make a graph plz [08:21:03] ping zz_YuviPanda [08:21:06] @notify Coren [08:21:06] This user is now online in #wikimedia-labs. I'll let you know when they show some activity (talk, etc.) [08:28:13] is here someone who understand php? :P [08:41:14] petan, /me [08:41:47] MaxSem cool, you know about tools project right? [08:42:04] vaguely [08:43:00] aha, anyway we have there a page, which is displaying a status of grid, it's a php script, which executes a shell process and parses its output [08:43:49] it works on production but on beta site it doesn't, the php function that calls the shell command just don't work. I can see it fail (according to php.net if it's NULL it is in error) but I don't know why [08:43:53] is there a way to find out why? [08:44:15] when I copy the command to shell it runs just fine [08:44:20] but php can't start it :/ [08:44:39] php isn't really very descriptive when it comes to errors... [08:47:14] which function, exec()? [08:47:36] yes that one and also this: $rawhosts = toarray(simplexml_load_string(`PATH=/bin:/usr/bin /usr/bin/qhost -F h_vmem -xml`)); [08:48:55] petan: have you checked the paths? [08:49:03] valhallasw what you mean? [08:49:13] valhallasw: the command itself works when I start it by hand [08:49:23] IDK why paths are defined there, coren wrote this [08:49:25] petan: *including* the PATH=....? maybe qhost shells out [08:49:35] does it work from every directory and for every user? [08:50:02] lemme check, but even if it didn't work it should throw specific error [08:50:04] not just "error" [08:50:25] when I do sudo su www-data [08:50:30] and execute this command [08:50:31] it's fine [08:50:43] again, including PATH? [08:50:47] why set PATH at all? [08:50:55] ask Coren I didn't write this [08:51:06] second thing to check is what happens if you echo `PATH=/bin:/usr/bin /usr/bin/qhost -F h_vmem -xml 2>&1` [08:51:06] valhallasw what you mean? [08:51:13] whether you included PATH=/bin:/usr/bin or not [08:51:30] included how? [08:51:38] I sent you exact line [08:51:39] in the command you ran in the shell [08:51:44] no [08:51:50] in shell I don't need to change PATH [08:51:53] ... [08:51:56] I don't know why it's in php script [08:52:49] my idea: your normal path includes a directory that's not in the PATH above, and qhost tries to shell out and fails [08:53:06] valhallasw good that command you sent me produced some output [08:53:07] :o [08:53:55] valhallasw yes I understand what you mean, but as I said, a) the PATH as it is in default shell, is correct enough because it works b) I don't know how to change PATH in php [08:54:28] the error I receive is weird indeed [08:57:05] lol I am an idiot [08:57:07] I found it [09:14:57] !log deployment-prep stopping squid and pruning cache [09:15:00] Logged the message, Master [09:17:13] !log deployment-prep recreated squid swap directories with `squid -z` restarting squid [09:17:15] Logged the message, Master [09:18:57] petan: so, what was it? [09:19:07] webserver wasn't submithost [09:19:13] so master was rejecting it [09:19:26] http://tools-beta.wmflabs.org/?status [09:19:29] here we go o/ [09:19:33] new feature :D [09:19:52] now I need to fight gerrit [09:20:01] and get it there [09:20:45] @search gerrit [09:20:46] Results (Found 6): gerrit, whitespace, git-puppet, gerritsearch, ryanland, gitweb, [09:20:50] meh [09:22:30] !log deployment-prep Squid restarted properly, that fixed some stalled resource loader entries that were causing some outdated Javascript modules to be served. Fixed at least an inconsistency such as {{bug|49911}} [09:22:33] Logged the message, Master [09:22:51] New patchset: Petrb; "implemented check for free v_mem" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/70603 [09:24:55] petan awesome graphs might be coming soon ;p [09:24:59] ok [09:30:05] New patchset: Petrb; "implemented check for free v_mem" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/70603 [09:35:15] !log tools petrb: updated status.php to version which display free vmem as well [09:35:17] Logged the message, Master [09:45:51] any bot-operator here? [09:47:10] yes [10:19:33] lbenedix: yes :P [10:36:42] greay [10:36:45] great [10:37:13] I'm looking for an historical event where a bot went amok and did a lot of unwanted edits [10:38:15] something like an edit war between a human and a bot.. [10:49:07] lbenedix: I've had a war against bots when trying to fix interwiki links [10:50:52] lbenedix: what about bot-bot edit wars? [10:51:41] I had to resort to {{nobots}} until someone suggested using form to defeat interwiki.py [10:58:04] I'm looking for an example for a talk [11:05:59] New review: Tim Landscheidt; "(1 comment)" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/70603 [11:36:08] any idea? [11:44:51] * fale is slowly having success with Gerrit. 3 errors fixed, at least another to go -.- [11:53:26] fale: you can setup gerrit to let you push directly, btw. [11:53:31] no need to use git review if you don't want it [11:53:36] tell ^demon and he'll do it :) [11:54:20] YuviPanda: The thing is that I've a different anchestor than the one present in gerrit... [11:55:02] fale: ah [11:55:14] hmm, would have to reset and rebase. [11:55:20] i'll keep quiet and not confuse you :) [11:56:11] YuviPanda: I was thinking about checking out the empty git, copy all the files and create a single huge commit [11:56:18] whyyyy [11:56:23] but that's not so good :( [11:57:06] some of the commits I have in my repo have been committed before the gerrit repo have been created [11:57:22] fale: link to your repo and the gerrit repo? [11:57:44] YuviPanda: https://gerrit.wikimedia.org/r/#/admin/projects/labs/tools/lists and https://github.com/Fale/lists [11:58:01] hmm, let me see [11:58:07] YuviPanda: thanks :) [11:58:33] * YuviPanda clones [11:59:13] fale: aah, so the problem, is that when you requested the repo, it should've been created with an 'import' of your github one [11:59:22] anyway, this is still fixable, i think [11:59:36] YuviPanda: by Coren? [11:59:38] fale: yes [11:59:42] this is all Coren's fault :P [11:59:50] you should've also mentioned it to him, I think [12:00:03] YuviPanda: I guess it's my fault since I did not mentioned it [12:00:08] heh :) [12:01:43] fale: okay, so I have your changes rebased on top of Coren's [12:01:51] and we should be able to get this on gerrit :) [12:02:07] YuviPanda: Awesome :) [12:02:19] fale: but it'll be easier if we have someone enable direct push for your gerrit repo [12:02:28] fale: can you wait until ^demon or someone turns up and can change that/ [12:02:37] else we'll have a large number of commits we'll have to manually +2 [12:02:48] fale: plus in general I think you'd want direct push, no? [12:03:36] YuviPanda: If is possible to keep some user to direct push and other to gerrit it would be perfect... but maybe this i not what you meant [12:03:53] fale: yeah, but definitely for you it'll be direct push, right? [12:04:02] fale: yeah, I think that is possible (so you can push directly, others need to use review) [12:04:05] YuviPanda: if possible, definitely yes :) [12:04:23] fale: so can you wait for a few more hours before the gerrit folks turn up? :) [12:04:51] YuviPanda: it's fine for me, even because I can continue coding on the github repo :) [12:05:02] YuviPanda: thanks a lot :) [12:05:23] fale: https://dpaste.de/LefYq/ [12:05:37] fale: sequence of commands that'll put your github commits on top of coren's commits, so the parent issue is fixed :) [12:05:52] YuviPanda: thanks :) [12:05:59] :) [12:07:08] YuviPanda: -.- That's why I was srewing it. I always started from the github repo and merged the gerrit one instead of the opposite :( [12:08:07] fale: :D [12:08:22] fale: it doesn't really matter where you start from in terms of cloning, only in terms of rebasing [12:08:45] YuviPanda: yep, always rebasing in the opposite order [12:09:38] :) [12:09:48] New review: Tim Landscheidt; "The change from tools to misctools was not reflected outside debian/control, so the generated miscto..." [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/67884 [12:10:48] YuviPanda: with gerrit, even if I'm in the repo owner-group I should have the +2? [12:11:00] you would already have +2 if you are the repo owner [12:11:25] YuviPanda: https://gerrit.wikimedia.org/r/#/admin/groups/516,members <-- it seems I'm [12:11:45] apparently I don't have permission to view that page [12:12:43] YuviPanda: is a confidential info the people who are repo owner, lol [12:13:00] :D [12:13:01] apparently [12:27:34] New review: coren; "Doh! And it 'seemed' to work since we already had a version installed so the files were not obvious..." [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/67884 [12:27:44] Warning: There is 1 user waiting for shell: Jboogie (waiting 0 minutes) [12:30:19] Coren: can you change fale's repository to allow direct push? [12:30:31] or do we need to summon demon for that? [12:30:40] I think I can. Lemme see real quick. [12:33:36] fale: Please to try? I /think/ I have you permission to just push. [12:34:05] Loaded module modules/mediawiki_thanks.bin [12:40:39] Coren: no error reported :) [12:40:58] fale: pushed? [12:41:02] That either means it worked or that it failed quietly. :-) [12:41:24] YuviPanda, Coren: now gitglit confirms that it woked :) [12:41:28] thanks a lot :) [12:41:31] yeah, I did a fetch, it works [12:41:33] woo :) [12:41:40] This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.10.8.15 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [12:42:59] petan: Where is your source? I'm not listed as a reviewer. [12:43:05] fale: see slso: https://github.com/wikimedia/labs-tools-lists [12:43:05] :) [12:43:14] Coren on github [12:43:25] petan: url? [12:43:27] https://github.com/benapetr/take [12:43:41] YuviPanda: yep :) going to delete mine copy [12:43:47] fale: :) [12:43:56] fale: want me to enable pull requests integration for that repository? [12:44:00] but given the amount of haters of this new take, I am fine if you implement all these new features to existing take [12:44:28] I am kind of tired of these never ending "your code suck" messages with 0 explanation what suck on it [12:44:39] YuviPanda: will they send me an email or some other kind of notification? [12:45:22] petan: did you ever get to reading the multitudes of links I gave you? [12:45:27] just asking. [12:45:45] petan: Well, having it in a place where it would be possible to actually do an inline review would help. :-) [12:45:46] fale: it'll submit a gerrit patchset, and I think you can configure gerrit to send you an email when that happens [12:45:54] Coren: you can do inline review on github [12:46:02] YuviPanda: then yes, please :) [12:46:07] fale: moment [12:46:19] Oh. Well, I don't know how. Lemme try to poke around and find the feature. :-) [12:46:22] YuviPanda did you ever get to reading: all issues mentioned in these links are handled in my code? [12:46:36] ok :) [12:46:46] Coren: you have to do that per commit tho :P [12:48:01] it's fine if you just tell me https://github.com/benapetr/take/blob/master/src/Take.cpp#L52: I don't like this... etc. [12:48:53] Coren: also please note I am actually passing the path in some places, but it's always just so that it can be in debug log [12:48:54] * fale gets no answer from tools-login.wmflab.org. Some network issue? [12:49:10] fale: wmflabS [12:49:14] not wmflab [12:49:26] petan: -.- thanks [12:49:29] :P [12:49:33] fale: done! try sendint that github repo a pull request now [12:49:57] petan: well, for one, you're using STL in a program intended to be SUID root. That's an automatic fail, but I'll pretend that it isn't the case for the purposes of this exercise. [12:50:43] what is wrong on doing that o.O [12:51:39] hmm, one of my bots seems to be leaking memory [12:51:40] * YuviPanda looks [12:52:04] petan: ... when was the last time you did a security audit of the STL code? Hint: it's never been done because it's impossible to do given the template layering. [12:52:44] I am almost sure even sudo is using STL [12:52:59] lemme find source code of sudo [12:53:20] :D [12:53:30] * YuviPanda pushes fix for bot, goes back to other things. [12:56:06] petan: I can guarantee sudo doesn't. For one, it's not in C++. :-) [12:56:13] indeed... [12:56:35] petan: In fact, I can guarantee that *no* program in any distribution ever uses the STL for anything intended to run as root. [12:57:03] petan: Problem 2: you're using ftw(). ftw() does not guarantee that it holds the directories open, only that it will try to. [12:58:00] but you realize, that every file / directory provided by ftw() is then open, its FD is hold, then it's checked and after check overtaken, and after then the FD is released? [12:58:18] so it shouldn't matter at all [12:58:33] YuviPanda: thanks :) [12:58:37] results of ftw() as treaten just as reliable as arguments provided by user [12:58:42] * are [12:59:58] petan: ... No they are not. You're only doing so depth first. By the time ftw() returns to one of the top levels it released the fd to to continue enumerating the contents, you could have substituted one of the intermediate directories with something else. [13:00:11] petan: Do an strace, check the sequence of system calls. [13:00:34] petan: You're relying on a function that does not have deterministic behaviour. [13:00:36] yes I know, but again.. ftw is providing a list of files, this list is treaten just as a list of files provided by user in parameters [13:00:50] these files are open again and checked [13:01:16] petan: I *know* it's providing a list of files. That's the /problem/. You should not recurse with names at all, ever. [13:01:28] Unless, at every chown, you revalidate the /entire/ path. [13:01:32] (Which you don't do) [13:01:33] even if its content or structure changed meanwhile, it wouldn't matter, the take would see it [13:01:36] (And you wouldn't want to) [13:01:41] Coren of course I do [13:01:50] You do? Show me where. [13:01:55] Coren: https://github.com/benapetr/take/blob/master/src/Take.cpp#L115 [13:02:04] https://github.com/benapetr/take/blob/master/src/Take.cpp#L141 [13:02:08] https://github.com/benapetr/take/blob/master/src/Take.cpp#L148 [13:02:22] You're only checking the file; not every component of the path to it. [13:02:38] this is callback function that is called for /every/ component of the path [13:03:15] if you had a folder with 2000 000 files this function would be called 2000 000 times [13:03:23] You're not understanding me. At that callback, the only thing you know is that /at some point in the past/ the containing directory was okay. [13:04:03] Maybe. [13:04:20] New patchset: SuchABot; "Add license" [labs/tools/lists] (master) - https://gerrit.wikimedia.org/r/70624 [13:04:22] given your use of ftw(), for all you know, you had /no/ fd open on the directory at all. [13:04:52] I don't see how it makes it less secure, if containing directory was changed in your version of take during run, you wouldn't prevent it anyway, because you would hold outdated FD and file it refers to would be likely deleted once you release it [13:04:54] Because the caller may well have purposefully opened every single available fd but one or two to cause the degenerate behaviour, and ftw() won't even tell you. [13:05:30] linux != windows - you can hold a fd of deleted / changed file... it's just a fd of original one and once you release it, it's deleted physically [13:05:34] petan: But you couldn't substitude the directory with something /else/. Do like Yuvi told you and read up on race condition exploits, please. You need to understand those things. :-) [13:06:13] " linux != windows"... you /do/ realize I've been doing system-level programing on Unix before you were born? :-) [13:06:29] unix yes, but surely not linux :) [13:06:36] because linux was created when I was born :D [13:07:00] it's inode/vnode and not fd :) [13:07:02] petan: Linus et. al. worked very hard to make Linux match SVID, SUS and POSIX semantics for a good reason. :-) [13:07:22] saper: the fd is your handle to the inode in userspace. :-) [13:07:28] hmm, at least you told me which part is wrong [13:07:31] saper: You don't get to play with inodes outside the kernel. :-) [13:08:08] Coren next topic: [13:08:15] there are problems with vmem on some exec nodes [13:08:23] tools? [13:08:28] I thought fd is just an offset in my process table :( [13:08:30] http://tools.wmflabs.org/?status [13:08:44] Coren I implemented Free vmem into that status.php script [13:08:51] as you can see free vmem is terribly low on exec 05 and 06 [13:08:58] that is because grid doesn't count its swap [13:09:18] YuviPanda: it worked properly :) (even if I've not yet figured-out how to actually merge the code :D) [13:09:24] I saw the patch, I was wondering why the extra qstat. [13:09:25] why is that? how I make it count the swap just as it does on other nodes? [13:09:42] Coren because it is /only/ qstat :P you did qhost -F h_vmem [13:10:03] petan: Oh. Probably just forgot to update the node definition when adding them. [13:10:13] how to fix it? [13:10:19] petan: Ah, right: complex_values h_vmem=7G,slots=32 [13:10:24] fale: so you've to merge it on Gerrit, and it should auto-close it on GitHub [13:10:32] except there's a bug in the bot preventing auto-close, but i'll fix it soon :) [13:10:35] anyway if you have a better idea how to implement that check in php fix it, I did it myself just because that bug was open for weeks and rather important [13:10:42] qconf -me tools-exec-*.pmtpa.wmflabs [13:10:44] fale: in the meantime, I'm admin on github for wikimedia/* so I can close them for you! Feel free to ping me :) [13:10:49] IMHO free vmem is one of most important information in that table [13:10:52] YuviPanda: dunno why, but I thought that gerrit would have merged it by itself :D [13:10:52] you want h_vmem to start at 31G and not 7G [13:11:04] * Coren fixes it now. [13:11:16] fale: nah, then anyone can send a pull request and automagically merge things for you, no? code you didn't want, in your repo! [13:11:28] good point [13:11:32] ok [13:11:48] petan: Fix't. [13:11:59] can the status.php file have autorefresh? :) [13:12:10] where is it at? [13:12:29] YuviPanda @/data/project/.system/public_html [13:12:45] it lives on both production and beta, but for some reason this vmem check is borked on beta [13:12:47] petan: on gerrit, I mean [13:12:55] eh [13:12:58] please tell me it is on version control :D [13:12:59] Change merged: Fale; [labs/tools/lists] (master) - https://gerrit.wikimedia.org/r/70624 [13:13:01] I thin somewhere in www [13:13:04] I don't remember :/ [13:13:07] yeah, found it [13:13:56] YuviPanda: thanks a lot :) very cool [13:14:01] fale: :D [13:14:14] \o/ :) [13:14:14] YuviPanda when you are in fixing it, can you make load and free vmem go red when load is more than 90% and free vmem less than 5gb [13:14:28] !log wikidata-dev wikidata-dev-9 update LocalSettings.php "$wgWBRepoSettings['siteLinkGroups'] = array ('wikipedia', 'wikitravel', 'wikisource');" [13:14:30] Logged the message, Master [13:14:33] is there any reason why this is in PHP and not a saner language? :P [13:14:42] YuviPanda such as? [13:14:44] python? XD [13:14:45] python [13:14:48] :D [13:14:48] LOL [13:14:50] I was joking [13:14:56] I clearly am not :P [13:15:00] Coren: also, WSGI? [13:15:04] PHP is sane language [13:15:13] it's my favorite language when it comes to web pages :P [13:15:15] YuviPanda: It's not wat #3 on my TODO [13:15:24] * hashar kicks petan out of the room [13:15:31] :/ [13:15:36] New review: Tim Landscheidt; "(1 comment)" [labs/toollabs] (master) - https://gerrit.wikimedia.org/r/70603 [13:15:40] YuviPanda: I find PHP saner than Python when it comes to making simple web pages. [13:15:41] hashar you also want to rewrite mediawiki to python? [13:16:03] I find PHP saner than python when it comes to making any web pages [13:16:29] rm -rf python [13:16:31]