[06:51:43] legoktm: hi. what toollabs stats do you think would be useful in http://charcoal.wmflabs.org/? [06:51:46] (guest/guest) [06:52:02] nginx stats from proxy, redis stats, and grid stats of some sort... [06:52:15] users on tools-login and tools-dev [06:54:49] I don't really know [06:55:15] hehe [06:55:17] fine [06:55:43] YuviPanda: do you know why http://tools.wmflabs.org/legobot/cgi-bin/reflinks.py?url=http%3A%2F%2Fwww.fijitimes.com%2Fstory.aspx%3Fid%3D105005 is failing? [06:55:54] it's just a simple urllib2.urlopen call, which works fine from the CLI [06:56:01] but not from the webserver [06:56:14] I copied the script from the toolserver where it works fine [06:56:29] are there limitations on web tools? [06:58:23] legoktm: trying [07:01:53] legoktm: hmm, unsure. maybe the thing is dead? [07:02:01] it's not! [07:02:15] YuviPanda: using that url from cli works perfectly fine [07:02:27] also works fine on TS [07:02:58] >>> import urllib2 [07:02:58] >>> foo = urllib2.urlopen('http://www.fijitimes.com/story.aspx?id=105005') [07:02:58] >>> [07:04:30] legoktm: I wonder if tmp is full... [07:04:34] legoktm: which node is this on? [07:04:47] umm [07:04:50] tools-webgrid? [07:09:05] legoktm: there are so many webgrid nodes :) [07:09:33] I don't know! [07:09:34] 1886036 0.25056 lighttpd-l tools.legobo r 06/27/2014 04:04:31 webgrid-lighttpd@tools-webgrid 1 [07:09:38] that's what qstat says [07:11:04] legoktm: ah, hmm [07:56:31] hi valhallasw [07:56:54] I've noticed over the past few days, reviewer-bot is super delayed [07:56:58] like, days behind [08:18:49] 3Wikimedia Labs: Migrate dynamicproxy module to work with the nginx module - 10https://bugzilla.wikimedia.org/67188 (10Yuvi Panda) 3NEW p:3Unprio s:3normal a:3None Currently we are hand-managing nginx, which is bad. We should use the nginx module instead. This requires the following features from the... [08:52:46] hi, I just mirated a tool from the toolserver to labs - it works as long as I ran it from a shell, but it gets no database connection if I start it as cronjob - any ideas? [08:52:47] 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/randomarticle.php to Tool Labs - 10https://bugzilla.wikimedia.org/60871#c6 (10Silke Meyer (WMDE)) I picked project = commons, all namespaces, categories = wildmills, include subcategories = yes, depth = 0, I am redirected to either a... [08:53:02] "migrated" [08:54:46] "Can't connect to MySQL server on 'dewiki.labsdb' (110)" [08:55:58] aka_: try copying your tools "replica.my.cnf" to ".my.cnf" [08:56:28] it should be in your tool's home directory [08:56:38] I'll try it [08:58:05] it does not work [08:58:36] the error message says still "Can't connect to MySQL server ..." [08:58:53] hmm [08:58:56] so i think this has nothing to do with the login information in .my.cnf [08:58:58] can you pastebin the code you're using? [08:59:41] I'll make a little test script for this [09:08:16] http://pastebin.com/hSnhee8U [09:08:26] if I start it from a shell, it works [09:08:54] if a cronjob starts it, it "Can't connect to MySQL server" [09:09:51] hmm, I don't know [09:49:12] @notify scfe_de [09:49:12] I'll let you know when I see scfe_de around here [09:49:59] * YuviPanda waves at petan [09:50:00] around? [09:56:47] 3Wikimedia Labs: Migrate dynamicproxy module to work with the nginx module - 10https://bugzilla.wikimedia.org/67188#c2 (10Yuvi Panda) Doing this should automatically setup diamond metrics for dynamicproxy. [10:53:47] 3Tool Labs tools / 3Erwin's tools: Migrate https://toolserver.org/~erwin85/randomarticle.php to Tool Labs - 10https://bugzilla.wikimedia.org/60871#c7 (10Andre Koopal) ahh, you leave family (the dropdown box) to wikipedia. Strange that works, I will debug a bit more later. Not sure if I manage quickly, otherw... [12:04:32] 3Wikimedia Labs: Implement ability to search wikitext of current Wikimedia wiki pages with regular expressions (regex) - 10https://bugzilla.wikimedia.org/43652#c11 (10Waldir) The patch has been merged and should be deployed (according to [[mw:MediaWiki 1.24/wmf10]] and looking at [[Special:Version]]), but the... [12:58:48] 3Wikimedia Labs: Implement ability to search wikitext of current Wikimedia wiki pages with regular expressions (regex) - 10https://bugzilla.wikimedia.org/43652#c12 (10Helder) E.g.: https://pt.wikipedia.org/w/index.php?title=Wikip%C3%A9dia:P%C3%A1gina_principal&action=edit contains "mf-artigos-destacados" but t... [13:35:18] 3Wikimedia Labs: Implement ability to search wikitext of current Wikimedia wiki pages with regular expressions (regex) - 10https://bugzilla.wikimedia.org/43652#c13 (10Nik Everett) 5PATC>3RESO/FIX This one requires that the index be rebuilt before it'll work. I tried to make that clear in my email to ambas... [14:26:07] Damianz: hey! around? [14:45:21] valhallasw: someone asks, shouldn't undoHTMLEntities be called earlier? https://github.com/WikiTeam/wikiteam/pull/124/files [15:27:58] Nemo_bis: I.. don't know? page.__init__ makes sure html entities are parsed correctly, I think [15:30:32] ok [15:31:49] 3Wikimedia Labs / 3tools: install GD perl lib - 10https://bugzilla.wikimedia.org/67199 (10Aka) 3NEW p:3Unprio s:3blocke a:3Marc A. Pelletier Intention: Steps to Reproduce: perl -e "use GD;" should work without errors, currently it does'nt find the GD package. Reproducible: Didn't try [17:02:50] Coren: I still don't quite understand what /srv/dumps is on labstore1001, but it is now full. What should I do about that? [17:03:10] * Coren groans. [17:03:18] It holds, unsurprisingly, the dumps. :-) [17:03:27] But the dumps are getting larger all the time [17:04:03] There's an RT ticket about finding alternatives; I think I'll have to improvise something in the meantime though. [17:06:34] Yeah, I remember we've talked about it a few times. And reached no conclusion :( [17:37:19] Coren: any progress on getting a /latest alias? [17:38:15] Betacommand: Hm? Sorry, that seems to have slipped between the cracks. [17:39:04] let me dig up the bug number [17:39:58] Coren: bug 45646 [17:41:29] Lemme poke Ariel. [17:46:34] Coren: can you run sql commonswiki_p "select img_name from image;" > commonsfiles.txt ? im getting an OOM message [17:47:18] o_O you want a file with the list of all image names? [17:47:59] Betacommand: yeah, that's gonna kill a few things, I think [17:48:06] Heh, no surprise that that causes an OOM :) [17:48:11] Coren: yeah [17:48:14] I think the sql client is too dumb for this and tries to fetch the entire result set at once before displaying it; you'll need to script something with a cursor. [17:48:21] Betacommand: if you really want it you should write a script that takes a cursor and iterates through them one at a time [17:48:29] damit [17:48:56] Coren: not sure if you knew, http://charcoal.wmflabs.org/ now is picking up metrics from all of labs, including tools ;) tools-redis is already sending stats there, will add for the proxy soon [17:50:38] I didn't. Neat. [17:51:04] Coren: I'll email labs-l soon [17:59:25] Coren: also we're planning on setting up http://grafana.org/ as a frontend as well, since graphite's UI sucks [17:59:48] I've seen worse. :-) [18:00:05] Coren: true, but :D [18:00:21] Coren: prod is also going to set that up soon (there's a fairly complete patch already from ori), so I'll clean it up a bit and set it up on labs first [18:02:48] 3Wikimedia Labs / 3Infrastructure: Create -latest alias for dumps - 10https://bugzilla.wikimedia.org/45646#c7 (10Ariel T. Glenn) The dumps script does this at time of dump creation, leaving other symlinks in the directory untouched. Someone would have to write a short script that goes through and updates li... [18:16:29] Earwig: around? I found what was causing the windows installer generators to bork, so I can build binary packages now. What Python versions should I build for? [18:16:38] uhmm [18:16:53] 2.6, 2.7, 3.2, 3.3, 3.4? [18:17:06] https://github.com/earwig/mwparserfromhell/blob/develop/setup.py#L63 [18:17:07] these [18:17:08] I just realized there's 32 and 64 bit versions of them, too. Ugh. [18:17:13] yeah, it's a pain. [18:17:23] what did you fix? [18:17:36] forcing __version__ to be str instead of unicoded [18:17:37] -d [18:17:47] ...weird. [18:18:33] the msi generator wants str; the wininst generator concatenates it to the readme [18:19:16] but the readme contains non-ascii, of course, so everything breaks. It's still sort-of broken, as wininst is of course ansi, so Σ's name gets borked [18:19:25] but hey, no-one reads the readme in installers anyway :-p [18:19:33] :/ [18:19:42] that is painful [18:22:31] Yes, it is. [18:57:36] valhallasw: what about on python 3? because I can just remove the `from __future__ import unicode_literals` line in __init__.py, that might be better. [18:58:28] Earwig: except __version__ is imported from mwparserfromhell.py [18:58:40] there is no mwparserfromhell.py [18:58:46] er [18:58:48] wait, __init__.py [18:58:49] I'm talking about the __init__.py that defines it [18:58:50] you're rightg [18:58:50] yes [18:59:07] in any case, in python 3, str() is a no-op; I'm not sure if it builds (I don't have VC2010 installed atm) [18:59:50] but I think it's more appropriate to have hacks closer to the problem [19:00:06] in this case, the crappy distribute (or setuptools or wtf it's called these days) implementation [19:00:35] but the question is how python 3 will handle unicode (i.e., what it calls strings) that are not ascii [19:00:45] with regards to your windows build thing [19:01:48] I don't know :-) [19:01:59] but I would be surprised if everything had to be bytes there [19:03:09] oy [20:42:19] Earwig: ok, I'm going for a slightly less elaborate target: win32 (my VM is a 32 bit windows 7 >_<) [20:42:33] so like [20:42:39] what do I need to do here [20:42:40] I'm confused [20:42:50] there are too many different formats involved [20:44:00] Earwig: I'm going to make a) .whls and b) .exe installers for win32 2.6 2.7 3.2 3.3 3.4 [20:44:10] why do we need both? [20:44:23] I guess we can just let people use pip, too [20:44:33] but I'm not sure if pip is available on a default windows python install... [20:45:05] I think python 3.4 or greater yes, less than that no [20:45:07] I think we should assume it exists [20:45:25] I mean, if that makes our job easier [20:45:33] yeah [20:45:39] no reason to expend extra effort if people won't bother installing it [20:46:01] ok, works for me [20:46:17] just wheels it is [20:49:05] Earwig: last thing I'd need is upload access to the pypi page, but I'll use the test page first to make sure everything works [20:49:15] okay [20:49:31] I can do that [21:55:54] Earwig: good news: you also get free test coverage on windows, which immediately spits out some bugs :-p [21:56:03] :/ [21:56:49] basically, you load in an utf-8 file without specifyign it's utf-8 [21:56:54] let me see how to reproduce on linux... [22:02:27] Earwig: just a quick fix in _test_tokenizer, so it's not too bad :-) [22:07:21] Earwig: and now another real bug, probably due to the new internal unicode representation in 3.3 [22:07:27] but 2.6, 2.7 and 3.2 are working [22:07:31] what's broken? [22:08:30] https://github.com/earwig/mwparserfromhell/issues/78 [22:08:54] uhm [22:08:58] hm, okay [22:09:07] ...what the hell, actually [22:09:18] what could that be about? [22:10:47] 3.3 has a new internal string representation [22:11:11] I know, but it works on OS X / linux [22:11:22] uhuh. [22:11:28] I'm not sure, really. [22:11:42] just entering "\U00010332" works without issues [22:14:48] is there a logstash for labs? [22:15:18] yurikR: https://logstash-beta.wmflabs.org/ for beta. No general one for labs at the moment [22:15:50] bd808, awesome, thanks, exactly what i needed! :) what's my login? [22:16:25] Read the prompt. It gives a page on officewiki to read for the user/pass [22:16:33] Totally lame I know [22:16:57] bd808, lol, i saw it was bdavis's page, so i figuerd i might as well ask you directly [22:17:12] I'd have to find the page myself ;) [22:18:01] Earwig: anyway, shall I upload the wheels for 2.6 .7 and 3.3? [22:18:23] hm [22:18:32] you mean 3.2? [22:18:36] sorry, 3.2 [22:18:51] 3.4 also doesn't work? [22:18:52] bd808, thx, got it [22:18:54] nope [22:18:55] same error [22:21:01] okay [22:21:02] well [22:21:12] shall I add you as a maintainer on pypi? [22:21:16] yeah, sure. [22:21:19] 'valhallasw'? [22:21:22] yep [22:21:39] alright [22:21:41] give it a shot. [22:27:46] yesssss! [22:27:47] https://pypi.python.org/pypi/mwparserfromhell [22:27:56] ñow let's see if I can pip install it here :-) [22:28:02] (workstation, not the VM) [22:29:06] yep, works perfectly [22:32:16] hmm [22:33:38] okay, cool [22:33:40] good work [22:33:49] https://github.com/earwig/mwparserfromhell/pull/79 [22:35:02] should probably have put it in scripts/ [22:35:23] also, you're using tabs [22:35:26] I'll fix that. [22:36:08] oh, crap, you're right [22:36:14] (or you can) [22:36:15] that's because it was a batch file originally [22:37:56] if you want it in /scripts/, go ahead, but I can't do that from the web interface :-p [22:37:58] ah, you broke it [22:38:15] you dedented too much [22:38:37] well, crap. [22:38:47] leave it alone and I'll fix the stuff [22:38:47] it's clearly too late to be coding :-p [22:38:53] thanks for your help [22:39:14] Thanks & you're welcome :-) [22:39:31] please let me know when you release a new version, I'll fire up the VM and build some stuff then [22:39:37] will do. [22:39:39] or when you need help debugging that 3.3/3.4 issue... [22:39:58] I'll try to get my own VM set up at some point so I can do this myself [22:40:10] there's free windows vm's from MS at modern.ie [22:40:30] hmm [22:40:59] other than that, you need VC++ 2010 EE from http://www.visualstudio.com/downloads/download-visual-studio-vs#DownloadFamilies_4 [22:41:28] and python3.4, pip, setuptools and wheel [23:28:10] bd808, how do i do a sync-file PrivateSettings on beta? [23:28:28] updated it on depl-bastion [23:29:07] yurikR2: /usr/local/bin/wmf-beta-scap [23:29:46] bd808, thx!