[00:10:55] Project WikipediaMobile - Nightly builds build #361: SUCCESS in 13 sec: https://integration.mediawiki.org/ci/job/WikipediaMobile%20-%20Nightly%20builds/361/ [00:10:55] jan_christiansen: Add entry for sharing page via mail to messages-qqq [00:15:09] we should have github ping IRC whenever someone commits to WLM master [00:16:03] yuvipanda: it's easy to configure [00:21:20] preilly: I don't have admin access. Can you do so if you do? [00:25:02] yuvipanda: I can do it in the morning [00:25:55] preilly: thanks :) [00:48:45] time for college. [01:04:22] preilly i was just looking over things for updating the redirector code for wikimedia.org domains [01:04:39] it occurred to me that we need to not let commons.wikimedia.org redirect to .m yet [01:05:52] i noticed a fancy php file for testing the redirector, which i presume will help me make sure i get the regex change to capture all wikimedia.org except commons correct [01:06:09] but i don't actually know how to properly compile redirector.c [01:09:05] preilly i installed libpcre3 and ran 'gcc redirector.c' but i'm getting undefined reference errors [01:09:46] preilly: http://dpaste.com/772436/ [01:11:54] awjr: to compile see To compile: gcc -O3 -o redirector -lpcre redirector.c [01:12:16] awjr: at the top of the file [01:12:24] p [01:12:27] derp [01:12:29] thanks preilly [01:12:37] np [01:15:58] awjr: are you using this repo ssh://gerrit.wikimedia.org:29418/operations/debs/squid.git [01:16:08] preilly yeah [01:16:27] so why don't you want to redirect commons? [01:16:42] I mean http://commons.m.wikimedia.org/wiki/File:Nord-Ostsee-Kanal_Grundsteinlegung_01.jpg is a valid URL for example [01:17:35] commons redirection isn't supposed to happen until august [01:18:44] at least according to http://www.mediawiki.org/wiki/Mobile_default_for_sibling_projects#Timeline [01:19:10] awjr: okay [01:41:16] awjr: probably don't want to match bits [01:41:30] ? [01:41:54] e.g., ^http:\/\/(?!commons)(\w+)\.(wikimedia|wikipedia|wiktionary|wikinews|wikisource|wikiquote|wikibooks|wikiversity)\.org[:\d]*\/wiki\/(.*) [01:42:22] you mean bits.wikimedia.orG? [01:42:47] preilly ^ [01:43:03] yeah [01:43:07] it won't have the /wiki/ [01:43:12] yeah you're right [01:43:15] but still just to be safe [01:43:31] yeah i'llf ix it [01:43:47] thanks [01:45:36] np [16:44:06] good morning, mobilistas [16:44:26] yo [16:56:19] awjr, I've reviewed your patch [16:58:11] MaxSem cool thanks [16:58:15] what do you think? [16:58:55] does bulk-read of the whole table work faster than the one that retrieves just what needed with GROUP BY adm0, adm1, adm2, adm3, adm4 ? [16:59:36] a LOT [17:00:01] those joins and group by's are VERY expensive [17:00:33] okay [17:00:39] i tried one of those inserts for adm3 - it never finished before i killed it after ~700s [17:00:59] my script, in non-verbose mode, does the whole thing in under 10s :) [17:01:04] at least on my local machine [17:01:16] next question: you don't escape anything in SQL queries [17:02:25] i thought the values in the where clause get escaped [17:02:47] $query = "INSERT INTO admin_tree ( `level`, name, parent ) VALUES ( 0, '$adm0', NULL )"; [17:02:58] oh in the import script [17:02:59] that's true [17:03:06] $insert_query = 'INSERT INTO admin_tree ( `level`, name, parent ) VALUES ( %u, "%s", %u )'; [17:03:11] $query = sprintf( $insert_query, $level, $adm, $parentId ); [17:03:25] it's assuming non-evil data in the originating db [17:03:30] but perhaps that is short-sighted [17:03:31] also, double quotes are non-standard [17:03:57] 1) this data contains single quotes [17:04:27] 2) fileds get pulled from the wiki, so there's nothing preventing a directed SQL injection attack [17:04:35] good point [17:04:48] i can clean that up pretty quick [17:06:24] another nitpick: because you weren't checking return codes, you didn't notice that some queries failed due to unescaped data [17:07:48] doh [17:08:40] ok, i'll fix those up now - anything else? [17:11:12] you didn't use the Database class because it lacked abstractions for the write commands? [17:11:28] in the import script? [17:11:33] i didn't use it because i hate it. [17:11:59] but feel free to talk me into it [17:13:23] had you used the abstractions, you would have noticed the script malfunctions because it would have thrown a nice exception. I'm not insisting though [17:13:58] yeah and we'd get the escaping for free [17:15:42] though with mysqli you get prepared statements [17:17:18] that is true [17:17:22] although i've never used those before [17:53:28] MaxSem: just sent you an updated patch [18:10:12] * yuvipanda looks around [18:13:29] awjr, looks good ,will commit with a few fixes [18:14:16] thanks MaxSem [18:24:20] awjr, problem: memory_limit on TS is only 128M :P [18:25:17] that is a problem [18:25:45] MaxSem for cli scripts as well as web? [18:25:57] yup [18:26:16] trying ini_set [18:26:29] crap [18:26:34] i forgot i had memory_limist = -1 [18:28:36] mmm, it's running for several minutes for me already [18:29:01] what did you wind up setting for the memory limit? [18:29:04] * MaxSem looks for swapping, as he has only 16 gigs RAM [18:29:13] lol [18:29:24] that is way more than i have [18:30:45] "Inserted 13214 admin zones in 314.70701098442 seconds." [18:31:22] MaxSem if you are a non-believer, try doing it with just MySQL queries [18:31:29] it should convert you pretty quickly :) [18:31:33] I know) [18:31:38] also, it will go faster if you disable verbosity [18:32:14] once upon a time, query killer killed monuments_all rebuild because iran for over an hour [18:32:23] yay to spatial indexes [18:32:25] ugh [18:32:58] since then, it uses my tiled search from GeoData [18:33:09] query killer killed monuments_all because of iran? [18:33:12] ok [18:33:26] lulz [18:33:45] * yuvipanda is on GPRS until he gets broadband here, might not be too useful for a few days [18:52:13] MaxSem: you should reply to rt 3221 - our window for getting it taken care of is shrinking quickly! [18:52:30] ouch, got carried away [18:52:45] awjr, [18:52:52] preilly when you've got a minute can you look at https://gerrit.wikimedia.org/r/#/c/16000/3? [18:52:54] [18:52:59] [18:53:21] MaxSem: yeah i know. i mentioned that in my notes about the patch i sent you yesterday. [18:53:38] i can take care of that, but it will take a fair amount of work [18:53:59] the API code is very inflexible [18:54:45] if we can live it, it's probably best to leave it as-is for now [18:54:57] but if it will cause problems, i can try to get it resolved [18:54:58] I vote for the latter [18:55:06] fixing it? [18:55:14] eh, the former [18:55:16] heh [18:55:19] aye [18:55:44] it offends my sensibilities but it will be a pita to fix [19:01:11] awjr, one problem: what if there will be different trees of the same name and level? [19:01:28] e.g. Main street in different cities [19:02:01] or Boston, Mass vs. Boston, UK [19:02:33] hmm [19:03:26] well they would have different parents, right? [19:03:50] oh MaxSem are you talking about for the API rather than the import script? [19:03:59] yes [19:04:03] awjr, also update schema on your machine, you still have the old admin_tree [19:04:19] what is the new schema? [19:04:29] without `country` [19:04:43] that is what i am using [19:05:04] but your .sql file has a different (and correct) definition [19:05:12] see erfgoedbot/sql/create_table_admin_tree.sql [19:05:16] I had to fix the API thus [19:05:36] `id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY, [19:05:36] `level` INT NOT NULL, [19:05:36] `name` VARCHAR(255) NOT NULL, [19:05:36] `parent` INT [19:06:08] okay, I'm committing because I actually had to fix some things [19:06:23] MaxSem oh i see on line 237 of the api i left the country field in there [19:06:26] that can be changed to adm0 [19:06:31] but I think we need to expose ids for the tree and allow searching by them [19:06:47] adm0 is what I, tried, yes [19:06:57] however the right one will be 'name';) [19:07:16] oh right [19:07:17] derp [19:07:36] r759 [19:08:03] i can add support for searching by and display ids [19:08:05] should be quick [19:08:23] okay, deploying to TS, then back to RT [19:08:36] cool - should i add the id support? [19:09:00] yes [19:09:03] ok [19:09:06] i'll do that after lunch [19:09:25] and add the new function to API help, as it's not autogenerated [19:09:37] jdlrobson: i spent some time looking into the cookie problem and i have a theory [19:09:44] MaxSem ok [19:10:44] jdlrobson the stopMobileRedirect cookie is not getting unset when the user gets a cached page, which i thought we had dealt with a long time ago, but maybe we didn't, or something changed [19:11:11] i think a fix would be to move view toggling to a special page [19:11:28] hmm, populate_adm_tree.php dies quietly on TS [19:11:37] :( [19:12:00] MaxSem maybe up the error reporting [19:12:08] yup [20:13:07] awjr_lunch: that sounds like it could work. [20:17:12] awjr_lunch: You should have a toolserver account now [20:17:56] wheeee [20:25:18] awjr_lunch & preilly, could you take a look at https://gerrit.wikimedia.org/r/#/c/15905/ ? [20:25:42] awesome thanks multichill! [20:26:09] Thank DaBPunkt :-) [20:28:19] doh i'm getting 'connection closed' when i try to ssh in [20:28:34] multichill ^ [20:28:51] You need to use your private key [20:29:21] You can edit your ~/.ssh/config to include: [20:29:45] Host *.toolserver.org IdentityFile ~/.ssh/ User [20:29:57] multichill: i am using my private key [20:31:11] You might want to check the manual and if that doesn't work join #wikimedia-toolserver [20:42:28] ok problems solved [20:48:43] preilly: https://gerrit.wikimedia.org/r/#/c/16000/3 [20:49:04] whee, just opened my nexus 7 and it already wants to install a system update. now, where's my jelly bean for my galaxy nexus *grr* [20:49:05] asher told me a few days ago he'd be deploying this today but i haven't heard back from him about when exactly [20:49:33] brion judging by verizon's past update schedule, i'm not holding my breath :p [20:51:05] true :P [21:08:06] New patchset: MaxSem; "Add MediaWiki:Mobile.css and Mobile.js to mobile view" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/14419 [21:13:55] MaxSem: https://gerrit.wikimedia.org/r/#/c/15905/3/api/ApiQueryGeoSearch.php [21:14:00] ~line 170 [21:14:37] i dont really know much about sphinx or its queries but i noticed you're diong $query = implode( ' | ', $vals ); and then a few lines later, $query .= ' '. implode( ' | ', $vals ); [21:15:07] is there supposed to be another | or some other delimiter between the two sets of vals, or is a space sufficient? [21:15:13] $vals got reset between them [21:15:27] yes but im more concerned about $query [21:15:33] yes, space means an implicit AND [21:15:36] oh ok [21:15:37] cool [21:15:58] I've actually uploaded a new patchset that explains how this query works:) [21:16:07] oh he [21:16:10] ok cool [21:17:06] MaxSem in intRange(), can granulatiry == 0? [21:17:29] judging by the math the follows probably not [21:18:16] well, if users set it to zero in config it's their problem. they might as well set wrong DB credentials [21:22:24] preilly, we didn't finish the GeoData discussion yesterday [21:27:29] awjr, re "I do not have any meaningful way to test": I'll set it up on Labs [21:27:35] :D [21:28:26] remind me, where's autoexec.bat on Linux? :P [21:28:38] >_< [22:12:21] awjr, http://mobile-geo.wmflabs.org/w/index.php/Special:ApiSandbox#action=query&list=geosearch&format=txt&gspage=San%20Francisco&gsradius=10000&gslimit=1000 [22:17:04] MaxSem i dont get any output from that [22:17:23] tried clicking on "Make request"? [22:17:32] i dont see that :( [22:17:51] oh [22:17:55] im in the mobile view [22:17:58] that's why [22:18:47] cool! [22:19:31] wow this is really sweet [22:21:22] MaxSem: merged [22:21:38] thanks! [22:22:33] preilly: sandbox/jdlrobson/ds [23:18:37] jdlrobson et al: about https://bugzilla.wikimedia.org/show_bug.cgi?id=34201 - do we have our own OSM tile server already?