[00:01:27] there's supposedly a 'scheduled power outage' for majority of the day today [00:01:32] i'll sleep through it. yay! [00:01:36] YuviPanda: nice [00:02:14] :) [00:55:28] dev ops borat is so right https://twitter.com/DEVOPS_BORAT/status/213065811240296448 [00:55:35] YuviPanda: --^ [00:55:56] hah [00:56:00] and double hah at 'can not able to' [00:56:20] actualy, triple hah at 'can not able do' [00:57:07] tfinc: i've setup the ant file so I can make release builds from the commandline, and don't have to use eclipse for that :P [00:57:17] so i've never actually had to use eclipse in a long time now. Big win! [01:05:54] i think it is bright enough for me to sleep [01:05:55] gnite everyone [13:44:21] YuviPanda, what will happen with the existing WLM database - will it be moved to our server or discarded and regenerated? [13:44:36] MaxSem: the bot regenerates the database every day. [13:44:46] from scratch? [13:44:49] MaxSem: primary data source is wiki, so it iwll be regenerated [13:44:51] MaxSem: yes [13:44:53] takes a few minutes [13:45:10] it does have some heuristics to not regen from unchanged pages though [13:45:11] (IIRC) [13:45:23] but regeneration was taking small enough time for it to not matter [13:45:32] if it's just a few minutes, it doesn't make sense to integrate it with our main DBs [13:45:50] 'integrate'? [13:46:47] yeah, as in mysql_connect() [13:47:54] MaxSem: i'm unclear on how our mysql cluster is setup, but I assume we'll just create a new database in the cluster and use it for this? [13:48:12] I've seen some unused definitions for accessing the TS's copy of commonswiki though [13:48:31] MaxSem: yeah, they're unused and multichill had no idea why they were there [13:48:59] this thingie should be preferrably isolated from the main DBs because it's a mess [13:49:21] DAL looks sane though [13:49:34] hmm, so run own instance of mysql server on machine [13:50:12] yeah [13:50:37] one server should be enough for web+mysqld+bot [13:51:06] but the schema is one big fucking WTF [13:51:33] :D [13:51:43] why the hell create separate databases for different countries? [13:51:58] aren't those separate tables/ [13:51:58] ? [13:52:07] meh, tables, right [13:52:09] still wtfy [13:52:30] makes no difference [13:52:46] poor man's sharding when no sharding is needed [13:53:21] :D [13:53:48] i'm guessing it was just setup for netherlands, and when it expanded to europe, people just went 'I know what i'll do - i'll just copy paste!' [13:58:40] is there some kind of big picture of this? [14:00:10] MaxSem: of the code? [14:00:11] no [14:00:25] of WTF it's supposed to be used for [14:00:53] btw, http://mobile-wlm.wmflabs.org/api/index.php [14:01:14] MaxSem: hmm, use cases. i'm unsure if tehre are any [14:01:15] *there [14:01:16] let me check [14:01:22] eh. looks down. [14:01:52] okay: in two words, what does our app need it for? [14:02:53] MaxSem: it actually lists the monuments so that people can figure out *what* they can take photographs of [14:04:40] so everything except the actual uploading will be via this. [14:04:42] this is the 'browse' [14:11:29] what the TS instance of it is being used for? [14:12:18] MaxSem: hmmm, not sure. [14:12:49] so the question is whether we're looking at forking or replacing it [14:14:39] MaxSem: platonides should have a better idea of what it is being used for (multichill isn't online) [14:14:42] poking him [14:23:19] brb food [14:23:54] MaxSem: also, respond on the email thread about forking vs replacing? I think that'd be the best place [16:05:54] jdlrobson, YuviPanda - I tried to watch a .ogv video on my iPhone and it didn't like it, are we working on a player to make videos work on phones? [16:06:42] Thehelpfulone: not at the moment, no [16:07:13] is it on a list somewhere? [16:08:43] Thehelpfulone: nope. you can file a bug! [16:09:22] sure, as part of mobile overall right? [16:09:34] yes [16:09:51] but i'm unsure how much support it will get from the browser directly [16:10:00] iirc it isn't supported on safari, and probably will never be [16:36:39] YuviPanda, in which channels Multichill is usually present? [16:36:56] MaxSem: #wikilovesmonuments, but he isn't around right now [16:37:06] i think we can also ask platonides, but he didn't respond to my earlier ping [16:57:11] mm MaxSem the watermark logo in the footer of wikipedia mobile seems to have disappeared.. [16:59:21] jdlrobson, I suspect Arthur's 40c8303682f0a457c489684d5ae6ff8deca23c42 [16:59:32] ok [16:59:44] also apparently a few people are reporting the WP7 bug is still there [16:59:59] which is strange as on my physical WP7 device I don't see the problem anymore [17:01:03] mmm https://bugzilla.wikimedia.org/show_bug.cgi?id=37328#c4 [17:01:14] varnish? [17:04:39] jdlrobson: forwarding you another win7 report [17:04:55] possibly MaxSem [17:05:02] heads up we're likely to need to push a change today [17:12:51] YuviPanda|Food, regarding tfinc's latest priorities: what's currently slow? [17:13:14] MaxSem: filtering is super slow [17:13:17] on order of minutes [17:13:23] let me get the exact query [17:14:23] so... our varnish change took out blackberries.. [17:14:30] replicated it on mobile-geo [17:15:27] MaxSem: along the likes of https://toolserver.org/~erfgoed/api/api.php?action=search&format=json&srname='%hi' [17:19:21] hmm, that is actually not so slow anymore, despite not actually returning results [17:19:47] are the single quotes required? [17:20:14] no, that was a typo [17:20:14] https://toolserver.org/~erfgoed/api/api.php?action=search&format=json&srcountry=es&srname=%25hi%25 [17:20:16] is fine [17:20:23] as in, fast, but zero results [17:20:40] https://toolserver.org/~erfgoed/api/api.php?action=search&format=json&srcountry=es returns a ton of them [17:21:29] https://toolserver.org/~erfgoed/api/api.php?action=search&format=json&srcountry=es&srname=Castell%20dels%20Comtes%20de%20Ribagor%C3%A7a%20(Estopany%C3%A0) works [17:21:49] https://toolserver.org/~erfgoed/api/api.php?action=search&format=json&srcountry=es&srname=C% doesn't [17:22:37] maybe LIKE was never there, and the slowness brion experienced was something else? [17:22:39] * YuviPanda looks at the code [17:22:42] C% is broken percent-encoding [17:22:57] so we need to catch brion? [17:23:57] MaxSem: bad copy pasta. https://toolserver.org/~erfgoed/api/api.php?action=search&format=json&srcountry=es&srname=C%25 doesn't work [17:24:16] and the code *Does* have a LIKE clause that is triggered when there's a % in it [17:24:19] which doesn't seem to be working [17:24:58] MaxSem: well, filters needed to be sped up, but now they seem to be not working at all! [17:25:21] well, that's certainly fast [17:26:49] :P [17:27:46] * YuviPanda goes to file a bug [17:34:14] preilly_: two important bits 1) jdlrobson needs you for a push 2) these changes need review https://gerrit.wikimedia.org/r/#/q/status:open+project:mediawiki/extensions/Scribunto,n,z [17:38:43] preilly: > https://gerrit.wikimedia.org/r/11309 [17:40:33] pffft, this API is too flexible to work reasonably in all cases. you can throw a few indexes at it to cover some use cases you need (if these use cases are reasonable), but overall users can shoot themselves on foot with nice unindexed queries, bringing mysql to a crawl [17:41:34] jdlrobson: it will be live in like 15 seconds [17:41:39] awesome [17:41:46] jdlrobson: okay it should be live now [17:42:01] MaxSem: hmm, which in particular is problematic? [17:43:16] two LIKEs = partially indexed query, at best. LIKE other than foo% = unindexed filesearch [17:44:15] if someone asked me how to make it perform fast, I'd say redesign the shit out of it to leave just a small subset of possible queries [17:45:22] MaxSem: mysql's fulltext index? [17:45:33] ?? [17:45:44] https://dev.mysql.com/doc/refman/5.0/en/fulltext-search.html [17:45:49] oh [17:45:50] only myisam [17:45:51] nevermind [17:46:05] well, they use myisam already, ololo [17:46:26] I know what fulltext search is, what do you want with it? [17:46:49] no, i was wondering if %foo% would be indexed when using that? [17:46:51] * YuviPanda is database noob [17:47:40] no, a like '%foo%' would i think not benefit from indexing [17:47:47] like 'foo%' would becuase we have a fixed prefix [17:48:03] i don't think there's any smarts for handling like with fulltext indexes [17:48:43] oh, here he is! [17:48:51] hi brion [17:48:55] hey brion [17:49:01] yo [17:49:30] brion, we wondered what were the deathly slow WLM queries you encountered? [17:50:15] so those would have included a 'sr_name like "%foo%"' or something (i forget the field names) [17:50:22] the comment in the source was for 'data.srname = '%' + filter + '%';' [17:50:30] it's super fast now, except that it doesn't work [17:50:32] (always empty) [17:50:33] heh [17:51:22] well, %foo% is %foo% [17:52:15] MaxSem: any way of getting %foo% at all without having to go solr/similar? [17:52:38] any way of living without it?:P [17:53:07] ok MaxSem - WP7 problem is still in the non-beta [17:53:10] could use fulltext index and use it explicitly instead of the like, but i've never had good experiences with it at scale [17:53:10] hasn't been flushed [17:53:22] I suspect it is the same problem as we had with the settings page.. [17:53:34] plus that means using myisam (eww) [17:53:54] brion, you'll laugh but these tables are already MyISAM [17:53:59] *sob* [17:54:35] well, looking at some of those names I don't think prefix-only search would do [17:56:26] MaxSem: i think we need to reach a fork / replace decision before we can move on [17:56:38] * YuviPanda goes to poke platonides again [18:00:27] YuviPanda: ready to chat? [18:00:37] tfinc: yup! skype/gchat? [18:00:41] Skype is fine [18:01:17] tfinc: okay, i'm on [18:16:48] tfinc: > https://twitter.com/sayonical/statuses/213333101403832320 [18:16:56] woot! [18:18:57] also reply from Nick over email -> "Yep, just gave it a try on the same article and it worked an absolute treat! Thank you for the very speedy fix and reply! :)" [18:24:04] \o/ [18:25:40] jdlrobson: does this also fix the issues on black berrys ? [18:25:49] blackberrys should be fixed yes [18:25:51] i'll message [18:26:39] jdlrobson: tweet out so that people know [18:26:52] preilly: thanks for the cove reviews [18:28:12] code* [18:32:56] Platonides, yo [18:33:48] hi MaxSem [18:34:39] Platonides, me and YuviPanda had been discussing the WLM API and the possibility of its forking/rewriting [18:34:56] I'm reading the backlog [18:35:33] I saw you complaining that it was too flexible :P [18:35:48] hehe [18:35:52] well, the reason for that is that when we did it we had no idea what would it be used for [18:36:10] not that I have more idea ;) [18:36:24] I don't know if someone really used its full potential [18:37:37] why the hell create separate databases for different countries? [18:37:48] different countries = different fields [18:38:02] that's why there's one table per country [18:38:05] it's generalisable [18:38:19] using many NULL columns? [18:38:38] e.g. `municipi` AS `adm1`, [18:38:39] feel free to mock it up [18:38:53] why not call it adm1 in the first place? [18:39:05] that was added later [18:39:11] `adresse` AS `address`, [18:39:42] country tables usually have more columns than those copied at the monuments_all [18:40:17] also there are some odd cases [18:40:39] such as half monuments using a kind of level1 subdivision, other another type of level 1 [18:40:45] due to coming from different lists [18:41:09] some monuments appear on different sources and thus we can get both subdivision types [18:41:16] how do you handle that? :( [18:41:36] e.g. monuments_dk-bygninger_(da) vs. monuments_dk-fortidsminder_(da) [18:41:38] ? [18:42:17] but that's not the main problem [18:42:47] the main problem is flexibility [18:42:50] I was thinking in monuments_dk_(dk) vs monuments_es (es) [18:42:53] yo preilly [18:43:01] and LIKE %foo% [18:44:22] I know that's not efficient [18:44:30] but I don't know the use cases [18:44:53] well, brion had such use case [18:45:19] what was he using substring search for? [18:45:22] so we might have hundreds or thousands of entries in a city/region [18:45:32] it's nice to be able to slap in a string search to go for something specific [18:45:32] Platonides: the mobile app would definitely want a substring query [18:45:35] yeah [18:46:20] so you are at the same time complaining that it's too flexible allowing substring search and saying it's needed? :) [18:46:49] hehe [18:47:03] it's also conceivable that the query plan has to be forced to do substring search after other prunings [18:47:04] well, except https://toolserver.org/~erfgoed/api/api.php?action=search&format=json&srcountry=es&srname=C%25 doesn't work now :P [18:47:06] yeah, current API simply can't be fast :P [18:47:18] and when brion tried it a while ago, it worked but took a looong time (~minutes) per query [18:49:29] there's no reason for queries for text beginning with a single letter to be slow [18:50:25] Platonides: just an example. It is broken right now, ofcourse. [18:50:29] https://toolserver.org/~erfgoed/api/api.php?action=search&format=json&srcountry=es&srname=Castell%20dels%20Comtes%20de%20Ribagor%C3%A7a%20(Estopany%C3%A0) works, however [18:50:35] there is: SELECT * FROM foo WHERE a = 1 AND b LIKE 'foo%' [18:50:54] you can't fully index this query [18:51:40] and if MySQL's stupid query optimizer decides to use an index on a, b will be unindexed [18:51:43] AFAIK [18:51:49] in theory the a=1 etc should cut us down a lot, so the like only has to operate on a few hundred or thousand rows, which shouldn't be too bad [18:52:00] either the country/municipality or the bounding box [18:52:16] but if it's doing the like search on the full table first, …….. gonna be slow as crap [18:52:25] can we get an EXPLAIN on the actual query and see what's going on? [18:52:52] the C% query? [18:52:55] yeah [18:53:14] the working query, not the one we have now:) [18:53:19] hehe [18:54:17] ok, so https://toolserver.org/~erfgoed/api/api.php?action=search&format=json&srcountry=es&srname=C%25 maps to [18:54:21] SELECT `country`, `lang`, `id`, `country`, `lang`, `id`, `adm0`, `adm1`, `adm2`, `adm3`, `adm4`, `name`, `address`, `municipality`, `lat`, `lon`, `image`, `source`, `monument_article`, `registrant_url`, `changed` FROM `monuments_all` WHERE (`country` IN ('es')) AND (`name` IN ('C%')) ORDER BY `country`, `lang`, `id` LIMIT 101 [18:55:06] * Platonides mumbles about why so much fields [18:55:20] ah, for the order by [18:55:45] the explain says: 1 | SIMPLE | monuments_all | ref | PRIMARY | PRIMARY | 32 | const | 11310 | Using where [18:56:19] for me it says Impossible WHERE noticed after reading const tables [18:56:22] hmm [18:56:31] but my table is empty [18:56:35] hehehe [18:56:41] ok we've got 'name in' instead of 'name like' [18:56:46] I wonder why it turned the LIKE into an IN [18:56:47] so that explains why that don't work [18:58:26] the proper query would be SELECT `country`, `lang`, `id`, `country`, `lang`, `id`, `adm0`, `adm1`, `adm2`, `adm3`, `adm4`, `name`, `address`, `municipality`, `lat`, `lon`, `image`, `source`, `monument_article`, `registrant_url`, `changed` FROM `monuments_all` WHERE (`country` IN ('es')) AND (`name` LIKE 'C%') ORDER BY `country`, `lang`, `id` LIMIT 101 [18:59:01] that's | 1 | SIMPLE | monuments_all | ref | PRIMARY | PRIMARY | 32 | const | 11310 | Using where | [18:59:06] country should probably be an = rather than ain in also [18:59:07] drdee: DeviceMapLogCapture [18:59:10] yes [18:59:16] shouldn't make a difference in theory but [18:59:25] I think I tried to make it IN if several, = if 1 [18:59:29] just as in mediawiki [18:59:47] ouch [18:59:54] if ( count( $value ) > 0 ) { instead of > 1 [19:00:08] :) [19:00:18] preilly: Create initial empty commit => yes / no? [19:00:43] drdee: no [19:00:48] ok [19:01:37] wait wait: complete fork of the api? [19:01:51] bbiab [19:04:22] joancreus, we need a stable source of information for the WLM app. one possibility to make a fast and reliable one would be to fork whatever we currently have on TS to the WMF cluster and tweak it to our needs. but that;s just one possibility we're discussing [19:04:44] stname is entering as array(1) { [0]=> string(2) "C%" } [19:04:57] that's why it's not passing the is_string( $value ) filter [19:04:59] oh i see. why not send those changes back to the toolserver svn? [19:05:17] preilly: hold on, looking at it [19:05:31] for example because it would break backwards compatibility [19:05:53] and because TS could be unpredictably slow [19:06:26] but I'm not saying that we WILL do it [19:06:39] we are discussing all the options [19:07:09] there could be a switch to disable expensive functions [19:07:54] there was actually a misunderstanding yesterday when I understood this task as 'move this service from TS to WMF' [19:08:11] but no, it's not what we need [19:08:30] we need just a working API for the app [19:08:49] what does the app need? [19:09:04] I haven't seen a list of requisites... [19:09:15] me neither [19:09:48] just that it's terribly slow with LIKE [19:10:12] if we want little little more speed, switch from connect to pconnect? [19:10:15] what the API has is more than enough - the only thing missing was the filtering by name. [19:10:18] it reuses a previous mysql connection [19:10:31] YuviPanda: filtering by name? fuzzy selectors are there afaik [19:10:43] srname=%foo% [19:10:47] they're slow [19:10:54] which is, I believe, the entire point being discussed [19:11:02] joancreus, connect() --> pconnect() is milliseconds [19:11:11] LIKE %foo% is seconds2 [19:15:26] tfinc: how is today looking for shipping wiktionary? [19:29:17] pfhayes: in a meeting [19:29:34] tfinc: okay no problem [19:34:46] the conversation seems to have died [19:55:21] New patchset: L10n-bot; "Localisation updates from http://translatewiki.net." [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/11449 [20:05:10] New review: preilly; "(no comment)" [mediawiki/extensions/MobileFrontend] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/11449 [20:05:13] Change merged: preilly; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/11449 [20:07:37] jdlrobson: ping [20:07:48] yo preilly [20:07:54] just about to have a chat with tomasz [20:09:22] jdlrobson: okay [20:11:15] * preilly — wow, NY Times reports that Nokia said on Thursday that it would slash 10,000 jobs, or 19 percent of its work force, by the end of 2013 as part of an emergency overhaul that includes closing research centers and a factory in Germany, Canada and Finland, and the departures of three senior executives. [20:12:31] * preilly — http://www.nytimes.com/2012/06/15/technology/nokia-to-cut-10000-jobs-and-close-3-facilities.html?_r=1 [20:12:39] ouch [20:13:37] * preilly — Nokia, posted a loss of €929 million, or $1.2 billion, in the first quarter as sales plummeted 29 percent. [20:15:51] yeah that windows phone bet is paying off in spades so far eh [20:16:05] brion: yeah, totally [20:16:38] brion: I'm curious if they announced a switch or at least support for android if that would make a big difference at all [20:17:27] android would probably sell better but the common wisdom is that there's still not much profit in selling android phones [20:17:56] now if they sold android phones and promised software updates... [20:17:58] i'd fucking buy one [20:18:10] still no android 4 update on my galaxy tab [20:18:13] heh heh [20:18:18] it's pissing me off man [20:18:22] wow [20:18:27] that's a bit surprising [20:18:38] they said q2 i think…. it's getting kinda late [20:19:05] they're probably wasting time porting TouchWiz [20:19:10] i don't want touchwiz man i want android [20:19:32] yeah, totally [20:20:45] "Mr. Jeronimo said Microsoft would be the most likely suitor for Nokia, whose share price has fallen by 75 percent since the company announced its Windows strategy last year. " ouuuuuch [20:24:37] did you see the chart in the apple wwdc keynote comparing version uptake on ios vs android? very sad [20:24:47] 7% on ics vs like 77% on ios 5 [20:24:50] yeah it was sad [20:25:09] I think that Google could really work to market things a bit better [20:25:25] and if they could update more frequently that would probably help as well [20:26:44] if more frequent releases keep the phone mfgs from customizing the system and making it hard to upgrade, i'm all for it [20:27:53] android was kind of ad hoc to start with, whereas iOS was more fully conceived at the outset, but its limitations are more baked in [20:31:34] btw, there is more profit in selling Android phones, that's why Samsung (and others) jumped on it [20:35:28] not as much profit as there is in iphones, but apple's got a seriously big profit margin :) [20:40:06] ICS distribution has been a disaster [20:40:17] i've seen less then a dozen phones that support it [20:40:21] and its been out how long? [20:40:22] FAIL [20:40:30] serious fail [20:40:36] where as think how fast iOS devices get upgraded [20:40:39] all the damn time [20:40:56] that's what Brion was saying: 7% ICS vs 77% iOS 5 [20:41:25] and thats because barely anyone android 2.3.x devices can get upgrade to ICS [20:41:26] i still can't test Chrome on my tablet because it's stuck on 3.2 [20:41:31] fail [20:41:50] companies are making it too easy for apple to dominate the world [20:41:55] its really not that hard [20:41:58] my old iphone 3gs will be upgradable to ios 6 on launch day, for god's sake [20:42:44] that's what happens by controlling end to end [20:43:27] i can also update my computers to new windows or ubuntu releases on launch day, and they're not vertically integrated [20:43:48] i don't see why it should be harder for phones and tablets [20:44:01] device lifetimes relative to updates suck on Android. simple as that [20:44:04] yea.. android fragmentation is a mess [20:44:32] not you could say that we never got updates back in the day on flip phones but i think we should be past that [20:44:42] but with my android device .. i honestly don't expect it to get any better then when i bought it [20:44:49] that's not the case with an iOS device [20:45:08] i got a few good updates on my nexus 1 before they abandoned it at 2.3 [20:45:17] and that was the best possible case: the official developer phone [20:45:34] ... Mozilla is doing good work here ... part of their boot to gecko efforts will also include installable android apps using a stable / updated web engine [20:45:43] \o/ [20:58:18] brion: phone makers don't make money upgrading the OS, but I suspect the issue is more about architectural changes as Android progresses, or drivers [20:59:17] phone makers make money, in theory, by building brand loyalty. if you give me updates i'm more likely to buy your future phone when it finally comes time to replace it [20:59:38] philinje: wouldn't phone makers make more money if they made more stock phones that sold for years on end ? [20:59:49] their model of constantly putting out crap actually makes them less money [20:59:54] due to extended rd and support costs [21:00:08] apple's actually still producing the iphone 3gs as its low-end model [21:00:10] where as if they would just put out 1-3 devices every couple of years … and those devices were good [21:00:16] then they would make a killing [21:00:21] which is what they want in the end [21:00:23] yeah, good points, but fundamentally they don't control the OS [21:00:58] well we'll see if the google-motorola purchase bears any interesting fruit :D [21:01:18] also driven a lot by carriers - unlike Apple [21:01:27] yeah, that was a while ago [21:01:41] carriers are evil :) so is apple, but they're an evil i understand better ;) [21:02:30] the devil you know… well, Apple does have usability as a primary goal [21:02:49] preilly: what is the next step with DeviceMapLogCapture? [21:04:14] whereas carriers have bad usability as a prime goal, at least as far as i can tell from verizon's website ;) [21:05:19] when i upgraded marti's iphone i gave up on ordering through verizon and ordered straight from apple instead [21:05:42] Platonides, how do I commit to the WLM repo? [21:06:42] MaxSem: i think you need to ask for commit perms on ts jira [21:06:49] brion: the other phone makers are slaves to the carriers, that was why iPhone scared the carriers so much [21:06:51] booring [21:07:16] tfinc: should we poke somebody about https://rt.wikimedia.org/Ticket/Display.html?id=2970 [21:07:20] i love the idea of separating the carriers from the phones more [21:07:35] i think it is a very US concept [21:07:37] it blows my mind that i can walk into a grocery store in germany and pick up a SIM card, stick it in my phone and go [21:07:51] brion, then move to RUssia where there are no locked phones [21:07:56] :) [21:08:00] same in India, but after a week of beaurucracy that are absolutely pointless :) [21:08:06] hehe [21:08:19] Reliance tried bringing on carrier locked phones, but they've migrated now to unlocked ones [21:09:00] 10 years ago, carriers tried to sell locked phones, but people simply unlocked them to get a cheaper unrestricted phone [21:09:16] GSM was a big success because of the openness, but funny enough in Japan they have had plenty of success with lcking to the max [21:09:30] locking [21:10:24] YuviPanda, I've fixed the empty result bug, need to commit it. now we're in the land of filesorts again [21:10:29] MaxSem, are you in toolserver erfgoed project? [21:10:34] no [21:11:16] I think you need to be in order to commit [21:12:12] Platonides, here's the fix meanwhile https://gist.github.com/2932978 [21:13:03] but why is it an array? [21:13:04] Platonides, do I need an endorsement or something in my request to join? [21:13:15] yes, but I can give it [21:13:29] joancreus could, too [21:14:25] I think the problen appeared when adding those parameters as ApiBase::PARAM_ISMULTI [21:20:52] Platonides, https://jira.toolserver.org/browse/TS-1409 [21:22:24] the bug was in r588 [21:26:29] MaxSem, https://toolserver.org/~erfgoed/api/api.php?action=search&format=json&srname=C%25 [21:26:55] whee [21:26:56] I don't know how could brion have made a LIKE to work, given that such feature seems broken since April.... [21:27:13] wow, DaB was fast [21:27:36] it never worked, it just died :) [21:28:03] Platonides, and the EXPLAIN for it: Using where; Using filesort [21:30:59] the query is SELECT `country`, `lang`, `id`, `country`, `lang`, `id`, `adm0`, `adm1`, `adm2`, `adm3`, `adm4`, `name`, `address`, `municipality`, `lat`, `lon`, `image`, `source`, `monument_article`, `registrant_url`, `changed` FROM `monuments_all` WHERE (`name` LIKE 'C%') ORDER BY `country`, `lang`, `id` LIMIT 101 [21:31:10] and the explain: | 1 | SIMPLE | monuments_all | index | NULL | PRIMARY | 141 | NULL | 101 | Using where | [21:31:39] even %C% is not bad [21:31:44] | 1 | SIMPLE | monuments_all | index | NULL | PRIMARY | 141 | NULL | 101 | Using where | [21:32:51] the order by is by the primary key [21:33:04] that is probably avoiding filesorts [21:33:09] Platonides, I'm still getting filesort [21:33:27] I don't know how it manages to only view 100 rows, though [21:33:51] I'm on 5.5.8 [21:34:56] have you tried it on the toolserver? [21:35:07] MaxSem: it is possible that the db on toolserver has indexes that one created solely from sql/ might not have. multichill was telling me that such discrepancies might exist [21:35:10] I don't know how I could have made this wrong [21:35:11] mysql> explain SELECT `country`, `lang`, `id`, `country`, `lang`, `id`, `adm0`, `adm1`, `adm2`, `adm3`, `adm4`, `name`, `address`, `municipality`, `lat`, `lon`, `image`, `source`, `monument_article`, `registrant_url`, `changed` FROM `monuments_all` WHERE (`name` LIKE '%C%') ORDER BY `country`, `lang`, `id` LIMIT 101; [21:35:39] a show create table shows the following indexes: [21:35:40] PRIMARY KEY (`country`,`lang`,`id`), [21:36:00] KEY `adm0` (`adm0`), [0 to 4, lat an lon] [21:36:25] and wty is it in MyISAM? :S [21:38:33] yeah, it's a regression in 5.5 [21:38:57] mysql Ver 14.14 Distrib 5.5.12, for solaris10 (i386) using 6.2 [21:39:20] ah, Server version: 5.1.59 Source distribution [21:39:34] exactly [21:40:04] MaxSem, were you trying on InnoDB or MyISAM? [21:40:14] myisam [21:40:27] we can't blame the engine [21:40:36] these definitions have engine explicitly set [21:41:21] I wonder why [21:43:07] has been like that since the beginning, r22 [21:52:31] New patchset: Jdlrobson; "serve mobile site to mobile browsers based on user agent (bug 36894)" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/11569 [21:52:59] New review: Jdlrobson; "Need help with phpunit several tests broken" [mediawiki/extensions/MobileFrontend] (master) C: -1; - https://gerrit.wikimedia.org/r/11569 [22:34:36] brion: what is your skype handle? [22:34:54] brionv [22:35:22] othx [22:36:28] time for that mtg? [22:36:58] yeah [22:37:18] i just added you as a contact on skype [22:37:26] approved [22:46:27] YuviPanda & brion, I think we should solve problems as they arise, so could you compile an approximate list of API requests for me so that I could analyze/fix the performance in these cases [22:46:53] MaxSem: will do. [22:47:10] but functionality wise, as I mentioned, the only thing that's currently problematic is the filter. [22:47:30] so I'm guessing fixing that would be good enough for now? [22:47:44] tfinc mentioned trying to spend minimal effort on the backend... [22:49:16] yeah, once we decide which fields will need complex filters, I'll patch it to use fulltext search for them [22:51:44] next thing I'm worried about are selects with multiple conditions but let's see how severe it can be in practice [22:55:52] most of the conditions we get should be either: [22:56:11] 1) country = X, municipality = Y [22:56:28] 2) a > lat > b, c > lon > c [22:56:39] and then a filter condition (on names, etc) [22:57:02] s/c/d [22:57:43] heh geospatial stuff [22:58:10] since it's myisam anyway... [22:58:31] a real rtree, at last:P [22:58:32] not sure if the geo indexes help (do they have geo indexes or just functions?) [22:58:38] worth looking at though [22:58:50] be nice to get things back ordered by distance [22:58:51] yes, myisam has real spatial indexes [22:58:56] \o/ [22:59:26] no, ordered by distance === filesort in SQL or sort in PHP [22:59:37] poop, that sucks either way [23:00:02] I studied these stuffs for GeoData