[01:04:52] jrobson: i see you liked my check in at southpaw. its a quality venue. been going there for ages. [14:47:50] Hi preilly, were you ever sent my GitHub information for the j2me repository? My username is caxthelm. [15:57:24] nice ! https://play.google.com/store/apps/details?id=wikokit.kiwidict.enwikt [16:59:14] hangout is up https://plus.google.com/hangouts/_/4697be442be7c85d91be834e9af4ff9a63021a89?authuser=1&hl=en [17:00:26] MaxSem: --^ [17:03:06] preilly, your beard! [17:06:39] MaxSem: ha ha [17:17:18] MaxSem: with the basic plumbing of the WLM app in place we need to look at next steps for its backend [17:17:34] yeah [17:17:39] what's the current state of the data, api, etc on tool server? [17:18:18] 1) I haven't really touched data collection so far, this is a question to Maarten [17:19:33] 2) API is sorta stable. We discussed the API with Yuvi yesterday, came to some vague conclusions, I'll look at implementing them Monday, nothing difficult here [17:20:01] MaxSem: have you heard from Yuvi today? [17:20:11] nope [17:20:21] ok [17:20:46] was further discussion needed about sorting? [17:20:47] 3) Etc. TS doesn't look stable or fast, I'm almost sure that it will not handle the load if the app will become popular [17:21:05] is the api a mw extension or its own beast? [17:21:24] philinje, I'll take a look which sorting is feasible for our needs performance-wise [17:21:31] tfinc, separate thing [17:22:03] though it resembles (and even reuses parts of) the MW API [17:22:23] ok, will copy you on an email to Yuvi about admin0-4 [17:25:38] MaxSem: we need to move it then. i'd feel more comfortable if it was running on the cluster like anything else. what would it take to move it to a mw extension ? [17:28:29] mmm, first of all, even though it has borrowings from the MW API, I'm not sure it will be possible to maintain API output format if switched to our API formatters [17:29:33] second, in its present state, this thing is not runnable on cluster's replicatable DB infrastructure, it'll require a dedicated DB server [17:32:00] third, with its present numerous possibilities to generate atrocious load on the server, I'm not comfortable with running it on anything but standalone app/DB servers [17:33:03] so if we seriously want to make it an extension, I'd say we'll have to fork and reingeneer it [17:34:03] to keep only the API queries that are guaranteed not to melt the DB [17:35:31] MaxSem: wow [17:35:41] hmm. thats a lot of risk and work for the move [17:35:47] cause currently it's a lego: you can make anything out of it, but it will collapse under any significant load. I mostly made it work properly with the kinds of requests our app will be making, but I don't want our cluster to have loopholes for DoS [17:35:49] can we make it scale if we don't move it to the cluster? [17:36:13] philinje: whats the projected usage of the app ? [17:36:20] huge [17:36:33] tfinc, we estimated it at 7 requests/second [17:36:41] philinje: actual numbers would be better [17:36:42] ok [17:36:42] let's take city of Cologne - 9000 monuments [17:36:48] we can run it on cluster. just do it on a standalone server not tied to the overall infrastructure [17:37:12] this way, if it goes down, the overall site is unaffected [17:37:14] MaxSem: a single machine is still a single point of failure. we want redundancy [17:37:51] then a standalone pool of machines [17:38:26] sounds also like other apps or sites could cause it to collapse, even if our app is being careful [17:38:47] the alternative is to fork and leave only stuff needed by the app [17:38:59] philinje: what other sites and/or apps are we talking about? we can easily limit this infrastructure to just our app [17:39:04] at least for launch [17:39:20] ...which means tha the TS version will have to continue existing for general use [17:39:20] i meant the Layar app or even Wikipedia [17:40:11] i saw we limit it to our own app. not layer not even wikipedia for the first round. if its successful then we'll open it up. but taking the time to scale it if we don't know its worth it is risky [17:40:36] yes, the point is that TS is accessible to anyone [17:40:48] MaxSem: if we fork it, would the data stay in sync? [17:40:52] and it will continue to be [17:41:09] MaxSem if we split it off would you imagine that we keep a slave copy of the data? [17:41:11] we don't need writes [17:42:06] well, the data evolves [17:42:48] I'm not sure how much but I can imagine it to become outdated during the campaign [17:43:23] there are also social issues with running it on WMF [17:43:31] social issues ? [17:43:51] we don't have to run WLM . i'm just talking about keeping a predictable and replicated copy of the data [17:43:54] interaction with volunteer WLM devs [17:44:58] running on WMF === sources in Git, strict code review, "no, you can't have this feature even if you want it, it's too slow" [17:45:51] MaxSem: can we stabilize the data model enough to keep a replicated copy ? [17:45:58] up to date replicated copy [17:46:23] we can, I think [17:46:42] live replication from TS to pmtpa looks scary though:) [17:47:11] what's brittle about it ? [17:47:41] Actually, there are several ways of running replication. [17:48:03] other than "OMG we haven't done it before"... [17:48:45] ...there's also a problem with the DB being bulk-updated once per day. this is not quite replication-friendly [17:50:13] so yeah, the picture currently looks like this: [17:50:44] we borrow one table from the TS, store it in WMF [17:51:13] we have an app-oriented API to it [17:51:49] we definitely shouldn't collect the data ourselves or maintain a generic-use API [17:54:24] if we can come up with reasonable use cases, I think we can make it past Asher [17:55:02] interesting thing here is the distributuion between different query types [17:55:58] we estimated the projected peak load as 160k pics last year * 10 times possible increase in contributions * 10 API requests per pic [17:57:46] the question here is whether users will make 9/10 of this as full-text searches (slow) or primitive enumerations by adm structure (fast) [18:17:12] hey guys, which servers are serving mobile traffic? [18:20:52] drdee: all of them [18:21:15] drdee: we've got 4 varnish boxes that talk directly to the backend apaches [18:21:49] thx, trying to see if i can find anything unusual period may 4th - may 23rd regarding drop in wikipedia zero traffic [18:32:59] [WikipediaMobile] brion pushed 2 new commits to master: http://git.io/zBaKEA [18:32:59] [WikipediaMobile/master] Kill overlay on searchbar hiding the spinner on ICS - YuviPanda [18:32:59] [WikipediaMobile/master] Merge pull request #263 from yuvipanda/remove-search-overlay - Brion Vibber [18:33:21] Project WikipediaMobile - Nightly builds build #354: SUCCESS in 13 sec: https://integration.mediawiki.org/ci/job/WikipediaMobile%20-%20Nightly%20builds/354/ [18:33:48] MaxSem: i like we that we have to make it past asher :) [18:34:28] our main focus is making the best wlm discovery and photo taking app. anything else is not a good use of our limited time [18:34:47] MaxSem: so how would you like to 'borrow' the table? [18:35:49] direct replication would choke from batch changes [18:36:29] so we could start with manual mysqldump import and then figuring out a method of smooth upadate [18:38:11] even mysqldump once per week isn't that bad [18:38:20] jrobson: https://office.wikimedia.org/wiki/Engineering/Mobile/Testing_Devices#Phones still needs an update [18:44:50] MaxSem: how much does their data change day to day? [18:45:50] tfinc, I thought it doesn't change much and designed smart updates with this in mind. next day, 50k rows were changed [18:46:14] most of these changes are minor, I think [18:46:27] like a changed revision in permalink [18:47:30] the question is how often are the changes we care about: name/location/address/new monuments [18:49:16] * MaxSem just asked WLM guys about it [19:01:48] jrobson: i'm ready when you are [19:57:54] hey preilly, please correct me if i am wrong: opera mini traffic goes through a proxy (always) and that's why those proxies need to be separately configured for wikipedia zero to ensure that people using zero and opera mini are still properly detected, correct? so my question why does opera_mini in varnish conf have pretty big ranges instead of the proxy server ip addresses? for example 82.145.208.0/20 is a pretty big range. [20:05:24] MaxSem: and what did they say about how quickly data changes ? [20:05:39] nothing [20:06:02] no multichill around:( [20:11:56] MaxSem: if it helps, one big issue is that a number of countries' monuments are not yet appearing in the database, so I imagine that they will get sumped in as we get close to the contest date [20:12:04] dumped [20:13:07] multichill is difficult to reach, so it was a nice surprise he popped in yesterday [20:13:43] it may be better to ask Elke the question, then she can talk to Maarten [20:14:56] MaxSem: we'll need to decide on the migration plan and start executing it next week [20:15:22] i'd like you to own that piece and let me know what you need help with [20:18:15] MaxSem: just let me know if you need help contacting WLM people [20:18:51] philinje, it's actually easier for me to contact them cause we're all in Europe [20:20:51] sure [20:21:27] did you send something yesterday about the blog post for the new API? [20:23:28] philinje, tfinc: I started drafting it at http://etherpad.wikimedia.org/API-blog-post - wanted to know if I'm moving in the right direction [20:23:50] * tfinc goes to read it [20:25:11] its a fine start! [20:26:13] lets add a short intro paragraph to frame this with the app newest releases [20:26:40] MaxSem: thanks, do you want to keep going, or should i start editing/adding? [20:27:09] philinje, I think I'll add some more first, thanks [20:27:23] ok [20:35:56] MaxSem: the audience is not really app developers, though they are the ones who will care about this [20:43:21] pchang__: why wouldn't this post be for developers ? [20:43:47] it is for developers and also a broader audience, so it has to be interesting to both [20:44:20] pchang__: we want more apps, devs, etc to use the api. casual readers will benefit from an the app being faster but the main driver is to get more people to use the api [20:44:33] pchang__: what points are you looking to hit for the broader audience ? [20:45:17] usually things like how this ties into the broader mission, but also using language that is basic and not technical, etc. [20:45:23] stuff I learned from Tilman [20:47:09] New patchset: Jdlrobson; "support position fixed nav bar" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/12979 [20:52:41] pchang__: interesting. i'm eager to see what you come up [20:54:10] pchang__: how is the next batch of sibling project main pages looking ? what kind of feedback are we seeing? [20:54:28] its too quiet for me [20:55:50] let me take a look, the notice went out this morning [20:56:05] we'll need the weekend then to really get pickup [20:56:13] pchang__: why don't we get the signpost to pick this up? [20:58:40] tfinc: hi. Looks like the nightly builds have been broken since may :-/ They magically fixed themselves when we upgraded Jenkins [20:58:53] so I am merely making you aware of it :-D [20:58:58] pchang__: do i lose anything important while redoing http://www.mediawiki.org/wiki/Wiki_Loves_Monuments_mobile_application/ToDo ? [20:59:13] HASHAR: full of lolz [20:59:26] i think Tilman dealt with the Signpost but let me check [20:59:31] pchang__: thanks [21:00:11] tfinc: lose anything? are you removing lots of stuff? [21:48:40] tfinc: do you know how many wikipedia zero carriers are supported at this moment? [21:49:33] because it seems from the varnish conf that a quite a few are not actively running [21:52:18] drdee: Patrick should know [21:52:35] drdee: can you talk on Skype? [21:52:44] preilly: do you know how many wikipedia zero carriers are supported at this moment? [21:52:45] yes [21:54:49] great [22:22:11] preilly: Sir are you around? [22:54:34] caxthelm: yes [22:54:44] caxthelm: I added you to https://github.com/wikimedia/WikipediaMobileJ2ME [22:57:31] Thank you [22:57:50] caxthelm: np [22:58:33] I'm pleased with this first commit, It isn't fancy but the basic flow and screens are in place. [23:02:12] caxthelm: okay cool — I'll take a look shortly [23:03:34] caxthelm: the code looks clean [23:04:30] Thank you, Did you want me to commit a jad/jar to be able to preview? [23:07:51] caxthelm: yeah let's do that for sure [23:26:59] preilly: After a little break to help a co-worker the preview build is committed. I also added in some libraries that were ignored for the first commit. [23:28:36] Good night all, have a good weekend. [23:33:54] tfinc: https://github.com/wikimedia/WikipediaMobileJ2ME/tree/master/dist