[09:10:45] morning [09:27:00] benestar: did you manage to get in touch with jdlrobson yesterday? [09:27:52] nope, we missed each other :S [09:28:16] * benestar is just messing again with mobile wikidata and there is some stuff left [09:29:40] boo and also yay [09:53:14] o/ [09:53:29] phuedx: hey, are you working on https://phabricator.wikimedia.org/T116707 ? [09:55:54] bmansurov: hey, if you don't mind [09:55:59] if you've picked it up then i'll put it down [09:56:14] no, i haven't. i just was going to ask about something [09:56:50] i see jon's patch (https://gerrit.wikimedia.org/r/#/c/250866) is further depending on MF. I thought we were trying to not depend on MF [09:57:07] I wouldn't want to re-write the code for desktop skins [09:59:33] further depending on mf? [09:59:43] i think he's added a reference to mw.mobileFrontend when he shouldn't have [10:00:25] mapPageData function in https://gerrit.wikimedia.org/r/#/c/250866/1/resources/ext.relatedArticles.readMore.bootstrap/index.js tries to conform to mobile.startup/Page [10:02:22] it maybe because that was the way before the patch, but it maybe better to depend on Cards before adding more features to RelatedArticles [10:02:35] hrrm [10:02:47] that function has been moved from the minerva-specific part to the general part [10:03:00] so, yarrrp, you're right [10:13:17] wait, sorry, i misread that [10:15:29] so mapPageData just massages the data into an obvious format [10:16:22] yes, and that's because mobile.startup/Page can use it, right? [10:16:31] nope, so i can read it :P [10:17:27] convertPages in minerva/index.js maps over the intermediate page objects again [10:18:54] the reason i wrote the former is because i didn't think "page.terms && page.terms.description && page.terms.description.length" was worth repeating [10:19:03] by one or more other things [10:19:16] it's just an intermediate object that can be removed, if you'd prefer [10:19:25] over-engineering on my part [10:31:30] morning folks [10:33:47] hey joakino [10:36:11] phuedx: np if we're not depending on MF [10:36:14] joakino: mornign [10:37:02] bmansurov: i'm rewording that docblock as part of the follow-on patch [10:37:30] ok [10:38:54] hi bmansurov ! [12:37:38] bmansurov: what's the status of the cards extension? [12:38:05] it's usable, but doesn't have watchstars [12:38:25] my patches to add watchstars need some discussion [12:38:58] okie poke -- i do agree that we need to move away from mf asap [12:39:19] that'd be nice, plus we can clean up parts of it [14:15:36] out for lunch [14:28:54] back [14:29:18] took longer than i thought to half complete a diy project [15:26:36] gah -- sorry was afk for a while [15:26:43] son's having to go to the doctor >.< [15:53:13] i'm here! [15:53:31] third time this year working at a cafe [15:53:34] feels nice [15:55:33] nice [15:56:14] nice joakino :P I actually hear through the list of songs I heared today morning in the radio, because I missed to write down the time or/and the name of the song :( [15:57:18] ooh [15:57:35] how are you doing FlorianSW [15:57:44] long time since we talked in lyon [15:58:25] joakino: I'm doing well, hope you, too? Yeah, unfortunately :( Too long :P [15:58:53] FlorianSW: how's the contributing going [15:59:07] anything cool you've engaged in lately? [15:59:52] it's... yeah, good :P The problem is, that I've so many things to do (or at least things I want to do, not only MediaWiki, but a big part) and I don't have as much time as I would love to have ;) [16:01:15] little things like the "back to top button" ( I really really like this very little change :P), font changer and some things in core and other extensions :) [16:07:06] "some things" [16:07:13] FlorianSW contributes /all the things/ [16:07:18] FlorianSW: back to top button is cool, but a floating menu is better ;) [16:07:51] bmansurov: yeah, I know, but that's a great step in the right direction, I would like to have the ToC like in the app :P [16:07:54] nice! [16:07:57] phuedx: not all, only some :D [16:08:18] FlorianSW: can't wait to see that feature in MF too [16:09:07] also I always think that back to top is discrimination against back to middle [16:09:22] or back to bottom [16:10:17] phuedx: do apps use cirrus search's more like feature for read more too? [16:10:34] lol [16:10:44] i'd love a "somewhere in the middle of the page" button [16:10:54] bmansurov: yup, as does gettingstarted [16:11:03] ok [16:11:04] but gettingstarted wrapped it in an api that everyone can use! [16:11:10] XD [16:17:20] fire alarms going off randomly! [16:18:23] poltergeist [16:20:38] is there a way to make an API request for a title using a template? [16:20:47] e.g. Template:TFA_title expands to today's featured article [16:26:06] nvm got it [16:26:15] jdlrobson: hey, where can I get an API key for http://wpt.wmftest.org/ ? [16:26:19] or not [16:35:39] bgerstle: what are you trying to do exactly? i'm not sure if i'm misreading your question [16:35:59] phuedx: run an API query for some information on today's featured article on EN wiki [16:36:08] the TFA title template expands to the title of today's featured article [16:36:29] was wondering if there's some clever way to use that as the title parameter for an API request [16:36:48] e.g. https://commons.wikimedia.org/w/api.php?action=query&generator=images&prop=imageinfo&iiprop=url|size|comment|metadata|user|userid&continue=&titles=Template:Potd/2013-06-09 [16:40:56] dbrant: should this be tagged for the android app? https://phabricator.wikimedia.org/T117631 [16:41:34] dbrant: i don't understand how googlebot is being counted. MobileWikiApp* should be Android only AFAIK [16:42:30] phuedx: i might've figured it out: https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&prop=extracts|images|pageprops|pageterms&format=json&titles=Wikipedia%3AToday's%20featured%20article%2FOctober%2030%2C%202015 [16:42:42] niedzielski: i'm not sure how that's happening... or what we can do about it :( [16:43:39] dbrant: i don't understand how it would happen either. it seems a little alarming, especially if the numbers are large [16:43:55] dbrant: i'll follow up on the ticket and mention the confusion [16:44:17] bgerstle: nice [16:46:52] meh, not quite what i was looking for [16:47:03] i can get the content of that page, but not properties of the actual title [16:47:08] e.g. page image, wikidata desc [16:50:37] brb [16:57:16] bmansurov: you'll need to ask peter [16:57:30] he'll need you brief you on a few things - so setup a meeting with him :) [16:57:38] ok [16:57:49] bgerstle: doh sorry i joined this conversation late [16:58:07] jdlrobson: the one about querying pages based on template expansion? [16:58:10] https://github.com/jdlrobson/pushipedia/blob/master/public/workers/tfa/worker.js [16:58:14] ^ todays featured article [16:58:19] that's how i was doing it for pushipedia [16:58:22] it's a pain [16:58:31] unless you don't want to use JSON and use an atom feed [16:59:15] jdlrobson: what's the base URL for "/api/articles/tfa"? [16:59:18] bmansurov: phuedx|afk i'm free to talk before standup if you both are (but guessing sam isn't) [17:00:09] sure i'm free too [17:01:11] jdlrobson: so you were doing https://github.com/jdlrobson/pushipedia/blob/7f20a5ce81897e3f4b64943403e2d2a464ccaa5e/index.js#L169 [17:03:01] jdlrobson: that's basically what i've tried https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&prop=extracts|pageimages&format=json&exchars=200&explaintext=&piprop=thumbnail&pithumbsize=120&titles=Wikipedia%3AToday's_featured_article%2FNovember%204%2C%202015 [17:03:09] but you can't get wikidata desc (AFAIK) [17:03:27] and i'm not getting page image, can only get the ref to the image that's transcluded [17:03:36] bgerstle: mmm yeh you can't [17:03:41] you'd need to do a second api hit [17:03:45] yep [17:04:11] if you could make query use an expanded template as a generator [17:04:25] i.e. expand "Template: TFA title" and use the output as the title [17:04:42] then you could do any other query as you normally would [17:06:30] jdlrobson: also, i can only get image if i do: https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&prop=extracts|images|pageimages&format=json&exchars=150&explaintext=&piprop=thumbnail&titles=Wikipedia%3AToday's_featured_article%2FNovember%204%2C%202015, and even then i only get the File: name [17:06:40] meaning i'd need to do an imageinfo request to get the resolutions i want [17:07:13] (see rant about a restful API to get an _image_ for a given resolution w/ only knowledge of filename) [17:10:33] bmansurov: minor fixup suggessted on your patch [17:12:20] jdlrobson: so, there is one way... [17:12:21] https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&prop=extracts|pageimages&format=json&exchars=200&explaintext=&titles=Wikipedia%3AToday's_featured_article%2FNovember%204%2C%202015&generator=links&gplnamespace=0&gpllimit=1&gpldir=descending [17:12:34] use "links" as a generator on the TFA page and grab the first result [17:12:43] _usually_ TFA has the title of the featured article as a link [17:12:50] it's the first word in the blurb [17:20:24] jdlrobson: nevermind [17:20:31] links are listed alphabetically, not in order of appearance [17:21:55] bgerstle VS mw-api, FIGHT!!! http://media.giphy.com/media/8iNovqRs0sN5C/giphy.gif [17:22:35] 😵 [17:22:48] haha [17:32:32] free now [17:32:38] jdlrobson, bmansurov ^ [17:34:17] bgerstle: https://github.com/jdlrobson/pushipedia/blob/master/index.js#L169-L176 [17:34:40] phuedx: yeah, thinking that's what i'll do for now [17:34:41] phuedx: hey [17:34:43] tracy island? [17:34:48] suresies [17:34:53] jdlrobson: sec [17:39:36] dr0ptp4kt: do you know why we would get eventlogging events with Googlebot in the useragent? I vaguely remember talking about this before... [17:40:50] dbrant: they do automated app runs. there are several known reasons for them, but one of them i think has to do with validating similarity of app presentation with webpage presentation for app indexing [17:42:24] that ^ [17:42:55] dr0ptp4kt: so if i understand correctly, Google runs the app and changes the useragent on all network requests to be Googlebot? [17:51:11] dbrant: yeah, that's what i remember. there's some inferential material at https://developers.google.com/app-indexing/support/faq?hl=en and https://developers.google.com/app-indexing/introduction?hl=en. you may be able to look at a small slice of time and reverse lookup the ip address to verify it's part of the google ip blocks if there's a question. lemme [17:51:11] know if you want me to open a thread with goog to get more info [17:53:38] dr0ptp4kt: will do; i'd just like to know if we can "detect" if we're being run by Google, and not send any events in that case. [17:55:37] dbrant: as opposed to having to customize mysql queries, right? that part, i don't so much know about. i anticipate they use some sort of fancy proxy or aop shim to override it, so i imagine anything we'd do would be a hack and subject to change. maybe with any luck you'll find some sort of tip in that documentation...fingers crossed [17:55:55] holding breath [17:56:22] cautiously optimistic [17:56:26] optimistically cautious [18:00:54] jdlrobson: phuedx etc, tryign to connect... [18:04:42] hey nuria do you know if there are any plans to blacklist UAs from EL for which we know they are bots? cc dbrant [18:05:01] nuria: see preceding messages for some context [18:12:23] dbrant: are we waiting on anything to merge the donation patch (https://gerrit.wikimedia.org/r/#/c/250750/)? i thought i should ask before +2ing [18:13:01] niedzielski: let me just confirm with Fundraising that the URL is exactly what they want. [18:26:29] bgerstle: thanks for fixing my description. Most folks who mess around in the docroots won't have any idea what that file is for and since we can't add comments (/me shakes fist at json) having a good commit message is the next best thing [18:27:20] bd808: sure, if there's anywhere else we should document it, let me know [18:27:36] but if people are savvy enough to do git log/blame, then i guess we're fine ;-) [18:27:52] deployers generally are :) [18:30:58] * phuedx makes an action item to get everyone to alias praise to blame [18:36:09] niedzielski: this is the test failure i'm getting with some regularity and doesn't appear at all valid: https://integration.wikimedia.org/ci/job/apps-android-wikipedia-test/662/testReport/junit/org.wikipedia.test.perf/PageLoadPerformanceTests/testLoadPages/ [18:36:15] niedzielski: unless i'm missing something [18:36:35] dr0ptp4kt: looking [18:36:38] niedzielski: i did a little looking around your idea yesterday of testing against a more recent sdk version [18:37:11] niedzielski: it seems like a good idea (especially if not that much of a pain to set up, although it looked like potentially a pain to set up) -- how did we come to be testing against 4.0.4 currently? [18:39:02] gtg get the kids to bed [18:39:29] dr0ptp4kt: I see, no, we have no code to that effect, seems it should be easy to add client side (rather than filtering events server side) [18:44:04] dr0ptp4kt: now, if we are talking about this ticket: https://phabricator.wikimedia.org/T117631#1780378 [18:44:12] dr0ptp4kt: your problem is a different one [19:05:24] nuria: yeah, that's the problem to which we refer. i've wondered if there was some generalizable way to deal with this sort of stuff. for sure there are bots on the web executing js that can trigger el as well. unfortunately, it seems hard to actually detect in the app if it's google running the app and that suppress el [19:06:06] nuria: should we label that card so that it's a generalizable approach instead of a one-off? [19:06:13] dr0ptp4kt: that is on your android app not the web, how is a web bot crawling your app content? [19:06:18] dbrant: ^ what do you think? [19:06:24] nuria: it's voodoo. [19:06:37] dr0ptp4kt: it is not a web issue, do you have remote webviews? [19:07:33] dr0ptp4kt: how is the google bot (for web, not apps) sending staff via your app, overriding your UA? [19:07:53] dr0ptp4kt: that cannot be, you must be seeing that through something lese that is not your apk code [19:07:55] nuria: nope, those messages can only come from native code, not a webview... [19:08:12] dbrant: how is the UA overriden? from the one you set? [19:08:27] nuria: from the looks of it they actually have the app installed, and use some sort of proxy or aspect oriented shim. so it's hard to handle the stuff. a webview is used, but we actually control what's injected. ironically, we munge all kinds of things, including the user-agent [19:08:32] nuria: the only explanation is that Google somehow overrides the ua... no idea how. [19:08:39] nuria: dbrant i gotta go. bd808 and i are meeting [19:09:08] dbrant: on your native code? (how?) we can talk about this latter, need to run to a meeting , let me ping you [19:09:17] nuria: sure anytime [19:17:27] dr0ptp4kt, dbrant: which events are you seeing logged? [19:19:27] dbrant: Fix top padding in Nearby fragment." [19:19:37] phuedx: the schema is "MobileWikiAppDailyStats_12637385" [19:19:55] okie poke [19:20:07] dbrant: Fix top padding in Nearby fragment T116081 - was deployed to production? [19:20:15] thought i'd ask because there are events logged server-side [19:20:40] etonkovidova: yep, should be [19:21:02] dbrant:that what I thought :) I do not see the issue on production Lollipop device [19:21:21] phuedx: afaik, the android app (native code) is the only sender of those events [19:22:09] dbrant: what other UAS you have on table? did you look? [19:23:45] nuria: besides googlebot, only from the Android app. [19:23:48] dbrant: i see, from ticket is googlebot or android, so no general web access [19:24:45] dbrant: now, in order to fix issue we need to see what is going on, this petitions are not coming from your apk if every eventlogging send comes with your "homemade" UA [19:27:43] dbrant: and we are sure none of your webviews is remote , like , say a ToS? [19:29:40] nuria: totally sure. And, again, those particular messages don't come from a webview. They come from a single place (literally) in native code. [19:31:35] niedzielski: donation patch is good to go [19:32:35] dbrant: right, so big mistery then, cause the post of info includes your "custom" user agent every time, correct [19:32:37] ? [19:33:14] nuria: yep, that's right [19:36:42] dbrant: there are other uas that are not android on: select distinct(userAgent) from MobileAppLoginAttempts_5257721 WHERE userAgent NOT LIKE '%Android%' ; [19:37:03] dbrant: do these makes sense: [19:37:06] https://www.irccloud.com/pastebin/Hw6LZ0xc/ [19:39:38] dbrant: no google uas on taht table [19:39:41] *that [19:40:05] nuria: that table is used by both Android and iOS apps. I'm guessing that what those are from. [19:40:26] dbrant: do you have another table only used by android? [19:41:50] nuria: here's one - MobileWikiAppSessions_14031591 [19:43:10] nuria: another point is that messages to MobileWikiAppDailyStats_12637385 are sent unconditionally when the app is launched, as opposed to other events that require user action. [19:43:20] dbrant: i cannot find any googlebot on those. but then, in order to fix this we need to understand how googlebot is making it to the post of data as -on my opinion - this should be fixed on clients. [19:48:57] dbrant: do a bit of research on your end and let me know if you find something, it might be of help to look in all your android tables in EL and see if "googlebot" only shows up on this one. [19:49:29] nuria: will do! [19:49:51] dbrant: thank you. [20:01:24] dr0ptp4kt: good talk. Maybe next step is a sketch of the VCL and see how badly Brandon pukes [20:06:21] bd808: cool. and ha. sorry, connection drop [20:06:34] yeah no worries [20:17:21] jdlrobson: could be caching but on wikidata, I get "Error: Module not found: mobile.watchstar/WatchstarApi Error" [20:17:26] on special nearby :/ [20:17:43] aude: it got renamed [20:17:48] ah [20:17:55] It's mobile.watchstar/WatchstarGateway now [20:17:57] to the gateway thing? [20:17:59] ok [20:18:12] what are you doing in wikidata that uses it? [20:18:26] loading Special:Nearby ? [20:18:36] same issue on mediawiki.org ? [20:18:44] we just deployed so very well can be caching [20:19:00] seems ok now [20:25:42] mdholloway: i think you picked up the test card this sprint. were you planning on fixing this repeated testLoadPages failure as part of that work? if not, i was thinking of taking a whack at it even though it's not in the sprint because it's failing almost constantly now [20:26:58] niedzielski: Not sure, really. I've thought/Googled about it a bit, but if you have any ideas on how to fix it, then by all means go ahead [20:27:18] mdholloway: i got nothing. i was jsut going to dink with it [20:30:11] niedzielski: it would definitely be nice to get that solved. it's becoming enough of a distraction that it probably wouldn't be overkill for both of us to be working on it. [20:31:00] niedzielski: oh, nice, you +2'd my utils patch! o/ [20:31:06] mdholloway: cool i'll check it out and let you know if i find anything [20:31:32] niedzielski: just saw it. my laptop has been kind of crashy lately :/ [20:31:51] mdholloway: yeah, i know that's been sticking around for a while. i had +1'd it a while back in case anyone else wanted to review it. seemed like enough time had passed for a +2 [20:32:44] mdholloway: :( i just switched to ubuntu werewolf over the weekend. i've been surprised to find only improvements so far [20:57:54] bearND dbrant|brb mdholloway: oof, i was just looking at upgrading some of our libs when i noticed this deprecation notice: https://github.com/mapbox/mapbox-android-sdk-legacy. i think we're supposed to use this now: https://github.com/mapbox/mapbox-gl-native [21:05:35] niedzielski: Cool. I haven't looked it in detail but I tend to favor moving to the new SDK. [21:07:02] and... it looks like they've changed just about everything. [21:11:33] jhobs: want to pair on getting a patch ready for deploying QUickSurveys so you are familiar with process? [21:17:18] jdlrobson: yeah. One sec, though, in a call with dfoy atm [21:17:24] k grab me when ready [21:24:38] jdlrobson: ready. Gonna hop in tracy island [21:28:50] jhobs: k [21:46:03] kaity_: did you get a chance to look at https://phabricator.wikimedia.org/T116810 (http://en.m.wikipedia.beta.wmflabs.org/wiki/Related_test) [21:46:28] jdlrobson: waiting to see when its ready [21:46:38] kaity what are you waiting for..? [21:46:54] jdlrobson: I thought it was broken? [21:47:05] waiting to hear back from you [21:48:50] i pinged you yesterday i guess you didnt get it [21:49:02] but yeh it was fixed shortly after [21:49:58] jdlrobson: ok cool [21:50:09] jdlrobson: will look [21:56:28] niedzielski: this is interesting (though supposedly fixed): https://stackoverflow.com/questions/17016011/junit-test-times-out-despite-executing-quickly [21:57:04] niedzielski: it seems like something's failing immediately and it's being falsely reported as a timeout/TimeoutException [21:57:28] mdholloway: hm, i wonder if this effects instrumentation tests. [21:57:45] niedzielski: not sure. [22:04:41] MaxSem: hey, have time for an API question? [22:05:10] sure [22:05:31] so, i want to do something w/ _today's_ featured article [22:05:37] i'm aware we have https://en.wikipedia.org/wiki/Wikipedia:Today%27s_featured_article/November_4,_2015 [22:05:43] and https://en.wikipedia.org/wiki/Template:TFA_title [22:06:14] ideally, i'd be able to do an action=query for the title of today's article and get the information i need [22:07:16] my current thinking is to do a TextExtracts query on TFA_title to extract the title, then do a query with it [22:07:20] (as a separate request) [22:08:48] or wait, i guess the TFA_title approach won't work? [22:08:54] or i have to pass the date?.. [22:09:11] yeah [22:09:12] https://en.wikipedia.org/wiki/Template:TFA_title/November_4,_2015 [22:09:43] specifically: en.wikipedia.org/w/api.php?action=query&prop=extracts&format=json&exintro=&explaintext=&exsectionformat=plain&titles=Template%3ATFA_title%2FNovember%204%2C%202015 [22:25:25] this has likely been asked and answered. when if ever will development start again for the windows platform? [22:26:05] MaxSem: i take your silence as a sign that there isn't really a good way to do this? [22:26:50] bgerstle, action=featuredfeeds [22:27:26] s/feeds/feed/ [22:27:30] MaxSem: that just gives you a feed of HTML, right? [22:27:36] i can't use that to get the titles, afaik [22:27:38] yup [22:27:50] i mean, an atom/RSS feed, whose entries are HTML blobs [22:28:10] yeahhhh [22:28:41] there's a snippet, but where the link to the actual, full article to it is completely user-defined [22:29:01] IOW https://en.wikipedia.org/wiki/Special:FeedItem/featured/20151104000000/en gives me the same as the Wikipedia:TFA/... [22:31:26] the feed, however, is supported by a ton of projects, unlike these URLs that are project-specific [22:32:11] i didn't think featuredfeed was widely supported [22:33:03] hm, i guess it is [22:37:01] wish they had the article titles somewhere in the feed [22:39:47] MaxSem: is there any kind of lower-level API for the feed? [22:40:02] no