[00:00:23] New patchset: Jdlrobson; "Avoid api request to lookup watchlist contents state (bug 44878)" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48580 [00:00:24] New patchset: Jdlrobson; "Prevent looking up whether an article can be watched on special page" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48581 [00:02:02] New patchset: JGonera; "Add progress bar to photo upload progress notification (#379)" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48050 [00:02:18] [Commons-iOS] brion pushed 1 new commit to master: http://git.io/gtxs0g [00:02:18] Commons-iOS/master 54ed82b Brion Vibber: basic pull to refresh, drop refresh button [00:03:32] New patchset: JGonera; "Run jshint on the whole javascripts directory" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48582 [00:03:40] Change merged: Jdlrobson; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48050 [00:04:46] jdlrobson: https://meta.wikimedia.org/wiki/Schema:MobileBetaWatchlist [00:07:37] New patchset: MaxSem; "Bug 36894: Mobile device detection for third parties" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48583 [00:07:38] my magnum opus for today^^^:) [00:07:45] New patchset: awjrichards; "(mingle 330) Styles upload count on dashboard" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48584 [00:10:20] YuviPanda|away: where was the reference page for the event tracking stuff? [00:10:28] jdlrobson: pm [00:11:28] i see https://www.mediawiki.org/wiki/Commons_App/Analytics but it's…. not very detailed [00:12:33] it's got a reference to https://meta.wikimedia.org/wiki/Schema:MobileUploads …. which doesn't seem to cover everything mentioned there [00:18:34] New patchset: Jdlrobson; "update event logging for watchlist star to use new schema" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48586 [00:19:23] Change merged: Jdlrobson; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48582 [00:45:08] awjr i'm out of my mtg. [00:45:13] how can i help? [00:45:26] hey munaf, just wanted to dbl check about the design asset for https://mingle.corp.wikimedia.org/projects/mobile/cards/330 [00:45:30] lemme send you a screenshot of what i did [00:46:57] munaf: just sent the screenshot [00:47:00] cool [00:47:39] question awjr [00:47:46] yah [00:47:48] in implementation notes it says "bolded sans serif" [00:47:57] yeah i wondered about that [00:48:04] i think that may have been a typo [00:48:20] it seemed totaly incongruous with the image, so i wasn't sure [00:48:23] this looks fine to me, and i believe it is what vibha intended [00:48:28] i'm guessing that was just a typo [00:48:31] i just did the easier of the two :) [00:48:32] ok cool [00:48:39] however i may speak to her about changing the design [00:48:40] i mean, easy enough to change if we need to [00:49:13] yup, but it works for me as-is :-) [00:49:17] cool thanks! [00:53:39] New patchset: Jdlrobson; "Avoid api request to lookup watchlist contents state (bug 44878)" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48580 [00:53:39] thanks for your blog post edits, awjr! [00:53:49] np Maryana, looks good :) [00:54:28] Change merged: JGonera; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48580 [00:55:29] New patchset: Jdlrobson; "Prevent looking up whether an article can be watched on special page" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48581 [00:55:58] Change merged: Jdlrobson; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48581 [01:42:47] ok awjr finally drafted that mail [01:42:55] sent [01:43:01] that was painful [02:23:32] Maryana: http://english.stackexchange.com/questions/67720/how-did-the-pronunciation-of-the-word-derby-evolve [02:25:52] Maryana: http://en.wikipedia.org/wiki/Ca%C3%B1o_Negro_Wildlife_Refuge [04:12:52] tfinc: I archived and have a wip on the merge [04:12:59] YuviPanda|away: thanks [04:13:20] tfinc: I'll respond on that email thread once done [04:13:23] k [07:32:30] * Debloper needs a list of keyboards which are (or needs to be) in "beta status" (at least a couple) - wonders if there's any list for it. [07:32:33] https://github.com/wikimedia/jquery.ime/issues/14 [11:04:43] j #mediawiki-i18n [13:04:32] [java-morelangs] yuvipanda pushed 2 new commits to master: http://git.io/eJ5nQA [13:04:32] java-morelangs/master a8ac866 YuviPanda: Generate string labels for scripts too [13:04:32] java-morelangs/master 1e7c1c3 YuviPanda: Updated to latest jQuery.IME [16:28:43] New review: MaxSem; "Patch Set 3: Code-Review-1" [mediawiki/extensions/MobileFrontend] (master) C: -1; - https://gerrit.wikimedia.org/r/48164 [16:42:04] New review: MaxSem; "Patch Set 1: Code-Review-1" [mediawiki/extensions/MobileFrontend] (master) C: -1; - https://gerrit.wikimedia.org/r/48598 [17:08:01] * awjr waves [17:09:05] whee [17:12:10] hello brion [17:12:15] heyy [17:12:26] brion: https://www.mediawiki.org/wiki/Apps/Commons#Reports [17:12:52] ooh [17:13:12] for some reason the css is failing on wiki for me :D [17:13:15] it feels very retro [17:13:43] i started on an EventLogging implementation for obj-c but got distracted redoing all my async stuff using deferreds [17:13:53] brion: me too :P [17:13:57] hehe [17:14:02] brion: ooo objc has deferreds? [17:14:12] easy enough to make them now that it has blocks [17:14:29] ah [17:14:30] right [17:14:31] i'm just hoping i'm not fucking up memory management ;) [17:14:33] * YuviPanda curses Java [17:14:42] you can have anonymous inner classes ;) [17:14:53] they suck [17:15:06] haha [17:15:09] verbose [17:15:16] aoh well [17:15:23] brion: but on the plus side, background tasks :P [17:15:27] \o/ [17:16:08] so far i'm just working on the main thread using the event loop, any threading is hidden in the background for me :) [17:16:18] eventually i might have to move some encoding/resizing stuff to a thread tho [17:16:29] oh. async network with callbacks? [17:16:36] yeah [17:16:37] I think I've a *lot* of threads running around [17:16:41] hehe [17:17:20] brion: I'm sure that they'll bite me sooon [17:17:34] brion: did you see that section on the page? [17:17:45] YuviPanda: so for the event logging are we using the same schema as the mobile web uploads? or are we going to have our own [17:17:55] brion: we're going to have our own I think [17:17:59] excellent [17:18:00] i've been thinking of the schema [17:18:07] and thinking of how separate / together they should be [17:19:38] we have a lot of distinct stuff, like the different entry workflows [17:19:46] probably makes sense to have our own [17:19:57] then it's up to the data guys if they want to put them together :) [17:20:18] brion: yeah, about that. I dunno how much 'data guys' time we can get :P [17:20:23] heh [17:20:59] brion: also me splitting up the reports by things we can pick up from categories [17:21:01] vs eventlogging [17:21:08] *nod* [17:21:28] most anything on successful uploads can be got from the actual pages yeah [17:21:35] it's mostly failures and behavior leading up to them that we want to do with event logging i guess [17:22:04] brion: yeah [17:23:00] brion: it'll also be interesting to see how to do 'joins' across data from two types [17:23:08] like, 'number of failed uploads per user per successful upload' [17:23:10] or soemthing like that [17:23:24] heh [17:25:20] awjr: what say you on these last commits [17:25:34] jcmish: one sec [17:25:38] k no prob [17:25:39] of Jon's? [17:29:59] notnarayan: hello? [17:30:07] brion: saw pau's response? [17:30:59] haven't read it yet [17:31:23] jcmish: sorry, there's an ops meltdown and there was some speculation it's mobile, but i think they're beyond that now [17:31:39] oh no! [17:31:43] was it? [17:32:26] bits cache sending 503s, i think current theory is something related to wikidata deploy yesterda [17:32:26] or today, ratehr [17:32:33] ah well we haven't deployed anything [17:33:47] fun morning already [17:34:01] YuviPanda: hey [17:34:09] although mobile requests to bits are still being pointed at as causing the overload [17:34:13] argh [17:34:39] rather, they are not helping [17:35:02] mark made one of the varnish caches serve 503s to help bring the load down, so we may hear some complaining [17:35:36] and testwiki has been disabled in varnish config [17:36:47] awjr, any thoughts on https://docs.google.com/a/wikimedia.org/document/d/13WASDu7PJKVY_DDfz4z9gir8_nIAAInUi-az4qBm4NA/edit ? [17:37:18] ah yeah, MaxSem, probably - i will look at it later [17:41:25] ok, the apache load still hasn't sufficiently dropped, mark is pushing a change that will cause all requests that have a referrer containing 'modules=mobile.startup' to 503 [17:41:37] there will be mobile site breakage [17:41:51] (https://gerrit.wikimedia.org/r/#/c/48645/1/templates/varnish/bits.inc.vcl.erb) [17:43:30] awjr, are you just lurking in #-ops or you have another source of information? [17:43:38] MaxSem: security channel [17:44:33] o_0 [18:01:02] MaxSem: there? [18:01:13] yup [18:02:21] awjr: there too? [18:02:26] apparently a problem with mobile right now [18:02:43] jdlrobson: yo, yeah there's an outtage [18:02:46] jdlrobson, it's a known meltdown [18:03:00] the ops sacrificed us [18:03:13] preilly is asking about doing mobile deployment earlier [18:04:14] why? [18:04:39] jdlrobson: yeah we're chatting - i was reporting about the outtage in here a while ago but havent emailed yet - i'll send a note [18:05:21] i've joined a state of confusion so am trying to work that out myself [18:10:50] ok so preilly is going to unblock mobile requests in about 15 minutes [18:11:09] and see if the meltdown goes - they are redeploying wikibase first [18:11:13] ^ preilly MaxSem [18:12:00] We've unblocked one box [18:12:07] Doing the next one right now [18:13:55] are mobile load.php requests so frequent that they're the easiest target to disable? [18:14:14] that was the impression i got [18:14:41] YuviPanda: https://twitter.com/ademianczuk/status/301372527354195968 [18:15:15] I'm not going to click that since that'll clearly be about how the app that nobody works on sucks and the mobile web part that about 5 kickass people work on is superior [18:15:17] :P [18:15:18] jdlrobson: ^ [18:15:46] YuviPanda: what is up? [18:15:51] ok i understand YuviPanda [18:15:54] let me quote it for you "Sometimes a mobile site is just superior to a native app. I'm looking at you, #wikipedia." [18:16:29] jdlrobson: you *do* realize that once we kick your ass that I'm not going to let any of this pass? [18:16:38] and you'll just have to hide behind 'but we have more page views!' [18:16:39] ? [18:16:41] :P [18:16:56] YuviPanda: I like your attitude I just don't believe the content [18:17:15] * YuviPanda considers what level of trash talking is appropriate [18:17:41] considering that if I trash talk mobile web that's about... 6 people? And I do not know how they'll take it (hopefully in the proper way) [18:18:08] and that trash talking the app is only one guy and the meanest thing I'll do is steal half a glass of special lassi from you :P [18:26:32] jdlrobson: are you in the office? is tfinc around? [18:26:45] YuviPanda: not yet [18:26:55] oh [18:26:58] ok [18:27:07] mobile requests to bits have been unblocked - ops is keeping an eye on load for the moment [18:27:19] jdlrobson: did brion come in? [18:27:19] (sorry to bother) [18:27:22] YuviPanda: nope [18:27:26] ah ok [18:27:28] thanks jdlrobson [18:32:33] so awjr MaxSem apparently our startup script includes device which is causing the fragmentation (http://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=mobile.startup%2Csite%2Cproduction-jquery%2Cproduction-only%7Cjquery.hidpi%7Cmediawiki.hidpi%7Cmobile.device.webkit&only=styles&skin=mobile&version=1360687729&*) [18:32:55] yeah - does it need to? [18:33:11] so on staging i'm seeing http://staging.wmflabs.org/load.php?debug=false&lang=en&modules=mobile.styles%7Cmobile.device.webkit&only=styles&skin=mobile&version=1360636444&* and http://staging.wmflabs.org/load.php?debug=false&lang=en&modules=startup&only=scripts&raw=true&skin=mobile&version=1360636444&*&target=mobile [18:33:27] so we seem to have fixed this already somewhat - but we can probably include the styles one to separate them into 2 requests [18:33:59] support [18:34:25] want me to look into that now? [18:34:27] we can even squeeze it in into today's deployment [18:34:32] or do you want to take control MaxSem ? [18:34:58] ok, I'll look [18:35:32] thanks MaxSem [18:35:43] i'll take a look as well to see if i can find some pointers [18:36:37] ok $headLinks[] = $this->resourceLoaderLink( $moduleNames['top'], 'styles', $target='mobile' ); is what's rendering the device [18:36:49] i'd suggest adding a hook that adds $headLinks[] = $this->resourceLoaderLink( 'device name', 'styles', $target='mobile' ); [18:37:56] MaxSem: it seems device is defined twice? [18:38:04] in hooks and body [18:38:38] getEnabledModules should not run $headModuleNames[] = $device->moduleName(); [18:42:48] ugh [18:43:12] jdlrobson: does the device name get appended in the appropriate place elsewhere? [18:43:55] at the moment it's treated like a head module that is static [18:43:58] that appears to be the only place it's invoked [18:44:01] i can throw a quick fix if you want [18:44:07] it's pretty clear to me how to fix this [18:44:14] jdlrobson: let's try it [18:44:44] ok, I'm lost in this insanity [18:44:52] :p [18:45:01] a bit proud that I started it:P [18:45:07] lol [18:45:15] ok i'm almost done with a fix [18:46:39] jdlrobson: it looksl ike some of those device-specific css files are empty [18:46:44] can we just exclude those as well? [18:47:02] android, iphone2, webkit [18:47:14] awjr, this will not help to remove the modules themselves [18:47:39] is there a reason to have those device specific modules though if we're not sending anything for them? [18:48:03] mmm push notifications from gerrit are broken [18:48:04] :( [18:48:16] awjr: MaxSem https://gerrit.wikimedia.org/r/#/c/48656/ [18:49:44] jdlrobson: can we run that only if there is actually device-specific css to return? [18:50:06] awjr: are you referring to empty files? [18:50:11] jdlrobson: yes [18:50:39] awjr: potentially in future we could make device return false when no file exists $device->getModuleName() [18:50:54] e.g. if ( $device->getModuleName() ) { //add module }} [18:51:44] yeah that's what i was thinking jdlrobson - that should be quick to do. we could just remove the css_file_name key/val for the specific devices and update getModuleName() to return false if there's nothing set [18:52:05] that would further reduce fragmentation, particularly for what i assume are the most common devices [18:52:10] feel free to do that - i don't feel that comfortable doing that right now so close to deployment [18:52:21] awjr: it shouldn't be such an issue [18:52:33] MaxSem: im happy to do this real quick - what do you think? ^ [18:52:43] at the end of the day as long as they are separate you are just caching an empty response [18:53:07] sure - I'll merge Jon's rev so you can build on top of it [18:53:12] MaxSem: groovy [18:53:29] testing [18:53:34] jdlrobson: true, but there are already so many things that cause cache variance. the more we can do to reduce that, the less impact we'll have particularly on the backend apaches [18:56:00] especially since the most popular devices don't have their own tweaks.. [18:57:03] New review: MaxSem; "Patch Set 1: Verified+2 Code-Review+2" [mediawiki/extensions/MobileFrontend] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48656 [18:57:06] Change merged: MaxSem; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48656 [18:57:11] here we go awjr [18:57:14] thanks MaxSem [18:58:53] jcmish, we'll need to deploy ^^^ today, it's a matter of performance [19:02:37] MaxSem: jdlrobson awjr I've been following along [19:02:53] you have my blessing [19:03:57] eek just noticed editing is broken :( [19:03:58] Uncaught Error: Unknown dependency: mediawiki.action.edit [19:05:09] RL magic breaking edit page MaxSem :( [19:05:15] editing is beta, soo.... [19:05:30] yeh but it's still annoying especially when people are using beta to edit [19:05:45] cheaters! [19:06:10] not sure why it is even a dependency [19:06:18] grabbing trevor [19:09:04] ugh, we broke unit tests meanwhile - fixing [19:10:02] heads up: the problem returned [19:10:32] arggh [19:10:51] Wikibase is very semantic lol [19:11:59] MaxSem: Stop swearing [19:12:12] hahahha [19:12:46] ok MaxSem so we can fix edit page easily by adding a hook [19:19:00] arggggghh damn you core [19:19:41] notnarayan: you got dc [19:19:51] browser crashed [19:20:00] that happens :) [19:22:10] notnarayan: :) [19:23:45] so awjr MaxSem turns out there is some horrible code in EditPage.php which assumes a certain module is available [19:27:41] notnarayan: i'm going to head to sleep now [19:27:59] YuviPanda: ah good to hear that :) [19:28:35] YuviPanda: good night! [19:30:01] notnarayan: :) [19:37:26] MaxSem, jdlrobson: https://gerrit.wikimedia.org/r/#/c/48659/ [19:38:55] awjr, can we return false instead of empty string? [19:39:04] MaxSem: yes [19:39:09] any reason for the preference tho? [19:39:48] and do you want the device definition to be 'css_file_name' => false and/or the return for moduleName() to be false? [19:40:06] semantics: string is a module name, false is no module needed [19:40:36] yeah that's fine with me [19:40:59] New review: MaxSem; "Patch Set 1: Verified+2 Code-Review+2" [mediawiki/extensions/MobileFrontend] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48659 [19:41:01] Change merged: MaxSem; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48659 [19:41:11] scrw it, can be bikeshedded later [19:41:26] :p [19:41:44] awjr, fancy a push right now?:P [19:41:52] MaxSem: was just about to ask the same thing [19:42:00] lol [19:42:07] MaxSem: lemme take a quick look at the other patchset [19:42:13] doesn't look like it's going to get any worse... [19:42:40] MaxSem i think we can safely cherry-pick those two patchsets [19:43:06] go for it [19:43:25] MaxSem: my local copies of deployment branches are way out of date and it might take me a while to git pull - are yours fresh? [19:43:55] 1 week old [19:44:02] updating [19:44:09] looks like wmf8 and wmf9 are in prod [19:44:17] Maryana: editing is broken in beta [19:44:24] :( [19:44:27] awjr, can you cherrypick these 2 into MF production branch? [19:44:46] should i disable the edit button or should we apply a horrible hack? [19:45:04] MaxSem: yes [19:45:07] Maryana: presenting horrible hack to https://gerrit.wikimedia.org/r/#/c/48663/ - awaits MaxSem to spit in disgust [19:45:47] jdlrobson, it has a dependency [19:46:22] jdlrobson, can you simply stick it into JS? [19:46:52] yeaj im getting a merge conflict [19:46:54] MaxSem: ? [19:46:57] ok, my branches are ready - will update as soon as prod is ready [19:47:05] it's an inline script MaxSem so it needs to be in the output page there :( [19:47:16] preilly has just disabled mobile scripts again [19:47:32] hmm this is an ugly merge conflict [19:47:32] we've had a huge amount of mobile edits today - https://en.wikipedia.org/w/index.php?title=Special:RecentChanges&tagfilter=mobile+edit would be a shame to disable those :( [19:47:36] awjr, can we just push master? [19:47:47] MaxSem: eg deploy early? [19:47:52] yes [19:48:07] i am comfortable with that if folks are able to commit to sticking around to help [19:48:12] jcmish, jdlrobson, jgonera, Maryana ^ [19:48:32] i'm more than happy to start deploying early [19:48:35] awjr: yup I'm here [19:48:37] right now? sure. has everybody gotten food? [19:48:37] this would be to deploy *now* [19:48:48] i haven't but i can wait [19:49:04] ok let's try it [19:49:14] on it [19:49:15] jdlrobson: jgonera : i'm going on a food run soon. do you want something ? [19:49:16] I can probably wait like 1hr, then I'll be starving ;) [19:49:17] maybe someone in the office can order sandwiches or whatever for folks? [19:49:31] i know that you guys are working and am hoping to pick something up [19:49:31] so before doing this 3 things - https://gerrit.wikimedia.org/r/#/c/48598/ didn't get merged Maryana meaning we won't get an accurate number for account creation data [19:49:50] horrible hack better than nothing :) [19:49:59] jdlrobson: we can push that later [19:50:03] edit page is broken [19:50:06] tfinc_, I was going to get something quick today, probably a sandwich from working girls + maybe a salad [19:50:08] :o [19:50:19] Maryana: are you ok with broken edit page going out now? [19:50:20] and we're not tracking isStable for watchlist usage https://gerrit.wikimedia.org/r/#/c/48586/ [19:50:20] boo, unfortunate, jdlrobson. but it's ok [19:50:24] we can push small bug fixes later [19:50:37] jgonera: jdlrobson: give me your order and i'll take care of it [19:50:38] ok let's do this then [19:50:38] yeah, i asked jdlrobson to just disable the edit button for now, awjr [19:50:45] cool [19:51:01] Maryana: i can't disable the edit button yet [19:51:08] that would have to be a small bug fix pushed later [19:51:46] brb food [19:53:33] ok we doing this awjr MaxSem ? [19:53:51] jdlrobson: yes, looks like MaxSem is merging to prod as we speak [19:55:55] MaxSem: im trying to confirm with matthias that it's ok for us to proceed (he has the current window) [19:56:33] awjr, it's an emergency [19:56:40] MaxSem: he's clear [19:56:49] pushing [19:57:15] going to update the code first, scap to get newest messages later [19:58:40] MaxSem: here if you need me [20:00:21] okaym we're live [20:01:04] jcmish, jdlrobson, jgonera, Maryana_food please test ^ [20:01:08] will do [20:01:14] on production or test? [20:01:15] on it [20:01:17] test.m? [20:01:21] assuming test? [20:01:35] production [20:01:40] oh [20:01:42] MaxSem: do we need a cache flush? [20:01:43] okay [20:01:51] seeing broken styles [20:02:01] awjr, lolwut? the cluster's already melting:P [20:02:06] :p [20:02:40] you may see broken/old messages [20:02:45] i'm not seeing new js either [20:02:49] although js is running now [20:02:59] MaxSem: touch/sync? [20:03:12] awjr, touched before syncing [20:03:15] k [20:03:17] it's funny seeing the star in the middle of the search box [20:03:22] @_@ [20:03:37] it may take time to settle [20:03:37] ha [20:03:47] yup i see that lovely star jdlrobson awjr [20:03:52] lol i like it [20:04:01] at least it works :p [20:04:26] looks like 503s are still coming from bits [20:05:06] logging in broken for me too [20:05:27] jdlrobson, the logging changes never went live [20:05:38] the config, I mean [20:05:51] sorry that should be login [20:05:58] it autocorrected me [20:05:58] i can't sign in [20:06:07] i can sign in jdlrobson [20:06:16] hmm i cant [20:06:18] we probably need that startup script enabled again [20:06:22] preilly is not here though [20:06:31] this is on prod right? [20:06:48] yeah i am still seeing 503s from varnish - were requests to bits from mobile disabled? [20:07:09] hmm well my log in is working :D [20:07:13] I'll keep moving along [20:07:20] jcmish: are you in stable or beta? [20:07:23] i'm testing stable only [20:07:25] awjr, see #-ops [20:07:33] log in works in beta [20:07:52] uh oh. but the settings page buttons aren't showing up [20:07:56] beta [20:07:56] so i can't get out of beta.. [20:08:32] Maryana: hahaha you're right [20:08:35] I can't get out either :D [20:08:51] ooh, they just came back for me. try now, jcmish [20:09:19] \o/ Maryana they're back [20:11:27] YuviPanda: https://www.mediawiki.org/wiki/File:Commons-Android-test-capture.webm \o/ [20:11:34] awjr, jdlrobson, I'm going to deploy https://gerrit.wikimedia.org/r/48669 asap [20:12:39] awjr, jdlrobson - objections? cause otherwise I'll merge in 60 seconds:) [20:12:51] looking MaxSem [20:12:54] temporary fix? [20:13:01] do it MaxSem [20:13:09] should be permanent actually, just need to clean up [20:13:17] ah right [20:13:24] thanks to RL [20:13:29] go for it [20:14:02] New review: awjrichards; "Patch Set 1: Verified+2 Code-Review+2" [mediawiki/extensions/MobileFrontend] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48669 [20:14:03] Change merged: awjrichards; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48669 [20:21:00] so MaxSem looks like it's only fixed on wmf9 [20:21:01] MaxSem: awjr jdlrobson am I waiting for this before trying testing again? [20:21:03] (possibly) [20:21:15] on french wikipedia mobile i'm seeing old style/script urls [20:21:20] on english not seeing any problems [20:21:20] jcmish, the meltdown is still in progress [20:21:27] jdlrobson: max hasn't pushed to wmf8 yet [20:21:31] jdlrobson, wmf8 is still updating [20:21:47] here we go [20:22:01] asher says things are looking better [20:22:05] so things are promising [20:22:08] good [20:22:29] gotcha [20:22:38] I really must read more about our deployment process, all this just seems too complicated [20:22:44] seems certain pages look normal as well - http://en.m.wikipedia.org/wiki/New_South_Wales_Conference_League_North [20:22:47] suggesting cache problem [20:22:57] jgonera: heh yes. [20:22:59] i'm still unable to login though MaxSem [20:23:08] has anyone else hit that problem in stable? [20:23:22] "There was an unexpected error logging in. Please try again. If the problem persists, it may be because you have cookies disabled, and you should check that they are enabled in your browser settings." [20:23:31] yes, I'm getting the "whoops" message [20:23:32] jdlrobson: yes, let's worry about that in a bit [20:24:38] 99.7% cache hit :D \o/ [20:24:54] jdlrobson: due to RL ? [20:25:14] i would call that an improvement :) jdlrobson i asume this is for bits/ [20:25:19] well things are more stable now - we're organizing our requests better [20:25:22] awjr: yup [20:25:28] binasher: MaxSem: i'm seeing a 99.7% hitrate on bits for reqs with an m domain referrer now [20:25:34] awesome [20:25:38] good job everyone :) [20:26:18] now let's fix the functionality [20:26:21] not done yet awjr [20:26:32] awjr: so old styles are still appearing [20:26:37] login is broken [20:26:49] yep [20:26:54] jdlrobson: yes i know, i wanted to get out of meltdown mode before we worried about that stuff [20:27:01] but the watchlist star is in the right place now! :) [20:27:07] haha yes it is [20:27:08] Maryana: only sometimes.. [20:27:15] haha [20:27:17] ha it moved [20:27:22] actually hard refresh fixes it [20:27:32] it's like a shooting star :) [20:27:38] ok yeh i'm seeing stable styles and scripts now [20:28:10] the login thing is weird, works in beta, not in stable [20:28:34] awjr: so steven walling raised this bug a while ago [20:28:48] which bug jdlrobson? [20:28:52] the login error [20:29:06] that was before we allowed login on beta, though [20:29:08] the old login mechanism was totally different, but i guess could be related - do you have the # handy? [20:29:23] also our first bug: https://twitter.com/ColonelKernel/status/301427675573129216 [20:29:30] issue https://twitter.com/ColonelKernel/status/301427675573129216 [20:29:32] hehe [20:30:09] heh maybe due to the star in the search bar? [20:30:17] yup [20:30:26] i've tweeted at him [20:30:45] so why would beta login be different from stable login? [20:31:16] on the plus side, this: https://twitter.com/ademianczuk/status/301372527354195968 [20:31:17] :) [20:33:55] oh that again. [20:34:01] in yo face [20:34:14] * YuviPanda scrolls back to copy paste what he told jdlrobson [20:34:37] i am baffled by this login problem. [20:35:05] you guys chattering about this has made my snide responses to jdlrobson disappear for now, so I'll just let it rest :) [20:35:39] but it was something about how once the only better thing that web has is page view numbers, I'll never stop rubbing it in :) [20:35:52] awjr: we need access to logs [20:36:00] awjr, do we have access to some production logs or something that would help us debug this? [20:36:28] (it also had something about comparing an abandoned app built by 1 guy over 6 months vs a mobile web experience built by a team of 6-7 people over years was incredibly fair) [20:36:41] we found a regression :( https://gerrit.wikimedia.org/r/#/c/48733/ [20:36:57] yuvipanda: i know :) just planting some inspirational competitive seeds [20:37:12] New review: JGonera; "Patch Set 1: Code-Review+2" [mediawiki/extensions/MobileFrontend] (master) C: 2; - https://gerrit.wikimedia.org/r/48733 [20:37:13] i can't wait for you guys to overhaul those abandoned apps [20:37:33] jgonera, jdlrobsonyeah we can see apache logs on fenari [20:37:42] Maryana: you bet. [20:38:09] but im not seeing any errors/warnings that look pertinent [20:38:25] awjr, you can ;) [20:38:38] jgonera: do you not have access to fenari? [20:39:28] awjr, I don't think so, I only have access to stat1 [20:40:03] "Permission denied (publickey)." [20:40:08] jgonera, well, at least you can see eventlogging data:) [20:40:17] even though it's not terribly relevant at the moment [20:40:47] awjr, jdlrobson, jgonera - we could add some debugging [20:41:00] the error log itself looks ok right now [20:41:05] and redeploy? [20:41:09] huh i just logged in successfully [20:41:18] oh nm was still in beta thoughi thought i was out [20:41:32] d'oh [20:42:15] this is so strange [20:42:32] ha [20:42:36] I was getting excited! [20:43:17] ok, why would even login care about mobile's site mode? [20:43:31] it shouldnt [20:46:05] there's something weird going on with the tokens [20:46:07] awjr: is a cookie not getting through? [20:46:17] jdlrobson: something along those lines i think jdlrobson [20:47:13] it seems no cookies are received on stable [20:47:39] yeah, the tokens are not getting perserved [20:47:41] varnish fun? [20:47:59] i dont know why it would happen in stable and not beta [20:49:37] centralauth_User [20:49:41] and centralauth_Session [20:50:01] as well as enwikiUserID and enwikiforceHTTPS get sent in beta but not in stable [20:50:32] the centralauth cookies will get set after the user's been logged in to the actual wiki [20:50:37] which isn't even happening [20:51:10] oh i wonder if the session cookie is not getting through [20:51:23] for token matching [20:51:29] * MaxSem yells VARNISH VARNISH VARNISH [20:51:41] yeah i think you're right MaxSem [20:53:06] so how do we change this? [20:53:09] MaxSem: do you understand what's happening around line 488 of mobile-frontend.inc.vcl.erb? [20:53:18] hahaha, after I disabled beta, testwiki stopped to recognise me as a signed in user [20:53:23] * MaxSem looks [20:53:40] do we need to add something like || req.http.Cookie ~ "session" [20:54:08] because all other cookies are getting unset before hitting the apaches [20:54:20] (all cookies aside from optin and disable) [20:54:45] awjr: that _would_ make sense.. [20:55:32] also note sign up is broken awjr [20:55:44] not sure if it uses same session cooie [20:56:06] jdlrobson: yes [20:57:56] so are those cookie-stripping rules different for stable and beta? [20:58:01] (in Varnish) [20:58:08] yes [20:58:13] why? [20:58:46] so question - why does the setting page work? [20:59:00] does the setting page not actually check the token before submitting? [20:59:11] it doesn't use the session cookie I guess [20:59:25] that is funny because it sets one ): [20:59:58] isn't the session cookie simply set by any page? [21:01:07] seems settings check the token and then just log it and ingore it... [21:01:09] if ( $request->getVal( 'token' ) != $context->getMobileToken() ) { [21:01:09] wfDebug( __METHOD__ . "(): token mismatch\n" ); [21:01:10] //return; // Display something here? [21:01:10] } [21:03:01] seems "return" was commented months ago [21:04:08] New patchset: Jdlrobson; "regression: Fix broken phpunit tests" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48735 [21:05:02] so awjr MaxSem anything i can help with? [21:12:59] MaxSem: and awjr it scares me when you are silent ;-) [21:13:11] jdlrobson: sorry, we're getting this fixed :) [21:13:13] yeah, we don't know what's going on ;) [21:13:18] ok cool that's all i wanted to check [21:13:25] one sec i'll catch you up [21:13:27] ok [21:13:47] awjr: any problems with merging fixes now? [21:13:57] let's hang on til we get this sorted [21:14:04] e.g. https://gerrit.wikimedia.org/r/#/c/48735/ [21:14:55] ok so, in MW, we use tokens in forms to help prevent CSRF attacks [21:15:28] so when you go to the login form for instance, a token gets set in a hidden form field that's some sort of hash of a unique identify that gets stored in a session cookie [21:15:49] so when the form gets submitted the token can be compared to the hash of the identiier in your session cookie [21:16:15] right now, varnish is only sending cookies that contain 'disable' or 'optin' in their names to the apaches [21:16:37] which means, the session cookie is not being sent to the apaches - the errors we're seeing are a result of the token mismatch [21:16:47] i just did: https://gerrit.wikimedia.org/r/#/c/48738/1/templates/varnish/mobile-frontend.inc.vcl.erb [21:16:52] but why is this different for stable and beta? [21:16:55] New review: Jdlrobson; "Patch Set 1:" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48598 [21:17:13] jgonera: because if you're in beta, you have the 'optin' cookie set, which means your requests totally bypass the varnish cache [21:17:30] I see [21:17:51] so what's the solution? we certainly don't want to bypass varnish in stable, right? ;) [21:18:12] (actually we probably don't want to bypass it in beta either) [21:18:16] okay - do we need to deploy any fixes? it's our deployment window after all, hahaha [21:18:32] jgonera: the solution is to do the same thing that is done in production: bypass cache if you have a session cookie set [21:18:50] I see [21:18:56] you should only have that session cookie set if you use a form provided by MW, or log in [21:19:11] jgonera: one funky thing though is that we actually set a session cookie on the settings page [21:19:29] so i thought this wouldn't be a problem - that somehow the session cookies were getting through in some other way [21:19:33] but that turned out to be a false assumption [21:19:43] right [21:19:54] the reason that we haven't seen problems with the settings page, is because even though ew set the token, we don't actually check to see if it's valid [21:20:11] anyway, we're not checking the token in settings (which is bad but not very dangerous because we don't have any sensitive information there) [21:20:18] yep [21:20:29] yah [21:20:36] so we should fix that one way or another [21:20:42] either not set the cookie, or actually check it [21:20:55] because with the change that just went out, if you have the session cookie set, you WILL bypass varnish cache [21:21:22] (that change will be live in less than 30 minutes, as puppet runs) [21:21:28] excellent [21:21:36] ok [21:21:48] we might consider prioritizing getting ESI working in MF higher now [21:21:53] awjr: seems like we should change the behavior of this and explicitly name cookies to let through? seems like a problem we could easily run into again [21:22:01] or that :) [21:22:07] jdlrobson: i think ESI is the real answer [21:22:12] awjr: ^ [21:22:15] :) [21:22:20] https://gerrit.wikimedia.org/r/#/c/48598/ < so awjr any idea how to fix this? [21:22:52] MaxSem, we should include this: https://gerrit.wikimedia.org/r/#/c/48733/ [21:22:58] i'm now thinking about making sure login works in stable [21:23:01] *logging fs [21:23:03] jdlrobson: i desperately need to eat something (it's 230 here) [21:23:09] im going to make a sandwich and i'll take a look [21:23:11] what's ESI? [21:23:15] ~10 mins [21:23:18] jgonera: http://en.wikipedia.org/wiki/Edge_Side_Includes [21:23:34] awjr: YOU CANNOT EAT! :P [21:23:40] go go go i'll grab max [21:24:06] ty [21:24:23] I agree, partial caching is the answer [21:24:31] MaxSem, are you here? [21:24:33] New review: MaxSem; "Patch Set 1: Verified+2 Code-Review+2" [mediawiki/extensions/MobileFrontend] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48735 [21:24:43] yes [21:24:54] Change merged: jenkins-bot; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48735 [21:25:12] New review: MaxSem; "Patch Set 1:" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48733 [21:25:15] ok, just checking if we have someone who can actually deploy ;) [21:25:38] New patchset: MaxSem; "regression: hide watchlist star on recent changes view" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48733 [21:27:22] New review: MaxSem; "Patch Set 2: Verified+2 Code-Review+2" [mediawiki/extensions/MobileFrontend] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48733 [21:27:39] Change merged: jenkins-bot; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48733 [21:28:51] jgonera, can you review https://gerrit.wikimedia.org/r/#/c/48663/ ? [21:30:16] ok, wait [21:30:47] jgonera: also Maryana: [21:30:50] https://gerrit.wikimedia.org/r/#/c/48586/ [21:31:35] New review: JGonera; "Patch Set 2: Code-Review+1" [mediawiki/extensions/MobileFrontend] (master) C: 1; - https://gerrit.wikimedia.org/r/48663 [21:31:55] ^ MaxSem [21:32:02] only +1?:P [21:32:16] it's really ugly ;) [21:32:24] but it's good for now [21:32:42] so +2? [21:32:51] yeah, merge it [21:33:06] New review: MaxSem; "Patch Set 1: Verified+2 Code-Review+2" [mediawiki/extensions/MobileFrontend] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48586 [21:33:23] Change merged: jenkins-bot; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48586 [21:33:24] New review: Cmcmahon; "Patch Set 1: Verified+2 Code-Review+2" [mediawiki/extensions/MobileFrontend] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48627 [21:33:26] Change merged: Cmcmahon; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48627 [21:33:44] MaxSem: is there anyway we can get this fixed and deployed? https://gerrit.wikimedia.org/r/#/c/48598/ [21:34:03] we're going to get skewed data without it ori-l Maryana [21:35:14] MaxSem: ? [21:35:19] looking [21:35:32] thanks MaxSem [21:36:01] SIGN IN IS WORKING :D [21:40:11] jdlrobson, aha: ["action"]=> string(91) "/w/index.php?title=Special:UserLogin&action=submitlogin&type=login&returnto=fadsgvfraswdrvg" [21:40:19] booya! login! yay! [21:41:16] MaxSem: we have to parse data['action']? [21:41:23] meh [21:41:29] will fix it shortly [21:41:32] New review: Jdlrobson; "Patch Set 2: Verified+2 Code-Review+2" [mediawiki/extensions/MobileFrontend] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48663 [21:41:33] Change merged: Jdlrobson; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48663 [21:41:53] jgonera: can you review https://gerrit.wikimedia.org/r/#/c/48662/ please ? [21:42:08] i didn't realize this was going to be such a deployment danger zone ;-) [21:42:40] heh yeah this has been an exciting one [21:43:59] so it seems sometimes the login cookie doesn't get through [21:44:05] try viewing watch list and switching between views awjr [21:44:11] sometimes it shows the please login message [21:44:23] both me, _jgonera and _Maryana have noticed this [21:44:55] jdlrobson: it's possible puppet hasn't finished running on all the backends yet [21:45:01] awjr: ok cool that's reassuring [21:45:25] it's been about 25mins since asher said it would take ~30 [21:45:32] let's keep an eye on it [21:45:49] so the last 4 patches merged to master, https://gerrit.wikimedia.org/r/#/c/48598/ and https://gerrit.wikimedia.org/r/#/c/48060/ still need to be deployed [21:45:57] ori-l also needs to redeploy event logging extension [21:47:53] we've also need to refresh the messages [21:47:54] New patchset: MaxSem; "on login form pass returnto and returntoquery parameters to create account" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48598 [21:47:56] looks better now - haven't gotten any random login CTAs from the watchlist star as a logged in user [21:48:01] jdlrobson, awjr ^^^ [21:48:03] many messages in beta/alpha are not showing [21:48:15] jdlrobson, that will be fixed by scap [21:48:16] i dont think the localisation caches were refreshed [21:48:24] New review: Jdlrobson; "Patch Set 2: Code-Review+1" [mediawiki/extensions/MobileFrontend] (master) C: 1; - https://gerrit.wikimedia.org/r/48598 [21:48:37] MaxSem, reviewing [21:48:56] okay boys and girls: I'd like to start scapping by 2 PM [21:49:36] what is left that needs review? [21:50:05] awjr: MaxSem Maryana anything left to test [21:50:15] before scap? [21:50:16] everything jcmish_food ;-) [21:50:28] haha k let me do that before I grab food :D [21:50:39] New review: JGonera; "Patch Set 1: Code-Review+2" [mediawiki/extensions/MobileFrontend] (master) C: 2; - https://gerrit.wikimedia.org/r/48662 [21:50:42] MaxSem: did we deploy https://gerrit.wikimedia.org/r/#/c/48060/? [21:50:56] MaxSem: https://gerrit.wikimedia.org/r/#/c/48662/ would be nice but not a big deal [21:50:59] Change merged: jenkins-bot; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48662 [21:51:16] jdlrobson, does it have "merged" anywhere?;) [21:51:53] MaxSem: will action actually give you the query string in https://gerrit.wikimedia.org/r/#/c/48598/2/includes/skins/UserLoginMobileTemplate.php ? [21:51:53] https://gerrit.wikimedia.org/r/#/c/48598/ still needs to be merged [21:52:04] awjr, yes [21:52:12] neat [21:52:24] New review: Jdlrobson; "Patch Set 2: Verified+2 Code-Review+2" [mediawiki/extensions/MobileFrontend] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48598 [21:52:25] Change merged: Jdlrobson; [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/48598 [21:53:11] OK [21:53:13] has anybody tested account creation? [21:53:14] looks like everything merged [21:53:22] jcmish_food ^ [21:53:42] Maryana: I'm on that right now [21:53:57] account creation wfm in prod [21:54:19] ori-l: poke [21:54:26] hey [21:54:35] oh although it returned me to Special:UserLogin [21:54:49] you should be able to deploy EventLogging soon when MaxSem says so [21:55:15] awjr: i guess you clicked login then create account? [21:55:17] cool. [21:55:20] jdlrobson, I've merged your config change - should it be pushed before or after Ori's update? [21:55:34] MaxSem: it shouldn't make a difference [21:55:41] alright, I'll scap it then [21:55:42] jdlrobson: yeah but i think something we weird - i tried again after clearing cache, etc and it worked as expected (from main page, sent back to main page) [21:55:51] awjr: mm weird [21:56:03] awjr: make a note of it and check it after all this is done [21:56:07] aye [21:56:09] sounds like a small bug we can live with [21:56:15] are we set to go? [21:56:33] i think so MaxSem [21:56:34] scap the shit out of it! [21:56:40] wait! [21:56:47] have we tested changes on test/ [21:57:02] MaxSem: % [21:57:03] er [21:57:04] ^ [21:57:18] (after deploying latest changes) [21:57:25] lol, we tested most of them in production [21:57:47] let's test this next push on testwiki before we go totally live [21:58:51] alrighty [21:58:53] Maryana: all my create test cases worked [21:58:58] all set [21:59:01] sweeeet! [21:59:10] yeah, i can see a bunch of accounts being created via mobile [21:59:18] a lot of them coming from the watchlist star! [21:59:29] hehehe [21:59:44] \o/ love it when people use our stuff! [22:02:53] ori-l, I accidentally updated EventLogging in wmf9 - is it safe to push? [22:03:49] awjr: worth noting by delaying deployments we are not logging accounts created after a user trying to login first via the watch list star [22:04:10] MaxSem: should be, yes, but let me take a quick look around on testwiki first. [22:04:50] jdlrobson: i know - but i dont think that's a good enough reason for us to keep being all cowboy for this deploy and not stage first [22:05:18] cowboy has got us this far ;-) [22:05:21] hehehe [22:05:52] http://blogs.msdn.com/cfs-filesystemfile.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-01-32-02-metablogapi/8054.image_5F00_thumb_5F00_35C6E986.png [22:06:11] MaxSem: yes, lgtm. [22:06:20] ori-l, thanks! [22:07:37] awjr: just seems like a waste of time for 5 patches but hey.. [22:08:08] jdlrobson: especially since max is going to run scap we should make sure we're not about to introduce more breakage [22:08:38] it wont take long for us to verify [22:09:53] alright, we're live on testwiki [22:10:09] ok let's test quick jdlrobson jgonera jcmish_food Maryana ^ [22:10:18] k [22:10:29] kk [22:11:28] ok works for me! [22:12:02] account creation is not working for me, i got a 503 error [22:12:10] oh, actually, i think we saw this before. [22:12:11] fully tested [22:12:16] everything seems to work for me [22:12:54] oh sweet, y'all killed the extraneous watchlist stars in modified view [22:12:59] :) [22:13:10] DEPLOY DEPLOY DEPLOY [22:13:31] login no longer sends you back to the page you were on [22:14:02] hmm, you're right [22:14:06] returnto appears to have been stripped out of the URL [22:14:10] jdlrobson: where can i see the real time data? [22:14:41] tfinc: stat1! [22:14:44] I think everything's fine [22:14:53] & http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&title=EventLogging&vl=events+%2F+sec&x=&n=&hreg[]=vanadium.eqiad.wmnet&mreg[]=%5E%28client-generated-raw%7Cserver-generated-raw%7Cvalid-events%29%24>ype=stack&glegend=show&aggregate=1 [22:14:53] tfinc: Maryana [22:14:54] no fancy visualizations or anything, just live stream of events [22:15:01] except for in the case of login/account creation via watchlist start [22:15:09] uh [22:15:10] oh, sweeeet :) ori-l, how long have we had that? [22:15:16] Maryana: are you ok with deploying with login from sidebar kinda broken? [22:15:29] Maryana: not very [22:15:32] how's it broken? [22:15:33] where can i see the event specific data/graphs ? [22:15:35] awjr? [22:15:40] tfinc: stat1 [22:15:48] MaxSem: http://cdn.memegenerator.net/instances/250x250/25501002.jpg [22:15:50] Maryana: logging in/account creation from sidebar will always return you to main page rather than the article you were on [22:16:09] awjr: it's always been that way by design [22:16:10] ori-l: is there a non account way of seeing the data ? [22:16:18] @_@ [22:16:21] really?! [22:16:21] yeah, it's not ideal and we should fix it soonish, but i think that's fine for now, awjr [22:16:29] wow, ignore me [22:16:40] yes the original motivation from munaf was that you can't be sure what the user wanted - as clicking login from there had no initial intention [22:16:51] tfinc: no, deliberately not, but you can stop by my desk when you're done with the workshop if you like [22:17:05] ori-l: k [22:17:08] so shouldn't we put the user back where they were instead of sending them somewehre totally different? [22:17:17] meh nevemrind [22:17:32] jcmish_food: are you satisfied for deployment? [22:17:37] awjr, ideally we should kill the post-login centralauth page entirely & just take the user back [22:17:42] checking one last thing awjr [22:17:43] kk [22:17:44] awjr: http://cdn.memegenerator.net/instances/250x250/25501002.jpg [22:17:44] then I'll let you know [22:18:11] i'm fine with making it redirect to the referring page if we're seeing that behavior. [22:18:36] hehehe jdlrobson [22:19:06] jdlrobson: the fundraising instance of jenkins used to have a sweet chuck norris plugin [22:19:23] it included a cool chuck norris watermark on some of the pages plus awesome chuck norris quip generation [22:19:56] ori-l, should I also deploy EventLogging on wmf8? [22:20:09] MaxSem: is anything still on it? [22:20:18] Maryana: awjr okay looks good to me [22:20:23] I tested beat and not beta [22:20:40] ori-l, I simply pulled and it shows that this submodule was updated [22:20:50] wait [22:20:52] my bad [22:20:55] grrrrrrrrrrrrrrrrrr [22:20:59] Maryana: resisting urge to reply to this with citation needed > https://twitter.com/Just_CallMe_E/status/301392975395696640 [22:21:01] ? [22:21:02] I'm overheating [22:21:23] ori-l, ignore me [22:21:43] * jdlrobson head explodes [22:21:58] weather report for moscow says it's −3C outside MaxSem [22:22:12] awjr, it's at least +25 inside [22:22:30] wow! [22:22:48] MaxSem: jcmish_food said everything looks OK - Maryana are you happy to push? [22:22:48] ori-l: so the EventLogging module isn't being loaded client side as it doesn't set a mobileTarget [22:23:12] which module, specifically? [22:23:23] * jdlrobson looks [22:23:25] awjr, happy! [22:23:30] w00t [22:23:36] jdlrobson: and what changed from dev -> prod? [22:23:42] ext.eventLogging [22:23:52] congrats, everyone, for staying cool through this crazy deployment [22:23:57] ori-l: we started using a better version of ResourceLoader [22:23:58] * Maryana high fives [22:24:11] previously we manually added EventLogging [22:24:41] if ext.eventLogging doesn't have 'mobileTargets' => array( 'alpha', 'beta', 'stable' ), it will default to 'mobileTargets' => array() [22:24:44] and not show up anywhere [22:25:00] awjr: I'm gonna grab a bite now :D [22:25:07] :) thanks jcmish_food [22:25:10] jdl, but it does [22:25:14] mm [22:25:32] MaxSem: is $wgMFLogEvents = true ? [22:26:17] > print_r( $wgResourceModules['ext.eventLogging']['mobileTargets'] ); [22:26:17] Array [22:26:17] ( [22:26:19] [0] => alpha [22:26:21] [1] => beta [22:26:23] [2] => stable [22:26:25] ) [22:26:50] jdlrobson, yes [22:26:53] huh [22:27:04] what about the schema module itself? [22:27:20] ori-l: well currently on mobile mw.eventLog is undefined [22:27:29] schema module is set in a config file [22:27:37] w/mobileTargets, etc? [22:27:47] in case y'all missed it, max started scap about 5 mins ago [22:27:54] ori-l: correct [22:28:12] * awjr steps out for some air [22:28:18] mm not sure why event logging wouldn't be registerd [22:28:31] jdlrobson: did this work in dev? [22:28:35] yup [22:31:11] ori-l: i can't think what might be wrong.. [22:31:13] jdlrobson: MaxSem: Hi, just a random thought as I saw a merge occur in #mediawiki, I noticed that MF is always being pushed to both the "old" and the "current" wmf branch. [22:31:48] This feels slightly dangerous / risky, because it is hard to verify things are working properly as there is no merge conflict potential (since it is updating a submodule) [22:31:51] jdlrobson: it's not in $wgResourceModules [22:32:02] It has caused issues at least once in the past when core changed between the two states [22:32:16] Krinkle, yes - this is our classic workflow [22:32:30] Does it make sense to simply refrain from updating the old branch and focussing only on the new one? Deployment train is already every 2 weeks. Afaik most projects do this already, except for mobile. [22:33:36] Krinkle, mobile deployments involve a lot of testing and are generally much harder than even core updares - we prefer to update everything ourselves [22:34:05] jcmish_food: jdlrobson i can't unwatch a page from watchlist modified articles view [22:34:07] it just spins [22:34:14] MaxSem: did you sync wmf-config/mobile.php btw? [22:34:21] tfinc: MaxSem is deploying a fix for that [22:34:31] tfinc: you shouldn't be able to do it from that screen [22:34:50] MaxSem: I don't see how that is relevant [22:34:55] brb [22:35:15] MaxSem: What I mean is, core has a 2 week deployment cycle. During that time all wikis are migrated from wmf A to wmf B, so backporting an extension update to the old wmf branch seems an unnecessary risk [22:35:34] MaxSem: As the wikis on that branch are already on schedule to be updated to the newer wmf branch within a few days [22:35:43] and I doubt you test against the old master state, do you? [22:35:59] Krinkle: I don't mean to be rude, but they're mid-deployment at the moment, so it might be best to wait a bit until they're done before discussing overall strategy, unless you think this current deployment should be stopped. [22:36:05] ori-l, you mean https://gerrit.wikimedia.org/r/#/c/48060/ ? yes [22:36:07] I know it only caused issues once (when a change was backported that uses RL target , which didn't exist yet in the old wmf branch of core) [22:36:18] jcmish_food: we need better wrapping on the watch list view. long running titles run right into the start on android 4.0 [22:36:29] ori-l: Sure, any time. [22:39:02] * Maryana obsessively refreshes server-side account creation table... [22:40:48] Maryana: client-side events should work too once MaxSem is done scapping [22:41:08] oh, nice! thank you so much for helping out, ori-l :) [22:41:22] I think it's just that the 'mobileTargets' change to EL did not go out prior to the current deploy, which is probably my fault. [22:41:24] * ori-l brbs. [22:48:37] jcmish_food: jdlrobson bug filed https://bugzilla.wikimedia.org/show_bug.cgi?id=44919 [22:52:36] yay, looks like changes are finally officially live! [22:54:35] and now i can add real screenshot to the blog post, and the world will see the strange and potentially embarassing things i have on my watchlist… such as [[Cry Me a River (Justin Timberlake Song)]] [22:57:48] jdlrobson: you'll be back in the office 26 Feb right? [22:58:17] wheee [22:59:11] scap done [22:59:18] \o/ [22:59:54] sample screen recordings from iPod touch & iPad mini: https://www.mediawiki.org/wiki/Screencasts#HDMI_capture [23:00:01] in future those blog posts can have video :D [23:00:27] im still seeing untranslated messages - like last modified [23:00:29] in eta [23:00:32] *in beta. [23:01:25] awjr: like "Last modified {{PLURAL:31|31 minute|31 minutes}} ago" or totally without translation? [23:01:31] yeah like that brion [23:01:47] yeah that's odd, for some reason it must not be parsing the message properly [23:02:05] event logging is in [23:02:08] I think I saw this before today's deployment [23:02:18] (the last modified message not being parsed) [23:02:41] I'm seeing MobileBetaWatchlist events [23:03:03] is that somehow related to RL message handling? [23:03:10] hmmmmmm, is that being done in client-side JS? [23:03:17] (no, sorry.) [23:03:17] maybe plural support isn't in the live version of core [23:04:09] hmm enwiki is running wmf9 [23:04:27] ori-l: event logging going crazy now :D [23:04:48] crazy in a good or bad sense? [23:04:52] brion was it a recent change to core? [23:05:19] jgonera: https://bugzilla.wikimedia.org/show_bug.cgi?id=44851 [23:05:22] brion: ^ [23:05:50] i think we are just not running parse - instead running mw.msg [23:06:18] good work MaxSem http://cdn.memegenerator.net/instances/400x/21977439.jpg [23:06:32] MaxSem: crazy good from our end - shows it's getting attention :) [23:06:44] message = mw.msg( 'mobile-frontend-last-modified-' + delta.unit, delta.value ) [23:07:34] yeah try changing that to mw.message(...).parse() [23:07:38] seems to work in console [23:07:52] that's in mf-last-modified.js [23:09:55] for some reason I don't see some of the upload messages [23:10:06] and the upload progress bar also doesn't seem to be there... [23:10:56] did we merge and deploy that progress bar commit? [23:11:15] (https://gerrit.wikimedia.org/r/#/c/48050/) [23:11:22] jgonera: yes we did [23:11:34] so I don't understand what's going on [23:12:13] i see a progress circle and "" :( [23:13:07] MaxSem: ^ ? [23:13:51] ugh [23:13:59] hmm [23:14:08] photo uploads are broken [23:14:09] X-Cache:MISS from cp1019.eqiad.wmnet [23:14:09] X-Cache-Lookup:NONE from cp1019.eqiad.wmnet:80 [23:14:10] X-Squid-Error:ERR_ACCESS_DENIED 0 [23:15:19] yeah, but the api returns HTTP 403 [23:16:04] :) https://twitter.com/michaelmorey1/statuses/301469661298581504 [23:17:11] weird things with photo upload: 1) api returns HTTP 403 on upload, 2) messages are not there [23:19:20] are the messages in question loaded via JS? [23:19:27] yes [23:19:44] but not all of them are missing [23:19:49] awjr: even with debug=true they don't seem to load [23:19:58] huh [23:20:03] can you give me an example? [23:20:09] mobile-frontend-image-uploading-wait, mobile-frontend-image-uploading-long and mobile-frontend-image-uploading-cancel are missing [23:20:31] well, just try uploading a photo (it will fails anyway because of 403) [23:20:43] all of those messages are in the photo upload progress notification [23:20:51] hmm [23:21:55] awjr: was it only this deployment we started forcing https [23:21:58] or was that last deployment? [23:22:16] jdlrobson: whenever we enabled login in beta - i think that was… last deployment or the one before [23:22:29] i can look at the deployment logs - why? [23:25:19] the msg thing is weird - i dont really know enough about how RL handles the message loading to have many ideas. might have things gotten cached somewhere before max ran scap (which should have refreshed the message cache)? [23:28:51] did we touch the i18n file? [23:28:51] hint: those three messages are new, they were not present before (so they're not old updated messages) [23:29:15] yeah, that's why im thinking the message keys got cached somewhere along the line without the messages [23:29:17] also, there was a message called mobile-frontend-image-uploading which we no longer use (so without -wait -long or -cancel) [23:31:13] if messages are being loaded via RL, are the messages cached along with the rest of the modules? [23:32:45] awjr, they should be [23:35:04] how long do those resources stay cached? i thought RL cache was 5 mins but cant really remember [23:35:35] it is, the startup module is cached for minutes [23:35:47] *5 minutes [23:35:49] brion: Hey Brion, can you let me know when you have 5-10 minutes to go over partner stuff? [23:35:51] the message exists on enwiki: http://en.wikipedia.org/wiki/MediaWiki:Mobile-frontend-image-uploading-wait [23:36:17] so if it's only cached for 5 mins then those messages should have refreshed b now [23:36:45] is it on en or test? [23:36:49] en [23:37:34] actually [23:37:36] test as well [23:38:06] ?debug=true has no effect [23:38:26] mw.msg() works (in console at least) [23:38:46] * awjr shrugs [23:39:39] awjr, debug=true actually fixes the lastmodified message for me [23:40:30] it does for me for a second, then i see languages expand, then collapse, then last modified goes back to being broken :p [23:41:09] riight [23:41:53] awjr, mw.msg() works for the upload messages for you? [23:42:08] Krinkle: know anything about CORS implementation on wikipedia/commons ? [23:42:13] jgonera: no, sorry, i meant mw.msg() is available and works for other messages for me in console [23:42:22] ok [23:42:26] jdlrobson: Not much, other than that it doesn't exist yet afaik. [23:42:41] it does exist.. it used to work but now doesn't :/ [23:42:46] But even if it is enabled, many browsers don't support CORS [23:42:47] i'm trying to get hold of Roan but he's busy [23:42:55] what are you using it for? [23:42:59] Krinkle: sure but we don't need to worry about the browsers that don't [23:43:03] Krinkle, photo upload [23:43:17] k [23:44:32] mw.msg('mobile-frontend-image-uploading-cancel') < MaxSem awjr - if that's not returning something then something is wrong with message delivery on MobileFrontend [23:45:05] jdlrobson, "" [23:45:06] it returns something but not the message itself [23:45:19] exactly - broken [23:45:29] it's registered but not reading from the i18n file for some reason [23:45:38] jdlrobson: MaxSem: What does mw.messages.get() yield? [23:45:47] Krinkle: ^ that [23:45:49] Because the <> message can either be created by mw.msg if it finds nothing [23:45:51] mw.msg doesn't read form the i18n file though does it? i thought it reads from some array of messages that get loaded [23:46:03] but if it is in mw.messages.values that means the server-side couldn't find it [23:46:11] (as opposed to it not being there at all on the client side) [23:46:22] mw.messages.get('mobile-frontend-image-uploading-cancel') [23:46:22] "" [23:46:34] awjr: RL exports the messages in load.php from the localization cache in mediawiki (which in turn loads from i18n.php) [23:46:51] it exports it as an object (not an array) in load.php to mw.loader.implement to mw.messages.set [23:47:01] which is what mw.msg uses to get the messages [23:47:04] gotcha so if mw.messages.get() doesn't return a message, it was unable to pull it form i18n [23:47:10] depends [23:47:22] if mw.msg('xxxx') yields it can mean 1 of 2 things [23:47:35] YuviPanda: whats your ticket for stat1 access ? [23:47:45] 1) It wasn't exported from the server, mw.messages.get('xxx') would give you undefined, and mw.msg falls back to < key > [23:48:25] mobile-frontend-image-uploading-cancel "" [23:48:56] 2) It was exported from the server mw.messages.get('xxx') literally gives , in that case it means the server side got when doing wfMsg('xxx'), in other words, the i18n.php file used at the time was outdated (or rather, since we disable i18n file loading on the cluster in favour of manually refreshed i18n cache, the i18n cache was outdated) [23:49:24] Krinkle, it was scapped [23:49:35] this can happen if you run sync without updating i18n cache afterwards (in that order) [23:49:40] so either l18n update during scap is borky [23:49:56] afaik scap does it in the right order [23:50:16] or there's some cache that persisted through the scap [23:50:19] MaxSem: What happens if you run mwscript somewiki eval.php and run wfMsg('') ? [23:50:22] (for that message) [23:50:22] Krinkle:does the mw.messages.values object get cached in some way that may not have been refreshed inspite of scap, etc? [23:50:32] awjr: no [23:51:12] does it return the right value? If not, then it is a server-side issue. Stop looking at js and RL :) [23:51:29] try running i18n update or scap again, not sure what to do in that case. Outside my area. [23:51:29] Krinkle, it works [23:51:30] Krinkle: we had a funky deployment today because ew simultaneously had to respond to an emergency. so files were sync'd out before message cache was refreshed, and things stayed in that state for a while. scap was run a while later to pick up a few more changes and refresh the message cache [23:51:57] awjr: Aha, in that case the problem is more likely to be cache related indeed [23:52:06] but where would things get cached? [23:52:46] awjr: What I think happened is that the files were sync, but i18n not updated yet. Then someone viewed the site, making a load.php request, which then sees the files have been updated, re-generates the module cache for that module and saves it for the max(timestamp) [23:53:10] Then when the i18n cache was updated again later it had no affect because the max(timestamp) of everything is still the same [23:53:25] so.. touch/sync? [23:53:48] Krinkle: ^ [23:53:55] Sure, that'll do. Touch 1 js or css file in the module with the broken i18n message in it [23:54:00] and sync that file [23:54:13] then wait between 1-5 minutes for bits to expire. [23:54:48] ok, touched javascripts/specials/donateimage.js and just ran sync-file [23:54:55] (on wmf9) [23:55:27] awjr: You may want to file a bug against LocalizationCache to investigate a potential bug. When i18n update is ran, the last-mod timestamps of those messages shouldn't be the same as before the i18n update. [23:55:42] yeesh [23:55:43] Especially if the messages, you know, actually were chanced. [23:55:45] changed* [23:55:50] That's the whole point of the cache timestamp :) [23:56:59] awjr, is there any trick I could use in production _not_ to be redirected from http to https? [23:57:10] jgonera: not if you're logged in [23:57:21] if you're logged out you should just be able to use http [23:57:38] yes, but then I can't access uploads ;) [23:57:52] Krinkle, is there a way to force a RL update by e.g. nuking the cached timestamp information? [23:57:53] interesting is that what's causing the 403? [23:58:11] does protocol have an effect on cors policy or something? [23:58:35] that's what I'd like to test [23:58:43] MaxSem: This information is not relevant [23:58:53] there's nothing else that comes to my mind [23:59:01] MaxSem: the startup module discards that info every 5 minutes and re-calculates it [23:59:26] MaxSem: The problem is the files, as before, an update was made that did not cause any files or module references to change [23:59:27] also SUL appears to not be working with mobile login [23:59:59] where are the cors rules defined?