[01:27:50] <phuedx>	 jdlrobson: yt?
[01:28:01] <jdlrobson>	 yes why are you here. Far too late for you
[01:28:15] <phuedx>	 can't sleep
[01:28:30] <jdlrobson>	 phuedx: :( those browser tests will keep you up all night
[01:28:32] <phuedx>	 lisa woke me up a while ago
[01:29:02] <phuedx>	 http://en.m.wikipedia.beta.wmflabs.org/wiki/Quick_survey_test_page_stub?quicksurvey=true works on a tablet for me
[01:29:30] <phuedx>	 the fix i pushed for the qunit tests should also cover this case
[01:29:53] <phuedx>	 that being said, the tests are still failing!?
[01:30:04] <jdlrobson>	 it doesn't for me for some reason..
[01:30:17] <jdlrobson>	 something funky is going on
[01:30:36] <phuedx>	 are the browser tests hitting mfbeta and seeing the opt in panel?
[01:30:43] <phuedx>	 that'd stop the survey from showing /anywhere/
[01:30:59] <phuedx>	 sorry -- the nightly browser tests
[01:31:37] <jdlrobson>	 wait what.. why do we check minerva.betaoptin ?
[01:31:51] <phuedx>	 don't want to show ALL THE PANELS
[01:32:12] <phuedx>	 the placement of a quick survey in mfbeta was way too spammy with the beta opt in panel
[01:33:09] <phuedx>	 i dunno -- possible red herring
[01:34:47] <jdlrobson>	 i can't replicate it locally but is issue on beta labs
[01:35:42] <phuedx>	 i'm not sure that we can do much more without help from release engineering
[01:36:07] <phuedx>	 as in, someone sitting down beside me/you and figuring out what's going on in the test environment
[01:36:51] <jdlrobson>	 what if mw.trackSubscribe( 'minerva.betaoptin', is being registered before mw.track( 'minerva.betaoptin' is called?
[01:37:28] <phuedx>	 late subscribers get notified iirc
[01:38:57] <phuedx>	 jdlrobson: data point: i see a survey at <= 768 px on minerva
[01:39:02] <phuedx>	 but /not/ above
[01:39:05] <phuedx>	 on the beta cluster
[01:39:10] <phuedx>	 on the stub article
[01:39:12] <jdlrobson>	 yeh that's what i'm saying :)
[01:39:48] <phuedx>	 that's not what your task says
[01:39:59] <phuedx>	 "at 768px" is not "above 768px"
[01:40:03] <phuedx>	 :P
[01:40:30] <phuedx>	 (it's late)
[01:41:37] <jdlrobson>	 oh i see ;-) I thought i just said tablet mode
[01:42:09] <jdlrobson>	 it's all very weird
[01:45:15] <phuedx>	 jdlrobson: i think beta doesn't have an updated asset
[01:45:30] <jdlrobson>	 i suspect so. Hopefully it will pass tomorrow
[01:45:34] <jdlrobson>	 it doesn't make much sense
[01:45:49] <jdlrobson>	 we've had numerous issues with cached RL modules historically
[01:47:00] <phuedx>	 jdlrobson: http://en.wikipedia.beta.wmflabs.org/static/master/extensions/QuickSurveys/resources/ext.quicksurveys.lib/lib.js
[01:47:03] <phuedx>	 olde olde olde
[01:50:39] <bmansurov>	 insomnia, phuedx?
[01:50:53] <phuedx>	 yarrrp
[01:51:15] <bmansurov>	 :( don't know how you'll get up at 6
[01:51:50] <phuedx>	 easily
[01:52:49] <bmansurov>	 good then
[01:53:29] <bmansurov>	 it's outsomnia for me, gotta go soon
[01:53:45] <phuedx>	 ever have nights where you can't sleep
[01:53:50] <bmansurov>	 did I just enrich the english language vocabulary by one word?
[01:53:58] <bmansurov>	 nope ;)
[01:54:01] <phuedx>	 you're a regular shakespeare
[01:54:10] <phuedx>	 i'll admit that the browser test thing has been bugging me
[01:54:22] <bmansurov>	 ohh, that's why you can't sleep?
[01:54:33] <bmansurov>	 what a commitment!
[01:55:17] <bmansurov>	 the first step to good sleep is to admit that we're helpless when browser tests fail :P
[01:56:44] <kristenlans>	 "outsomnia". Classic baha
[01:56:57] <bmansurov>	 hey kristen!
[02:01:26] <phuedx>	 alright it's 2 am
[02:01:29] <phuedx>	 i'm going to try again
[02:01:33] <phuedx>	 zzZ
[02:01:39] <phuedx>	 wait, who's kristenlans again?
[02:02:52] <phuedx>	 BURN
[02:03:23] <kristenlans>	 phuedx: cold man, cold
[03:00:58] <MaxSem>	 heh, even our own apps are affected by this kerffuffle
[04:33:18] <tgr>	 anyone around who can tell me how to get a HTTP traffic log of the android app?
[04:39:03] <niedzielski>	 tgr: hey! i'm trying to get you the info you want. that part of the app is using our older networking layer that doesn't support logging very well
[04:40:34] <niedzielski>	 tgr: i can tell you we're unexpectedly getting the NeedToken response here https://phabricator.wikimedia.org/diffusion/APAW/browse/master/app/src/main/java/org/wikipedia/login/LoginTask.java;50037ff167aedde52d18e7b15041a8f47317d321$53
[04:40:59] <tgr>	 niedzielski: https://phabricator.wikimedia.org/T124252#1954389 and https://phabricator.wikimedia.org/T124252#1951112 gives you an idea of what's *probably* going on
[04:41:04] <niedzielski>	 tgr: the first login post comes back NeedToken as expected but the so does the second (unexpected)
[04:41:18] <tgr>	 or it could be a bug in the API response but so far we couldn't verify one
[04:54:11] <tgr>	 niedzielski: thanks! so the problem is that the app doesn't handle expired cookies correctly
[04:55:03] <tgr>	 enwikiToken=deleted -> that's something like Set-Cookie: enwikiToken=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; secure; httponly
[04:55:19] <tgr>	 expires is in the past so it should not be sent back
[04:55:37] <tgr>	 I assume there is no chance to fix that in the app within a reasonable timeframe?
[04:57:45] <niedzielski>	 tgr: hrm, i can look into fixing it tonight but it takes up to 24 hours to be available in the app store :/
[04:59:21] <tgr>	 I'll see if we can change the bahvior on the API side, that won't be out before tomorrow morning either though
[04:59:58] <niedzielski>	 tgr: one thing i'm not grokking is i cleared the app cookies manually and i still see this issue
[05:00:45] <tgr>	 the deleted cookie is set on the API response in the first leg of the login sequence, so that won't help
[05:01:13] <tgr>	 when you start the login, the API tries to delete any pre-existing login
[05:02:17] <tgr>	 if it's not too much work, you could check what Set-Cookie headers you are getting with the API responses, just to makes sure this analysis is correct, but I'm pretty sure that's what's going on
[05:02:47] <niedzielski>	 tgr: ah, ok. well, i'll do what i can on my side. offhandedly it looks like the way this was handled is to catch unexpected exceptions and delete all cookies wholesale
[05:03:42] <niedzielski>	 tgr: sure, give me a couple minutes and i'll follow up. thanks for all your helo
[05:03:45] <niedzielski>	 help* :)
[05:51:26] <niedzielski>	 ok i think i might it sorted in the app. i'll cc you on the patch
[05:53:07] <niedzielski>	 oh and thanks again for your help and patience :)
[07:54:52] <phuedx>	 jdlrobson: yt?
[08:37:21] <joakino>	 morning
[08:57:34] <phuedx>	 morning joakino 
[13:15:28] <phuedx>	 going out to do some bank-related stuff (boo)
[14:49:10] <mdholloway>	 dbrant: i see i missed some excitement last night.  i suppose we're doing releases sometime today?  anything for me to do right now?  otherwise i'll get back to work on the app-side changes for the wiktionary update and try to squeeze that into the beta release.
[14:50:34] <dbrant>	 mdholloway: yeah, the release today will be just the latest production tag, with the login patch cherry-picked. Not much else to do there.
[14:51:16] <mdholloway>	 dbrant: gotcha
[15:14:29] <dbrant>	 niedzielski: mdholloway: quick batcave session? (assuming it's not too early)
[15:14:38] <mdholloway>	 omw
[15:16:07] <niedzielski>	 omw
[15:42:56] <niedzielski>	 dbrant: shall i add a line like "* Fix for issue preventing login."? we have 90 chars to spare from the last release
[15:45:55] <dbrant>	 niedzielski: yep, sure
[16:09:06] <phuedx>	 bmansurov_gone: could you include your browser and its version in your review please?
[16:09:21] <bmansurov_gone>	 ah sorry, i will
[16:22:22] <phuedx>	 bmansurov_gone: nm -- it's not browser-based
[16:22:34] <phuedx>	 it's faulty logic on my part
[16:23:36] <phuedx>	 (read: sorry)
[16:26:33] <phuedx>	 bmansurov_gone: there's a linear gradient mixin?
[16:26:41] <phuedx>	 all i see is the vertical-gradient mixin
[16:28:32] <phuedx>	 brb
[16:59:29] <bmansurov_gone>	       /nick back
[17:00:43] <bmansurov>	 phuedx: you may be right, sorry for the confusion
[17:09:52] <dr0ptp4kt>	 phuedx: joakino can i do more (or less) on that image loading phabricator task? any other tasks you'd like me to get written up? for example, anything references related.
[17:22:38] <phuedx>	 dr0ptp4kt: i haven't had time to dig deeper into the references stuff
[17:22:46] <phuedx>	 i've been working on getting the stories in the sprint into sign off
[17:23:02] <phuedx>	 i'm confident that most of monday will be spent investigating
[17:23:08] <phuedx>	 so sync at monday's standup?
[17:24:14] <dr0ptp4kt>	 phuedx: cool cool. monday's standup is  fine. i'm just wondering if there's anything i can do to jumpstart things today. i have fewer meetings than i have all week, although i can spend the time reviewing tasks that are already in the sprint 64 board and acting on a pile of emails
[17:24:25] <dr0ptp4kt>	 sorry, i said 'standup'
[17:25:14] <dr0ptp4kt>	 phuedx: i meant to say monday's reading web story prioritization meeting is a good place to power through some part of the story writing / decomposing
[17:25:28] <phuedx>	 okie poke
[17:26:23] <dr0ptp4kt>	 phuedx: but if you get a head start on it before then, cool. i'm mainly thinking about my own work here this fine rainy friday
[17:27:06] <phuedx>	 dr0ptp4kt: SPIKE: What information is necessary to reconstruct the references HTML?
[17:27:10] <phuedx>	 (or something along those lines)
[17:27:47] <phuedx>	 the internal representation has a lot of info, i'm wondering if we can shrink it down at all while being able to reconstruct the html
[17:29:32] <dr0ptp4kt>	 phuedx: are you saying i should create that?
[17:30:00] <phuedx>	 maybe, maybe not -- hopefully i might have an answer on monday
[17:35:10] <phuedx>	 dr0ptp4kt: sorry, that wasn't helpful
[17:35:10] <phuedx>	 hrrm
[17:35:16] <phuedx>	 ok -- i need to get the boys to bed
[17:35:19] <phuedx>	 shall be back later
[17:36:04] <dr0ptp4kt>	 phuedx: ttyl
[18:00:23] <joakino>	 dr0ptp4kt: I can't think of anything right now, I'll have a look during monday from the epics and tasks and fill up any gaps I can think of
[18:02:04] <dr0ptp4kt>	 joakino: cool. hey, how do you guys normally treat fridays? is it commonly a 10% time day, or does it depend on conditions in the sprint when 10% time is used up? i know some times dedicate one day per sprint for 10% time, which makes it consistent. but i also know a lot of teams (and i've been told at least anecdotally) tend to crank out a lot of pure
[18:02:05] <dr0ptp4kt>	 sprint work product on fridays, even mid sprint fridays
[18:04:05] <joakino>	 dr0ptp4kt: there's no set rule, depending on how busy the sprint is we may do experimentation or not. traditionally I think people tend to do it at the end of the week if they do
[18:06:22] <dr0ptp4kt>	 joakino: cool. thanks. i'd been meaning to ask about that. my preference really is for a consistent 10% time allocation in a given sprint, although i'm indifferent to whether that's eight one hour blocks, two four hour blocks, or one eight hour block, or some such other thing :)
[18:07:21] <joakino>	 dr0ptp4kt: I usually try to do around half day on fridays for experimentation every week, since every day there's work stuff to do, so that gives me time to do it and a some time every week to clear mind and do something different (instead of once every 2 weeks)
[18:09:15] <dr0ptp4kt>	 joakino: cool - sounds like a nice way to do it. for me i kind of alternated between splitting on two days across the sprint versus taking a full day on a sprint. never did really figure out which one i liked more.
[18:12:07] <joakino>	 yeah I guess it depends on the workload of every week
[18:16:06] <mbinder>	 kaity_: android standup
[18:44:40] <Deskana>	 dbrant, bearND|afk, mdholloway, niedzielski: I was poking around in the app last night, thinking about A/B testing, and got a bit confused about what one of the methods was for. I wrote a patch to improve the documentation for that method: https://gerrit.wikimedia.org/r/#/c/265784/
[18:45:36] <Deskana>	 dbrant, bearND|afk, mdholloway, niedzielski: I might be wrong about what the method is for though, so I'd appreciate review. :)
[18:46:19] <niedzielski>	 Deskana: thank you for your continued patchwork!! will review in a bit (we're all in meeting right now)
[18:46:33] <Deskana>	 niedzielski: Thanks! And yeah, no rush.
[18:55:42] <lzia>	 hi bmansurov. a clarification question: is Nathaniel from Research working on part of https://phabricator.wikimedia.org/T123696 ?
[18:56:31] <lzia>	 I'm a  bit confused. :-) I see your name in the task, which is great. I just want to make sure we track things closely so we don't end up waiting for each other given that the time is tight. :-)
[18:56:38] <lzia>	 bmansurov, ^
[19:09:11] <bmansurov>	 lzia left?
[19:27:56] <mhurd>	 https://usercontent.irccloud-cdn.com/file/j1R457N7/Screen%20Shot%202016-01-22%20at%2011.27.21%20AM.png
[19:28:09] <mhurd>	 ^ umm... what?
[19:28:10] <mhurd>	 lol
[19:30:32] <mhurd>	 went to revert and someone beat me to it
[19:39:33] <dbrant>	 mhurd: rofl
[19:44:13] <etonkovidova>	 mhurd: is there any particular reason that all articles started with HMS give 404 in android?
[19:44:56] <mhurd>	 etonkovidova: hmm shouldn't be... dbrant are you seeing that?
[19:45:53] <dbrant>	 etonkovidova: mhurd: working fine for me...
[19:47:19] <etonkovidova>	 dbrant: mhurd interesting ... the latest build on Nexus 5 all articles  with HMS in title - e.g. HMS Hood, HMS Furious 
[19:48:09] <niedzielski>	 mdholloway: so, when you say the patch doens't work. what do you mean? i'm checking the logcat and i see the response was kept. the map size is 6 for cat
[19:48:58] <mdholloway>	 I'll try your patch again, but last time I got: 01-22 14:45:50.540 11071-11071/org.wikipedia.dev E/org.wikipedia.wiktionary.WiktionaryDialog$1: failure():141: Wiktionary definition fetch error: retrofit.RetrofitError: 501 Not Implemented
[19:49:05] <mdholloway>	 same as always
[19:50:25] <niedzielski>	 mdholloway: well, i hardcoded my url to http://192.168.1.3:6927/en.wiktionary.org/v1 so it wouldn't keep putting an en.m in it
[19:50:30] <niedzielski>	 (under dev settings)
[19:50:37] <niedzielski>	 i thought this was a different issue
[19:50:55] <etonkovidova>	 dbrant: reinstalled - all is fine now But it was really interesting to observe such selectiveness for 404 ...
[19:55:05] <mdholloway>	 niedzielski: sure enough, making that change works.
[19:55:56] <mdholloway>	 niedzielski: but why would that be happening now?  or have been happening the whole time, but worked before and not now?  :|
[19:56:26] <niedzielski>	 etonkovidova: well keep in mind that you have a 50% chance of Content Service (RESTBase option) or MediaWiki backend service on reinstall of alpha / beta / dev
[19:56:38] <niedzielski>	 mdholloway: not sure. you know i'
[19:56:49] <niedzielski>	 i'm always having trouble / complaining about endpoint problems
[19:57:51] <mdholloway>	 niedzielski: etonkovidova: actually we're up to 100% in RESTBase on pre-production builds/releases as of yesterday
[19:58:31] <mdholloway>	 sorry, a bit sloppy, i should 100% say content service (though RB also true)
[19:58:47] <mdholloway>	 s/100%/say
[20:01:14] <niedzielski>	 whoopsies
[20:02:43] <niedzielski>	 mdholloway: are you good on this patch then or do you want me to check out this other en.m issue?
[20:03:28] <mdholloway>	 niedzielski: yeah, i'm digging in on the en.m issue if you want to move onto something else.  thanks for your help digging into this!
[20:03:46] <niedzielski>	 mdholloway: np. sounds good.
[20:07:00] <mbinder>	 coreyfloyd: join us for offsite chat hangout
[20:24:48] <mdholloway>	 niedzielski: so here's a curious thing. injecting the .m has apparently happened since time immemorial due to 'final String domain = site.getApiDomain();' on L40 of RbEndpointsCache.java.  I just tried changing it to site.getDomain() and that fixes the problem; everything works. 
[20:25:15] <mdholloway>	 niedzielski: so why would we have been purposely putting in the .m before?
[20:25:31] <niedzielski>	 mdholloway: is there someone we can talk to about rb routing?
[20:25:38] <niedzielski>	 maybe there was a change
[20:25:45] <mdholloway>	 niedzielski: i think that would be bearND|afk ...
[20:27:34] <mdholloway>	 niedzielski: ah, in restbase, you mean.  mobrovac and gwicke in #services would be the experts there
[20:28:25] <mdholloway>	 niedzielski: i initially thought you were talking about a change to RbEndpointsCache, and there's been no change there
[20:29:03] <gwicke>	 niedzielski: IIRC, varnish strips the .m bit
[20:30:16] <niedzielski>	 hm
[20:30:57] <niedzielski>	 mdholloway: is prod RB usage 404 for every page or just me?
[20:31:17] <gwicke>	 https://github.com/wikimedia/operations-puppet/blob/b622f6eaa6ac15edfa2f05f2fa0092051e4ff517/templates/varnish/text-backend.inc.vcl.erb#L63
[20:31:32] <niedzielski>	 gwicke: thanks!
[20:31:36] <mdholloway>	 niedzielski: just you, i think... (did you change your hardcoded en.wiktionary.org domain back to %2$s?)
[20:32:17] <mdholloway>	 niedzielski: ah, prod, nm
[20:32:34] <niedzielski>	 mdholloway: yeah. uninstalled. every RB page is 404 now
[20:36:06] <mdholloway>	 niedzielski: on your patch i was getting results from prod but crashing when trying to pull up a definition :(
[20:36:20] <mdholloway>	 niedzielski: pulling up a clean install from master now
[20:38:26] <mdholloway>	 niedzielski: everything working fine for me from production RB/content service
[20:41:59] <niedzielski>	 mdholloway dbrant: if i delete the app, install the prod app from play store, force RESTBase on without changing the URI,all pages are 404s
[20:42:58] <dbrant>	 ಠ_ಠ
[20:45:33] <mdholloway>	 niedzielski: can't reproduce...
[20:45:49] <niedzielski>	 mdholloway: yikes. going to reboot phone
[20:48:04] <niedzielski>	 mdholloway: same issue. dbrant can you repro?
[20:49:54] <dbrant>	 niedzielski: mdholloway: working for me..
[20:50:18] <niedzielski>	 mdholloway: geez. what gives, fresh install of latest master beta build with no changes 404s even on the emulator for me
[20:50:25] <niedzielski>	 dbrant:^
[20:54:07] <niedzielski>	 dbrant mdholloway: ok it worked for a moment but now it's back to broken...
[20:54:22] <mdholloway>	 niedzielski: on the wiktionary change, i realized it's actually my service patch that's causing the 501 unsupported language error for en.m.*
[20:54:41] <mdholloway>	 the service should be agnostic to the presence or absence of m
[20:55:09] <mdholloway>	 so i'm going to make a change to the server side patch, mind testing once again in a moment once it's in gerrit?
[20:55:35] <niedzielski>	 mdholloway: sure i can test. prod is 99% broken for me right now though
[20:55:40] <niedzielski>	 not sure why only for me
[20:55:52] <mdholloway>	 tbh i'm still not sure why we're changing the URL to include the m but best not to change that before talking to bearND|afk 
[20:56:03] <mdholloway>	 niedzielski: yeah, that's really strange, i'll keep trying to repro here
[20:56:15] <mdholloway>	 niedzielski: you said you're getting 404s?
[20:56:22] <niedzielski>	 mdholloway: ok it just worked for one page but  then i got a 404 for the next click
[20:56:28] <niedzielski>	 mdholloway: yeah
[20:59:53] <gwicke>	 mdholloway, niedzielski: we generally key on the project domain without the .m, and have been testing without it so far
[21:02:02] <niedzielski>	 mdholloway dbrant: it seems to bounce back and forth between working and not more regularlly when clearing data and trying again
[21:02:14] <niedzielski>	 gwicke: thanks
[21:04:44] <niedzielski>	 gwicke: is it likely that RESTBase would be failing in the Colorado area but working elsewhere? i seem to be getting some sporadic but regular 404s
[21:05:22] <gwicke>	 niedzielski: do you have a URL that 404s for you?
[21:06:05] <gwicke>	 it's unlikely that the response would differ by area
[21:06:32] <mobrovac>	 hey guys
[21:06:36] <niedzielski>	 gwicke: https://en.m.wikipedia.org/api/rest_v1/page/mobile-sections-lead/Brazil
[21:06:57] <niedzielski>	 o/
[21:07:26] <gwicke>	 is that 404ing for you in the browser, too?
[21:07:31] <niedzielski>	 gwicke: yes
[21:08:06] <gwicke>	 could you check which IP en.m.wikipedia.org resolves to for you?
[21:08:38] <gwicke>	 also, is https://en.wikipedia.org/api/rest_v1/page/mobile-sections-lead/Brazil also 404ing?
[21:08:48] <niedzielski>	 gwicke: https://phabricator.wikimedia.org/P2520
[21:09:01] <niedzielski>	 gwicke: no, that second request works
[21:09:22] <gwicke>	 niedzielski: do you have a local host entry pointing to labs, or something like that?
[21:09:48] <niedzielski>	 gwicke: no
[21:09:52] <gwicke>	 what's the output of `host en.m.wikipedia.org` ?
[21:10:06] <niedzielski>	 en.m.wikipedia.org has address 208.80.153.236
[21:10:07] <niedzielski>	 en.m.wikipedia.org has IPv6 address 2620:0:860:ed1a::1:c
[21:10:19] <niedzielski>	 (from my pc)
[21:10:55] <gwicke>	 interesting, that's codfw
[21:11:10] <gwicke>	 mobile-lb.codfw.wikimedia.org
[21:11:50] <gwicke>	 this sounds like a matter for bblack
[21:13:09] <mobrovac>	 yup
[21:13:11] <niedzielski>	 gwicke: ok thanks. i'll ping him in the operations channel
[21:13:22] <mobrovac>	 i get the correct response on my laptop as well
[21:18:32] <gwicke>	 mdholloway, niedzielski: I think it's clear that this issue is specific to the .m. domain variant
[21:18:59] <niedzielski>	 mm
[21:19:00] <niedzielski>	 i mean hmm
[21:19:24] <gwicke>	 it should work consistently without the .m.
[21:21:34] <codezee>	 jdlrobson: when you have a moment, can you have a look at https://gerrit.wikimedia.org/r/#/c/265826/
[21:21:49] <jdlrobson>	 codezee: sure
[21:22:29] <jdlrobson>	 looks good codezee :)
[21:22:41] <mdholloway>	 niedzielski: i just took the debug logging back out of your Wiktionary patchset and pushed to Gerrit.  It's r4r from my end.  Mind giving it a spin with the latest service patch on https://gerrit.wikimedia.org/r/#/c/265655/?
[21:23:43] <niedzielski>	 gwicke: yeah, seems no problem without m
[21:24:24] <niedzielski>	 mdholloway: sure but i can only test by hardcoding "no m"
[21:24:34] <mdholloway>	 niedzielski: that shouldn't matter now
[21:24:51] <mdholloway>	 niedzielski: i updated the service patch to accept en.m.*
[21:25:02] <mdholloway>	 niedzielski: ah, but your stuff
[21:25:13] <mdholloway>	 :(
[21:26:06] <mdholloway>	 dbrant, would you mind doing a little wiktionary testing when you have a moment?
[21:27:14] <mdholloway>	 niedzielski: does the app work for you with standard API loading?  From looking earlier I would have that that used .m as well.
[21:27:49] <niedzielski>	 mdholloway: you mean mediawiki? yeah, that's great. everything restbase comes back as 404 unless i hardcode a non-m domain
[21:28:07] <mdholloway>	 niedzielski: yeah, the mediawiki api i mean
[21:28:18] <niedzielski>	 mdholloway: no issues
[21:28:23] <gwicke>	 so, the issue is specific to codfw
[21:28:41] <gwicke>	 it has some new varnish code that broke the rewrite
[21:30:06] <gwicke>	 niedzielski, mdholloway: Could you consistently use the main project domain instead of .m. ?
[21:31:20] <niedzielski>	 gwicke: i think we still need m for mediawiki api? we could update the RB stuff AFAIK
[21:31:46] <mdholloway>	 gwicke: I don't think that would cause any issues.  Like I said, I'm not sure why we're sending out requests with the .m. in the domain to begin with.  niedzielski, dbrant, anything you can think of that i'm missing?
[21:32:20] <gwicke>	 niedzielski: is the API behaving differently for .m. ?
[21:32:23] <niedzielski>	 mdholloway: right, i don't know the history of that either
[21:32:47] <mdholloway>	 niedzielski: yes, I could at least change the content service/restbase side if we wanted to be conservative about it
[21:32:48] <niedzielski>	 gwicke: i'm not sure about MW behaving differently for m or not but i don't know why we would have used it in the first place
[21:32:54] <niedzielski>	 mdholloway: right
[21:33:03] <mobrovac>	 mdholloway: let me know when you guys are ready to roll out the new page/definition format so that we do it on both ends at the same time
[21:33:18] <mobrovac>	 there's no diff with or without .m.
[21:33:23] <mobrovac>	 mediawiki never sees that
[21:33:27] <mdholloway>	 mobrovac: will do.  hopefully soon.  you're still in sf, right?
[21:33:32] <mobrovac>	 yup
[21:33:37] <niedzielski>	 mdholloway mobrovac: do we need a beta today?
[21:34:04] <mdholloway>	 yup.
[21:34:54] <gwicke>	 there are several downsides to using .m., including performance (cache fragmentation) and complexity (need to purge .m. separately), so if you could move all API requests to the main project, that would be great
[21:35:16] <niedzielski>	 mdholloway: your patch appears to work when i harcode the domain to .../en.wiktionary.org/...
[21:36:00] <mdholloway>	 niedzielski: if we want to stop .m in content service requests now, i could make that change in the app-side wiktionary patch before we merge it.  from my testing, doing that doesn't set the world ablaze (as we would expect it wouldn't)
[21:36:17] <niedzielski>	 mdholloway: two betas in one day!
[21:36:27] <gwicke>	 yay, Fridays!
[21:36:31] <mdholloway>	 niedzielski: achievement unlocked!
[21:37:15] <niedzielski>	 mdholloway /cc gwicke mobrovac: in my limited experience with the networking layer of our app, that code is convoluted. do we need to remove the m endpoint usage today or can it wait for some testing
[21:37:16] <niedzielski>	 ?
[21:38:19] <gwicke>	 niedzielski: bblack is looking into a temporary work-around that would make codfw work again; I do think though that dropping .m. is preferable and safe
[21:38:26] <mobrovac>	 niedzielski: it's not so urgent that needs to be done on a friday afternoon and possibly break beta for the whole weekend :)
[21:39:14] <gwicke>	 it's broken right now
[21:39:32] <gwicke>	 so the biggest risk is breaking it for more users
[21:40:29] <niedzielski>	 gwicke mobrovac mdholloway dbrant: hm. can we leave the MW stuff for later or does it all have to go in one shot? (it might be easier to do both but i want to know my options)
[21:41:21] <gwicke>	 niedzielski: no reason to do everything at once from my end
[21:41:50] <gwicke>	 doing only the REST API would un-break things for now
[21:42:46] <niedzielski>	 gwicke mobrovac mdholloway dbrant: hm, i can start working on it now but i want at least dbrant's blessing before we publish it. this part of our app is "between patches" right now :/
[21:43:23] * dbrant looks at chat log...
[21:43:28] <mobrovac>	 lol
[21:47:26] <dbrant>	 niedzielski: mdholloway: hmm, if we're only talking about Beta, and we're absolutely sure that removing .m is safe for MW API, then i would say go for it.
[21:48:14] <gwicke>	 for the MW API, the host header is rewritten to the main domain before sending the request to the backend
[21:49:39] <mobrovac>	 correct, as i said, MW never sees the .m. in the domain name
[21:53:02] * niedzielski working to remove m&.m's the code base
[21:55:35] <mdholloway>	 dbrant: did you want to test the wiktionary changes before we start merging/deploying/releasing? looks like i've got +1s from mobrovac on the service side and niedzielski on the app side
[21:55:49] * gwicke likes m&m's
[21:56:05] <dbrant>	 mdholloway: ah yes, i'll run through real quick
[21:56:31] <mdholloway>	 cool patches here https://gerrit.wikimedia.org/r/#/c/265768/ and here https://gerrit.wikimedia.org/r/#/c/265655/
[22:02:41] <dbrant>	 mdholloway: ok, all looks good to me
[22:03:11] <niedzielski>	 dbrant mdholloway: -m patch coming soon i think. running some tests over here
[22:03:26] <mdholloway>	 dbrant: awesome, thanks!
[22:04:19] <mdholloway>	 mobrovac: did you want someone else to review https://gerrit.wikimedia.org/r/#/c/265655/ or just +1 to wait for the app patch?  dbrant just tested and everything seems in order
[22:06:11] <niedzielski>	 mobrovac gwicke: since we're talking about RESTBase problems in Colorado area, would there be any reason I would get 503s very rarely? so far i've only seen them on Android Marshmallow devices but thought i'd ask
[22:07:06] <mobrovac>	 mdholloway: +1'ed it to wait for the app part
[22:07:19] <mobrovac>	 mdholloway: i'm happy to merge whenever you're ready
[22:07:54] <etonkovidova>	 niedzielski: nobody but me getting 502 for http://android-builds.wmflabs.org/ ? Today is kind of a day of surprises for me :(
[22:08:01] <mdholloway>	 niedzielski: dbrant: ok to start service side merge/deployment, or should we wait on the .m patch?
[22:08:10] <mobrovac>	 niedzielski: "very rarely" as in, when outside of colorado the 503s come more often?
[22:08:21] <niedzielski>	 etonkovidova: i get 502 for that as well
[22:09:05] <niedzielski>	 mdholloway dbrant: i say plow ahead with service deployment. we can't sync these things perfectly anyway
[22:09:14] <mobrovac>	 there was a round of labs restarts, niedzielski and etonkovidova, so you might need to log onto there and check things out
[22:09:24] <etonkovidova>	 niedzielski: sigh of relief...  
[22:09:28] <etonkovidova>	 ok
[22:10:42] <mdholloway>	 niedzielski: yeah, makes sense
[22:10:49] <mdholloway>	 mobrovac: ok!  let's merge.
[22:11:00] <mobrovac>	 k
[22:11:00] <mdholloway>	 mobrovac: do you want to deploy or should i?
[22:11:09] <mobrovac>	 as you wish mdholloway
[22:11:57] <mdholloway>	 mobrovac: i'll deploy
[22:12:01] <mobrovac>	 kk
[22:12:06] <niedzielski>	 mdholloway: i'm merging your app side patch now, cool?
[22:12:15] <mdholloway>	 niedzielski: sounds good!
[22:13:04] <mobrovac>	 btw guys, i'm doing a dump of frwiki for mobile-sections
[22:13:10] <mobrovac>	 (enwiki has been completed)
[22:20:28] <niedzielski>	 dbrant mdholloway: please check out that -m patch when you can. i'm going to start the beta process
[22:20:44] <dbrant>	 testing...
[22:21:31] <niedzielski>	 etonkovidova: http://android-builds.wmflabs.org/ is back
[22:27:03] <etonkovidova>	 niedzielski: thx!
[22:33:19] <dbrant>	 niedzielski: best of all, we won't even need to think about having a fallback from desktop to mdot...lol
[22:33:38] <dbrant>	 or the other way around, i mean
[22:35:27] <mobrovac>	 mdholloway: the amount of 404s has been significantly reduced!
[22:35:30] <mobrovac>	 yay
[22:35:40] <mdholloway>	 mobrovac: hooray!
[22:35:55] <mdholloway>	 i was just watching the log on scb1002
[22:36:01] <mdholloway>	 still getting some, though
[22:36:46] <dbrant>	 mdholloway: niedzielski: i'm ready to merge the mdot patch. any objections?
[22:36:57] <mdholloway>	 dbrant: none!
[22:37:02] <niedzielski>	 dbrant: 👍
[22:42:44] <niedzielski>	 mdholloway dbrant: release notes started in the usual place
[22:46:01] <mdholloway>	 mobrovac: hmm, i'm still getting the old style results from the production endpoint...
[22:46:04] <mdholloway>	 https://rest.wikimedia.org/en.wiktionary.org/v1/page/definition/horse
[22:46:32] <mobrovac>	 ah oh!
[22:46:53] <mobrovac>	 mdholloway: ofc, i need to clean up the db, so that it doesn't contain the old results
[22:46:56] <mobrovac>	 hang on, doing it
[22:50:54] <dr0ptp4kt>	 moizsyed: here's the modal. pretty sweet. https://phabricator.wikimedia.org/T123384
[23:13:55] <dbrant>	 niedzielski: added some notes; all looks good
[23:14:29] <niedzielski>	 dbrant: thanks :)
[23:17:50] <mdholloway>	 niedzielski: lgtm as well
[23:18:23] <niedzielski>	 mdholloway: cool thanks
[23:23:46] <mobrovac>	 mdholloway: {{done}}, only new defs from now on!
[23:24:13] <mdholloway>	 mobrovac: awesome!  thanks.
[23:24:14] <niedzielski>	 gwicke mdholloway dbrant mobrovac: correction, i get that 503 across devices. i'll get more info and cc you on the phab
[23:24:34] <niedzielski>	 (unrelated to today's changes AFAIK)
[23:27:43] <mdholloway>	 mobrovac: still more 404s coming in than i had hoped
[23:28:29] <mobrovac>	 mdholloway: touché
[23:34:31] <mdholloway>	 niedzielski: anything for me to stick around for re: the beta release?
[23:34:42] <niedzielski>	 mdholloway: no, thanks :)
[23:34:55] <mdholloway>	 ok!  i'd better run.  have a good weekend, everyone!
[23:35:19] <niedzielski>	 mdholloway: you too, thanks :)
[23:38:04] <gwicke>	 niedzielski: okay
[23:57:47] <kaity_>	 hey dbrant I've been following the village pump discussion on gather
[23:58:08] <kaity_>	 do you know more about the WMF reply next week? https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(proposals)#A_WMF_reply_next_week