[01:53:34] <aude>	 jdlrobson: around?
[01:53:47] * aude wondering about wikidata descriptions in mobile
[07:57:37] <phuedx>	 morning o/
[07:58:23] <joakino>	 morning phuedx 
[09:59:24] <hashar>	 o/
[12:09:28] <phuedx>	 o/ hashar 
[12:09:30] <phuedx>	 how're you?
[12:10:08] <hashar>	 sleep deprived as usual :D
[12:10:11] <hashar>	 but overall fine!
[12:10:14] <phuedx>	 kids?
[12:10:16] <hashar>	 yeah
[12:10:20] <phuedx>	 KIDS
[12:10:25] <phuedx>	 (i know the feeling)
[12:10:33] <phuedx>	 my youngest wakes up at ~5 am at the momen
[12:10:34] <phuedx>	 *t
[12:10:36] <hashar>	 and lot of over hours last two weeks (mw train deploy is in our evening)
[12:10:36] <phuedx>	 teeth
[12:10:43] <phuedx>	 bleh
[12:10:47] <hashar>	 ah for teeth
[12:11:26] <hashar>	 for our first we timed paracetamol with the nightly  milk at 2am
[12:11:42] <hashar>	 so would give her paracetamol at like 8pm and again at 2am :}
[12:12:10] <hashar>	 but yeah it is terrible :/
[12:13:49] <hashar>	 anyway the secret is to have naps whenever possible :d
[12:14:45] <phuedx>	 ^
[12:14:45] <phuedx>	 :D
[12:14:58] <phuedx>	 sleep whenever the baby sleeps was the advice that my mum gave me
[12:15:05] <phuedx>	 i followed it to the letter ;)
[12:15:41] <phuedx>	 hashar: work related: where can i find the ci config for the mwext-mw-selenium job?
[12:15:50] <phuedx>	 the environments.yml specifically
[12:16:44] <phuedx>	 or does it use the one from the repo?
[12:16:46] <phuedx>	 hrrm
[12:16:50] <phuedx>	 maybe it does
[12:17:45] <phuedx>	 hrrm -- jenkins apparently sets up MEDIAWIKI_URL
[12:18:18] <phuedx>	 hashar: basically just sit there and be my rubber duck while i figure out why a build is failing ;)
[12:18:21] <phuedx>	 :D :D :D
[12:18:23] <phuedx>	 thanks!
[12:18:46] <hashar>	 phuedx: should use the one from the repo
[12:18:49] <hashar>	 per design :D
[12:19:04] <hashar>	 the idea behind that environments.yml is to delegate the configurations being run in Jenkins to the devs
[12:19:40] <hashar>	 zeljkof has all the details
[12:33:18] <phuedx>	 hashar: it seems like either the mediawiki_url that jenkins is //setting// or the mediawiki_url that mediawiki-ruby-api is //expecting// is wrong
[12:33:31] <phuedx>	 raise a bug for zeljko and dduvall?
[12:33:43] <hashar>	 yeah  most probably
[12:33:51] <hashar>	 and poke zeljkof about it in #wikimedia-releng
[12:34:05] <hashar>	 he is better than me at debugging such issues
[12:34:13] <hashar>	 it is probably a simple mistake
[12:34:28] <hashar>	 (and maybe you need the gems to be updated)
[13:29:53] <phuedx>	 olliv: yt?
[13:30:07] <olliv>	 hello!
[13:30:39] <olliv>	 phuedx: good afternoon
[13:32:17] <phuedx>	 olliv: and the same to you
[13:32:55] <phuedx>	 i just wanted to clarify that, afaict from the config changes, wikidata descriptions are enabled everywhere (apart from as taglines on mediawikiwiki and metawiki)
[13:33:21] <olliv>	 phuedx: yes, I noticed - do you know why?
[13:33:47] <phuedx>	 no
[13:34:28] <olliv>	 do you think it's possible to turn them off at least for the top 6 asap or should we wait for jonr?
[13:38:07] <phuedx>	 olliv: we've missed the mid-day swat window
[13:38:23] <phuedx>	 but i can line up a config change for this evening (1800 utc)
[13:41:16] <olliv>	 phuedx: okay - let's do that
[15:08:16] <olliv>	 joakino: backlog grooming?
[16:00:17] <bearND>	 coreyfloyd: will be a few minutes late
[16:00:34] <coreyfloyd>	 bearND: k - just ping me when ready
[16:35:35] <bearND>	 coreyfloyd: we're waiting for you in the next meeting
[17:01:09] <maxbinder>	 olliv: nzr standup
[18:04:30] <coreyfloyd>	 dbrant: you doing credit or do you have some free time?
[18:04:55] <dbrant>	 coreyfloyd: credit. but free time afterwards.
[18:05:15] <coreyfloyd>	 dbrant: going to toss something on your calendar
[18:05:20] <coreyfloyd>	 thanks
[18:32:06] <matanya>	 niedzielski: i don't know what happened, but the app crashes on me at every launch
[18:32:26] <matanya>	 niedzielski: i have an adb logcat: https://dpaste.de/EMNG
[18:32:29] <matanya>	 if that helps
[18:32:30] <niedzielski>	 matanya: hm
[18:33:01] <niedzielski>	 matanya: you're browsing in hebrew?
[18:33:06] <matanya>	 yes
[18:33:51] <niedzielski>	 matanya: ok, i can repro :] will file a ticket with the details and cc you. thanks for the report and log!
[18:34:26] <matanya>	 good niedzielski but how can i use the app now? go back to prev version ?
[18:35:11] <niedzielski>	 matanya: i think it's likely a bug in the output of the content service backend and an incorrect assumption in the app. let me get a little more detail and see if i can provide a workaround
[18:35:29] <matanya>	 niedzielski: thanks!
[18:35:48] <niedzielski>	 matanya: thank you!
[18:35:52] <matanya>	 :)
[18:39:39] <niedzielski>	 matanya: hm. the feature article card is coming back as an empty object which i think is unexpected. the content service shouldn't send an empty card
[18:40:06] <matanya>	 niedzielski: is that sca ?
[18:40:19] <niedzielski>	 matanya: if you're in a bind, i can give you a hacked alpha build that will account for the failure, otherwise you will have to wait until the bug is fixed in the content service
[18:40:40] <niedzielski>	 matanya: sorry, what do you mean by sca?
[18:41:09] <matanya>	 contect service is what is also known as SCA ?
[18:41:56] <matanya>	 niedzielski: I have 68 apks of the app on my phone, i can always revert to prev version if it is better than getting one from you
[18:41:57] <niedzielski>	 matanya: i'm not sure. some folks call it MCS or mobile content service but i think this is a horrible thing to call it since everyone, not just mobile devices, are supposed to use it
[18:42:14] <matanya>	 I see
[18:42:22] <niedzielski>	 matanya: i think you'd have to go pretty far back or hope that tomorrow there is a featured article that is digested by the content service
[18:42:49] <niedzielski>	 matanya: this bug is probably also on prod :|
[18:42:59] <matanya>	 Is content service in the hands of marco ?
[18:43:33] <matanya>	 also niedzielski : https://phabricator.wikimedia.org/T144960
[18:43:44] <niedzielski>	 matanya: marco contributes but its owned by android
[18:44:11] <niedzielski>	 matanya: could be related. not enough info to say unfortunately
[18:44:28] <matanya>	 niedzielski:  just to understand, you are the owner of this service, or another team ?
[18:45:00] <niedzielski>	 matanya: i don't own it personally. the android team owns it of which i am a member. most contributions on the team have been made by bearND or mdholloway
[18:45:27] <matanya>	 I see, thanks for the clarification 
[18:48:56] <niedzielski>	 matanya: T144990
[18:48:56] <stashbot>	 T144990: [CRASH] Content Service shouldn't send empty objects - https://phabricator.wikimedia.org/T144990
[18:49:01] <mdholloway>	 ^looking into this
[18:49:11] <matanya>	 thanks niedzielski and mdholloway
[18:49:33] <mdholloway>	 matanya: you're welcome, thanks for the report!
[18:50:36] <niedzielski>	 mdholloway: i just added the logcat output which i should have done in the first place
[18:52:38] <niedzielski>	 mholloway: i'm guessing the fix is just to call mUtil.isEmpty(tfa) and only set the field if false
[18:53:19] <matanya>	 why would it be false anyway, we have a tfa every day
[18:53:51] <niedzielski>	 mdholloway: crash in production too :/
[18:54:16] <niedzielski>	 matanya: sometimes the wiki pages are difficult to parse
[18:54:30] <mdholloway>	 affects german as well
[18:54:43] <mdholloway>	 crashing on prod
[18:54:43] <bearND>	 oh oh, this probably came in with this mornings deploy
[18:54:55] <mdholloway>	 i'd actually call this an unbreak now level bug
[18:55:26] <bearND>	 yes. /cc: mobrovac 
[18:56:09] <mobrovac>	 euh?
[18:56:12] <bearND>	 I think this is probably caused by https://gerrit.wikimedia.org/r/#/c/309004/, which I had not seen yet
[18:56:28] <bearND>	 mobrovac: the app is crashing since the deploy this morning
[18:56:37] <mobrovac>	 ^^^ fixed a crash i witnessed in prod
[18:56:48] <bearND>	 now it sends an empty object for tfa instead of omitting it
[18:56:53] <mobrovac>	 ah!
[18:56:54] <mobrovac>	 ok
[18:56:59] <mobrovac>	 bearND: lemme fix it
[18:57:02] <bearND>	 ok
[18:57:04] <mobrovac>	 bearND: 
[18:57:06] <mobrovac>	 hm
[18:57:08] <mobrovac>	 actually
[18:57:16] <mobrovac>	 there is a reason why i put the empty object
[18:57:24] <mobrovac>	 oh no ok
[18:57:37] <mobrovac>	 i know how i can fix it to make everybody happy
[18:57:42] <mobrovac>	 at least, i think
[18:57:47] <mobrovac>	 lemme do a patch and we can discuss
[18:58:08] <bearND>	 mobrovac: What was the issue that prompted you to add it?
[18:58:38] <mobrovac>	 if you do res.json(null), the client sees that as an abrupt end of the conn and fails
[18:59:10] <mobrovac>	 but that's true for /v1/page/featured/
[18:59:18] <mobrovac>	 which i guess is not called by the app at all
[18:59:23] <mobrovac>	 but it is called by RB
[18:59:28] <bearND>	 but that's only for the microservice. The aggregated endpoint should omit the property altogether
[18:59:54] <mobrovac>	 yeah but the individual endpoints are called bt RB now
[18:59:58] <mobrovac>	 s/by/by/
[19:00:00] <mobrovac>	 damn
[19:00:03] <mobrovac>	 :P
[19:00:21] <bearND>	 mobrovac: can we revert this for now until we get a solution that works for both?
[19:00:50] <mobrovac>	 no, because if we revert this clients will see 500s so it;s no good either
[19:01:09] <mobrovac>	 gimme 10 mins
[19:01:13] <bearND>	 ok
[19:06:56] <matanya>	 Once it is over i think it deserves an outage report
[19:09:00] <mdholloway>	 matanya: yes, that seems fair
[19:12:08] <dbrant>	 bearND: mdholloway: we can cancel Grooming, if you guys are in the thick of it
[19:13:09] <bearND>	 dbrant: mdholloway yes, we might need to.
[19:13:28] <mdholloway>	 yes, postponing or cancelling seems best
[19:14:02] <dbrant>	 alright!
[19:20:16] <mobrovac>	 bearND: mdholloway: https://gerrit.wikimedia.org/r/309075
[19:20:26] <mobrovac>	 need to fix RESTBase too, patch coming up
[19:21:29] <bearND>	 ok, ty
[19:28:17] <mobrovac>	 bearND: mdholloway: https://github.com/wikimedia/restbase/pull/667
[19:31:19] <mobrovac>	 bearND: if you're k with the patch, i can force-merge it
[19:31:39] <mobrovac>	 oh, wait, need to remove a comment
[19:32:10] <mobrovac>	 bearND: kk, good to go now, shall i +2 ?
[19:32:16] <mobrovac>	 ps3
[19:32:19] <bearND>	 mobrovac: yes, I'm ok with it. Just looked at the code. Unfortunately I cannot repro the issue when just running against my local server. I think it's the interaction with RB that plays an important role in this.
[19:32:29] <mobrovac>	 yes
[19:32:37] <bearND>	 yes, +2
[19:32:38] <mobrovac>	 you need a rb server locally too to see that
[19:33:01] <bearND>	 in mw-vagrant?
[19:33:18] <mobrovac>	 that works too
[19:33:39] <bearND>	 ok, will check this out later
[19:33:56] <mobrovac>	 bearND: just verified locally that curl localhost:7231/fr.wikipedia.org/v1/feed/featured/2016/09/06 does not give an empty tfa with https://gerrit.wikimedia.org/r/309075 and https://github.com/wikimedia/restbase/pull/667
[19:37:34] <mobrovac>	 bearND: deploying https://gerrit.wikimedia.org/r/#/c/309079/
[19:39:08] <bearND>	 mobrovac: want to deploy the rb change first?
[19:39:19] <bearND>	 seems more innocuous to me
[19:39:26] <mobrovac>	 yup
[19:39:53] <mobrovac>	 the RB fix works with the current mcs code in prod too
[19:40:04] <bearND>	 yes
[19:42:21] <dbrant>	 coreyfloyd: ready now, if you like
[19:42:34] <coreyfloyd>	 dbrant: sure
[19:42:47] <mdholloway>	 yeah, i was wondering if the mcs change was even needed so long as rb checks for and doesn't send empty objects in the response
[19:43:02] <coreyfloyd>	 dbrant: heading to the hangout on the invite
[19:43:52] <mdholloway>	 s/send/include
[19:49:56] <mobrovac>	 mdholloway: right, but better to be precise :)
[20:00:26] <mobrovac>	 bearND: mdholloway: ok, in 10 mins all should be back to normal 
[20:01:51] <bearND>	 mobrovac: thanks for the deploys. Can't wait for this to return back to normal. App still crashes on startup. Even cleared the cache.
[20:02:21] <mobrovac>	 bearND: yeah, we need to wait for the content to expire on the server side before the new version is computed
[20:02:28] <mobrovac>	 the ttl is set to 10 mins for that
[20:04:30] <mdholloway>	 bearND: mobrovac: thanks for the fixes, mobrovac -- he, de, fr all working again
[20:05:15] <mobrovac>	 oh good!
[20:05:26] <bearND>	 now it's working for me, too. :)
[20:05:27] <mobrovac>	 i''l try to clear the RB cache to be on the safe side
[20:07:31] <bearND>	 I guess I'll respond to some of the of app store reviews in a couple of minutes when things should be cleared up
[20:10:00] <mdholloway>	 bearND: mobrovac: cool.  i'll start writing up an incident report.  this outage probably started with the restbase deploy at 13:31 UTC, correct?
[20:10:24] <mobrovac>	 bearND: mdholloway: cache cleared, we should be good now
[20:10:41] <mobrovac>	 mdholloway: yup
[20:10:54] <mdholloway>	 mobrovac: cool, thanks again.
[20:11:03] <mobrovac>	 bearND: mdholloway: i need to go now, but if something's wrong, please send me a mail directly and i'll come online asap
[20:11:48] <mdholloway>	 mobrovac: ok!  forgot about the time difference.  thanks again and good night!
[20:11:55] <bearND>	 mobrovac: ok, sounds good. Thank you!
[20:12:00] <mobrovac>	 not yet, not yet
[20:12:05] <mobrovac>	 first, bowling time! :)
[20:12:13] <mobrovac>	 np
[20:12:29] <bearND>	 at 10pm?
[20:12:40] <mobrovac>	 yeah well, i'm already late :P
[20:12:58] <mobrovac>	 quite late actually
[20:12:59] <mobrovac>	 hahah
[20:49:26] <niedzielski>	 mhurd coreyfloyd joewalsh: o/ would one of you devs mind reviewing this patch when you can? https://gerrit.wikimedia.org/r/#/c/304801/
[20:50:54] <mdholloway>	 niedzielski: ^ ah, thanks for pinging, i was meaning to do that this p.m. but got distracted by the app outage
[20:51:39] <niedzielski>	 mdholloway: np :] it will be nice to see it merged
[20:53:45] <coreyfloyd>	 niedzielski: sure
[20:54:00] <mhurd>	 niedzielski: coreyfloyd: checking it now...
[20:54:15] <coreyfloyd>	 mhurd: oh cool… i defer to you
[20:54:18] <coreyfloyd>	 thanks!
[20:54:18] <niedzielski>	 mhurd coreyfloyd: thanks!
[20:54:35] <mhurd>	 coreyfloyd: niedzielski: np
[21:00:01] <mhurd>	 niedzielski: coreyfloyd: mdholloway: the css patch looks good. i +1ed it
[21:01:31] <niedzielski>	 mhurd: yayyy!
[21:01:33] <mdholloway>	 mhurd: thanks for the review! :)
[21:01:45] <niedzielski>	 mdholloway: going to +2 now
[21:01:49] <mhurd>	 mdholloway: niedzielski: :)
[21:02:00] <mdholloway>	 niedzielski: cool
[21:02:48] <coreyfloyd>	 mhurd: niedzielski awesome thanks!
[21:12:16] <mhurd>	 coreyfloyd: one that patch is merged/deployed we'll be able to remove the redundant styles from our "misc.less" the next time we sync with upstream css
[21:12:23] <mhurd>	 *once
[21:12:29] <coreyfloyd>	 sweet
[21:12:38] <mhurd>	 coreyfloyd: every little bit :)
[21:12:58] <mhurd>	 coreyfloyd: i'll make a ticket to remind us
[22:06:35] <mobrovac>	 mdholloway|afk: did you start an incident report?
[22:10:44] <mdholloway|afk>	 mobrovac: yep, i've got a draft going, i'll send your way
[22:13:31] <mobrovac>	 mdholloway|afk: cool!
[22:14:10] <mobrovac>	 mdholloway|afk: you can put it as a subpage of https://wikitech.wikimedia.org/wiki/Incident_documentation and we can complete it async
[22:14:31] <mobrovac>	 you can just copy the template at https://wikitech.wikimedia.org/wiki/Incident_documentation/Report_Template
[22:16:34] <mdholloway|afk>	 mobrovac: cool, i thought there must be a template somewhere, thanks for the links
[22:16:44] <mobrovac>	 heh np
[22:17:02] <mobrovac>	 hopefully you won't need it too often :)
[22:26:58] <mdholloway|afk>	 lol, i hope not!  
[22:28:52] <mdholloway|afk>	 created https://wikitech.wikimedia.org/wiki/Incident_documentation/20160907-Android
[22:34:54] <jdlrobson>	 niedzielski: bearND fyi in case you didnt see.. i just submitted a bunch of mobile content service patches
[22:35:16] <niedzielski>	 jdlrobson: sweeet
[22:57:02] <mobrovac>	 mdholloway|afk: the report looks really good
[22:57:17] <mobrovac>	 in the informative sense, not that it's good that it needed to be written
[22:57:18] <mobrovac>	 :P
[22:58:07] <mobrovac>	 btw, it's customary to announce the report on the relevant lists, mobile-l and ops-l are probably good targets
[22:58:24] <mobrovac>	 (ops-l is kind-of obligatory for this sort of things)