[00:43:15] <grrrit-wm>	 (CR) Krinkle: [C: -1] "This module does not account for changes in messages in the maxtime." [extensions/MobileFrontend] - https://gerrit.wikimedia.org/r/74684 (owner: Jdlrobson)
[00:44:00] <grrrit-wm>	 (CR) Krinkle: "Merging it seems to make that situation worse, though not introducing the problem. If you want to do that later, feel free to ignore my -1" [extensions/MobileFrontend] - https://gerrit.wikimedia.org/r/74684 (owner: Jdlrobson)
[02:43:11] <wikipedia-github>	 [Commons-iOS] montehurd synchronize pull request #100: Restyled settings page to match rest of app styling more closely. (master...settingsStyle)  http://git.io/p4VAjQ
[03:00:03] <wikipedia-github>	 [Commons-iOS] montehurd synchronize pull request #100: Restyled settings page to match rest of app styling more closely. (master...settingsStyle)  http://git.io/p4VAjQ
[03:48:59] <wikipedia-github>	 [Commons-iOS] montehurd opened pull request #101: Sharing via email was overlaying the nav bar. Looked weird. (master...detailsShare)  http://git.io/2vyHEw
[04:17:50] <wikipedia-github>	 [Commons-iOS] montehurd synchronize pull request #101: Sharing via email was overlaying the nav bar. Looked weird. (master...detailsShare)  http://git.io/2vyHEw
[04:22:24] <wikipedia-github>	 [Commons-iOS] montehurd synchronize pull request #101: Sharing via email was overlaying the nav bar. Looked weird. (master...detailsShare)  http://git.io/2vyHEw
[06:50:37] <wikipedia-github>	 [Commons-iOS] montehurd opened pull request #102: Made app start fresh - avoids pre-update memory cruft issues. (master...startFresh)  http://git.io/I-gxAg
[06:53:32] <wikipedia-github>	 [Commons-iOS] montehurd closed pull request #85: Customized the navigation bar to *not* use classic gradient look (master...customNavBar)  http://git.io/s-mv5g
[07:49:01] <wikipedia-github>	 [Commons-iOS] montehurd synchronize pull request #100: Restyled settings page to match rest of app styling more closely. (master...settingsStyle)  http://git.io/p4VAjQ
[08:21:09] <wikipedia-github>	 [Commons-iOS] montehurd synchronize pull request #100: Restyled settings page to match rest of app styling more closely. (master...settingsStyle)  http://git.io/p4VAjQ
[08:37:24] <wikipedia-github>	 [Commons-iOS] montehurd synchronize pull request #100: Restyled settings page to match rest of app styling more closely. (master...settingsStyle)  http://git.io/p4VAjQ
[15:47:35] <wikipedia-github>	 [Commons-iOS] siebrand pushed 1 new commit to master: http://git.io/6-PjMg
[15:47:35] <wikipedia-github>	 Commons-iOS/master 4d6904b Siebrand Mazeland: Localisation updates from http://translatewiki.net.
[15:47:46] <wikipedia-github>	 [WikipediaMobile] siebrand pushed 1 new commit to master: http://git.io/nQRyYw
[15:47:46] <wikipedia-github>	 WikipediaMobile/master da32016 Siebrand Mazeland: Localisation updates from http://translatewiki.net.
[15:59:21] <grrrit-wm>	 (PS3) Jdlrobson: Enable login for older phones [extensions/MobileFrontend] - https://gerrit.wikimedia.org/r/75058
[17:29:21] <YuviPanda>	 mhurd: did we finally find a time for the design syncup that works for everyone?
[17:29:49] <mhurd>	 YuviPanda: the soonest i could find was next monday i believe
[17:29:54] <YuviPanda>	 oh wow
[17:29:54] <YuviPanda>	 ok
[17:30:05] <YuviPanda>	 sigh.
[17:31:36] <mhurd>	 YuviPanda: :( sorry i couldn't find sooner time
[17:32:09] <mhurd>	 YuviPanda: did you get invite?
[17:32:20] <YuviPanda>	 if it is of any help, I'm usually up until 5pm PST every day these days :)
[17:32:34] <YuviPanda>	 looking
[17:32:57] <YuviPanda>	 mhurd: i see it
[17:33:01] <YuviPanda>	 i just was assuming it is this week
[17:34:13] <mhurd>	 YuviPanda: lots of people are offsite for product strategy this week
[17:34:23] <YuviPanda>	 yeah, guessed
[17:34:26] <YuviPanda>	 i guess it'll have to wait
[17:38:59] <jcmish>	 MaxSem: Is there any issue you see with keeping the deployment at 1pm PT on Wednesday? Are you avaialble?
[17:39:10] <MaxSem>	 yes I am
[17:39:47] <MaxSem>	 no issues, as long as we don't steal anyone else's time
[17:40:41] <jcmish>	 k sweet
[18:01:46] <YuviPanda>	 mhurd: look at the history?
[18:01:48] <YuviPanda>	 on the right?
[18:04:31] <mhurd>	 YuviPanda: no luck thus far
[18:08:30] <jgonera>	 jcmish, do we have the QA/Design review today?
[18:10:22] <mhurd>	 YuviPanda: i could be wrong, but i'm wondering if trello may have had some data loss…
[18:10:38] <YuviPanda>	 mhurd: I... doubt that. Would've been publicized in some form
[18:10:42] <YuviPanda>	 mhurd: write to support?
[18:10:47] <YuviPanda>	 we've a business subscription and all
[18:15:04] <mhurd>	 YuviPanda: they've changed the link format recently i believe. the link i was able to recover (from a google hangout chat) looks like this: https://trello.com/card/login-page-interaction-update/5112b893d0cd43cb370062bd/57 but the only trello card i can find with that title has a link which looks like this: https://trello.com/c/jMJojx5e/14-login-page-interaction-update. thought maybe when they made this switch something may have not carried over
[18:25:47] <YuviPanda>	 mhurd: probably
[18:25:50] <YuviPanda>	 mhurd: still poke support?
[18:25:54] <YuviPanda>	 they'll be able to sort it out
[18:25:56] <mhurd>	 YuviPanda: writing them now
[18:26:06] <YuviPanda>	 sweet
[18:26:06] <YuviPanda>	 brb
[19:38:07] <grrrit-wm>	 (PS1) MaxSem: Fix CentralAuth logins [extensions/MobileFrontend] - https://gerrit.wikimedia.org/r/75155
[20:55:43] <jcmish>	 jgonera: can you test something for me?
[20:55:54] <jgonera>	 jcmish, I can try, what is it?
[20:55:57] <jcmish>	 http://en.m.wikipedia.beta.wmflabs.org/wiki/San_Francisco
[20:56:04] <jcmish>	 if you hit it do you get an error?
[20:56:36] <jgonera>	 yes, error 500
[20:56:41] <jcmish>	 k thanks
[20:56:46] <jcmish>	 I kept hearing it's fixed
[21:08:44] <chrismcmahon>	 (02:06:04 PM) hashar: chrismcmahon: got the stack trace :)
[21:08:44] <chrismcmahon>	 (02:06:25 PM) hashar: [22-Jul-2013 21:05:53] Fatal error: Call to a member function removeChild() on a non-object at /data/project/apache/common-local/php-master/extensions/MobileFrontend/includes/formatters/HtmlFormatter.php on line 256
[21:09:31] <chrismcmahon>	 jgonera: jcmish ^^
[21:10:19] <MaxSem>	 chrismcmahon, how to repro?
[21:10:43] <jcmish>	 MaxSem: just click on the link above
[21:10:50] <jcmish>	 http://en.m.wikipedia.beta.wmflabs.org/wiki/San_Francisco
[21:10:51] <MaxSem>	 awwwwsum
[21:10:55] <jcmish>	 and you get the errror
[21:10:58] <jcmish>	 *error
[21:16:06] <hashar>	 hi :)
[21:16:23] <MaxSem>	 yo hashar
[21:18:32] <hashar>	 MaxSem: chrismcmahon told me there is stack trace on beta mobile cache
[21:18:33] <hashar>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=51840
[21:19:00] <MaxSem>	 awesome, thanks!
[21:19:09] <hashar>	 http://en.m.wikipedia.beta.wmflabs.org/wiki/San_Francisco gives a stack trace
[21:19:17] <hashar>	 I think michelle mentioned it to me last week
[21:19:27] <hashar>	 but Imust have forgotten to fill in a bug for it
[21:19:32] <hashar>	 [22-Jul-2013 21:11:53] Fatal error: Call to a member function removeChild() on a non-object at /data/project/apache/common-local/php-master/extensions/MobileFrontend/includes/formatters/HtmlFormatter.php on line 256
[21:19:33] <hashar>	 :(
[21:19:43] <jcmish>	 Thanks hashar  I was just going to create a bug and link it
[21:19:58] <hashar>	 nowadays lot of issues are related to PHP code :-]
[21:20:26] <MaxSem>	 hashar, maybe we should depart from reproducing prod wrt error handling and install xDebug and remove WMFErrors?:)
[21:20:47] <hashar>	 eheh
[21:21:15] <hashar>	 or logs all fatals in a database and fill bug automatically :-]
[21:21:38] <chrismcmahon>	 hashar: that's actually a pretty good idea greg-g ^^
[21:22:15] <hashar>	 ori-l wrote something to consume exception.log / fatal.log
[21:22:22] <hashar>	 but it is more a pet project ;)
[21:22:52] <hashar>	 the idea is to regex the files, extract the fields, do an http query to event logging and that then log that in a database
[21:23:08] <hashar>	 so potentially we could create a dashboard out of the data logged there
[21:24:38] <YuviPanda>	 hashar: we should jsut setup logstash :P
[21:24:45] <YuviPanda>	 or something. should be easy-ish.
[21:25:44] <chrismcmahon>	 YuviPanda: and here you were just now talking about monitoring tool labs, we could use some beta labs monitoring also
[21:26:05] <YuviPanda>	 chrismcmahon: hah! I guess they're fundamentally different things, though.
[21:26:21] <hashar>	 YuviPanda: definitely :-]
[21:26:28] <MaxSem>	 hashar, but the problem here is that Michelle should know what happened if she sees an error - a PHP bug or Labs down again:)
[21:26:50] <MaxSem>	 to know whom to poke
[21:26:56] <hashar>	 MaxSem: could you train her to get access on bastion and do the first level check ? such as looking at the log maybe
[21:27:01] <MaxSem>	 otherwise, shell poke you;)
[21:27:34] <chrismcmahon>	 hashar:  I don't even remember where the logs are on bastion any more :)
[21:28:15] <hashar>	 chrismcmahon: I have symlinked /data/project/logs  in my homedir
[21:28:21] <YuviPanda>	 hashar: I'd love to help out with helping other people out with betalabs, just so you know :)
[21:28:24] <hashar>	 chrismcmahon: so I just log on deployment-bastion and do:  cd logs
[21:28:37] <hashar>	 YuviPanda: go ahead!  ton of doc can be written :-]
[21:28:46] <chrismcmahon>	 hashar: awesome; write that down somewhere?
[21:29:01] <hashar>	 YuviPanda: and you will be more than welcome there :]  It is a good step to learn about the prod cluster as well
[21:29:03] <MaxSem>	 okay jcmish, teaching time!:P
[21:29:12] <YuviPanda>	 hashar: indeed, that's my aim. of sorts.
[21:29:26] <hashar>	 YuviPanda: so yeah beta would help as a training platform
[21:29:35] <jcmish>	 ha
[21:29:52] <jcmish>	 I'm good with the learning
[21:29:55] <hashar>	 https://wikitech.wikimedia.org/wiki/Beta
[21:29:56] <jcmish>	 asked for it before :D
[21:30:01] <YuviPanda>	 chrismcmahon: isn't there already a labs icinga?
[21:32:21] <jcmish>	 MaxSem: wanna get together some time tomorrow?
[21:32:24] <jcmish>	 how long would it take?
[21:35:44] <MaxSem>	 3 minutes?:P
[21:36:47] <jcmish>	 ha
[21:36:53] <greg-g>	 hashar: chrismcmahon I want ori's pet project to get real legs
[21:37:18] <jcmish>	 greg-g: what do we do to help it get legs
[21:37:20] <jcmish>	 I'm in
[21:37:21] <jcmish>	 :D
[21:37:57] <MaxSem>	 jcmish, okay - log into labs
[21:38:01] <jcmish>	 MaxSem: at a high level I know how to grep logs etc I think really all I need is to know how to differentiate between something beta labs vs a read MF bug
[21:39:24] <jcmish>	 *not read real MF bug
[21:39:57] <greg-g>	 jcmish: provide dev time :)
[21:40:21] <jcmish>	 ha :D sure I'll magically create a few for ya greg-g
[21:40:42] <grrrit-wm>	 (PS1) MaxSem: Bug 51840: removeChild() on a non-object in HtmlFormatter line 256 [extensions/MobileFrontend] - https://gerrit.wikimedia.org/r/75252
[21:40:57] <hashar>	 greg-g: well you can talk with ori about logging fatals in a db, maybe get analytics people involved to finally have a dashboard for event logging data
[21:41:14] <MaxSem>	 jgonera, can you review ^^ quickly, please?
[21:41:26] <jgonera>	 MaxSem, looking
[21:41:34] <hashar>	 MaxSem: you are fast :)
[21:41:58] <MaxSem>	 no I'm not - I grabbed a sandwich first
[21:42:02] <chrismcmahon>	 jcmish:  the Varnish error you see in the browser has a root cause.  If the root cause is from a certain extension it's a code problem  [22-Jul-2013 21:11:53] Fatal error: Call to a member function removeChild() on a non-object at /data/project/apache/common-local/php-master/extensions/MobileFrontend/includes/formatters/HtmlFormatter.php on line 256
[21:42:09] <hashar>	 MaxSem: a nicer fix would be to have parentNode a stub object but he… we are in php world
[21:42:14] <jcmish>	 ha MaxSem
[21:42:18] <jgonera>	 MaxSem, we can merge that, but I'm just wondering, why an element would have no parent node?
[21:42:27] <jcmish>	 k chrismcmahon thanks
[21:43:02] <MaxSem>	 because parent has already been removed?
[21:43:11] <chrismcmahon>	 jcmish: and the stack trace is in the log, and I think we finally have the logs in a fairly stable place
[21:43:22] <jcmish>	 alrighty that I can do
[21:43:32] <jcmish>	 now that I know where they are etc
[21:43:57] <jcmish>	 which extensions can affect other extensions?
[21:44:16] <jcmish>	 I know for example an issue with Central Auth can cause problems for lots of other extensions
[21:44:25] <jcmish>	 are there others that are that way?
[21:44:39] <jcmish>	 of course some core related but any others/
[21:44:39] <jcmish>	 ?
[21:45:11] <MaxSem>	 almost everything can cause stack traces belonging completely to other parts of code
[21:45:31] <jgonera>	 ooh, so removing a parent doesn't automatically remove all their children? that's kind of odd, eh HtmlFormatter is a mess ;)
[21:45:56] <grrrit-wm>	 (CR) JGonera: [C: 2] Bug 51840: removeChild() on a non-object in HtmlFormatter line 256 [extensions/MobileFrontend] - https://gerrit.wikimedia.org/r/75252 (owner: MaxSem)
[21:46:09] <MaxSem>	 jgonera, the problem is that we collect the nodes first, then remove them in batch
[21:46:16] <grrrit-wm>	 (Merged) jenkins-bot: Bug 51840: removeChild() on a non-object in HtmlFormatter line 256 [extensions/MobileFrontend] - https://gerrit.wikimedia.org/r/75252 (owner: MaxSem)
[21:46:28] <jgonera>	 yeah, I see
[21:49:51] <MaxSem>	 jcmish, so when do we chat?
[21:53:35] <hashar>	 MaxSem: http://en.m.wikipedia.beta.wmflabs.org/wiki/San_Francisco fixed apparently :)
[21:54:14] <jcmish>	 wooho hashar MaxSem tomorrow morning okay?
[21:54:18] <hashar>	 closed da bug. Thank you MaxSem
[21:54:21] <jcmish>	 what time do you usually get online?
[21:54:32] <hashar>	 in your timezone? I have no idea
[21:54:54] <hashar>	 I should be connected from 9am to 5pm, then I got a call and then I am leaving
[21:55:00] <hashar>	 (UTC+2)
[21:55:21] <jcmish>	 sorry hashar the woohoo was for you... the what time do we meet is for MaxSem
[21:55:40] <MaxSem>	 jcmish, ping each other when both are online?
[21:55:46] <jcmish>	 yup that works
[21:56:38] <hashar>	 I am off :) see you tomorrow
[21:56:50] <MaxSem>	 sweet dreams hashar
[21:56:55] <jcmish>	 take care hashar
[21:57:05] <hashar>	 thank you !
[22:19:37] * jgonera  still waiting for a package... probably will get to the office late just to grab my laptop.