[10:48:21] <grrrit-wm>	 (PS10) Nuria: Add ability to tag a cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/133091 (owner: Terrrydactyl)
[13:27:27] <grrrit-wm>	 (PS11) Nuria: Add ability to tag a cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/133091 (owner: Terrrydactyl)
[13:30:26] <grrrit-wm>	 (CR) Nuria: "Please see patch #11 that fixes a number of issues. Mostly to do with encoding." (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/133091 (owner: Terrrydactyl)
[14:00:12] <grrrit-wm>	 (PS12) Nuria: Add ability to tag a cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/133091 (owner: Terrrydactyl)
[14:59:57] <qchris>	 nuria: You were mentionining EventLogging issues and where to report incidents.
[15:00:04] <qchris>	 https://wikitech.wikimedia.org/wiki/Incident_documentation
[15:00:09] <qchris>	 ^ looks like a good place to me.
[15:00:18] <nuria>	 but aren't those tier-1 services reports for availability?
[15:00:29] <qchris>	 We already have an EventLogging entry there.
[15:00:30] <nuria>	 this is more like issues with data
[15:00:48] <qchris>	 https://wikitech.wikimedia.org/wiki/Incident_documentation/20140318-EventLogging
[15:00:53] <nuria>	 for availability
[15:00:56] <nuria>	 not data
[15:01:40] <qchris>	 Let's ask greg.
[15:01:50] <qchris>	 Oh ... there is greg-g.
[15:02:18] <qchris>	 greg-g Should EventLogging issues generally go into https://wikitech.wikimedia.org/wiki/Incident_documentation
[15:02:19] <nuria>	 but i have no strong opinion really, we are just going to see a bunch of 'data' related reports next to availability reports
[15:02:59] <qchris>	 or should issues around data (like EventLogging only getting parts of the data for some hours) go to a separate place.
[15:25:45] <grrrit-wm>	 (PS13) Milimetric: Add ability to tag a cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/133091 (owner: Terrrydactyl)
[15:29:32] <greg-g>	 qchris: ideally yes
[15:30:04] <greg-g>	 (I'm everywhere!)
[15:30:22] <qchris>	 greg-g: Yes, we file it as incident report ... or Yes, they should go to a different place
[15:30:38] <qchris>	 (Sorry my question was stupid)
[15:30:40] <greg-g>	 oh, sorry, yes, incident report
[15:30:45] <qchris>	 Ok. Thanks.
[15:30:49] <greg-g>	 I missed the second line of your question :)
[15:31:30] <qchris>	 nuria: Are you reading along?
[15:34:31] <greg-g>	 My goal is that the incident reports are just mostly a free side product of what you already do in response to an outage, and then, once a quarter (reminder to self, I need to set that up for this quarter) I go through them, look at action items, set up a meeting with a few people if needed (if eg there were patterns we should address).
[15:34:41] <greg-g>	 so, what you already do plus me helping
[15:34:44] <greg-g>	 ideally.
[15:35:26] <qchris>	 Great.
[15:35:36] <qchris>	 Thanks for being awesome!
[15:35:41] <greg-g>	 I try :)
[15:47:35] <grrrit-wm>	 (Abandoned) Milimetric: Address parts of Bug 63680 [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/125102 (owner: QChris)
[15:47:48] <nuria>	 just read that thnks greg-g
[16:43:04] <ori>	 qchris, nuria: hey
[16:43:09] <ori>	 i'm still catching up with the backlog
[16:43:12] <qchris>	 Hi ori.
[16:43:15] <ori>	 it sounds like you guys caught and fixed a major bug
[16:43:40] <qchris>	 Not sure if the fix is a fix though :-)
[16:43:46] <qchris>	 And catching is due to Ironholds
[16:44:33] <Ironholds>	 I think the diagnosis was a CN bug rather than an EL bug, if I recall correctly?
[16:44:36] <ori>	 my son had a bellyache all night so i barely slept so it might take me an hour or two to respond coherently, still not fully up
[16:44:43] <Ironholds>	 :(. Hope he feels better!
[16:44:47] <Ironholds>	 (and that you do, too)
[16:44:52] <qchris>	 Sad to hear that :-(
[16:45:02] <ori>	 thanks. just a bellyache, no biggie :)
[16:45:20] <qchris>	 I get all crankie with a bellyache :-)
[16:45:41] <Ironholds>	 yeah, but with small children every illness is confusing. "This has never happened to me before and I don't know how to deal with it! Halp!"
[16:48:51] <ori>	 yep
[17:06:02] <nuria>	 all merit of fixing goes to qchris
[18:39:42] <milimetric>	 ottomata: or whoever is puppet-smart around here
[18:39:45] <milimetric>	 how do we take this:
[18:39:46] <milimetric>	     cron { "rsync_mobile_apps_stats":
[18:39:46] <milimetric>	         command => "python $command $config && /usr/bin/rsync -rt $rsync_from/* stat1001.wikimedia.org::www/limn-public-data/",
[18:39:46] <milimetric>	         user    => $user,
[18:39:46] <milimetric>	         minute  => 0,
[18:39:47] <milimetric>	     }
[18:39:55] <milimetric>	 and make it so it outputs its errors somewhere reasonable
[18:40:20] <milimetric>	 like /var/log/limn-mobile-data.error.log
[18:40:45] <milimetric>	 or just all output to /var/log/limn-mobile-data.log
[18:41:08] <ottomata>	 that's tougher because you ahve two commands there
[18:41:18] <ottomata>	 you have to redirect the output of both
[18:41:44] <ottomata>	 "python $command $config >> /path/to/file 2>&1 && /usr/bin/rsync -rt $rsync_from/* stat1001.wikimedia.org::www/limn-public-data/ >> /ppath/to/file 2>&1
[18:42:30] <milimetric>	 awesome
[18:46:32] <milimetric>	 ottomata: if you don't mind taking a look at https://gerrit.wikimedia.org/r/138884
[18:46:37] <milimetric>	 would be cool
[18:46:39] <milimetric>	 ok, gtg
[19:53:51] <qchris>	 mwalker: I see you scheduled to deploy CentralNotice's GeoIP fixes to today. Thanks!
[19:54:12] <qchris>	 Since I am totally new to SWAT ... is there anything we have to do?
[19:56:01] <mwalker>	 no; I'll need someone to verify that the bug is fixed for things that are not centralnotice
[19:56:07] <mwalker>	 e.g. eventlogging
[19:59:04] <qchris>	 Ok. I'll be around for that.
[20:40:30] <grrrit-wm>	 (PS14) Nuria: Add ability to tag a cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/133091 (owner: Terrrydactyl)
[20:57:36] <Ironholds>	 okay, these reducers have died four times in a single query. I give up.
[20:57:39] <Ironholds>	 sampled logs it is!
[21:56:34] <awight>	 Ironholds: u know if DarTar is coming back?  I want to bug him.
[21:58:45] <Ironholds>	 awight, I do not, I'm afraid
[22:59:25] <qchris>	 ori: About cleaning up the EventLogging country columns, do you think it's worth unfiddeling the value we have in the country columns and try to extract values from there?
[22:59:42] <qchris>	 Or is it fine to set everything to NULL that does not have LENGTH($COLUMN) = 2
[23:02:48] <qchris>	 Since the affected columns typically have two 'GeoIP=' parts, I do not trust any value we see there and would just set the affected country values to NULL.
[23:20:23] <Ironholds>	 qchris, !ori, but I'd go for setting to NULL
[23:20:40] <Ironholds>	 we're currently getting a pretty high number of values out of the successful geolocations, so missing that data shouldn't hit anything.
[23:20:48] <qchris>	 k.
[23:24:21] <ori>	 qchris, Ironholds: sorry, I missed the ping earlier
[23:24:27] <ori>	 I'd set it to NULL
[23:24:56] <qchris>	 So three votes for NULL. That's sufficient :-D
[23:25:03] <qchris>	 Thanks ori and Ironholds.
[23:25:07] <Ironholds>	 np :)