[00:42:40] <grrrit-wm>	 (CR) BearND: "Some minor things I noticed (besides the missing unit tests) but since you marked is as WIP, you can ignore for now." (2 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274644 (https://phabricator.wikimedia.org/T128612) (owner: BearND)
[01:52:19] <wikibugs>	 Quarry: Quarry should preserve the protocol when logging in through OAuth - https://phabricator.wikimedia.org/T92600#2101766 (Huji) Invalid>Open
[02:25:32] <wikibugs>	 Quarry: Quarry should preserve the protocol on redirects - https://phabricator.wikimedia.org/T92600#2101895 (Huji)
[06:36:58] <grrrit-wm>	 (CR) Joal: [C: 2] "Self merging before deploying to include ASAP." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/273889 (owner: Joal)
[06:39:40] <grrrit-wm>	 (PS1) Joal: Re-update changelog.md before deploying v0.0.27 [analytics/refinery/source] - https://gerrit.wikimedia.org/r/276102
[06:40:49] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] "Self merging for deploy." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/276102 (owner: Joal)
[06:46:18] <joal>	 !log Deploying refinery-source v0.0.27 to archiva
[07:14:46] <elukey>	 hey joal!
[07:14:50] <elukey>	 o/
[07:15:00] <joal>	 Hi elukey :)
[09:41:26] <grrrit-wm>	 (PS1) Joal: Add refinery-source v0.0.27 to artifacts [analytics/refinery] - https://gerrit.wikimedia.org/r/276120
[09:46:43] <grrrit-wm>	 (PS2) Joal: Add refinery-source 0.0.27 and camus-wmf7 jars [analytics/refinery] - https://gerrit.wikimedia.org/r/276120
[09:48:09] <grrrit-wm>	 (PS1) Joal: Upgrade camus-wmf dependency to camus-wmf7 [analytics/refinery/source] - https://gerrit.wikimedia.org/r/276122
[09:51:06] <grrrit-wm>	 (PS3) Joal: Add refinery-source 0.0.27 and camus-wmf7 jars [analytics/refinery] - https://gerrit.wikimedia.org/r/276120
[09:51:44] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] "Self-merging for deploy" [analytics/refinery] - https://gerrit.wikimedia.org/r/276120 (owner: Joal)
[09:53:43] <grrrit-wm>	 (PS1) Joal: Correct symbolic link in download_jars script [analytics/refinery] - https://gerrit.wikimedia.org/r/276123
[09:54:00] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] "Self merging bug." [analytics/refinery] - https://gerrit.wikimedia.org/r/276123 (owner: Joal)
[10:04:59] <grrrit-wm>	 (PS1) Joal: Upgrade webrequest refine to use 0.0.27 jar [analytics/refinery] - https://gerrit.wikimedia.org/r/276126
[10:05:36] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] "Self merging for deploy" [analytics/refinery] - https://gerrit.wikimedia.org/r/276126 (owner: Joal)
[10:15:58] <joal>	 !log Killing oozie jobs before deploy and restart with new error email configuration
[10:25:58] <wikibugs>	 Analytics-EventLogging: Send graphite metrics for Schema.<action-value> as well - https://phabricator.wikimedia.org/T95780#2102428 (yuvipanda) Open>declined I don't see this ever happening.
[10:45:23] <joal>	 !log Deploying refinery from Tin
[10:48:01] <joal>	 !log deploying refinery on hdfs from stat1002
[10:55:08] <joal>	 !log restart oozie job with newly dpeloyed refnery
[11:04:10] <mforns>	 hellooo
[11:04:17] <joal>	 Hi mforns :)
[11:04:29] <mforns>	 hey joal!
[11:16:24] <elukey>	 hellooo!
[11:50:24] <joal>	 mforns: Do yuou remember when we were in SF two month ago, we talked about go being a game at which very strong human players were far better than computers?
[11:50:37] <mforns>	 joal, aha
[11:51:17] <joal>	 mforns: Well, that's not true anymore !
[11:51:35] <mforns>	 joal, wow, the found an algorithm?
[11:51:39] <mforns>	 *thay
[11:51:44] <mforns>	 *they
[11:52:12] <joal>	 This morning, Alphago has beaten Lee Sedol, currently 5th best player worldwide
[11:52:50] <joal>	 Alphago is developped by DeepMind (bought by Google two years ago)
[11:53:55] <joal>	 DeepMind first started by showing how new machine learning techning (deep learning to be precise, also named convolutional networks) could be applied to automatically play (and win) old school computer games
[11:55:31] <joal>	 https://www.youtube.com/watch?v=V1eYniJ0Rnk
[11:56:56] <joal>	 After the atari demo, they went quiet for while --> they were actually applying there machine learning skills to go
[11:57:00] <joal>	 And they managed :)
[11:57:43] <joal>	 The game with Lee Sedol is on 5 matches, so it's not yet finished, but the fact that alphago won the first match is already a huge breakthrough
[11:58:46] <joal>	 mforns: --^
[11:58:51] <joal>	 Sorry, it was long explanation :)
[11:59:29] <mforns>	 joal, interesting!
[12:00:35] <mforns>	 it reminds me of the documentary: Game over, Kasparov and the machine
[12:01:18] <joal>	 indeed mforns, we are currently leaving the next step of AI :)
[12:08:32] <joal>	 mforns: http://www.theverge.com/2016/3/9/11184362/google-alphago-go-deepmind-result
[12:12:04] <mforns>	 awesome
[12:12:35] <joal>	 sorry to bug mforns, but I'll be all about that today and possibly for a couple of days :D
[12:13:02] <elukey>	 joal: do you think that we could apply deep learning to oozie too? :D
[12:14:04] <joal>	 oozie WIN
[12:14:12] <joal>	 :D elukey
[12:15:22] <mforns>	 joal, hehehehe
[12:15:53] <mforns>	 elukey, oh! that is a good idea :]
[12:27:53] <elukey>	 today I had to use netcat and openssl to transfer data, each time those tools impress me
[12:28:33] <joal>	 :)
[12:52:06] <grrrit-wm>	 (PS1) Joal: Correct bug in mediacount load job [analytics/refinery] - https://gerrit.wikimedia.org/r/276154
[12:52:43] <grrrit-wm>	 (CR) Joal: "Live patched in current refinery, no need to deploy right now." [analytics/refinery] - https://gerrit.wikimedia.org/r/276154 (owner: Joal)
[13:35:13] <wikibugs>	 Analytics, Graph, Pageviews-API: Unable to get pageviews for the title with ' in the name - https://phabricator.wikimedia.org/T129346#2102915 (Yurik)
[14:32:55] <ottomata>	 GOOD morning!
[14:33:02] <ottomata>	 joal:  shoudl we set overwrite = true on all the jobs?
[14:33:58] <joal>	 ottomata: why not :)
[14:34:09] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Correct bug in mediacount load job [analytics/refinery] - https://gerrit.wikimedia.org/r/276154 (owner: Joal)
[14:34:11] <joal>	 ottomata: didn't of it since they have no errors, but why not :)
[14:34:17] <ottomata>	 ja i think we should :)
[14:34:24] <joal>	 ok, let me patch
[14:34:30] <ottomata>	 k, then i will merge both of those
[14:36:49] <joal>	 ottomata: done
[14:38:05] <ottomata>	 hmm, eh?
[14:38:26] <joal>	 oh yes, done, but not pushed ... sorry
[14:38:26] <ottomata>	 there it is!
[14:39:52] <ottomata>	 moritzm: shall we do the EL ferm thang?
[14:40:33] <ottomata>	 joal:  puppet run on an27
[14:40:52] <joal>	 ottomata: ?
[14:40:58] <joal>	 Oh, you are doing so :)
[14:41:01] <joal>	 ok
[14:41:32] <joal>	 monitoring
[14:41:32] <elukey>	 ottomata: managed to run vk without VUT only using the varnish API
[14:41:49] * elukey dances
[14:43:06] <moritzm>	 ottomata: ok, let me prepare the patch
[14:43:50] <ottomata>	 oh nice elukey
[14:44:05] <moritzm>	 wrt django: for a python lib it's in fact not 100% sure to tell, but I usually checked lsof etc. and didn
[14:44:11] <moritzm>	 t find anything
[14:44:21] <moritzm>	 we can just leave it as-is, it's fine
[14:45:11] <ottomata>	 ah ok
[14:45:12] <ottomata>	 lsof
[14:45:16] <ottomata>	 nah, let's get rid of it
[14:47:05] <ottomata>	 merged the django thang
[14:47:30] <moritzm>	 ok, thanjs
[14:54:52] <ottomata>	 moritzm:  this one, ja? https://gerrit.wikimedia.org/r/#/c/274715/
[14:54:57] <ottomata>	 shall I rebase and merge?
[14:55:01] <ottomata>	 i can just do it and watch
[14:56:18] <moritzm>	 sure, please go ahead and merge
[14:56:28] <moritzm>	 I'll add some logging rules to catch eventual omissions
[14:57:05] <ottomata>	 ok
[15:04:47] <ottomata>	 joal:  as far as I can tell, camus runs look good, eh?
[15:05:02] <joal>	 I think so yes
[15:05:18] <joal>	 I'll still keep my eyes on it for a few hours, but sounds allright
[15:07:21] <joal>	 ottomata: Was reviewing hive auxpath stuff: seems clean
[15:07:31] <joal>	 can you confirm?
[15:08:17] <joal>	 in /hieradata/eqiad/cdh/hive.yaml --> auxpath only contains hcatalog jar
[15:09:41] <joal>	 rah actually no ... My bad ottomata
[15:09:48] <joal>	 Will look for it agaim
[15:15:41] <wikibugs>	 Analytics-Cluster, Analytics-Kanban: Remove refinery-hive.jar from hive-site.xml - https://phabricator.wikimedia.org/T114769#2103219 (JAllemandou) a:Ottomata>JAllemandou
[15:32:21] <moritzm>	 ottomata: both followup patches merged, I've re-enabled ferm after a puppet and re-added the logging rules
[15:32:42] <moritzm>	 will keep you posted, but so far only pxe/bootp
[15:36:21] <ottomata>	 cool, danke
[15:40:20] <moritzm>	 ottomata: still pxe/bootp, we're good here. I'll merge a patch to remove the conditional so that 2001 is also covered
[15:43:33] <mforns>	 milimetric, yt?
[15:44:30] <ottomata>	 k sounds good moritzm
[15:44:32] <ottomata>	 danke
[15:49:07] <grrrit-wm>	 (CR) Ottomata: "Looking good!" [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618) (owner: BryanDavis)
[15:53:11] <milimetric>	 mforns: yeah, hi
[15:53:23] <mforns>	 hey milimetric :]
[15:53:52] <mforns>	 if you have time before standup to discuss mocks, lets batcave, if not maybe after goals?
[15:54:27] <milimetric>	 let's batcave now, sure, mforns
[15:54:36] <mforns>	 ok give me 1 min
[15:59:15] <joal>	 ottomata: we have ariel approval for dataset cron :)
[16:02:11] <ottomata>	 ok i merge!
[16:02:16] <grrrit-wm>	 (CR) Joal: "I'm bad on names ... Maybe "data_type" ?" [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618) (owner: BryanDavis)
[16:08:22] <ottomata>	 joal:  i'm going to do a manual rsync run just to check that this works
[16:09:32] <joal>	 ottomata: sure
[16:12:03] <ottomata>	 joal:  looks good!
[16:12:20] <ottomata>	 UH oh
[16:12:23] <ottomata>	 typo joal?
[16:12:27] <ottomata>	  /data/xmldatadumps/public/other/uniques_devices/
[16:12:31] <ottomata>	 instead of unique_devices?
[16:12:47] <joal>	 YAYAYYY
[16:12:54] <joal>	 SHiiiiiit
[16:13:08] <ottomata>	 sok gimme patch, i merge, and i fix files
[16:13:14] <ottomata>	 but, other than that looks goood
[16:13:16] <ottomata>	 http://dumps.wikimedia.org/other/uniques_devices/2016/2016-01/
[16:13:21] <joal>	 awesome :)
[16:13:25] <joal>	 Thanks for spotting !
[16:17:13] <joal>	 ottomata: patch sent !
[16:18:00] <milimetric>	 MarkTraceur: I noticed http://multimedia-metrics.wmflabs.org/graphs/mmv_actions_global stopped updating and I wonder if it's related to some puppet changes we made.
[16:19:02] <joal>	 ottomata: camus run confirmed ok
[16:25:50] <ottomata>	 joal cool, there we go http://dumps.wikimedia.org/other/unique_devices/2016/
[16:27:34] <joal>	 Wonderfull ottomata :)
[16:27:58] <joal>	 Man that has been a productive day :)
[16:30:03] <ottomata>	 :)
[16:30:07] <joal>	 ottomata: would you mind restarting oozie? You have merged the SLA config patch, but it's not taken into account from what I see
[16:30:15] <madhuvishy>	 Hi a-team. I won't be at standup today. I'll be at the goals meeting after
[16:30:27] <ottomata>	 hmmm, joal i restarted it yesterday evening
[16:30:29] <ottomata>	 my eve
[16:30:34] <joal>	 hmmmmmm
[16:30:37] <joal>	 mwarf
[16:30:47] <joal>	 no good then, still missing something
[16:31:06] <ottomata>	 joal:  also, the purgeservice hasn't reduced the size of the; mysql db
[16:31:08] <ottomata>	 so i dunno
[16:31:29] <joal>	 there must be something wrong in our settings
[16:31:36] <joal>	 Will investigate
[16:31:55] <ottomata>	 joal:  piinnng
[16:32:04] <ottomata>	 batcave haiiii
[16:34:16] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Create file dumps for last access that can be downloaded publically {mole} - https://phabricator.wikimedia.org/T126767#2103408 (Nuria)
[16:36:56] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Make last access data public {mole} - https://phabricator.wikimedia.org/T126767#2103435 (Nuria)
[16:57:42] <grrrit-wm>	 (CR) Milimetric: [WIP] Requests that come tagged with pageview=1 in x-analytics header are considered pageviews (3 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274644 (https://phabricator.wikimedia.org/T128612) (owner: BearND)
[16:57:43] <ori>	 ottomata: do you know if there is anything left to do for https://phabricator.wikimedia.org/T127209 ?
[16:59:59] <joal>	 a-team: meeting in batcave, right ?
[17:01:08] <milimetric>	 a-team: yeah, we should just always meet in the batcave, too much inertia to change
[17:05:04] <elukey>	 mmmm hangouts is acting weird
[17:05:34] <nuria>	 argh, ...
[17:07:11] <milimetric>	 right - same
[17:07:15] <milimetric>	 bluejeans?
[17:07:41] <milimetric>	 oh - we don't have rights to make a meeting - ew
[17:07:58] <kevinator>	 I can create a bluejeans meeting
[17:08:04] <milimetric>	 ooh :)
[17:08:09] <milimetric>	 a-team: bluejeans then?
[17:09:30] <madhuvishy>	 milimetric: nuria was just able to join - still not working for you?
[17:09:55] <kevinator>	 can you try again?  everybody is in hangouts now
[17:10:16] <nuria>	 milimetric: try again after signin into google gmail
[17:10:26] <nuria>	 milimetric: my login had expired
[17:10:49] <milimetric>	 (trying)
[18:16:51] <milimetric>	 mforns: before you take off, I wanna ask a favor from you for tomorrow morning
[18:17:06] <joal>	 a-team, I call it day :)
[18:17:13] <joal>	 See you tomorrow !
[18:17:17] <madhuvishy>	 nite joal :)
[18:17:20] <mforns>	 milimetric, sure!
[18:17:21] <milimetric>	 let me know if you think it's worth it to continue with making a ko.timeseriesDataObs()
[18:17:23] <mforns>	 bye joal !
[18:17:46] <milimetric>	 I'll start it after Scrum of Scrums today, but I'd like your opinion and we can catch up tomorrow morning
[18:18:11] <milimetric>	 ok, nite mforns, don't think about it now, just tomorrow when you start work again :)
[18:18:18] <mforns>	 milimetric, hehehe
[18:18:20] <wikibugs>	 Analytics: Terms of use for Wikimetrics needs updating - https://phabricator.wikimedia.org/T129383#2103935 (ZhouZ)
[18:18:21] <mforns>	 ok
[18:18:24] * milimetric flashes Men in Black memory eraser
[18:18:26] <mforns>	 will do
[18:18:32] <mforns>	 hehehe
[18:19:02] <mforns>	 good night yalls!
[18:19:36] <madhuvishy>	 ottomata: want to finish up step 3 and 4 of EL deploy?
[18:20:03] <ottomata>	 i need to make lunch  madhuvishy but then yes!
[18:20:20] <madhuvishy>	 ottomata: cool, ping me when you're back
[18:25:30] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, DBA, Patch-For-Review: Add autoincrement id to EventLogging MySQL tables. {oryx} - https://phabricator.wikimedia.org/T125135#2104011 (Nuria) Ping @jcrespo , let us know if now that edit table is trimmed we can proceed with this.
[18:25:54] <nuria>	 madhuvishy: ping me when you are starting and i will keep you guys company.
[18:26:31] <elukey>	 byyyeeeee!!! Talk with you tomorrow!!
[18:26:52] <nuria>	 madhuvishy: BTW, I think we are Ok with mobileschema issues and IPs, please see: https://phabricator.wikimedia.org/T127789
[18:26:54] <nuria>	 ciao elukey
[18:27:04] <grrrit-wm>	 (CR) BryanDavis: "I picked "channel" because that is what it would be called on the MediaWiki side. These events are pumped out by our debug logging layer o" [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618) (owner: BryanDavis)
[18:27:38] <madhuvishy>	 nuria: Or we could just give up connection-type geolocation and instead rely on per-country data, which means we could use the varnish geolocation and wouldn't have to store the raw IPs, just a hash for pseudo-UUID generation. We'd lose the ability to contrast connection types, but it sounds like it'd be a lot less effort and a lot cleaner for everyone.
[18:27:48] <madhuvishy>	 What is varnish geolocation?
[18:28:52] <bd808>	 ottomata, joal: the other name that might make sense on both sides is "topic"?
[18:29:04] <nuria>	 madhuvishy: I think there it refers on geolocation on our webrequest dataset
[18:29:56] <madhuvishy>	 nuria: webrequest?
[18:30:03] <nuria>	 madhuvishy: as we do not that in varnish
[18:30:31] <nuria>	 madhuvishy: we could though, varnish can do the ip-> country translation using maxmind just like we do in java
[18:30:41] <madhuvishy>	 nuria: I dont understand - the data is collected through EL right?
[18:31:12] <nuria>	 madhuvishy: right, that is where the confusion came in, i think devs thought we also had geo-location in EL
[18:31:34] <madhuvishy>	 i'm still confused
[18:32:58] <nuria>	 madhuvishy: i think the final idea is that we are not going to use that aschema
[18:33:03] <nuria>	 *that schema as is
[18:36:03] <madhuvishy>	 nuria: okay
[18:56:02] <leila>	 milimetric: batcave?
[18:56:13] <milimetric>	 sure, omw
[19:01:28] <ottomata>	 madhuvishy: oook
[19:01:48] <madhuvishy>	 ottomata: hi hi
[19:02:29] <madhuvishy>	 ottomata: hangout here? https://plus.google.com/hangouts/_/wikimedia.org/a-batcave1
[19:02:37] <madhuvishy>	 dan is using the other one
[19:02:40] <madhuvishy>	 nuria: ^
[19:02:56] <nuria>	 madhuvishy: omw
[19:17:18] <grrrit-wm>	 (CR) Ottomata: "Hmm, ok then. I'm fine with channel. Can you add some comments explaining?" [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618) (owner: BryanDavis)
[19:20:46] <nuria>	 !log deploying new capsule version to eventlogging
[19:21:21] <nuria>	 !log removing IP encryption from eventlogging. Service restart.
[19:21:38] <grrrit-wm>	 (CR) EBernhardson: "after playing around with the avro-tools some more, i'm not sure if this is necessary. We can transform the idl into a schema directly, wi" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274307 (https://phabricator.wikimedia.org/T128530) (owner: EBernhardson)
[19:24:09] <legoktm>	 hello
[19:24:19] <legoktm>	 is there a list of wikis for which the pageview API is set up for?
[19:24:25] <legoktm>	 I noticed it doesn't work for test.wikipedia.org
[19:25:03] <nuria>	 legoktm: Is 'setup" for what are consider "knowledge" wikis, see:
[19:25:03] <legoktm>	 milimetric: ^
[19:25:17] <madhuvishy>	 legoktm: we have a whitelist
[19:25:23] <madhuvishy>	 let me get it for you
[19:25:27] <nuria>	 legoktm: https://github.com/wikimedia/analytics-refinery/blob/master/static_data/pageview/whitelist/whitelist.tsv
[19:25:44] <nuria>	 legoktm: those are wikis counted on "pageview" definition
[19:26:02] <icinga-wm>	 PROBLEM - Check status of defined EventLogging jobs on eventlog1001 is CRITICAL: CRITICAL: Stopped EventLogging jobs: processor/server-side-0 processor/client-side-11 processor/client-side-10 processor/client-side-09 processor/client-side-08 processor/client-side-07 processor/client-side-06 processor/client-side-05 processor/client-side-04 processor/client-side-03 processor/client-side-02 processor/client-side-01 processor/client-side-
[19:26:25] <legoktm>	 hmm, how hard would it be to add one of the test wikis?
[19:26:41] <madhuvishy>	 legoktm: they have to be part of the pageview definition
[19:26:53] <legoktm>	 I'm asking because I'm figuring out how we're going to deploy the WikimediaPageViewInfo extension (https://phabricator.wikimedia.org/T125917), and I'd like to turn it on a testwiki first
[19:27:28] <madhuvishy>	 ottomata: lost you
[19:27:36] <madhuvishy>	 legoktm: aah
[19:28:43] <legoktm>	 I can point it to en.wp.o for now or something, but I suspect other people will also want to use the API with test.wp in the future :)
[19:29:33] <icinga-wm>	 RECOVERY - Check status of defined EventLogging jobs on eventlog1001 is OK: OK: All defined EventLogging jobs are runnning.
[19:29:35] <legoktm>	 Also, is there a beta cluster version of the pageview API? I'm going to guess not...
[19:30:58] <nuria>	 !log restarted eventlogging after puppet fix: https://gerrit.wikimedia.org/r/#/c/276244/
[19:31:36] <nuria>	 legoktm: teh pageview APi holds waht are "knowledge" pageviews, we have plans to measure other requests to other wikis
[19:31:41] <madhuvishy>	 legoktm: no - we have a beta cluster version of the analytics cluster - but too much of a pain to maintain so far and the instances have been dying. so its even harder to get the full pipeline with cassandra setup
[19:31:46] <nuria>	 legoktm: but they are out of the api intentionally
[19:31:48] <ottomata>	 madhuvishy:  let's check back in a few days to really compare, but it look slike load average is down since no longer hashing ips
[19:31:51] <ottomata>	 https://grafana.wikimedia.org/dashboard/db/server-board?panelId=12&fullscreen
[19:31:55] <madhuvishy>	 nuria: but he does want to use something to just test
[19:32:03] <madhuvishy>	 ottomata: oh yay
[19:33:23] <nuria>	 madhuvishy: for which we should be able to point to a test version of pageview api... that ahem... we do not have
[19:33:29] <madhuvishy>	 nuria: yeah
[19:33:44] <nuria>	 madhuvishy: as prod is prod and testing is testing
[19:34:02] <madhuvishy>	 legoktm: may be you can use a project that is a little less used so less risky, but already on the definition
[19:34:28] * legoktm nods
[19:34:33] <legoktm>	 I can use the standard aa.wikibooks :P
[19:34:42] <nuria>	 legoktm: also given that this is a total test can't you "fake" results of http requests using a /etc/hosts kind of hack
[19:34:50] <madhuvishy>	 i know it's not ideal - but even though technically we can add test.wikipedia.org or something to the whitelist - i doubt doing something not confirming to the definition on prod is a good idea
[19:35:23] <legoktm>	 nuria: probably, but that would require making changes outside of the MediaWiki layer, which requires ops, etc.
[19:35:31] <nuria>	 madhuvishy: ya, we will not be doing that, that says: "we need a test endpoint with test data"
[19:35:38] <nuria>	 legoktm: i see, not easy
[19:37:03] <icinga-wm>	 PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [30.0]
[19:37:16] <nuria>	 ay ay
[19:38:14] <madhuvishy>	 ahem
[19:38:25] <nuria>	 madhuvishy: righttt, let's look at logs again
[19:38:43] <madhuvishy>	 nuria: i still see insertions etc
[19:39:25] <nuria>	 madhuvishy: ya, i do not see crazy validation errors
[19:39:36] <madhuvishy>	 ottomata: raw valid difference rate seems to be climbing
[19:39:58] <nuria>	 legoktm: let us knwo of your results with aa.wikibooks
[19:41:26] <legoktm>	 will do :)
[19:41:36] <nuria>	 madhuvishy: I would expect to see validation errors in: /srv/log/upstart#
[19:41:43] <nuria>	 but i do not see them there
[19:41:51] <nuria>	 or i just  see a few
[19:42:08] <madhuvishy>	 yeah
[19:42:09] <madhuvishy>	 me too
[19:42:27] <madhuvishy>	 nuria: it looks like the rate is falling
[19:42:37] <madhuvishy>	 may be unrelated to our deploy?
[19:43:37] <nuria>	 I think so
[19:43:52] <nuria>	 cause our "outage' emptied valid-mixed
[19:43:54] <nuria>	 cc madhuvishy
[19:44:04] <nuria>	 but not raw
[19:44:26] <madhuvishy>	 well no outage really
[19:44:41] <nuria>	 madhuvishy: our downtime while we had no processor
[19:44:46] <madhuvishy>	 right
[19:44:53] <madhuvishy>	 aah
[19:45:00] <nuria>	 so nothing is flowing into teh "valid" topic
[19:45:03] <nuria>	 makes sense?
[19:45:08] <madhuvishy>	 so you're saying all the invalid events showed up at the same time?
[19:45:14] <nuria>	 but stuff is flowing into raw
[19:45:23] <madhuvishy>	 causing temporary burst in the difference
[19:45:29] <nuria>	 no, I am saying that raw had events while valid was empty
[19:46:01] <nuria>	 as the processor was down and was not putting events into valid.. makes sense?
[19:46:06] <nuria>	 cc madhuvishy
[19:46:17] <madhuvishy>	 yup yup
[19:46:20] <madhuvishy>	 got it
[19:47:09] <nuria>	 if that is the case difference should be disappearing as events are processed, that explains the lack of logs too
[19:47:21] <nuria>	 as events are not invalid, rather no valid events are there
[19:47:27] <nuria>	 on the topic
[19:48:06] <nuria>	 madhuvishy: ya, that is what is going on
[19:48:21] <nuria>	 madhuvishy: line is going down
[19:48:24] <madhuvishy>	 yup
[19:48:38] <madhuvishy>	 back all the way down now
[19:49:19] <nuria>	 madhuvishy: we are good cc ottomata in case he sees ping from icinga in a bit
[19:49:52] <leila>	 madhuvishy: congratulations on dropping the hashed IP thingy. :D
[19:50:01] <madhuvishy>	 leila: thank you :D
[19:50:10] <leila>	 madhuvishy: I'm happy that you waited for our work to finish, this thing got done so quickly. :D
[19:50:53] <madhuvishy>	 leila: :D yeah! way better than the optional ip stuff i was working on
[19:51:43] <leila>	 haha
[19:52:00] <leila>	 madhuvishy: I have a funny news. I'll be in Berlin the same time y'all have an offsite.
[19:52:06] <madhuvishy>	 leila: yayyy
[19:52:12] <leila>	 I haven't told Lydia yet, but I will. :D
[19:52:24] <leila>	 if they let me in WMDE office, that'd be great. :-)
[19:52:30] <madhuvishy>	 nice! we should hang out! and go together
[19:52:47] <leila>	 I go earlier, mid-May since there is a conference in Cologne I should attend.
[19:52:53] <madhuvishy>	 ahhh
[19:53:00] <madhuvishy>	 and you'll come to wikimania after?
[19:53:04] <leila>	 I tried to convince milimetric that y'all come to Wikimania, it's the week after your offsite.
[19:53:13] <leila>	 yeah
[19:53:13] <madhuvishy>	 leila: me and milimetric are coming
[19:53:18] <leila>	 ooooooo, nice!
[19:53:37] <leila>	 definitely talk to Ellie, madhuvishy. I think there are few spots left. milimetric doesn't listen to me and he wants to pay himself. :D
[19:53:41] <milimetric>	 road trip! :)
[19:53:51] <madhuvishy>	 leila: we are both marked approved on the form
[19:53:55] <madhuvishy>	 milimetric: ^
[19:53:56] <leila>	 hahaha!
[19:53:57] <leila>	 :D
[19:53:57] <milimetric>	 we are?!
[19:54:00] <madhuvishy>	 yes
[19:54:03] <madhuvishy>	 nuria approved us
[19:54:06] <leila>	 c'mon, milimetric! :D
[19:54:09] <nuria>	 leila: do you have any suggestions of places for us to stay?
[19:54:16] <leila>	 where, in Berlin nuria?
[19:54:19] <milimetric>	 haha, that's news to me, leila
[19:54:26] <milimetric>	 I thought we left it at - nobody's gonna go
[19:54:37] <leila>	 milimetric: you really deserve a big hug. ;-)
[19:54:38] <milimetric>	 but yay
[19:54:39] <milimetric>	 :)
[19:54:43] <madhuvishy>	 :)
[19:54:49] <madhuvishy>	 leila: yeah in Berlin
[19:54:57] <leila>	 milimetric: which team are you in?! ;p
[19:55:11] <leila>	 madhuvishy, nuria: I'm doing research. I can let you know.
[19:55:25] <leila>	 I was hoping I can rent a room in Ellery's parents place. :D
[19:55:32] <leila>	 but I haven't talked to him about it yet.
[19:55:43] <nuria>	 ok, i saw couple places on berlin wmf events thus far but neither is appealing
[19:55:55] <leila>	 okay, I'm going to start an email with Lydia. nuria, madhuvishy: when are y'all arriving and how long you need the place? for how many?
[19:56:07] <nuria>	 I will cc you on thread we alreday have
[19:56:13] <nuria>	 with DE events@ group
[19:56:14] <leila>	 sounds good nuria.
[19:56:43] <madhuvishy>	 leila: week of june 13 is offsite. I might come on 11th though since its the weekend
[19:57:01] <leila>	 okay, let's say June 11-20
[19:58:13] <icinga-wm>	 RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0]
[20:14:50] * milimetric lunching
[20:20:57] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Ensure file.encoding is UTF-8 for all JVMs in the Analytics Cluster - https://phabricator.wikimedia.org/T128607#2104455 (Ottomata) Cool!  Now on  nodemanager restart:  ``` Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 ```
[20:24:41] <grrrit-wm>	 (PS3) Nuria: Requests that come tagged with pageview=1 in x-analytics header are considered pageviews [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274644 (https://phabricator.wikimedia.org/T128612) (owner: BearND)
[20:25:42] <grrrit-wm>	 (PS4) Nuria: Requests that come tagged with pageview=1 in x-analytics header are considered pageviews [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274644 (https://phabricator.wikimedia.org/T128612) (owner: BearND)
[20:26:34] <grrrit-wm>	 (CR) Nuria: "Please see patch #4. This is ready to be merged." (3 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/274644 (https://phabricator.wikimedia.org/T128612) (owner: BearND)
[20:33:55] <wikibugs>	 Analytics-Kanban: Eventlogging udp cleanup - https://phabricator.wikimedia.org/T129402#2104530 (Nuria)
[20:36:24] <wikibugs>	 Analytics, Analytics-EventLogging: Send raw server side events to Kafka using a PHP Kafka Client {oryx} - https://phabricator.wikimedia.org/T106257#2104543 (Nuria) Open>Resolved
[20:45:14] <grrrit-wm>	 (PS8) BryanDavis: Add initial oozie job for ApiAction [analytics/refinery] - https://gerrit.wikimedia.org/r/273557 (https://phabricator.wikimedia.org/T108618)
[20:49:31] <ottomata>	 !log restarted oozie server
[21:04:01] <wikibugs>	 Analytics-Kanban: Eventlogging udp cleanup - https://phabricator.wikimedia.org/T129402#2104653 (Nuria) p:Triage>Normal
[21:05:51] <wikibugs>	 Analytics-Kanban: Eventlogging udp cleanup - https://phabricator.wikimedia.org/T129402#2104530 (Nuria) Does perf team have any udp code we need to revamp/remove?
[21:12:18] <wikibugs>	 Analytics-Kanban: Eventlogging udp cleanup - https://phabricator.wikimedia.org/T129402#2104530 (Ottomata) No, they  use the ZMQ stream emitted by a forwarder
[22:25:56] <grrrit-wm>	 (PS1) Catrope: Add more wikis to the cross-wiki beta feature report [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/276356
[23:04:47] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Create reportupdater browser reports that query hive's browser_general table {lama} - https://phabricator.wikimedia.org/T127326#2105089 (Milimetric) Resolved>Open This is still not done, just the line endings part was done.  It's the task I was looking for this mo...
[23:04:53] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Create reportupdater browser reports that query hive's browser_general table {lama} - https://phabricator.wikimedia.org/T127326#2105091 (Milimetric) In general we shouldn't mark things done if they're not in the Done column.
[23:12:26] <wikibugs>	 Analytics, Analytics-Cluster, Deployment-Systems, scap, Scap3 (Scap3-Adoption-Phase1): Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2105139 (dduvall)
[23:29:24] <wikibugs>	 Analytics-Wikimetrics, Community-Wikimetrics: Uploading cohort or running a large report fails - https://phabricator.wikimedia.org/T87596#2105189 (NickK) stalled>Open I confirm that this behaviour still occurs. While attempting to work with a very large cohort I get the following errors:  1) https:...
[23:35:44] <wikibugs>	 Analytics-Engineering, Analytics-Wikimetrics: Usernames with commas not supported - https://phabricator.wikimedia.org/T129422#2105197 (NickK)