[02:05:41] <wikibugs>	 10Analytics, 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, 10WMDE-Analytics-Engineering, and 5 others: How often are new editors involved in edit conflicts - https://phabricator.wikimedia.org/T182008#3809775 (10Milimetric) @GoranSMilovanovic you can query either wmf_raw.mediawiki_user or wmf.mediawiki_user...
[02:53:24] <wikibugs>	 (03PS7) 10Milimetric: [WIP] Saving in case laptop catches on fire [analytics/refinery] - 10https://gerrit.wikimedia.org/r/408848 (https://phabricator.wikimedia.org/T184759)
[03:09:58] <wikibugs>	 (03PS8) 10Milimetric: [WIP] Saving in case laptop catches on fire [analytics/refinery] - 10https://gerrit.wikimedia.org/r/408848 (https://phabricator.wikimedia.org/T184759)
[06:58:23] <elukey>	 bearloga: cleaned! 
[06:58:57] <bearloga>	 elukey: thank you very much!!!
[07:21:40] <elukey>	 people we are rebooting db1108 (analytics-slave.eqiad.wmnet) to update mariadb and the linux kernel
[07:21:56] <elukey>	 !log reboot db1108 (analytics-slave.eqiad.wmnet) for mariadb+kernel updates
[07:22:02] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[07:33:49] <elukey>	 we'll do db1107 (el master) in a couple of hours
[08:29:37] <wikibugs>	 10Analytics-Kanban, 10User-Elukey: Reduction of stat1005's disk space usage - https://phabricator.wikimedia.org/T186776#3988334 (10elukey) We are trying to back up everything to HDFS before deleting the /srv/stat1002-a dir, but it would be great to delete data if possible to avoid wasting space:  ``` elukey@st...
[08:45:53] <wikibugs>	 10Analytics-EventLogging, 10Analytics-Kanban, 10Beta-Cluster-Infrastructure, 10User-Elukey: EventLogging broken in beta - https://phabricator.wikimedia.org/T185952#3988342 (10Krinkle)
[08:56:53] <wikibugs>	 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey, 10User-Joe: rack/setup/install conf1004-conf1006 - https://phabricator.wikimedia.org/T166081#3988353 (10elukey) Before proceeding, https://phabricator.wikimedia.org/T187022 needs to be closed.
[09:06:12] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Data mismatch for number of Wikidata editors - https://phabricator.wikimedia.org/T187806#3988363 (10ChrisPins) If I am right, this points out to a major difference between WikiStats and WikiStats v2:  - All sums in the different filters/ splits of WikiStats v2 seem to includ...
[09:40:03] <wikibugs>	 10Analytics, 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, 10WMDE-Analytics-Engineering, and 5 others: How often are new editors involved in edit conflicts - https://phabricator.wikimedia.org/T182008#3988380 (10GoranSMilovanovic) @Milimetric Thank you, Dan. The JsonRefine is what is [[ https://lists.wikime...
[09:50:20] <elukey>	 so I am currently trying to copy ellery's stat1002-a user dir to /wmf/data/archive/backup/misc/stat1002-a/user_dirs_from_stat1002/ellery to free space on stat1005
[09:50:33] <elukey>	 that is a bit short on space
[09:50:46] <elukey>	 using hdfs dfs -copyFromLocal, but it takes ages 
[09:55:32] <elukey>	 files are big (except for some small .git etc.. related ones) so the hdfs block overhead shouldn't be that heavy
[10:18:11] <wikibugs>	 10Analytics-Kanban, 10User-Elukey: Reduction of stat1005's disk space usage - https://phabricator.wikimedia.org/T186776#3988461 (10elukey) To free some space, now ellery's stat1002 user dir is available in /mnt/hdfs/wmf/data/archive/backup/misc/stat1002-a/user_dirs_from_stat1002/ellery
[10:50:35] <elukey>	 all right stat1005's srv dir is ~91% used now, much better
[11:22:04] <addshore>	 Hi A Team!#
[11:22:20] <addshore>	 I have an EventLogging question...
[11:23:03] <addshore>	 .. Some EventLogging that we introduced for edit conflicts and our new conflict resolution tool tried to record the text submitted by the user at point of edit conflict, this text blob / string can be quite big and when too big it doesn't seem to make it into the system
[11:23:25] <addshore>	 it is easy to see why it doesnt land in the mysql tables as they are VARCHAR(1024) or something similar
[11:23:34] <addshore>	 but they also don't seem to land in hadoop :(
[11:23:39] <addshore>	 Any thoughts? :D
[11:24:02] <elukey>	 addshore: o/ 
[11:24:10] <elukey>	 what is the schema name?
[11:24:52] <addshore>	 *looks*
[11:24:56] <addshore>	 https://meta.wikimedia.org/wiki/Schema:TwoColConflictConflict
[11:25:04] <elukey>	 also, what do you mean that it doesn't land in hadoop? You don't see it in Hive?
[11:25:14] <icinga-wm>	 PROBLEM - HDFS active Namenode JVM Heap usage on analytics1001 is CRITICAL: CRITICAL - hadoop-hdfs-namenode-heap-usaage is 5914120765 https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?panelId=4&fullscreen&orgId=1
[11:25:30] <elukey>	 yeah nothing to worry about --^
[11:25:32] <addshore>	 well, the schema is there, but for records that I assume the text is long, those records are not in hadoop
[11:25:33] <elukey>	 alarms too sensitive
[11:26:25] <elukey>	 addshore: yep, but how do you verify it? I am asking to tracking down where is the issue.. we pull raw data from kafka via Camus and then a tool called JsonRefine puts it into Hive
[11:26:36] <elukey>	 *to track down
[11:26:41] <addshore>	 So, https://phabricator.wikimedia.org/T182008#3978750 is the comment that I first noticed this in
[11:27:01] <addshore>	 I was wondering if the message is too big for kafka?
[11:27:25] <addshore>	 So the only way to verify that data is missing, is we should have a record in https://meta.wikimedia.org/wiki/Schema:TwoColConflictConflict for every record in https://meta.wikimedia.org/wiki/Schema:EditConflict
[11:27:29] <addshore>	 but we do not :)
[11:28:03] <addshore>	 hadoop seems to contain roughly the same number of rows as mysql does
[11:28:30] <addshore>	 I was kind of thinking maybe the mysql insert fails first and then it never attempts to do the hadoop insert? but as I don't know the system i have no idea :D
[11:28:45] <addshore>	 I can file a ticket if needed :) Dashing to a meeting now
[11:30:02] <elukey>	 addshore: it is possible that eventlogging is not validating correctly the event and it discards it
[11:30:14] <icinga-wm>	 PROBLEM - HDFS active Namenode JVM Heap usage on analytics1001 is CRITICAL: CRITICAL - hadoop-hdfs-namenode-heap-usaage is 5937156214 https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?panelId=4&fullscreen&orgId=1
[11:31:05] <elukey>	 another one? weird
[11:32:45] <elukey>	 addshore: yeah I can see some logs in the eventlogging processors rejecting some events for that schema
[11:33:07] <elukey>	 (Invalid \uXXXX escape: line 1 column 1615 (char 1614))
[11:33:34] <elukey>	 not a huge amount though
[11:35:38] <elukey>	 now alarms
[11:38:14] <icinga-wm>	 PROBLEM - HDFS active Namenode JVM Heap usage on analytics1001 is CRITICAL: CRITICAL - hadoop-hdfs-namenode-heap-usaage is 5873131070 https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?panelId=4&fullscreen&orgId=1
[11:41:14] <icinga-wm>	 PROBLEM - HDFS active Namenode JVM Heap usage on analytics1001 is CRITICAL: CRITICAL - hadoop-hdfs-namenode-heap-usaage is 5873832638 https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?panelId=4&fullscreen&orgId=1
[11:41:34] <elukey>	 ufff
[11:42:53] <elukey>	 merging a fix
[11:43:15] <icinga-wm>	 PROBLEM - HDFS active Namenode JVM Heap usage on analytics1001 is CRITICAL: CRITICAL - hadoop-hdfs-namenode-heap-usaage is 5873832638 https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?panelId=4&fullscreen&orgId=1
[11:44:03] <elukey>	 weird that this alarm keeps sending alerts
[11:44:06] <elukey>	 it should send only one
[11:45:21] <elukey>	 all right icinga should behave now
[11:50:05] <elukey>	 mmm maybe not
[12:07:23] <icinga-wm>	 PROBLEM - HDFS active Namenode JVM Heap usage on analytics1001 is CRITICAL: CRITICAL - hadoop-hdfs-namenode-heap-usage is 6030265576 https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?panelId=4&fullscreen&orgId=1
[12:10:04] <elukey>	 is icinga still out of sync?
[12:10:40] <elukey>	 lemme disable notifications
[12:10:41] <elukey>	 mah
[12:30:25] <mforns>	 helloooo
[12:30:58] <mforns>	 joal, yt? I have a problem with scala DataFrames... :[
[12:58:00] <joal>	 mforns[m]: wednesday is kids day for me, will be on later tonight
[12:58:21] <joal>	 elukey: wanted to double check with you the alerts were fake - looks ok :)
[13:01:59] <elukey>	 sadly yes, sorry for the spam
[13:07:32] <addshore>	 elukey: (Invalid \uXXXX escape: line 1 column 1615 (char 1614)) interesting
[13:11:53] <addshore>	 so infact it is not an issue with length but instead an issue with this character?
[13:12:56] <elukey>	 addshore: this is what I've found, but it might be a red herring, how many events do you think you are loosing?
[13:13:12] <addshore>	 hundreds or thousands probably
[13:13:32] <addshore>	 I can probably go and dig up a pretty exact number later
[13:13:51] <addshore>	 are these logs in logstash?
[13:14:29] <elukey>	 not sure, never checked there!
[13:14:49] <addshore>	 I had a hunt around in logstash before but couldnt find anything, where do you find these magical logs? :D
[13:15:00] <elukey>	 on the eventlog1001's host :)
[13:15:16] <elukey>	 addshore: do you have a timeframe also about those events?
[13:16:24] <addshore>	 so, they will have been happening since roughly 1/2 was through January up until now
[13:21:28] <elukey>	 ah ok ok!
[13:49:13] <wikibugs>	 10Analytics, 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, 10WMDE-Analytics-Engineering, and 5 others: How often are new editors involved in edit conflicts - https://phabricator.wikimedia.org/T182008#3989072 (10Milimetric) yes, @GoranSMilovanovic, that's it.  But my main point was about the mediawiki table...
[14:18:42] <addshore>	 elukey: I wront a post it not for me to write a ticket
[14:21:41] <ottomata>	 o/
[14:22:46] <ottomata>	 yar elukey kafkatee
[14:22:56] <ottomata>	 what should we do?  i can take some time right now to try to make it multi instance in puppet
[14:23:04] <elukey>	 ottomata: morning :)
[14:23:05] <ottomata>	 we could run a manual kafkatee on oxygen that consumes from jumbo for a while
[14:23:08] <ottomata>	 hiiii
[14:23:18] <ottomata>	 until all topics are switched over
[14:23:19] <elukey>	 I didn't have a lot of time to check since hhvm didn't behave well today
[14:23:23] <ottomata>	 oo nasty
[14:23:28] <elukey>	 yeah :(
[14:23:45] <ottomata>	 well, tldr kakfatee puppet needs a refactor if we are going to make it work with 2 kafka clusters at the same time
[14:23:51] <ottomata>	 which, i started working on
[14:24:00] <ottomata>	 but will probably keep us from doing webrequest misc today :(
[14:26:53] * elukey nods
[14:27:03] <elukey>	 okok.. I saw the code review, it seems fine! 
[14:27:30] <elukey>	 ottomata: I mean https://gerrit.wikimedia.org/r/#/c/413056/
[14:28:21] <ottomata>	 yeah, that just gets rid of the submodule
[14:28:34] <ottomata>	 can't see any reason to keep it as a submodule, no one is ever going to use our kafkatee puppet module
[14:28:41] <ottomata>	 and iwanted to use base::service_unit
[14:28:58] <elukey>	 I would argue that all our modules should become part of operations/puppet :D
[14:29:07] <elukey>	 so let's start with kafkatee
[14:29:15] <moritzm>	 \o/
[14:29:19] <ottomata>	 shhhhh
[14:29:25] <ottomata>	 not cdh
[14:29:25] <elukey>	 I'd be in favor of the multi instance solution rather than the manual one
[14:29:27] <ottomata>	 people use that one!
[14:29:30] <ottomata>	 or at least they used to
[14:29:40] <elukey>	 well even all but cdh would be a win :D
[14:29:41] <ottomata>	 ok elukey on it...
[14:59:16] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban: Refactor kafkatee module to support multi instance - https://phabricator.wikimedia.org/T187890#3989250 (10Ottomata) p:05Triage>03Normal
[15:16:19] <mforns>	 ottomata, hiiiii
[15:16:43] <mforns>	 do you have 5 mins to look at an x-files scala/spark problem?
[15:20:29] <mforns>	 ooohhhhh... I think I got that..
[15:23:14] <wikibugs>	 10Analytics-Tech-community-metrics, 10Developer-Relations: For new authors on C_Gerrit_Demo, provide a way to access the list of Gerrit patches of each new author - https://phabricator.wikimedia.org/T187895#3989396 (10Aklapper) p:05Triage>03Low
[15:24:39] <wikibugs>	 10Analytics-Tech-community-metrics, 10Developer-Relations: For new authors on C_Gerrit_Demo, provide a way to access the list of Gerrit patches of each new author - https://phabricator.wikimedia.org/T187895#3989396 (10Aklapper)
[15:26:26] <ottomata>	 mforns:  hiiyaaa
[15:26:30] <mforns>	 heyyyy
[15:26:40] <mforns>	 ottomata, I found the thing
[15:27:02] <mforns>	 I was getting a very weird error when creating a dataframe with sqlContext.createDataFrame
[15:27:09] <mforns>	 I learned that
[15:27:42] <mforns>	 you can not reference a dataFrame inside the distributed code that processes its elements
[15:27:47] <mforns>	 i.e:
[15:28:12] <mforns>	 sqlContext.createDataFrame(
[15:28:13] <mforns>	             dataFrame.rdd.map { row =>
[15:28:13] <mforns>	                 sanitizeRow(dataFrame.schema, whitelist, row)
[15:28:13] <mforns>	             },
[15:28:13] <mforns>	             dataFrame.schema
[15:28:13] <mforns>	         )
[15:28:19] <mforns>	 this fails
[15:28:25] <ottomata>	 oh huhh interesting
[15:28:32] <ottomata>	 mforns:  i guess you can put the schema in a val?
[15:28:37] <mforns>	 yeeees
[15:28:51] <mforns>	 that was the solution... that took me like 2 hours to figure out
[15:28:53] <ottomata>	 aye
[15:28:59] <ottomata>	 yeah good oollll spark
[15:29:07] <ottomata>	 only joseph knows its secrets
[15:29:43] <mforns>	 makes sense, but the error message was just a nullPointerException
[15:30:12] <mforns>	 thanks anyway
[16:05:01] <milimetric>	 someone wanna brainbounce about this sqoop import
[16:05:04] <milimetric>	 I'm spinning in circles
[16:06:12] <milimetric>	 joal / mforns ^
[16:06:33] <mforns>	 milimetric, yep! bc?
[16:06:37] <milimetric>	 omw
[16:06:51] <mforns>	 ok, be there in 1
[16:17:16] <wikibugs>	 10Analytics-Data-Quality, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Please review: new WMDE public data set on stat1005 - https://phabricator.wikimedia.org/T187606#3989618 (10Lea_WMDE) @GoranSMilovanovic could you provide me with a direct link to that file? Thanks!
[16:40:40] <wikibugs>	 (03PS9) 10Milimetric: Refactor sqoop jobs and add from_timestamp [analytics/refinery] - 10https://gerrit.wikimedia.org/r/408848 (https://phabricator.wikimedia.org/T184759)
[17:02:11] <elukey>	 urandom: another clash with our standup, just realized, sorry :(
[17:03:16] <urandom>	 elukey: yeah; i noticed today my note from last time that we needed to find a new time
[17:03:59] <elukey>	 urandom: totally my fault, didn't follow up :(
[17:05:47] <wikibugs>	 (03PS2) 10Mforns: [WIP] Add EL and whitelist sanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/412939 (https://phabricator.wikimedia.org/T181064)
[17:22:52] <wikibugs>	 (03CR) 10Milimetric: "poke @joal, this is ready for review then" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/408848 (https://phabricator.wikimedia.org/T184759) (owner: 10Milimetric)
[17:45:09] <joal>	 ottomata: https://gist.github.com/jobar/83845555a9ad0c2dddb531fbbea15c84
[17:57:46] * elukey off!
[17:57:53] <ottomata>	 elukey: !
[17:57:56] <ottomata>	 kafkatee patch ok with you?
[17:58:18] <elukey>	 argh sorry forgot to finish it
[18:01:23] <elukey>	 ottomata: jumbo-analytics? is it right?
[18:02:56] <ottomata>	 oh
[18:02:58] <ottomata>	 that was for testing
[18:03:02] <ottomata>	 no not right good catch
[18:03:04] <ottomata>	 will change it back
[18:03:11] <ottomata>	 next patch after will set up a different instance
[18:04:17] <elukey>	 ottomata: the only thing that I found a bit odd is the rsylog/logrotate config in the module
[18:04:35] <elukey>	 but it might be ok, the rest looks super fine
[18:04:52] <elukey>	 so if you have tested it and it works, please go ahead :)
[18:07:47] <elukey>	 gtg now for a couple of hours, will check later on! sorry :(
[18:08:10] <ottomata>	 ok thanks!
[18:17:35] <wikibugs>	 10Analytics-Kanban, 10User-Elukey: Reduction of stat1005's disk space usage - https://phabricator.wikimedia.org/T186776#3990211 (10Halfak) I've dropped my usage to 44GB
[18:18:35] <wikibugs>	 10Analytics-Kanban, 10User-Elukey: Reduction of stat1005's disk space usage - https://phabricator.wikimedia.org/T186776#3990218 (10Halfak) Woops.  Caught something else I don't need.  Down to 12GB
[18:54:36] <joal>	 Heya milimetric - We got pinged on T187374
[18:54:36] <stashbot>	 T187374: Measure how many unique people visit the Timeline  - https://phabricator.wikimedia.org/T187374
[18:54:50] <joal>	 milimetric: Do you have the right to configure Piwik?
[18:55:16] <milimetric>	 yes joal I’ll do that
[18:55:22] <joal>	 Many thanks :)
[18:55:32] <joal>	 milimetric: --^ :)
[18:55:42] <joal>	 milimetric: another question for you
[18:55:49] <milimetric>	 shoot
[18:56:13] <joal>	 milimetric: we have quarterly goals deps on us - I had no clue about where we are standing on that - Do you have anything for me?
[18:56:28] <joal>	 The two things are: 
[18:56:29] <joal>	 Improve, adjust, or create features geared at the needs identified in New Editors research project.
[18:56:33] <joal>	  New Editors Experience:Analytics
[18:56:57] <joal>	 Side-talk to ottomata in the meantime
[18:57:09] <joal>	 ottomata: I think we're good to go for spark 2.2.1
[18:57:30] <ottomata>	 greaaat
[18:57:46] <joal>	 ottomata: every wmf table has been "infered and saved", and if people have issues I know what it is, so I'm happy for us to move
[18:57:55] <milimetric>	 joal: the team that was doing the New Editors project put it on hold I thought, but we met with them and the ball was in their court to determine what they needed help with
[18:57:59] <ottomata>	 awesome, thanks joal
[18:58:28] <joal>	 ottomata: I'll actually put a note on a wiki page on this thing :)
[18:59:20] <joal>	 milimetric: Ok - Thanks for the update :)
[19:09:40] <joal>	 ottomata: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Spark
[19:10:01] <joal>	 ottomata: I let you merge yout jsonrefine branch?
[19:14:24] <ottomata>	 joal k ya will do 
[19:14:26] <ottomata>	 want me to do now?
[19:14:43] <joal>	 ottomata: if you do it before tomorrow I'll work on it tomorrow morning :)
[19:14:53] <joal>	 No rush for today though
[19:14:54] <ottomata>	 ok will do
[19:14:55] <ottomata>	 great
[19:25:39] <wikibugs>	 10Analytics, 10InteractionTimeline, 10Anti-Harassment (AHT Sprint 15): Measure how many unique people visit the Timeline - https://phabricator.wikimedia.org/T187374#3973443 (10Milimetric) @dbarratt Nuria's on vacation this week, here's your piwik snippet, you're site 14!  :)  ``` <!-- Piwik --> <script type=...
[19:28:16] <wikibugs>	 10Analytics, 10InteractionTimeline, 10Anti-Harassment (AHT Sprint 15): Measure how many unique people visit the Timeline - https://phabricator.wikimedia.org/T187374#3990516 (10dbarratt) @Milimetric Thank you so much! :)
[19:46:54] <wikibugs>	 10Analytics-Data-Quality, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Please review: new WMDE public data set on stat1005 - https://phabricator.wikimedia.org/T187606#3990607 (10GoranSMilovanovic) @Lea_WMDE No. The file is now found in the designated directory on the **stat1005** server. I will se...
[19:48:13] <wikibugs>	 (03PS13) 10Joal: Upgrade scala to 2.11.7 and Spark to 2.2.1 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/348207
[19:48:27] <joal>	 Gone for tonight a-team - See you tomorrow
[20:04:02] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move webrequest varnishkafka and consumers to Kafka jumbo cluster. - https://phabricator.wikimedia.org/T185136#3990685 (10Ottomata)
[20:23:10] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move webrequest varnishkafka and consumers to Kafka jumbo cluster. - https://phabricator.wikimedia.org/T185136#3990746 (10Ottomata)
[20:24:47] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move webrequest varnishkafka and consumers to Kafka jumbo cluster. - https://phabricator.wikimedia.org/T185136#3907314 (10Ottomata)
[20:43:17] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move webrequest varnishkafka and consumers to Kafka jumbo cluster. - https://phabricator.wikimedia.org/T185136#3990810 (10Ottomata)
[20:46:29] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move webrequest varnishkafka and consumers to Kafka jumbo cluster. - https://phabricator.wikimedia.org/T185136#3990815 (10Ottomata)
[20:46:43] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move webrequest varnishkafka and consumers to Kafka jumbo cluster. - https://phabricator.wikimedia.org/T185136#3907314 (10Ottomata)
[20:48:41] <ottomata>	 !log now running 2 camus webrequest jobs, one consuming from jumbo (no data yet), the other from analytics.  these should be fine to run in parallel.
[20:48:42] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[20:52:09] <wikibugs>	 (03PS1) 10Ottomata: Merge branch 'jsonrefine' [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/413244
[20:52:58] <wikibugs>	 (03CR) 10Ottomata: "Has been reviewed by joal in jsonrefine branch." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/413244 (owner: 10Ottomata)
[20:55:13] <wikibugs>	 10Analytics, 10TCB-Team, 10Two-Column-Edit-Conflict-Merge, 10WMDE-Analytics-Engineering, and 5 others: How often are new editors involved in edit conflicts - https://phabricator.wikimedia.org/T182008#3990862 (10GoranSMilovanovic) @Milimetric Will do. I've already made some use of the `wmf.mediawiki_history...
[20:58:11] <wikibugs>	 (03CR) 10Ottomata: [C: 032] Merge branch 'jsonrefine' [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/413244 (owner: 10Ottomata)
[20:58:28] <ottomata>	 heya mforns FYI, I just merged ^, which moves a lot of the json refine job classes around
[20:58:37] <ottomata>	 you might need to carefully rebase your stuff
[20:58:46] <mforns>	 ottomata, right
[20:58:57] <mforns>	 thanks, will do!
[20:59:05] <ottomata>	 also, the change smight be relevant to what you are working on
[20:59:14] <ottomata>	 since it is now input agnostic
[20:59:31] <mforns>	 sure, I will look into the transformation function
[20:59:32] <mforns>	 s
[20:59:34] <ottomata>	 you might be able to use DataFrameToHive directly to do your transforming
[20:59:40] <ottomata>	 e.g.
[21:00:51] <ottomata>	 targets = RefineTarget.find('data/path/', ...)
[21:00:51] <ottomata>	 targets.foreach (DataFrameToHive(spark.read.parquet(target.path), tranformFunctions = Seq(purgeFn))
[21:02:04] <mforns>	 ottomata, I think this will be awesome
[21:02:24] <ottomata>	 :D lemme know if you have qs
[21:03:35] <mforns>	 ok
[21:04:54] <mforns>	  don't think it will change a lot the code, but it will remove the writing code from WhitelistSanitization
[21:06:27] <mforns>	 ottomata, do you think WhitelistSanitization should become a transformFunction that is passed to DataFrameToHive by EventLoggingSanitization?
[21:07:01] <mforns>	 or rather should it stay an Object that is called by EventLoggingSanitization and calls on its turn DataFrameToHive?
[21:09:52] <mforns>	 mmmm, I think it makes sense for WhitelistSanitization to stay a callable Object, because it contains some RefineTarget handling code and also exception handling code (outside of the transform function) that could be shared by other sanitization jobs in the future.
[21:15:00] <ottomata>	 looking
[21:16:42] <ottomata>	 mforns:  a transform 'function' in this case is an object with a particular apply method. hm
[21:16:54] <ottomata>	 i wonder, joal and i were talkinga bout making JsonRefine itself not json specific
[21:16:58] <ottomata>	 but configurable
[21:17:06] <ottomata>	 you tell it what the input type of your stuff will be
[21:17:18] <ottomata>	 (or maybe we could make it figure it out from the path? that would be fancy..)
[21:17:21] <ottomata>	 but, if we did that
[21:17:34] <ottomata>	 then i think most of the stuff in your apply function is just duplicated code from JsonRefine, no?
[21:17:42] <ottomata>	 the parts that find and parallelize the targets?
[21:18:02] <ottomata>	 sanitizeDataFrame() is the only real new thing?
[21:19:04] <ottomata>	 we wanted to wait for spark 2 to do the JsonRefine genericize stuff, since it requires more params
[21:19:16] <ottomata>	 and/or we should use properties instead of params, (there are a LOT of CLI params now)
[21:20:35] <mforns>	 ottomata, yes, the parts that find and parallelize the targets are one thing
[21:21:34] <mforns>	 but also the logic of choosing which tables are sanitized and which are purged completely (each table has different RefineTargets and ergo DataFrames to be transformed)
[21:22:56] <mforns>	 and also some exception handling, and marking RefineTargets as done/falied
[21:23:00] <mforns>	 *failed
[21:24:08] <mforns>	 but yea, this might go temporarily in EventLoggingSanitization, until spark2 and JsonRefine is made generic
[21:24:49] <ottomata>	 why does each table have different targets?
[21:25:26] <ottomata>	 ah empty
[21:25:27] <ottomata>	 i see
[21:25:46] <ottomata>	 mforns:  you pass the whitelist to sanitizeDataFrame() anyway
[21:25:50] <ottomata>	 you could do that logic there
[21:25:59] <ottomata>	 val sanitizedDf = sanitizeDataFrame(...)
[21:26:01] <ottomata>	 and in sanitizeDataFrame
[21:26:18] <ottomata>	 oh tableName
[21:26:18] <ottomata>	 hmm
[21:26:19] <ottomata>	 i see
[21:26:23] <ottomata>	 nice to not have t know about table name there
[21:26:24] <ottomata>	 for sure.
[21:26:24] <ottomata>	 hm
[21:29:23] <wikibugs>	 (03CR) 10Ottomata: [WIP] Add EL and whitelist sanitization (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/412939 (https://phabricator.wikimedia.org/T181064) (owner: 10Mforns)
[21:30:10] <mforns>	 I see your comment, makes sense
[21:30:46] <mforns>	 I assumed that, because for now it was like that, don't think it will take a huge refactor if we want to make it more generic in the future
[21:31:18] <mforns>	 but yea, if we can use generic JsonRefine for that, it would be great
[21:37:12] <wikibugs>	 10Analytics, 10InteractionTimeline, 10Anti-Harassment (AHT Sprint 15): Measure how many unique people visit the Timeline - https://phabricator.wikimedia.org/T187374#3973443 (10dbarratt)
[21:44:47] <ottomata>	 mforns:  i think i can make it automatically detect file type, at least for limited file types
[21:44:55] <ottomata>	 json, sequence file json, parquet
[21:45:18] <mforns>	 aha
[21:46:45] <ottomata>	 its pretty  hacky
[21:46:47] <ottomata>	 kinda like:
[21:46:58] <ottomata>	 read it as a text file
[21:47:08] <ottomata>	 if the first line starts with PAR, SEQ, or {
[21:47:09] <ottomata>	 do stuff
[21:47:10] <ottomata>	 :p
[21:47:17] <ottomata>	 but would work for all our use cases i think
[22:07:07] <mforns>	 logging off, bye teaaam
[22:14:50] <wikibugs>	 10Analytics, 10InteractionTimeline, 10Anti-Harassment (AHT Sprint 15): Measure how many unique people visit the Timeline - https://phabricator.wikimedia.org/T187374#3991147 (10dbarratt) https://github.com/wikimedia/InteractionTimeline/pull/47
[22:16:58] <wikibugs>	 (03PS1) 10Milimetric: [WIP] start work on geowiki computing [analytics/refinery] - 10https://gerrit.wikimedia.org/r/413265
[22:31:32] <wikibugs>	 (03PS1) 10Ottomata: RefineTarget can now infer input file format [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/413267
[22:32:59] <wikibugs>	 (03CR) 10Ottomata: "Haven't yet tested, but this should work! :o" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/413267 (owner: 10Ottomata)
[22:40:12] <wikibugs>	 (03PS1) 10Amire80: WIP Analyzing failed ULS searches [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/413270
[23:11:38] <wikibugs>	 10Analytics, 10InteractionTimeline, 10Anti-Harassment (AHT Sprint 15): Measure how many unique people visit the Timeline - https://phabricator.wikimedia.org/T187374#3991320 (10dbarratt) a:03dbarratt