[07:30:17] morning elukey :) [07:32:14] morning :) [07:32:55] elukey: I have seen the message about the deploy being fixed, may I try? [07:36:51] yep it should work [07:37:43] ok, will try in minutes [07:57:31] !log Deploying refinery with scap - 2nd try [07:57:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:06:24] joal: all good? [08:22:02] Yes elukey - so far all good (canary done, deploying to others) [08:22:50] good :) [08:24:09] elukey: I'm not feeling super well today - I think I'm gonna deploy, restart needed jobs etc and go to bed [08:25:38] ah snap ok :( [08:25:54] elukey: anything you'd have like me to help? [08:27:06] elukey: deploy successful :) [08:27:14] Thanks for the fix :) [08:27:49] nono all good from my side [08:31:25] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) [08:31:28] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add interface::add_ip6_mapped { 'main': } to all the Analytics hosts - https://phabricator.wikimedia.org/T199180 (10elukey) 05Open>03Resolved [08:40:09] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) @ayounsi I am running tcpdump on stat1005 with ' ip6 and src 2620:0:861:108:10:64:53:30` on stat1005 but I see only traffic to... [09:37:41] just created https://wikitech.wikimedia.org/wiki/Analytics/Data_access#User_responsibilities [09:43:41] 10Analytics, 10Operations, 10Documentation: Remove data from Hadoop's HDFS as part of the user offboard workflow - https://phabricator.wikimedia.org/T200312 (10elukey) [11:46:16] * elukey lunch + errand! [13:43:50] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) nevermind, I saw it, it doesn't happen very often though. Will try to figure out its origin. [13:52:33] morning yall [13:55:19] hello my dear friend [13:55:29] how's your jetlag doing so far? [13:59:44] !log Start wikidata-coeditors job [13:59:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:03:21] !log Restart webrequest-bundle load job to pick new pageview definition [14:03:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:07:06] heh [14:14:04] Hi a-team, oozie alerts are me restarting some jobs after this morning deploy [14:14:22] roger [14:14:25] gotcha [14:21:51] ack [14:30:10] cool beans [14:37:45] qq - how can I read avro files via spark? [14:37:50] (spark2-shell I mean) [14:37:58] super newbie I know [14:38:19] but I was able to do some experiments wit parquet (spark.read.parquet.etc..) [14:38:25] and some SequenceFiles [14:38:43] for avro I tried to import the databricks module but it is not there of course [14:38:53] elukey: you need to "import com.databricks.spark.avro._" [14:39:26] joal: I tried but I get error: object databricks is not a member of package com :( [14:39:57] Ah elukey - start your shell with the refinery-job jar in the path: spark2-shell --master yarn --jars /srv/deployment/anlytics/refinery/artifacts/refinery-job.jar [14:40:25] ah there you go, the databricks jar is there [14:40:26] lovely [14:40:36] elukey: then import, then "val df = spark.read.avro(path)" [14:41:01] joal: I ask for patience and also mercy beforehand :D [14:41:09] I'll return the favor in beers [14:41:23] elukey: we need the avro module in mediawiki-history, so the databricks-avro-jars is bundled with refinery-job :) [14:42:41] elukey: I don't need beers as repayment, I like teaching :) I however enjoy having beers wih you folks, so it'll happen nontheless ;) [14:44:32] yeah sure sure [14:44:56] :) [14:58:18] (03PS3) 10Mforns: [WIP] Add ability to salt and hash to eventlogging sanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/446592 (https://phabricator.wikimedia.org/T198426) [15:04:18] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add ability to salt and hash to eventlogging sanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/446592 (https://phabricator.wikimedia.org/T198426) (owner: 10Mforns) [15:26:30] 10Analytics: Data governance for topics - https://phabricator.wikimedia.org/T200440 (10fdans) [15:26:45] 10Analytics: Data governance for topics - https://phabricator.wikimedia.org/T200440 (10fdans) [15:26:49] 10Analytics, 10Discovery-Search (Current work), 10Patch-For-Review: Create kafka topic for mjolinr bulk daemon and decide on cluster - https://phabricator.wikimedia.org/T200215 (10fdans) [15:30:30] 10Analytics, 10Operations, 10Services, 10Discovery-Search (Current work), 10Patch-For-Review: Create kafka topic for mjolinr bulk daemon and decide on cluster - https://phabricator.wikimedia.org/T200215 (10fdans) [15:30:44] 10Analytics, 10Operations, 10Services, 10Discovery-Search (Current work), 10Patch-For-Review: Create kafka topic for mjolinr bulk daemon and decide on cluster - https://phabricator.wikimedia.org/T200215 (10fdans) p:05Triage>03Normal [15:31:12] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Add app_install_id and other renamed fields to EL sanitization whitelist - https://phabricator.wikimedia.org/T200095 (10fdans) [15:31:25] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Add app_install_id and other renamed fields to EL sanitization whitelist - https://phabricator.wikimedia.org/T200095 (10fdans) p:05Triage>03High [15:32:04] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: Add app_install_id and other renamed fields to EL sanitization whitelist - https://phabricator.wikimedia.org/T200095 (10fdans) p:05High>03Unbreak! [15:36:56] 10Analytics, 10Analytics-Wikistats: Issues with page view map in Wikistats 2 - https://phabricator.wikimedia.org/T200070 (10fdans) p:05Triage>03Low [15:37:09] 10Analytics, 10Analytics-Wikistats: Issues with page view map in Wikistats 2 - https://phabricator.wikimedia.org/T200070 (10fdans) p:05Low>03Normal [15:46:00] sorry a-team I hung up instead of demuting :( dan wanna disable geowiki jobs? [15:51:43] milimetric: ^ sorry [15:58:39] elukey: found that: https://github.com/linkedin/WhereHows [16:03:07] mmm looks nicE! [16:49:05] joal: do we want the pageview_whitelist as a required property in the coordinator.xml as well? [17:03:44] fdans: yessir :) [17:04:42] fdans: enforcing properties in coordinators prevent failing in later stages if parameter is not present [17:06:48] !lof Restart mediawiki-history-reduced job after deploy [17:09:51] !log Restart mediawiki-history-reduced job after deploy [17:09:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:12:59] joal: niice [17:16:52] (03PS4) 10Fdans: Filter out unwanted wikis from wmf.virtualpageview_hourly [analytics/refinery] - 10https://gerrit.wikimedia.org/r/447665 (https://phabricator.wikimedia.org/T197971) [17:18:00] (03CR) 10Joal: [C: 031] "LGTM ! Let's have somebody else reviewing as well :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/447665 (https://phabricator.wikimedia.org/T197971) (owner: 10Fdans) [17:43:45] * elukey off! [18:47:47] (03PS4) 10Mforns: [WIP] Add ability to salt and hash to eventlogging sanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/446592 (https://phabricator.wikimedia.org/T198426) [19:31:39] heya team, someone that wants to brainstorm on creating/rotating salts for EL?