[08:19:00] 10Analytics: Deletion of limn-language-data repository - https://phabricator.wikimedia.org/T228975 (10fdans) @Amire80 ping on this, as we are planning on starting to clean up repos this week. [10:54:13] * elukey lunch! [13:26:13] ottomata: o/ [13:29:14] (03PS4) 10Aklapper: Remember recent queries filter last used by a user. [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/176506 (https://phabricator.wikimedia.org/T76084) (owner: 10Rtnpro) [13:29:44] (03CR) 10jerkins-bot: [V: 04-1] Remember recent queries filter last used by a user. [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/176506 (https://phabricator.wikimedia.org/T76084) (owner: 10Rtnpro) [13:31:15] 10Quarry: Make query URLs have a sluggified version of the title in them - https://phabricator.wikimedia.org/T75885 (10Aklapper) a:05rtnpro→03None @rtnpro: I am resetting the assignee of this task because there has not been progress lately (please correct me if I am wrong!). Resetting the assignee avoids th... [13:35:31] 10Analytics, 10Tool-Pageviews: Create new mediarequests table - https://phabricator.wikimedia.org/T229817 (10fdans) [13:37:20] (03PS1) 10Fdans: Add creation query for new nediarequests dataset [analytics/refinery] - 10https://gerrit.wikimedia.org/r/528134 (https://phabricator.wikimedia.org/T229817) [13:41:16] (03PS2) 10Fdans: Add creation query for new nediarequests dataset [analytics/refinery] - 10https://gerrit.wikimedia.org/r/528134 (https://phabricator.wikimedia.org/T229817) [13:44:12] 10Analytics, 10StructuredDataOnCommons, 10Tool-Pageviews, 10Patch-For-Review: Create new mediarequests table - https://phabricator.wikimedia.org/T229817 (10Tnegrin) [13:58:15] milimetric: you here already? [14:05:42] hey fdans, yes [14:06:00] milimetric: do you have a min in the batcave? [14:06:14] Omw [14:27:29] milimetric: this is the creation quey btw, we can merge that whenever https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/528134/ [14:30:34] 10Analytics: mediawiki-history-wikitext-coord job fails every month - https://phabricator.wikimedia.org/T228883 (10elukey) As FYI the last run succeeded: https://hue.wikimedia.org/oozie/list_oozie_workflow/0008208-190715143115257-oozie-oozi-W/?coordinator_job_id=0053331-190417151359684-oozie-oozi-C [14:49:54] (03PS1) 10Elukey: edit: remove hive.auto.convert.join from oozie coord's .hql file [analytics/refinery] - 10https://gerrit.wikimedia.org/r/528167 (https://phabricator.wikimedia.org/T227257) [14:50:37] /o\ --^ [14:57:14] (03CR) 10Ottomata: [C: 03+1] edit: remove hive.auto.convert.join from oozie coord's .hql file [analytics/refinery] - 10https://gerrit.wikimedia.org/r/528167 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [14:57:22] (03CR) 10Mforns: Add creation query for new nediarequests dataset (035 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/528134 (https://phabricator.wikimedia.org/T229817) (owner: 10Fdans) [14:59:33] sorry mforns didn't push the last patch [15:00:23] oh, fdans, sorry for rushing [15:00:36] mforns: nono my bad [15:01:04] having problems to join batcave... [15:08:22] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Sunset MySQL data store for eventlogging - https://phabricator.wikimedia.org/T159170 (10Ottomata) [15:13:38] (03CR) 10Milimetric: Add creation query for new nediarequests dataset (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/528134 (https://phabricator.wikimedia.org/T229817) (owner: 10Fdans) [15:33:42] (03CR) 10WMDE-leszek: "recheck" [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/526471 (https://phabricator.wikimedia.org/T214894) (owner: 10Ladsgroup) [15:34:09] (03CR) 10jerkins-bot: [V: 04-1] Use the internal WDQS endpoint instead [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/526471 (https://phabricator.wikimedia.org/T214894) (owner: 10Ladsgroup) [15:34:26] 10Analytics, 10Analytics-Kanban, 10StructuredDataOnCommons, 10Tool-Pageviews, 10Patch-For-Review: Create new mediarequests table - https://phabricator.wikimedia.org/T229817 (10Ottomata) [15:36:03] 10Analytics, 10Analytics-Kanban, 10StructuredDataOnCommons, 10Tool-Pageviews, 10Patch-For-Review: Create new mediarequests table - https://phabricator.wikimedia.org/T229817 (10Ottomata) p:05Triage→03High [15:37:05] 10Analytics, 10Analytics-Kanban: Add more dimensions to netflow's druid ingestion specs - https://phabricator.wikimedia.org/T229682 (10Ottomata) a:03elukey [15:37:15] 10Analytics, 10Analytics-Kanban: Add more dimensions to netflow's druid ingestion specs - https://phabricator.wikimedia.org/T229682 (10Ottomata) p:05Triage→03High [15:37:40] 10Analytics, 10Analytics-Kanban: Set up a deletion timer for netflow data set - https://phabricator.wikimedia.org/T229674 (10Ottomata) a:03mforns [15:37:54] 10Analytics, 10Analytics-Kanban: Set up a deletion timer for netflow data set - https://phabricator.wikimedia.org/T229674 (10Ottomata) p:05Triage→03High [15:38:18] (03CR) 10Ottomata: [C: 03+1] Add jar to cassandra jobs for compatibility with hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/527583 (https://phabricator.wikimedia.org/T229669) (owner: 10Mforns) [15:38:48] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Oozie queries that use 'reflect("org.json.simple.JSONObject"...' need refinery_hive jar - https://phabricator.wikimedia.org/T229669 (10Ottomata) p:05Triage→03High [15:39:35] 10Analytics: placeholder - https://phabricator.wikimedia.org/T229464 (10Milimetric) 05Open→03Invalid [15:41:42] 10Analytics, 10Analytics-Kanban: Add --skip-trash arg to refinery-drop-older-than calls in data_purge.pp - https://phabricator.wikimedia.org/T229436 (10Ottomata) a:03mforns [15:42:26] 10Analytics, 10Analytics-Kanban: Add --skip-trash arg to refinery-drop-older-than calls in data_purge.pp - https://phabricator.wikimedia.org/T229436 (10Ottomata) p:05Triage→03High [15:44:14] 10Analytics, 10Analytics-Kanban, 10Wikimedia-Portals: projectview-hourly-coordinator needs to alarm when in error - https://phabricator.wikimedia.org/T228747 (10Ottomata) a:05elukey→03None [15:46:52] (03PS1) 10WMDE-leszek: Fix pom.xml for SureFire issue [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/528180 [15:47:11] (03CR) 10jerkins-bot: [V: 04-1] Fix pom.xml for SureFire issue [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/528180 (owner: 10WMDE-leszek) [15:47:22] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: API Request for unique devices for all wikipedia families is only showing data up to November 2018 - https://phabricator.wikimedia.org/T229254 (10Ottomata) p:05Triage→03Unbreak! [15:53:15] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Rebuild spark2 for Debian Buster - https://phabricator.wikimedia.org/T229347 (10Ottomata) a:03Ottomata [15:58:10] ottomata: quick question..the swift/upload/complete schema gives the example "swift.example_container.upload-complete", but should that be prefixed with eqiad. and codfw.? [15:58:20] * ebernhardson thinks thats how eventgate works, but not entirely sure... [15:58:32] for the topic, i mean [16:00:30] 10Analytics, 10Product-Analytics, 10Readers-Web-Backlog: Reading_depth remove eventlogging instrumentation? - https://phabricator.wikimedia.org/T229042 (10Jdlrobson) Note there are many tasks to add features to it ( https://phabricator.wikimedia.org/T219212, https://phabricator.wikimedia.org/T200093, https:... [16:03:21] (03PS2) 10WMDE-leszek: Fix pom.xml for SureFire issue [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/528180 [16:04:14] (03CR) 10jerkins-bot: [V: 04-1] Fix pom.xml for SureFire issue [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/528180 (owner: 10WMDE-leszek) [16:05:54] (03PS2) 10WMDE-leszek: Use the internal WDQS endpoint instead [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/526471 (https://phabricator.wikimedia.org/T214894) (owner: 10Ladsgroup) [16:08:22] (03CR) 10jerkins-bot: [V: 04-1] Use the internal WDQS endpoint instead [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/526471 (https://phabricator.wikimedia.org/T214894) (owner: 10Ladsgroup) [16:16:17] (03PS3) 10WMDE-leszek: Fix pom.xml for SureFire issue [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/528180 [16:16:37] (03CR) 10jerkins-bot: [V: 04-1] Fix pom.xml for SureFire issue [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/528180 (owner: 10WMDE-leszek) [16:20:31] (03PS4) 10WMDE-leszek: Fix pom.xml for SureFire issue [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/528180 [16:26:00] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10ORES, and 4 others: Fix "Must provide the 'topic' parameter" in ORES /precache endpoint - https://phabricator.wikimedia.org/T228689 (10Halfak) [16:26:03] (03PS5) 10WMDE-leszek: Fixed CI build [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/528180 [16:26:10] (03PS3) 10WMDE-leszek: Use the internal WDQS endpoint instead [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/526471 (https://phabricator.wikimedia.org/T214894) (owner: 10Ladsgroup) [16:35:35] (03CR) 10jerkins-bot: [V: 04-1] Use the internal WDQS endpoint instead [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/526471 (https://phabricator.wikimedia.org/T214894) (owner: 10Ladsgroup) [16:38:45] (03PS4) 10WMDE-leszek: Use the internal WDQS endpoint instead [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/526471 (https://phabricator.wikimedia.org/T214894) (owner: 10Ladsgroup) [16:38:58] (03PS6) 10WMDE-leszek: Fixed CI build [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/528180 [16:39:02] (03PS5) 10WMDE-leszek: Use the internal WDQS endpoint instead [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/526471 (https://phabricator.wikimedia.org/T214894) (owner: 10Ladsgroup) [16:41:50] (03CR) 10jerkins-bot: [V: 04-1] Use the internal WDQS endpoint instead [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/526471 (https://phabricator.wikimedia.org/T214894) (owner: 10Ladsgroup) [16:41:55] (03CR) 10jerkins-bot: [V: 04-1] Fixed CI build [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/528180 (owner: 10WMDE-leszek) [16:43:48] (03CR) 10jerkins-bot: [V: 04-1] Use the internal WDQS endpoint instead [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/526471 (https://phabricator.wikimedia.org/T214894) (owner: 10Ladsgroup) [16:46:57] (03PS7) 10WMDE-leszek: Fixed CI build [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/528180 [16:47:26] (03PS6) 10WMDE-leszek: Use the internal WDQS endpoint instead [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/526471 (https://phabricator.wikimedia.org/T214894) (owner: 10Ladsgroup) [16:48:05] (03CR) 10jerkins-bot: [V: 04-1] Fixed CI build [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/528180 (owner: 10WMDE-leszek) [16:50:29] (03CR) 10jerkins-bot: [V: 04-1] Use the internal WDQS endpoint instead [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/526471 (https://phabricator.wikimedia.org/T214894) (owner: 10Ladsgroup) [17:05:02] ebernhardson: o/ [17:05:12] i just tested the swift upload without setting PATH [17:05:14] it seems to work for me [17:05:17] with just [17:05:18] 'swift' [17:08:56] ottomata: sec lemme make a minimal repro paste [17:11:36] ottomata: on stat1007.eqiad.wmnet in ~ebernhardson/repro [17:11:55] ottomata: run PATH=$PWD/bin:$PATH python3 test.py [17:12:11] OHHH because you need to set a custom path [17:12:13] ? [17:12:28] ottomata: i guess? i dunno. The path defined that way should be the same as exporting it [17:12:39] it just only defines it for that one command instead of the shell in general [17:12:49] but in general yes, the PATH needs to point to non-system directories [17:12:51] aye, mine works probably because it is just using default PATH [17:12:54] and swift is in it [17:22:46] ottomata: I checked on my previous notes for the testing cluster, and in the past I have used spark.executorEnv.LD_LIBRARY_PATH=/usr/lib/hadoop/lib/native [17:22:52] to make it work [17:23:09] that is strange since we don't need it in the analytics cluster [17:23:19] (I can see spark-env having it) [17:23:27] so now I am wondering what changes (or what I didn't se) [17:23:32] *set [17:28:39] hm [17:28:58] elukey: you are asying that is not et in analytics cluster? [17:29:02] set* [17:32:32] ottomata: yes basically there is something (apparently) not working in the testing cluster that needs the extra parameter to do things like read snappy compressed sequence files, but that is not required in our prod cluster [17:34:04] (03CR) 10Ottomata: swift-upload.py to handle upload and event emitting (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) (owner: 10Ottomata) [17:34:45] (03PS10) 10Ottomata: swift_upload.py to handle upload and event emitting [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) [17:35:34] (03PS11) 10Ottomata: swift_upload.py to handle upload and event emitting [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) [17:38:20] elukey: hm i dunnoooooo but maybe we can just set it anyway [17:38:31] we can set that in analytics cluster too, won't hurt; probabbly good to do [17:40:11] ottomata: yeah I am thinking the same, but at the same time I feel that it could be a symptom of something misconfigured for some reason [17:40:27] maybe I am too paranoid as always :D [17:50:58] I am also trying to read the NavTiming sequence files from pyspark and failing miserably [17:51:04] sigh I am such a n00b :D [17:51:19] I can see the records but they still seem compressed when printed [17:54:12] ah no with a data frame it works :) [18:04:58] haha ok! [18:05:20] elukey: am gonna bike home real quick, if you are still working on it when I get there we can work on it together [18:06:06] ottomata: I am going afk in a bit for dinner, let's do it tomorrow if you have time! thanks :) [18:06:12] I'll try to make more tests tomorrow [18:06:15] super weird though [18:06:59] * elukey off! [18:51:48] ok! [22:53:31] (03CR) 10EBernhardson: [C: 03+1] swift_upload.py to handle upload and event emitting [analytics/refinery] - 10https://gerrit.wikimedia.org/r/525435 (https://phabricator.wikimedia.org/T227896) (owner: 10Ottomata)