[06:22:03] (03PS1) 10Elukey: unique_devices: add missing jdbc_url for hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529859 (https://phabricator.wikimedia.org/T227257) [06:23:15] !log kill/restart oozie coord unique_devices per_project_family daily due to missing jdbc_url in coordinator.properties (for hive2 actions) [06:23:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:28:23] (03CR) 10Elukey: [V: 03+2 C: 03+2] unique_devices: add missing jdbc_url for hive2 actions [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529859 (https://phabricator.wikimedia.org/T227257) (owner: 10Elukey) [06:31:21] going afk for a bit for an errand, will check later! [07:24:04] (03PS6) 10Fdans: Add creation query for new nediarequests dataset [analytics/refinery] - 10https://gerrit.wikimedia.org/r/528134 (https://phabricator.wikimedia.org/T229817) [07:27:24] (03PS1) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [07:32:40] (03Abandoned) 10Fdans: Add access type to mediacounts hourly dataset [analytics/refinery] - 10https://gerrit.wikimedia.org/r/517426 (https://phabricator.wikimedia.org/T225910) (owner: 10Fdans) [07:57:42] 10Analytics, 10Analytics-Kanban, 10StructuredDataOnCommons, 10Tool-Pageviews, 10Patch-For-Review: Create new mediarequests table - https://phabricator.wikimedia.org/T229817 (10fdans) [07:57:47] 10Analytics-Kanban, 10Patch-For-Review: Add and backfill agent-type in mediacounts hive dataset - https://phabricator.wikimedia.org/T225910 (10fdans) [07:59:53] (03PS2) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [08:09:58] (03PS3) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [08:11:34] (03PS4) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [08:15:44] (03PS5) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [08:18:36] (03PS6) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [08:18:49] sorry for the git spam, this is what happens when you're clueless and you have to brute force things [08:21:15] (03PS7) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [08:21:51] fdans: ahahahah [08:22:17] no problem for the git notifications, I personally don't mind [08:22:31] except when there is an outage and people are trying to coordinate [08:24:11] https://hips.hearstapps.com/ghk.h-cdn.co/assets/cm/15/11/54ffe5266025c-dog1.jpg?crop=1xw:1.0xh;center,top&resize=768:* [08:25:24] (03PS8) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [08:28:22] (03PS9) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [08:28:27] I'm aiming for a new PS record [08:37:20] (03PS10) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [08:40:35] !log kill/restart mediawiki history metrics/reduced to pick up hive2 actions [08:40:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:40:40] (03PS11) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [08:40:55] !log kill/restart mediawiki geoeditors druid/monthly to pick up hive2 actions [08:40:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:41:29] (03PS12) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [08:47:14] !log kill/restart mediawiki geoeditors|history load, history wikitext to pick up chances to the repair workflow (hive2 actions) [08:47:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:49:28] ok so in theory, excluding the cassandra bundle, all the coordinators are running hive2 actions! [09:31:22] superset 0.34 seems in the release process [09:31:39] trying to build it now, let's see what breaks [09:31:40] sigh [10:06:25] (03PS13) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [10:17:39] brb [11:19:46] (03PS1) 10Elukey: [WIP] Release version 0.34.0rc1-wikimedia [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/529936 [11:40:20] * elukey lunch! [11:56:57] omg I so wish joseph was here right now [11:57:19] (03PS14) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [12:16:16] (03PS15) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [12:25:05] (03PS16) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [12:50:36] (03PS2) 10Elukey: Release version 0.34.0rc1-wikimedia [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/529936 [13:00:51] elukey: from now on you shall only refer to me as OOZIE WIZARD [13:02:50] :) [13:15:56] (03PS17) 10Fdans: [wip] Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [14:11:17] (03PS3) 10Elukey: Release version 0.34.0rc1-wikimedia [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/529936 [14:23:06] 10Analytics-Kanban: Upgrade superset to 0.34 - https://phabricator.wikimedia.org/T230416 (10elukey) [14:24:43] heyaaa :] [14:30:13] 10Analytics-Kanban: Upgrade superset to 0.34 - https://phabricator.wikimedia.org/T230416 (10elukey) Created https://gerrit.wikimedia.org/r/#/c/529936/ and deployed to an-tool1005, you can easily test it via: ` ssh -L 8088:an-tool1005.eqiad.wmnet:80 an-tool1005.eqiad.wmnet ` And then `localhost:8088` on your br... [14:30:52] mforns: o/ [14:31:04] o/ [14:32:45] 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics: Superset Updates - https://phabricator.wikimedia.org/T211706 (10elukey) [14:33:32] so an-tool1005 is ready for testing with the new superset release [14:33:52] that fixes (from my tests) the annoying bug of getting a stacktrace upon first login [14:34:03] (so we'll not need to manually add the user) [14:40:51] neilpquinn: o/ - qq - is there a repo for the oozie toledo_pageviews-daily-coord? I'd need to change the hive action to hive2, so if there is a repo I'll send a patch :) [14:56:02] 10Analytics, 10Product-Analytics: Streamline Superset signup and authentication - https://phabricator.wikimedia.org/T203132 (10elukey) I am testing Superset 0.34 in https://phabricator.wikimedia.org/T230416#5411856. The new version contains the fix for FlaskAppBuilder https://github.com/dpgaspar/Flask-AppBuild... [15:49:44] o/ [15:50:25] o/ [15:51:20] milimetric, mforns: I'll need your input for T224459#5406021 before I can proceed. (No rush on my end, just a note that I'm on the planes today and I can iterate based on your feedback. Once I get to Wikimania, it may be delayed for some time.) [15:51:21] T224459: Recommend the best format to release public data lake as a dump - https://phabricator.wikimedia.org/T224459 [15:51:27] elukey: how's it going? [15:51:36] was just looking leila, thanks for setting that up [15:51:51] milimetric: nice. thanks! :) [15:52:16] leila: all good! and you? [15:52:53] elukey: doing well. On my way to Newark and then Stockholm. If there are no delays, I get there at 7am, just in time for the hackathon. :D [15:53:24] wow :D [15:53:29] have a good flight! [15:53:37] quick question for druid experts. how up to date is section on deleting segments over at https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid ? trying to figure out if `http://localhost:8081` should be `http://druid1003.eqiad.wmnet:8090` for example [15:53:39] This is my motivation to sleep in the flight. :D [15:53:41] milimetric: o/ we can discuss https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/529795/4/modules/statistics/templates/stats.wikimedia.org.erb@21 after standup if you have time [15:53:47] elukey: thanks! [15:55:05] also running into some conflicting instructions with https://druid.apache.org/docs/latest/tutorials/tutorial-delete-data.html with respect to preparing segments for deep storage deletion [15:56:26] bearloga: so the instructions should be good IIRC, but "localhost" should be the actual coordinator leader [15:56:29] elukey@druid1001:~$ curl http://localhost:8081/druid/coordinator/v1/datasources/test -I -s | grep Location [15:56:32] Location: http://druid1003.eqiad.wmnet:8081/druid/coordinator/v1/datasources/tes [15:56:43] so in this case, druid1003 [15:57:15] 8090 should be the overlord [15:57:22] (that is used only for indexations) [15:57:40] what conflicting instructions are you reading? [15:57:47] (trying to understand if we have a gap in the docs) [15:58:21] elukey: thank you! the conflict is that druid docs are telling me to mark segments as unused [15:58:42] https://druid.apache.org/docs/latest/tutorials/tutorial-delete-data.html#disable-segments-by-interval [15:59:46] (meeting bbiab sorry!) [16:00:01] so maybe not really conflicting instructions but missing from https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid#Preparation_for_deletion [16:00:04] no problem! :) [16:03:01] in theory the -X DELETE should mark the segments as unused [16:03:14] (so not dropped from deep storage == HDFS) [16:03:55] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Move refinery to hive 2 actions - https://phabricator.wikimedia.org/T227257 (10Milimetric) [16:04:13] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Move refinery to hive 2 actions - https://phabricator.wikimedia.org/T227257 (10Milimetric) a:03elukey [16:14:59] elukey: follow-up question on druid1001 vs druid1003. should I send ingestion jobs to one specifically or either one is fine as long as I use port 8090? [16:20:56] bearloga: in theory you should follow the same procedure to find the overlord leader [16:21:10] basically we have one running on each host, but only one actively working [16:21:36] but, if you code follows HTTP redirects, you can issue the query to any of them [16:22:25] (03PS18) 10Fdans: Add mediarequests hourly oozie job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) [16:27:58] bearloga: ah nice our docs specifiy curl -L [16:28:02] that follows redirects [16:28:16] I added a note to use any of the druid hostnames if you don't have access to the host [16:32:15] (going afk for a bit but will read later) [16:32:45] elukey: thank you! [16:54:40] elukey: when you're back, can you please check that cchen_ (conniecc1 on wikitech, see T228447) is in the wmf group? she's having trouble logging in to SWAP and Superset [16:54:40] T228447: Requesting access to machines [stat1004, stat1007, stat1006, notebook1003 and notebook1004] and groups for cchen - https://phabricator.wikimedia.org/T228447 [17:11:05] bearloga: the user is already in the wmf ldap group, it was missing the account on superset, done! [17:11:21] (btw we are resting superset 0.34 in these day that should solve this issue) [17:11:38] elukey: thanks! yo cchen_ you should be good now :D [17:12:36] bearloga: np! [17:12:42] going off people o/ [17:14:06] thanks!! [19:18:30] 10Analytics, 10Anti-Harassment (The Letter Song), 10Patch-For-Review: Instrument Special:Mute - https://phabricator.wikimedia.org/T224958 (10dmaza) @Milimetric @nettrom_WMF correct me if I'm wrong but other than annotating that the schema is active[[[ https://meta.wikimedia.org/wiki/Schema_talk:SpecialMuteSu... [19:49:36] (03CR) 10Milimetric: Add mediarequests hourly oozie job (036 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/529911 (https://phabricator.wikimedia.org/T229817) (owner: 10Fdans) [19:51:00] @elukey Thank you! i am able to log into Superset and Turnilo now! another issue i have now is when i trying to log into SWAP, i freeze in the login page. do you have any idea about why this happen? Thanks again! [20:08:04] 10Analytics, 10Research: Recommend the best format to release public data lake as a dump - https://phabricator.wikimedia.org/T224459 (10Milimetric) @leila: I looked over the survey and made some edits to the first part. The questions look great to me. Thanks again for putting this together. [21:07:47] (03PS1) 10Mforns: Add Oozie job for mediawiki history dumps [analytics/refinery] - 10https://gerrit.wikimedia.org/r/530002 (https://phabricator.wikimedia.org/T208612) [21:08:23] (03PS2) 10Mforns: [WIP] Add Oozie job for mediawiki history dumps [analytics/refinery] - 10https://gerrit.wikimedia.org/r/530002 (https://phabricator.wikimedia.org/T208612) [21:08:44] (03CR) 10Mforns: [C: 04-2] "Still needs testing." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/530002 (https://phabricator.wikimedia.org/T208612) (owner: 10Mforns) [21:22:15] (03PS3) 10Mforns: [WIP] Add Oozie job for mediawiki history dumps [analytics/refinery] - 10https://gerrit.wikimedia.org/r/530002 (https://phabricator.wikimedia.org/T208612)