[00:38:58] 10Analytics, 10Research, 10Article-Recommendation: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10bmansurov) @Nuria, Oozie task is happening in parallel here: {T210844}. [01:34:13] 10Analytics, 10Analytics-Kanban: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Neil_P._Quinn_WMF) >>! In T213670#4878956, @Neil_P._Quinn_WMF wrote: > I actually had a query on dbstore1002 that had been running on Wikidata for 25 hours as of 23:30 UTC yesterday (T210807#4876169). Am I... [01:34:52] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Contributors-Analysis, and 2 others: Add change tag tables to monthly mediawiki_history sqoop - https://phabricator.wikimedia.org/T205940 (10Neil_P._Quinn_WMF) [01:34:55] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Replace the current multisource analytics-store setup - https://phabricator.wikimedia.org/T172410 (10Neil_P._Quinn_WMF) [01:48:56] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Replace the current multisource analytics-store setup - https://phabricator.wikimedia.org/T172410 (10Neil_P._Quinn_WMF) Product Analytics met with @elukey, @Milimetric, and @elukey on Monday to further discuss our need... [06:15:26] 10Analytics, 10Analytics-Kanban: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) 05Resolved→03Open dbstore1002 crashed last night again - this time, the mysqld process crashed. @Neil_P._Quinn_WMF how does your query look like? dbstore1002 is in a very bad state, we nee... [07:55:14] 10Analytics, 10Analytics-Kanban: Clean up staging db - https://phabricator.wikimedia.org/T212493 (10elukey) Thanks all for your clean up! So remaining tables to drop: ` | diego_tmp | | leila_artrecom_user_gender | | leila_artrecom_user_gender_en | | leila_design_research_particip... [08:27:50] 10Analytics, 10Analytics-Kanban: Clean up staging db - https://phabricator.wikimedia.org/T212493 (10elukey) [08:29:16] addshore: o/ - when you have time can you check the wmde_ prefixed tables listed in T212493 ? [08:29:18] T212493: Clean up staging db - https://phabricator.wikimedia.org/T212493 [08:29:31] if those tables are not needed we can probably drop them [08:31:36] 10Analytics, 10Continuous-Integration-Infrastructure (Slipway): Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10hashar) The job has to be refactored and generalized so we can use it on other maven repositories. mwdumper is one such use case ( T213874#4883... [09:11:48] elukey: *reads up* [09:14:19] 10Analytics, 10Analytics-Kanban: Clean up staging db - https://phabricator.wikimedia.org/T212493 (10Addshore) The wmde_prefixed tables are still in use and are written to every day :) | wmde_analytics_echoStatusNotif_users_yesterday | 2017-09-13 03:31:42 | NULL | 68616 | | wmde_analytics_e... [09:14:31] elukey: still used :) [09:14:31] gooood thanks :) [09:14:52] addshore: you know that we are going to migrate that db to another host before April? [09:15:09] I'll take a note to ping you to move your scripts to it [09:15:31] Year, I probably have to alter some scripts a bit [09:15:53] so that db / those tables will be on a different host to the wiki dbs? [09:16:52] https://phabricator.wikimedia.org/T210478 [09:17:08] Yeh, I'll need to do some script modifications [09:17:29] I guess I can only do them once the new setup is there then, but before the old setup is killed [09:17:36] the mapping is in https://phabricator.wikimedia.org/T210478#4794536 [09:17:43] sure [09:17:50] whats the timeline for the new place? :) [09:18:31] so we need to decom dbstore1002 in april due to Ubuntu Trusty going EOL [09:18:35] and the host being OOW [09:18:54] we are setting up the new hosts so I'd say in a month, more or less, they should be up [09:19:03] and usable [09:19:55] elukey: I made T213894 [09:19:55] T213894: Modify analytics/wmde/scripts repo to work with the new analytics multisource db setup - https://phabricator.wikimedia.org/T213894 [09:20:06] that should block the deocmm of the old setup [09:20:46] 10Analytics, 10Analytics-Kanban, 10DBA, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10Marostegui) >>! In T212487#4867178, @Marostegui wrote: > @elukey For those databases that... [09:22:47] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Convert Aria tables to InnoDB on dbstore1002 - https://phabricator.wikimedia.org/T213706 (10elukey) Skipping this table for the moment: ` MariaDB [(none)]> ALTER TABLE fawiki.ores_classification ENGINE=InnoDB; ERROR 1005 (HY000): Can't create table `fawiki`.`#... [09:23:13] are user properties in hadoop at all? [09:23:18] :P [09:23:38] then I could at least get rid of those 2 dbs in staging [09:23:49] addshore: this is a question for Joseph/Dan! I am super ignorant about it :( [09:23:53] :D [09:24:05] I wonder if Joseph is feeling any healthier today *reads up* [09:27:07] 10Analytics, 10Analytics-Kanban: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) I have re-imported wikishared.echo_unread_wikis as it was failing due to inconsistencies [09:28:06] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Convert Aria tables to InnoDB on dbstore1002 - https://phabricator.wikimedia.org/T213706 (10Marostegui) >>! In T213706#4883702, @elukey wrote: > Skipping this table for the moment: > > ` > MariaDB [(none)]> ALTER TABLE fawiki.ores_classification ENGINE=InnoDB;... [09:30:02] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Convert Aria tables to InnoDB on dbstore1002 - https://phabricator.wikimedia.org/T213706 (10elukey) Seems happening for `.ores_classification` tables: ` MariaDB [(none)]> ALTER TABLE hewiki.ores_classification ENGINE=InnoDB; ERROR 1005 (HY000): Can't create ta... [09:33:53] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Convert Aria tables to InnoDB on dbstore1002 - https://phabricator.wikimedia.org/T213706 (10elukey) Leaving also the following aside for a moment: ` ALTER TABLE information_schema.ALL_PLUGINS ENGINE=InnoDB; ALTER TABLE information_schema.COLUMNS ENGINE=InnoDB;... [09:35:42] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Convert Aria tables to InnoDB on dbstore1002 - https://phabricator.wikimedia.org/T213706 (10Marostegui) >>! In T213706#4883718, @elukey wrote: > Leaving also the following aside for a moment: > > ` > ALTER TABLE information_schema.ALL_PLUGINS ENGINE=InnoDB; >... [10:36:48] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Convert Aria tables to InnoDB on dbstore1002 - https://phabricator.wikimedia.org/T213706 (10Marostegui) This fixes it: ` root@dbstore1002.eqiad.wmnet[(none)]> ALTER TABLE hewiki.ores_classification ROW_FORMAT=COMPRESSED; ALTER TABLE hewiki.ores_classification E... [11:18:33] 10Analytics, 10Analytics-Kanban: Clean up staging db - https://phabricator.wikimedia.org/T212493 (10elukey) @Marostegui I don't see any more "easy" way to ping people about those tables, do you think that this clean up was ok (and the task can be closed) or that we'd need something more? [11:20:22] 10Analytics, 10Analytics-Kanban, 10DBA, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) [11:20:25] 10Analytics, 10Analytics-Kanban: Clean up staging db - https://phabricator.wikimedia.org/T212493 (10Marostegui) 05Open→03Resolved a:03elukey I think the clean up work was good. Most of the tables that are left are in use. Thanks! [11:27:57] 10Analytics, 10Analytics-General-or-Unknown: Kafkatee zero files having 10% less requests than udp2log zero files - https://phabricator.wikimedia.org/T66181 (10fgiunchedi) 05Open→03Invalid Zero is no more, resolving [11:37:10] * elukey lunch! [12:00:35] Hi addshore - Wednesday is usually kids day, and is actually kids and flu day today :) [12:00:41] I however have an answer for you: [12:00:46] :D [12:00:58] We don't have user properties in the cluster as of now [12:02:45] addshore: Creating a task would be the first step, and then if you feel like it you could add an SQL bit in this function of this file: https://github.com/wikimedia/analytics-refinery/blob/master/python/refinery/sqoop.py#L151 [12:03:21] Once the SQL to gather the data is ready, it's just matter of adding the table to the timer in puppet [12:04:12] joal: thanks, I'll think about writing a ticket :D [12:04:42] addshore: It'd be great of you do it as you're not the first one having asked for it [12:04:46] :-P [12:04:55] oooh, in that case I'll write it right now [12:04:58] * joal hates pushing work on other, but still do sometimes [12:06:25] 10Analytics: Add user_properties mysql table data to hadoop cluster - https://phabricator.wikimedia.org/T213910 (10Addshore) [12:06:27] done [12:06:36] Thanks addshore :) [12:10:18] Hi team - I'm gonna be off today as well, but plan to be on tomorrow - I feel the fight is getting to my advantage now [12:11:07] (03PS1) 10Fdans: [wip] Change email send workflow to notify of completed jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/484657 (https://phabricator.wikimedia.org/T206894) [13:45:14] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Replace the current multisource analytics-store setup - https://phabricator.wikimedia.org/T172410 (10Nuria) >Analytics agreed to work on providing such a tool before the shutdown of dbstore1002 (based of course on our... [13:50:40] joal: ack! Try to rest a bit today! [14:01:30] o/ [14:07:42] o/ [14:08:43] 10Analytics, 10Analytics-Kanban, 10DBA, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10elukey) @Marostegui I have now everything in my home gzipped, I'll move it to stat1007 and... [14:09:27] 10Analytics, 10Analytics-Kanban, 10DBA, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10Marostegui) cool, let me know when you think it is safe for me to delete my files [14:11:24] (03CR) 10Ottomata: [wip] Change email send workflow to notify of completed jobs (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/484657 (https://phabricator.wikimedia.org/T206894) (owner: 10Fdans) [14:12:05] 10Analytics, 10Analytics-Kanban: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) More errors appeared: ` Last_SQL_Error: Error 'Can't lock file (errno: 22 "Invalid argument")' on query. Default database: 'outreachwiki'. Query: 'INSERT /* ManualLogEntry::insert... [14:13:32] 10Analytics, 10Analytics-Kanban, 10DBA, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) Just for the record, I am spending an incredible amount of time fixing dbstore1002 on the last few days si... [14:13:38] 10Analytics, 10Analytics-Kanban: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) [14:13:42] 10Analytics, 10Analytics-Kanban, 10DBA, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) [14:14:27] 10Analytics, 10Analytics-Kanban, 10DBA, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10Marostegui) As spoken on IRC...I have deleted my files. [14:16:54] ottomata: thanks for the review, although it's still wip, I got a refactor coming, this was just to test in the stat machine [14:17:12] 10Analytics, 10Research, 10Article-Recommendation: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10Nuria) @bmansurov sounds good, let's start working on the code for oozie jobs, that work has to be done before we could transfer data into mySQL anyways. [14:17:19] ok great [14:17:25] does wip mean don't review yet? [14:21:16] fdans: i have to ask, what happened with superset deploy yesterday? [14:25:35] nuria: for some reason we still don't understand, we deployed superset 0.28 (with all its issues) instead of our version with the fixes, even though we swear we had the same wheels and the version tag in superset said 0.26.3-wikimedia1 [14:25:41] so we reverted to the original one [14:26:04] fdans: and ahemm.. do we have a plan to deploy the version we want? [14:27:12] nuria: the best course of action is to remove a variable from the problem, namely Luca :) Next deployment should be done by Andrew, I always feel that I am missing some bit if we keep slitting the work [14:27:37] or, even better, we don't deploy anymore until we have a staging environment [14:28:00] leaving superset broken to debug is not an option [14:28:11] and it is diffucult to understand what went wrong in this way [14:28:23] elukey: sounds like plan, there comes ottomata with the magic wand [14:28:44] oh hi i was supposed to help yal lyesterday! [14:28:45] let's dlo now... [14:28:56] elukey: ottomata wanna batcave? [14:29:03] or we can track on irc [14:29:06] let's bc [14:29:17] elukey: ok, staging environment is a must too, i agree, let's see if we can get that done this quarter [14:30:11] 10Analytics, 10Analytics-Kanban: Create staging environment for superset - https://phabricator.wikimedia.org/T213923 (10Nuria) p:05Triage→03High [14:31:15] 10Analytics, 10Analytics-Kanban: Create staging environment for superset - https://phabricator.wikimedia.org/T213923 (10Nuria) [14:34:05] elukey: we're in the batcave if you wanna join [14:35:00] fdans: doing something else, I'll join if necessary ok?? [14:44:07] (I am prepping the hadoop worker nodes for the testing cluster and fighting with dbstore1002 :( ) [15:00:40] elukey: if you have some minutes, will you come to bc? [15:01:27] ottomata: sure! [15:30:52] a-team are we doing today the first analytics deployment train? :) (if there is something to deploy) (sorry for double posting, didn't realize I was in the ops chan) [15:33:02] (03PS1) 10Ottomata: Update to 0.26.3-wikimedia2 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/484689 [15:34:23] there are a couple of things in ready to deploy for refinery [15:40:24] (03PS1) 10Ottomata: Fix create_virtualenv.sh to build from $wheels_dir directly [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/484691 [15:40:40] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Fix create_virtualenv.sh to build from $wheels_dir directly [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/484691 (owner: 10Ottomata) [15:44:57] (03PS2) 10Ottomata: Update to 0.26.3-wikimedia2 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/484689 [15:50:34] fdans: check superset labs [15:50:35] ya? [15:50:51] on it [15:53:36] ottomata: hmmm [15:53:36] http://localhost:9080/superset/explore/?form_data=%7B%22datasource%22%3A%224__table%22%2C%22viz_type%22%3A%22time_pivot%22%2C%22slice_id%22%3A2%2C%22granularity_sqla%22%3A%22date%22%2C%22time_grain_sqla%22%3A%22P1D%22%2C%22since%22%3A%227+years+ago%22%2C%22until%22%3A%22now%22%2C%22metric%22%3A%5B%5D%2C%22adhoc_filters%22%3Anull%2C%22freq%22%3A%22W-MON%22%2C%22show_legend%22%3Atrue%2C%22line_interpolation%22%3A%22li [15:53:36] near%22%2C%22color_picker%22%3A%7B%22r%22%3A0%2C%22g%22%3A122%2C%22b%22%3A135%2C%22a%22%3A1%7D%2C%22x_axis_label%22%3A%22%22%2C%22bottom_margin%22%3A%22auto%22%2C%22x_axis_showminmax%22%3Afalse%2C%22x_axis_format%22%3A%22smart_date%22%2C%22y_axis_label%22%3A%22%22%2C%22left_margin%22%3A%22auto%22%2C%22y_axis_showminmax%22%3Afalse%2C%22y_log_scale%22%3Afalse%2C%22y_axis_format%22%3A%22.3s%22%2C%22y_axis_bounds%22%3A% [15:53:36] 5Bnull%2Cnull%5D%2C%22url_params%22%3A%7B%7D%2C%22labels_outside%22%3Atrue%2C%22donut%22%3Afalse%2C%22pie_label_type%22%3A%22key%22%2C%22groupby%22%3A%5B%5D%2C%22color_scheme%22%3A%22bnbColors%22%7D [15:53:41] goddamn [15:54:27] loool [15:54:27] ah ottomata never mind [15:54:36] it works fine [15:55:35] ottomata I think we can proceed to prod [15:56:48] ok [15:57:00] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Update to 0.26.3-wikimedia2 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/484689 (owner: 10Ottomata) [15:57:17] deploying [15:59:09] fdans: deployed in prod [15:59:19] looks good to me! [16:00:45] checky check [16:00:54] nuria: presto meeting! [16:03:10] nuria: ottomata periodicity pivot working, all good :) [16:03:16] thank you andrew :) cc elukey [16:11:29] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade Superset to 0.28.1 - https://phabricator.wikimedia.org/T211605 (10elukey) a:05elukey→03fdans [16:11:46] fdans: assigned to you since you are coordinating the work and I have done not so much :) [16:11:56] so you can move it to done if we are good [16:13:51] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Set up a Analytics Hadoop test cluster in production that runs a configuration as close as possible to the current one. - https://phabricator.wikimedia.org/T212256 (10elukey) Today I have made all the partitions on the old hosts (wiping th... [16:30:26] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Bug: can't make a YoY time series chart in Superset - https://phabricator.wikimedia.org/T210687 (10fdans) This is now fixed in superset production, periodicity pivots are working correctly. [16:54:35] (03PS1) 10Fdans: [wip] Only use artificial ID if there is no page id in delete events [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/484706 [16:56:09] a-team i will miss standup! better use of data meeting today [16:58:53] 10Analytics, 10Contributors-Analysis, 10Product-Analytics, 10Epic, 10User-Elukey: Provide feature parity between the wiki replicas and the Analytics Data Lake - https://phabricator.wikimedia.org/T212172 (10Milimetric) Quick note to thank everyone very much for all these use cases. They're very useful fo... [16:59:00] (03PS1) 10Mforns: Adapt EventLogging/WhiteListSanitization to new way of storing salts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/484708 (https://phabricator.wikimedia.org/T212014) [17:00:01] ottomata: if you have time today can you check if https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/482275/ is good ? [17:01:00] (03CR) 10jerkins-bot: [V: 04-1] [wip] Only use artificial ID if there is no page id in delete events [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/484706 (owner: 10Fdans) [17:03:05] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10Ottomata) Great meeting! We realized that the same problem we are encountering with the available Cloud VPS infrastructu... [17:38:48] Heya team - I've survived the kids today, means I should be ok for tomorrow :) [17:40:32] joal: we are glad you have survived [17:41:41] nuria: anything in particular I shoujld focus on tomorrow morning? [17:41:54] joal: catching up! [17:42:05] right - [17:47:43] will do that - Thank you all folks for having covered [17:56:28] (03CR) 10Ottomata: [C: 03+1] Adapt EventLogging/WhiteListSanitization to new way of storing salts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/484708 (https://phabricator.wikimedia.org/T212014) (owner: 10Mforns) [18:12:14] * elukey off! [18:32:43] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Replace the current multisource analytics-store setup - https://phabricator.wikimedia.org/T172410 (10Neil_P._Quinn_WMF) >>! In T172410#4884315, @Nuria wrote: >>Analytics agreed to work on providing such a tool before t... [18:55:08] 10Analytics, 10Analytics-Kanban: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Neil_P._Quinn_WMF) >>! In T213670#4883416, @Marostegui wrote: > @Neil_P._Quinn_WMF how does your query look like? I posted it in T210807#4876169. You don't need to worry about me getting the data; I rewrot... [19:51:13] musikanimal: how are you guys doing on deploying beacon for piwik? [20:00:03] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10bd808) >>! In T207321#4885112, @Ottomata wrote: > A. Bryan noted that we won't want to use users regular LDAP accounts fo... [20:05:42] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10Ottomata) > I was trying to figure out how this will actually look for the end user It'd be in the JDBC connection, e.g... [20:11:33] 10Analytics, 10EventBus, 10serviceops: Include git in our alpine docker image on docker-registry.wikimedia.org - https://phabricator.wikimedia.org/T213963 (10Ottomata) p:05Triage→03Normal [20:17:05] nuria: we hopefully will deploy maybe latter today or tomorrow. Got a few unrelated things that need to be merged first [20:17:13] *later [21:00:36] 10Analytics, 10Knowledge-Integrity, 10Research, 10Epic: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969 (10bmansurov) Adding #analytics to give a heads up that EL will get busy when we deploy this. [21:07:24] musikanimal: k, let me know [21:26:29] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10bd808) >>! In T207321#4885829, @Ottomata wrote: >> I was trying to figure out how this will actually look for the end use... [21:29:06] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Replace the current multisource analytics-store setup - https://phabricator.wikimedia.org/T172410 (10Neil_P._Quinn_WMF) >>! In T172410#4885722, @Nuria wrote: >>I definitely understand that you can't make any ironclad g... [21:30:46] 10Analytics, 10Contributors-Analysis, 10Product-Analytics, 10Epic, 10User-Elukey: Provide feature parity between the wiki replicas and the Analytics Data Lake - https://phabricator.wikimedia.org/T212172 (10Neil_P._Quinn_WMF) >>! In T212172#4885088, @Milimetric wrote: > Quick note to thank everyone very m... [21:32:23] 10Analytics, 10Product-Analytics, 10Research, 10WMDE-Analytics-Engineering, and 2 others: Replace the current multisource analytics-store setup - https://phabricator.wikimedia.org/T172410 (10Neil_P._Quinn_WMF) [21:34:18] 10Analytics, 10Operations, 10Product-Analytics, 10ops-eqiad: Degraded RAID on dbstore1002 - https://phabricator.wikimedia.org/T206965 (10Neil_P._Quinn_WMF) [21:35:05] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10Ottomata) > That works for a java developer, but nobody else (including Quarry). Quarry is a Python app which is why I wa... [21:35:26] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Neil_P._Quinn_WMF) [21:59:14] 10Analytics, 10Discovery: Workflow to be able to move data files computed in jobs from analytics cluster to production - https://phabricator.wikimedia.org/T213976 (10Nuria) [22:00:03] 10Analytics, 10Research, 10Article-Recommendation: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10Nuria) [22:00:08] 10Analytics, 10Discovery: Workflow to be able to move data files computed in jobs from analytics cluster to production - https://phabricator.wikimedia.org/T213976 (10Nuria) [22:18:14] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline - https://phabricator.wikimedia.org/T211247 (10Ottomata) @fselles, I'm not able to get the [[ https://gerrit.wikimedia.org/r/#/c/operations/... [22:20:35] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Bug: can't make a YoY time series chart in Superset - https://phabricator.wikimedia.org/T210687 (10Nuria) See graph For YoY pageviews: https://bit.ly/2QRRiCw in superset cc @JKatzWMF Closing ticket [22:20:43] 10Analytics-Kanban, 10Product-Analytics: Superset Updates - https://phabricator.wikimedia.org/T211706 (10Nuria) [22:38:06] (03CR) 10Nuria: Adapt EventLogging/WhiteListSanitization to new way of storing salts (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/484708 (https://phabricator.wikimedia.org/T212014) (owner: 10Mforns) [22:38:17] thx nuria [22:38:22] lookin [22:42:38] (03CR) 10Mforns: Adapt EventLogging/WhiteListSanitization to new way of storing salts (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/484708 (https://phabricator.wikimedia.org/T212014) (owner: 10Mforns) [22:46:53] (03PS2) 10Mforns: Adapt EventLogging/WhiteListSanitization to new way of storing salts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/484708 (https://phabricator.wikimedia.org/T212014) [22:47:47] (03CR) 10Mforns: Adapt EventLogging/WhiteListSanitization to new way of storing salts (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/484708 (https://phabricator.wikimedia.org/T212014) (owner: 10Mforns) [23:05:18] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Bug: can't make a YoY time series chart in Superset - https://phabricator.wikimedia.org/T210687 (10Nuria) Trying link again: https://superset.wikimedia.org/superset/explore/?form_data=%7B%22datasource%22%3A%22352__druid%22%2C%22viz_type%22%3A%22time_pivot... [23:07:08] (03CR) 10Mforns: "I tested this with real event db data and everything seems to be working fine." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/484708 (https://phabricator.wikimedia.org/T212014) (owner: 10Mforns) [23:40:12] 10Analytics, 10Analytics-Kanban, 10DBA, 10User-Elukey: Review dbstore1002's non-wiki databases and decide which ones needs to be migrated to the new multi instance setup - https://phabricator.wikimedia.org/T212487 (10greg) >>! In T212487#4867200, @Marostegui wrote: > I asked @chasemp about `fab_migration`... [23:48:39] 10Analytics, 10EventBus, 10serviceops: Include git in our alpine docker image on docker-registry.wikimedia.org - https://phabricator.wikimedia.org/T213963 (10Joe) Do not use alpine as a base for your containers if you want to execute them in production. That is strictly limited to debian-based images, for wh...