[00:00:03] addshore: kinit and reload yeah [00:03:13] (03CR) 10Nuria: [C: 04-1] "Code changes look fine but prod build does not work, will debug further but I think language is not being set." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/585725 (https://phabricator.wikimedia.org/T199386) (owner: 10Fdans) [00:07:51] (03CR) 10Nuria: [C: 04-1] "Error is is numbro " missing mandatory key 'ordinal' "" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/585725 (https://phabricator.wikimedia.org/T199386) (owner: 10Fdans) [02:58:05] 10Analytics, 10Analytics-Kanban, 10Privacy Engineering, 10Product-Analytics, and 3 others: Drop data from Prefupdate schema that is older than 90 days - https://phabricator.wikimedia.org/T250049 (10sbassett) >>! In T250049#6052563, @nettrom_WMF wrote: > Is background information available in the parent tas... [04:15:33] (03CR) 10Nuria: [C: 04-1] "On my buildIi see for en_AU (on main.js file, once build):" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/585725 (https://phabricator.wikimedia.org/T199386) (owner: 10Fdans) [04:30:08] (03CR) 10Nuria: [C: 04-1] "Also, is en-US missing and that is why en-AU is packed? https://github.com/BenjaminVanRyseghem/numbro/tree/develop/languages" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/585725 (https://phabricator.wikimedia.org/T199386) (owner: 10Fdans) [05:44:27] 10Analytics: Add TLS to Kafka Mirror Maker - https://phabricator.wikimedia.org/T250250 (10elukey) [05:49:09] good morning :) [07:01:12] Morning!! [07:01:28] Morning [09:04:33] PROBLEM - aqs endpoints health on aqs1005 is CRITICAL: /analytics.wikimedia.org/v1/edits/per-page/{project}/{page-title}/{editor-type}/{granularity}/{start}/{end} (Get daily edits for english wikipedia page 0) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs [09:05:18] PROBLEM - aqs endpoints health on aqs1006 is CRITICAL: /analytics.wikimedia.org/v1/edits/per-page/{project}/{page-title}/{editor-type}/{granularity}/{start}/{end} (Get daily edits for english wikipedia page 0) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs [09:05:23] PROBLEM - aqs endpoints health on aqs1009 is CRITICAL: /analytics.wikimedia.org/v1/edits/per-page/{project}/{page-title}/{editor-type}/{granularity}/{start}/{end} (Get daily edits for english wikipedia page 0) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs [09:05:28] PROBLEM - aqs endpoints health on aqs1004 is CRITICAL: /analytics.wikimedia.org/v1/edits/per-page/{project}/{page-title}/{editor-type}/{granularity}/{start}/{end} (Get daily edits for english wikipedia page 0) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs [09:05:33] PROBLEM - aqs endpoints health on aqs1007 is CRITICAL: /analytics.wikimedia.org/v1/edits/per-page/{project}/{page-title}/{editor-type}/{granularity}/{start}/{end} (Get daily edits for english wikipedia page 0) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs [09:05:56] good morning [09:06:11] this seems to be a drop datasource event sigh [09:08:51] RECOVERY - aqs endpoints health on aqs1005 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs [09:09:12] !log restart druid brokers on druid100[4-6] - stuck after datasource deletion [09:09:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:09:35] grabbed a jstack, we really need to plan a druid upgrade :( [09:09:41] RECOVERY - aqs endpoints health on aqs1006 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs [09:09:47] RECOVERY - aqs endpoints health on aqs1009 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs [09:09:53] RECOVERY - aqs endpoints health on aqs1004 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs [09:09:59] RECOVERY - aqs endpoints health on aqs1007 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs [09:35:17] !log execute "create_virtualenv.sh ../venv" on stat1006, notebook1003, notebook1004 to apply new settings to Spark kernels (re-creating them) [09:35:20] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:35:33] !log restart jupyterhub too as follow up [09:35:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:37:49] joal: --^ [09:37:51] :) [09:57:21] 10Analytics, 10ContentTranslation, 10Language-Team (Language-2020-Focus-Sprint): Test Performance of Marian NMT translation in stat cluster - https://phabricator.wikimedia.org/T247245 (10santhosh) NOTE: I have documented the steps for doing this installation and performance test at https://wikitech.wikimedia... [10:10:16] 10Analytics, 10User-Elukey: Upgrade Druid to its latest upstream version (currently 0.17) - https://phabricator.wikimedia.org/T244482 (10elukey) [10:10:57] 10Analytics, 10User-Elukey: Upgrade Druid to its latest upstream version (currently 0.17.1) - https://phabricator.wikimedia.org/T244482 (10elukey) [10:14:55] * elukey errand + lunch! [12:26:29] 10Analytics, 10Analytics-SWAP, 10Product-Analytics, 10Patch-For-Review, 10User-Elukey: Upgrade all SWAP users to JupyterLab 1.0 - https://phabricator.wikimedia.org/T230724 (10elukey) If everybody agrees I'd call this task done. Migrating all users to jupyterlab 1.0 will be probably handled when we'll upg... [12:31:52] 10Analytics, 10User-Elukey: Standard partman recipe for druid hosts - https://phabricator.wikimedia.org/T245810 (10elukey) 05Open→03Resolved a:03elukey Closing since it is now done. Will re-open in case it doesn't work :) [12:58:10] 10Analytics: Add TLS to Kafka Mirror Maker - https://phabricator.wikimedia.org/T250250 (10Ottomata) > on Logging, pulling data from each other Is this true? I'm not so sure, I thought they used ElasticSearch replication instead of Kafka replication. [12:59:05] 10Analytics: Add TLS to Kafka Mirror Maker - https://phabricator.wikimedia.org/T250250 (10elukey) >>! In T250250#6058642, @Ottomata wrote: >> on Logging, pulling data from each other > Is this true? I'm not so sure, I thought they used ElasticSearch replication instead of Kafka replication. Didn't know it, rem... [12:59:14] 10Analytics: Add TLS to Kafka Mirror Maker - https://phabricator.wikimedia.org/T250250 (10elukey) [13:01:46] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Users having issues with presto sqllab on superset - https://phabricator.wikimedia.org/T249923 (10elukey) List of permissions for the `sqllab` role in Superset: ` can search queries on Superset, can sql json on Superset, can csv on Superset, menu access... [13:05:55] I don't get --^ [13:06:03] I mean, on Alpha I see some permissions for sqllab [13:06:25] what is the reasoning behind having that split? [13:06:28] so confusing [13:06:34] I'll open a task to upstream [13:15:20] https://github.com/apache/incubator-superset/issues/9543 [13:19:05] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Users having issues with presto sqllab on superset - https://phabricator.wikimedia.org/T249923 (10elukey) Opened the following to upstream https://github.com/apache/incubator-superset/issues/9543 [13:30:01] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230 (10Ottomata) In https://phabricator.wikimedia.org/T249261#6056644 I parked some thoughts about how to migrate SearchSatis... [13:54:20] (03CR) 10Mforns: ">> I don't expect this file to change anymore (significantly anyway)." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [13:55:53] (03CR) 10Ottomata: "> Patch Set 9:" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [13:58:03] (03CR) 10Mforns: [C: 03+1] "Code LGTM!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [14:02:12] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Users having issues with presto sqllab on superset - https://phabricator.wikimedia.org/T249923 (10elukey) In this case the problem seems to be reported by the JS console: `Failed to load resource: the server responded with a status of 401 (UNAUTHORIZED)`... [14:03:37] !log update Superset Alpha role perms with what stated in T249923#6058862 [14:03:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:03:43] T249923: Users having issues with presto sqllab on superset - https://phabricator.wikimedia.org/T249923 [14:39:24] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Users having issues with presto sqllab on superset - https://phabricator.wikimedia.org/T249923 (10elukey) @dr0ptp4kt I modified your permissions again, in theory now everybody should be able to use Presto without issues. Can you re-test and let me know if... [14:39:49] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Users having issues with presto sqllab on superset - https://phabricator.wikimedia.org/T249923 (10elukey) @Aklapper same thing for you :) [14:51:12] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10Nuria) [14:52:45] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Users having issues with presto sqllab on superset - https://phabricator.wikimedia.org/T249923 (10elukey) Created https://wikitech.wikimedia.org/wiki/Analytics/Systems/Presto#Presto_on_Superset [14:52:51] mforns: do you have time for a quick brain bounce before standup? [14:53:03] ottomata: sure [14:53:05] omw [14:56:37] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10Nuria) ping @srishakatux so she is subscribed to this ticket [15:00:19] (03PS3) 10Amire80: WIP Update for hive [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/579022 (https://phabricator.wikimedia.org/T223958) [15:00:36] a-team: sorry, I'm out getting groceries so I won't be in standup, will send an escrum in a bit [15:00:50] (03PS4) 10Amire80: WIP Update for hive [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/579022 (https://phabricator.wikimedia.org/T223958) [15:02:14] a-team: have a tunning session mmeeting now [15:02:22] a-team: will miss standup [15:02:32] a-team: and send e-scrum [15:02:40] fdans: let's talk about numbro when you can [15:03:35] nuria: I'd also gladly get some time if you have - No problem if later :) [15:03:46] joal: ya, let me put it on calendar [15:04:03] nuria: can also be later this week [15:14:50] wow! https://github.com/wikimedia/operations-software-druid_exporter/pull/7 [15:17:36] actually mforns docs says: Operators that are executed on the worker need to have their dependencies met in that context. For example, if you use the HiveOperator, the hive CLI needs to be installed on that box, [15:17:40] Weird [15:28:20] wow cool! [15:28:22] elukey: ! [15:31:56] ottomata: ops sync? [15:32:15] oh ya [15:32:16] coming [15:36:07] joal: https://airflow.apache.org/docs/stable/_api/airflow/models/connection/index.html [15:39:45] elukey: just opened superset sqllab and everything seems to be working [15:52:45] fdans: yt? [15:53:00] nuria: helooo, yes [15:53:22] fdans: so, on numbro, can you repro the errors i see on your setup [15:53:55] fdans: namely, things not working on the prod build? [15:54:42] nuria: yeah I the default locale is not being loaded correctly [15:55:12] basically in the en locale we don't pass a numbro locale because it's the one that already ships with the library, so the language isn't present in the languages dir [15:55:23] (which I think is a mistake, but whatever) [15:55:47] what I'm not sure it's why it was working for me when I pushed it [15:56:38] fdans: cause your browser defualt languge is not english? [15:57:28] nuria: hmmm it is [15:57:35] en_US? [15:57:36] wait it isn't [15:57:39] fdans: en-US? [15:57:44] fdans: or en-GB? [15:57:51] fdans: ya, that is why [15:58:13] nuria: more like: https://usercontent.irccloud-cdn.com/file/loPPRapN/%E3%82%B9%E3%82%AF%E3%83%AA%E3%83%BC%E3%83%B3%E3%82%B7%E3%83%A7%E3%83%83%E3%83%88%202020-04-15%20%E5%8D%88%E5%BE%8C5.57.42.png [15:58:37] 😑 [15:58:48] fdans: ok, ya, there are different ways to fix this, i cannot take another look until later but yeah, we need a default [15:59:09] nuria: nono there's already a way [15:59:22] it's just that the code doesn't glue it correctly [15:59:26] fdans: k, if you push apatch I can CR later on tonite [15:59:32] yep! [16:03:04] 10Analytics, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Multi-Content-Revisions (New Features), 10User-ArielGlenn: MCR: Import all slots from XML dumps - https://phabricator.wikimedia.org/T220525 (10daniel) Putting this con the CPT clinic duty board as... [16:03:32] lexnasser: thanks! [16:04:21] 10Analytics, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Multi-Content-Revisions (New Features), 10User-ArielGlenn: MCR: Import all slots from XML dumps - https://phabricator.wikimedia.org/T220525 (10Pchelolo) a:05Pchelolo→03None [16:05:18] * elukey off! [16:47:54] 10Analytics, 10CPT Initiatives (MCR Schema Migration), 10Core Platform Team Workboards (Clinic Duty Team), 10Multi-Content-Revisions (New Features), 10User-ArielGlenn: MCR: Import all slots from XML dumps - https://phabricator.wikimedia.org/T220525 (10Pchelolo) a:03CCicalese_WMF [16:59:05] (03CR) 10Joal: "Minor comments :)" (034 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [17:00:08] Yay, nuria , https://wikimediafoundation.org/covid19/data/ is the article I wanted someone to write :) [17:13:33] addshore: credit to kzeta 's team [17:15:09] Research I guess? :) [17:15:27] addshore: that was comms and the data analysts i think [17:15:49] So many teams [17:15:52] addshore: now, it is a bit sad that that is not written as a blogpost [17:16:28] I still plan on writing mine :) [17:17:07] Thank you! It was a group effort; couldn't have done it without the public APIs and collaboration between my team (Product Analytics; primarily snowick, mayakpwiki, and cchen_), dsaez on Research, and the Comms team [17:18:08] it might be interesting to write a behind-the-scenes blogpost on the technical processes behind all of this [17:22:13] Id love to know the technical bits [17:22:47] I wrote a notebook a few weeks ago grabbing the same data (basically), starting with 3 Wikidata statements, and expanding from that [17:23:14] Was aiming to write it up into a blog post in the coming week [17:44:05] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230 (10Ottomata) After brain bouncing this with Marcel today we found a way to exclude analytics/legacy schemas from the robu... [17:46:56] (03CR) 10Ottomata: Unify Refine transform functions and add user agent parser transform (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [18:10:31] joal: the configs are set in the UI and are stored directly in the airflow db! [18:11:53] mforns: maaaaaan - I hope elukey doesn't find about that ;) [18:14:49] (03CR) 10Joal: Unify Refine transform functions and add user agent parser transform (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [18:19:53] fdans: can you follow up here on whether we need to ingest from event or event_sanitized? event will only have the last 90 days of data, in teh case of ingesting from events_snitized we probably need to adjust the patch a bit [18:20:11] cc fdans and kaldari [18:26:46] (03PS5) 10Lex Nasser: Configure geoeditors monthly public Oozie job to work with geoeditors public monthly table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/576618 (https://phabricator.wikimedia.org/T244597) [18:43:53] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review, 10Wikimedia-production-error: EventLogging-based extensions cause errors on test2.wikipedia.org - https://phabricator.wikimedia.org/T196309 (10Krinkle) 05Open→03Resolved a:03Krinkle [18:48:25] (03CR) 10Lex Nasser: "This patch has been tested and successful. It has been modified in accordance with Mforns' CR. Also, accidental changes to oozie/cassandra" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/576618 (https://phabricator.wikimedia.org/T244597) (owner: 10Lex Nasser) [18:52:06] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230 (10Ottomata) > B. Each EL Schema will have a single corresponding stream, and each stream will be made up of (currently)... [19:20:47] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230 (10Ottomata) Alright, the changes I'm making today make the migration much simpler. By keeping the Kafka topics the same... [19:21:11] (03PS5) 10Joal: Update hive geocoded-data udf [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 [19:21:27] ottomata: if you want to have a look --^ [19:23:28] (03CR) 10Ottomata: Unify Refine transform functions and add user agent parser transform (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) (owner: 10Ottomata) [19:40:35] (03PS10) 10Ottomata: Unify Refine transform functions and add user agent parser transform [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/586447 (https://phabricator.wikimedia.org/T238230) [19:42:06] (03CR) 10Joal: "Tested on cluster with Spark and hive" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 (owner: 10Joal) [19:46:24] joal: i like that much better! [19:46:26] 2 thoughts [19:47:02] is there some way the impl classes could just override the database props + paths instead of implementing getReader? [19:47:11] then getReader could be concrete in parent class [19:47:43] other thought: maybe the AbstractDatabaseReader should have a factory method to do the singleton thing? [19:47:51] then maybe one could call [19:48:19] GeocodeDatabaseReader.getInstance().getResponse(ip) [19:49:00] then the only thing the impl clases like GeocodeDatabaseReader would have to implement is setting (or getting?) the database _prop and default path [19:49:19] maybe just implement a getDatabasePath abstract function? [19:49:25] instead of getReader? [19:50:02] also, should getResponse be abstract too? [19:50:06] so it is forced to be implemented? [19:57:06] 10Analytics, 10Analytics-Kanban, 10Pageviews-API: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10Milimetric) Well, if they're identical, then the speed of method 2 seems to me too nice to ignore. The only downside there is that as new characters... [19:57:28] ok will post on path [19:57:29] patch [20:03:49] (03CR) 10Ottomata: "I am liking this better!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 (owner: 10Joal) [20:33:57] 10Analytics: Superset: "Error while fetching database list" - https://phabricator.wikimedia.org/T249825 (10Milimetric) hm, according to https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Data_Dashboards._Superset_and_Turnilo_(previously_called_Pivot) you should have access with your Wikitech username and... [20:42:51] 10Analytics, 10Analytics-Kanban, 10Pageviews-API: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10Nuria) @lexnasser I think as a general paradigm i agree with a do-not-allow list being way easier to maintain than an "allowed" list, let's test metho... [20:45:10] 10Analytics, 10Analytics-Kanban, 10Pageviews-API: Pageviews missing for titles with emojis since April 23, 2019 - https://phabricator.wikimedia.org/T245468 (10Nuria) @lexnasser let's also test the "bad" pageview titles that should be excluded and that motivated @awight 's changes on 1st place [20:59:47] 10Analytics, 10Performance-Team, 10Readers-Web-Backlog (Tracking): Review referer configuration of origin/origin-when-crossorigin/origin-when-cross-origin - https://phabricator.wikimedia.org/T248526 (10Jdlrobson) [21:15:49] (03CR) 10Nuria: "Overall I think refactor is fine, i think however the code we added to the UDFs needs a bit of work" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 (owner: 10Joal) [21:17:11] (03CR) 10Nuria: Update hive geocoded-data udf (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 (owner: 10Joal) [21:18:09] (03CR) 10Ottomata: Update hive geocoded-data udf (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 (owner: 10Joal) [21:34:18] 10Analytics, 10Product-Analytics: SQL definition for wikidata metrics for tunning session - https://phabricator.wikimedia.org/T247099 (10jwang) [21:35:04] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: SQL definition for structure data in commons metrics - https://phabricator.wikimedia.org/T247101 (10jwang) [21:35:24] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Users having issues with presto sqllab on superset - https://phabricator.wikimedia.org/T249923 (10dr0ptp4kt) Yep, that works. [21:36:05] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Epic, 10Patch-For-Review: SQL definition for structure data in commons metrics - https://phabricator.wikimedia.org/T247101 (10jwang) [21:37:09] 10Analytics, 10Product-Analytics, 10Epic: SQL definition for wikidata metrics for tunning session - https://phabricator.wikimedia.org/T247099 (10jwang) [21:39:13] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Epic: Tech Tunning Session metrics - https://phabricator.wikimedia.org/T247100 (10jwang) [21:41:37] (03CR) 10Nuria: Update hive geocoded-data udf (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/588715 (owner: 10Joal)