[07:05:47] 10Analytics, 10Product-Analytics, 10Research, 10WMDE-Analytics-Engineering, and 3 others: Provide tools for querying MediaWiki replica databases without having to specify the shard - https://phabricator.wikimedia.org/T212386 (10elukey) As FYI to everybody, with Thursday's Analytics Refinery deployment we'l... [07:06:18] Hello people! [07:06:25] as FYI dbstore1002 is read-only [07:06:42] the procedure to dump staging is started, and it will likely finish tomorrow morning [07:07:01] Cc: Amir1,addshore --^ [08:06:30] good morning team! [08:08:12] o/ [09:04:09] 10Analytics, 10ExternalGuidance, 10Product-Analytics, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review: Measure the impact of externally-originated contributions - https://phabricator.wikimedia.org/T212414 (10santhosh) @chelsyx I still don't see a table for ExternalGuidance in db1108 log s... [09:11:18] elukey: thanks [10:21:11] 10Analytics, 10Operations, 10hardware-requests: GPU upgrade for stat1005 - https://phabricator.wikimedia.org/T216226 (10elukey) Thanks all for all the detailed info! One thought: I found this interesting use case https://www.amd.com/en/case-studies/school-42 among the case studies in the AMD website, that s... [10:24:21] (03CR) 10Joal: [V: 03+2 C: 03+1] Correct sqoop script for change_tag [analytics/refinery] - 10https://gerrit.wikimedia.org/r/490828 (https://phabricator.wikimedia.org/T205940) (owner: 10Joal) [10:55:31] (03PS4) 10Joal: [FUN] AQS for druid only [analytics/aqs] - 10https://gerrit.wikimedia.org/r/384113 [11:14:43] now that I think about it joal, we need to migrate AQS to nodejs 10! [11:14:46] (bonjour) [11:15:04] Bonjour elukey :0 [11:15:11] elukey: :)) [11:15:54] elukey: Sounds good to me - I thought we had planned for that? Must have been on Dan's list - Maybe I should take it? [11:16:32] it should be on Dan's list but I haven't heard any discussion about it recently, we'll need to do it as early goal for next Q (I think) [11:16:38] the deadline is first of april [11:17:04] OoooooK - I assume our JS belover fdans will probably be part of the game :) [11:17:04] I resumed a patch for puppet to allow the deployment of nodejs 10 [11:17:24] need to ping the sre services team to see if they like it [11:17:35] o/ [11:24:02] elukey: how should we proceed? [11:24:11] elukey: I assume we have a prefered 10 version [11:29:03] joal: so in theory if https://gerrit.wikimedia.org/r/#/c/477475/ is accepted, there is a way via puppet to deploy the nodejs 10 environment selectively. The idea is to enable it in cloud/labs, test and then move to rpdo [11:29:06] *prod [11:29:08] how does it sound? [11:29:50] elukey: all good for me - I also think texting locally first will be a necessary option :) [11:41:15] yep yep! [11:50:23] ah wait we already have it in Kanban [11:50:24] https://phabricator.wikimedia.org/T210706 [11:50:31] so probably it is my fault [11:50:33] argh [11:50:37] going to update it [11:54:11] ahhhh snap [11:54:21] the labs instances are jessie [11:54:24] we need stretch [11:55:26] 10Analytics, 10Analytics-Kanban: Move AQS to nodejs 10 - https://phabricator.wikimedia.org/T210706 (10elukey) Sorryyyy just realized that this task has been sitting here due to me!!! So the plan should be the following: * review/merge https://gerrit.wikimedia.org/r/#/c/477475/ (rebased today, going to ask to... [12:52:05] ok wonderful I need to re-create the aqs cluster in labs first too [12:52:31] * elukey lunch! [12:56:06] elukey: I have a patch for you in ops-chan for when you're back [13:11:29] 10Analytics: Check home leftovers of dartar - https://phabricator.wikimedia.org/T216410 (10MoritzMuehlenhoff) [13:46:24] (03PS1) 10Joal: Add mediawiki_wikitext_history to drop-script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491252 [14:01:44] (03CR) 10Joal: [V: 03+1] "Tested on cluster using dry-run" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491252 (owner: 10Joal) [14:07:59] 10Analytics, 10Analytics-Kanban: Purge wikitext snapshots - https://phabricator.wikimedia.org/T216414 (10JAllemandou) [14:09:07] 10Analytics, 10Analytics-Kanban: Purge wikitext snapshots - https://phabricator.wikimedia.org/T216414 (10JAllemandou) Taking advantage of currently existing strategy and job for mediawiki-oriented snapshots, I have provided a patch alowing to keep 6 parquet snapshots. It's probably more than needed, but preven... [14:09:59] 10Analytics, 10Analytics-Kanban: Purge wikitext snapshots - https://phabricator.wikimedia.org/T216414 (10JAllemandou) [14:10:25] (03PS2) 10Joal: Add mediawiki_wikitext_history to drop-script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491252 [14:10:46] 10Analytics, 10Analytics-Kanban: Purge wikitext snapshots - https://phabricator.wikimedia.org/T216414 (10JAllemandou) a:03JAllemandou [14:12:31] 10Analytics, 10Analytics-Kanban: Purge wikitext snapshots - https://phabricator.wikimedia.org/T216414 (10JAllemandou) Strategy for deleting raw data would be to use the new `refinery-drop-older-than` script: ` refinery-drop-older-than \ --base-path=/wmf/data/raw/mediawiki/xmldumps/pages_meta_history \... [15:22:31] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10elukey) Thanks to Moritz we have buster back on stat1005. I did the following: * Added `radeon.cik_support=0 amdgpu.cik_support=1` to grub.cfg... [15:31:35] joal: o/ [15:31:52] if https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/491246/ it is not urgent let's wait for milimetric to review the parameters, just to avoid typos etc.. [15:32:14] I am going to verify all of them, not because I don't trust you but because it is so easy to flip two of them and make a mess [15:32:22] would it work for you? [15:36:56] elukey: PLEASE :) [15:37:19] elukey: thank you for that :) [15:40:58] also joal I nuked the aqs cluster in deployment-prep, re-creating it with stretch [15:41:30] Thanks also for that elukey :) [15:48:18] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10elukey) Very promising: https://github.com/RadeonOpenCompute/ROCm/issues/702#issuecomment-461982554 > As noted in #691 and #640, Hawaii GPUs... [16:00:13] 10Analytics, 10ExternalGuidance, 10Product-Analytics, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review: Measure the impact of externally-originated contributions - https://phabricator.wikimedia.org/T212414 (10Nuria) @santhosh tables are created in hadoop, not mysql. [16:01:00] ping fdans joal standdduppp [16:07:55] 10Analytics, 10Operations, 10Research, 10serviceops, and 4 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10akosiaris) For the record, just saying pointing out that the question of a new VM versus mwmaint1002 is probably irrelevant here. We c... [16:14:18] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Contributors-Analysis, and 2 others: Add change tag tables to monthly mediawiki_history sqoop - https://phabricator.wikimedia.org/T205940 (10Nuria) a:05fdans→03JAllemandou [16:24:02] 10Analytics, 10Analytics-Kanban: Purge wikitext snapshots - https://phabricator.wikimedia.org/T216414 (10fdans) p:05Triage→03High [16:24:57] 10Analytics: Check home leftovers of dartar - https://phabricator.wikimedia.org/T216410 (10fdans) p:05Triage→03Normal [16:27:16] 10Analytics, 10Analytics-Data-Quality, 10Tool-Pageviews: Anomalous statistics results in eu.wikipedia siteviews - https://phabricator.wikimedia.org/T212879 (10fdans) @Theklan can you send us some url examples for us to understand? Thank you! [16:31:00] 10Analytics, 10DBA, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10JAllemandou) Hi @Isaac, I have generated some parquet data here `/user/joal/wmf/data/wmf/wikidata/item_page_link/20190204` with th... [16:32:03] 10Analytics, 10Operations, 10Wikimedia-Stream, 10Services (watching): Eventstreams build is broken - https://phabricator.wikimedia.org/T216184 (10fdans) [16:34:17] 10Analytics, 10Analytics-Kanban: yearly labels in wikistats say 2017 - https://phabricator.wikimedia.org/T216105 (10fdans) p:05Triage→03High [16:34:37] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10Cmjohnson) @elukey, it will affect how it's rack...10G racks have different switches but we are also limited in space for those racks. If 1... [16:36:45] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10elukey) >>! In T215231#4961942, @Cmjohnson wrote: > @elukey, it will affect how it's rack...10G racks have different switches but we are also... [16:37:33] 10Analytics, 10Analytics-Wikistats: Year total(2018) legend in the charts is misleading - https://phabricator.wikimedia.org/T216104 (10fdans) Yeah it should say "last 12 months" [16:37:54] 10Analytics, 10Analytics-Wikistats: Year total(2018) legend in the charts is misleading - https://phabricator.wikimedia.org/T216104 (10fdans) p:05Triage→03High [16:39:51] 10Analytics, 10Analytics-Data-Quality, 10Tool-Pageviews: Anomalous statistics results in eu.wikipedia siteviews - https://phabricator.wikimedia.org/T212879 (10Theklan) There you have some very absurd pageviews: https://tools.wmflabs.org/siteviews/?platform=all-access&source=pageviews&agent=user&start=2018-12... [16:40:18] 10Analytics, 10Analytics-Kanban, 10Discovery-Search (Current work): Spike. Load search data into turnilo to test whether exploratory data can do away with some of the dashboards - https://phabricator.wikimedia.org/T216058 (10fdans) a:03Nuria [16:40:30] 10Analytics, 10Analytics-Kanban, 10Discovery-Search (Current work): Spike. Load search data into turnilo to test whether exploratory data can do away with some of the dashboards - https://phabricator.wikimedia.org/T216058 (10fdans) p:05Triage→03High [16:41:48] 10Analytics, 10Operations, 10RESTBase, 10Traffic, and 2 others: Verify that hit/miss stats in WebRequest are correct - https://phabricator.wikimedia.org/T215987 (10fdans) [16:42:33] 10Analytics, 10Operations, 10RESTBase, 10Traffic, and 2 others: Verify that hit/miss stats in WebRequest are correct - https://phabricator.wikimedia.org/T215987 (10fdans) @BBlack do you have any concerns related to the hit/miss data sent to webrequest? [16:42:52] 10Analytics, 10Operations, 10RESTBase, 10Traffic, and 2 others: Verify that hit/miss stats in WebRequest are correct - https://phabricator.wikimedia.org/T215987 (10fdans) a:05JAllemandou→03None [16:49:30] 10Analytics, 10Data-Services: Rethink Cloud DB replicas - https://phabricator.wikimedia.org/T215858 (10fdans) p:05Triage→03Normal [16:52:48] 10Analytics, 10Analytics-Kanban, 10Operations, 10hardware-requests: GPU upgrade for stat1005 - https://phabricator.wikimedia.org/T216226 (10fdans) [16:52:55] 10Analytics, 10Analytics-Kanban, 10Operations, 10hardware-requests: GPU upgrade for stat1005 - https://phabricator.wikimedia.org/T216226 (10fdans) p:05Normal→03High [16:57:45] 10Analytics: Upgrade to Spark 2.4.0 - https://phabricator.wikimedia.org/T215043 (10fdans) [16:58:56] 10Analytics, 10Analytics-Cluster, 10Operations, 10Traffic: Respect X-Forwarded-For only from trustworthy sources - https://phabricator.wikimedia.org/T56783 (10fdans) [16:59:37] 10Analytics, 10Analytics-Cluster, 10Operations, 10Traffic: Respect X-Forwarded-For only from trustworthy sources - https://phabricator.wikimedia.org/T56783 (10fdans) @BBlack is this task finished? [17:20:37] 10Analytics, 10Operations, 10Research, 10serviceops, and 4 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10Nuria) >Should we btw stall this on T213976? yes, we need to resolve first where/how are binarie/data files s going to be moved to the... [17:26:49] 10Analytics, 10Scoring-platform-team: [Discuss] ORES model development and deployment processes - https://phabricator.wikimedia.org/T216246 (10Nuria) As we mentioned earlier, stats machines are not to be used to deploy to prod. There are models being trained in hadoop right now but as you said that process nee... [17:27:03] 10Analytics, 10Discovery, 10Operations, 10Research: Workflow to be able to move data files computed in jobs from analytics cluster to production - https://phabricator.wikimedia.org/T213976 (10jcrespo) We use [[ https://phabricator.wikimedia.org/T156462 | transfer.py ]] to transfer up to 12TB of data for da... [17:29:44] 10Analytics, 10Discovery, 10Operations, 10Research: Workflow to be able to move data files computed in jobs from analytics cluster to production - https://phabricator.wikimedia.org/T213976 (10Nuria) @jcrespo: have in mind that this is not only for data destined to mysql (although this is the particular ca... [17:35:46] 10Analytics, 10Discovery, 10Operations, 10Research: Workflow to be able to move data files computed in jobs from analytics cluster to production - https://phabricator.wikimedia.org/T213976 (10jcrespo) transfer.py works for: * Plain files from filesystem to filesystem * Online mysql/mariaDB databases It w... [17:36:31] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Move AQS to nodejs 10 - https://phabricator.wikimedia.org/T210706 (10elukey) Created the new cluster in deployment-prep: ` elukey@deployment-aqs01:~$ nodetool status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Jo... [17:38:28] 10Analytics, 10Discovery, 10Operations, 10Research: Workflow to be able to move data files computed in jobs from analytics cluster to production - https://phabricator.wikimedia.org/T213976 (10Nuria) @jcrespo seems something worth considering, I leave up to @fgiunchedi @Ottomata and @akosiaris to see if tra... [18:02:14] 10Analytics, 10Dumps-Generation, 10Wikidata: Update wikidata-entities dump generation to fixed day-of-month instead of fixed weekday - https://phabricator.wikimedia.org/T216160 (10Nicolastorzec) Hi Ariel et al., I don't think switching data dump generation from a pattern like "every Monday" to a pattern li... [18:02:42] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Move AQS to nodejs 10 - https://phabricator.wikimedia.org/T210706 (10elukey) So next steps: 1) Add some data to Cassandra in deployment prep (deployment-aqs0[1,2,3].deployment-prep.eqiad.wmflabs) 2) Verify that AQS works as expected 3) Add `profile::aqs::... [18:03:27] ok so the new aqs cluster in deployment-prep is ready [18:03:33] added all the next steps in the task [18:03:40] (for whoever is interested) [18:12:43] (03PS1) 10Framawiki: view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) [18:13:05] (03CR) 10jerkins-bot: [V: 04-1] view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) (owner: 10Framawiki) [18:13:34] 10Quarry, 10Patch-For-Review: Show query run date above outputs section - https://phabricator.wikimedia.org/T215831 (10Framawiki) a:03Framawiki [18:15:23] 10Analytics, 10Analytics-Kanban: Purge wikitext snapshots - https://phabricator.wikimedia.org/T216414 (10mforns) @JAllemandou You need to run the script once without the --execute flag (dry run). The checksum will be printed by the script at the end. To *really* execute the script, run it again with the same p... [18:26:36] (03CR) 10Zhuyifei1999: view.js: Show query run date above outputs section (031 comment) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) (owner: 10Framawiki) [18:46:29] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10elukey) Tried to purge rocm-dev 2.1 and install 1.9.2, same problem: ` [ 90.690958] BUG: unable to handle kernel NULL pointer dereference at... [18:52:35] * elukey failed another GPU attempt sigh [18:52:43] going off, see you tomorrow! [18:54:46] (03CR) 10Mforns: [C: 03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491252 (owner: 10Joal) [19:04:22] mforns: yt? [19:04:30] yep nuria [19:04:43] mforns: in order to specify transforms on the el2druid [19:05:11] mforns: do i do it directly on the dimensions line? [19:05:40] nuria, no, there's a specific syntax for transforms, they have their own --transforms arg [19:06:01] nuria, but THEN you also have to list them either as dimensions or metrics [19:06:06] ex: [19:06:13] mforns: k [19:06:43] --transforms "event.some_value / count as normalized_value" [19:06:53] --metrics "normalized_value" [19:07:12] nuria, ^ [19:08:01] mforns: i see, but a transform is not necessarily a metric , makes sense? could be transform (0,1) to (true, false) [19:08:03] if you specify more than 1 transform, you should separate them with semicolons (because druid expressions can contain commas) [19:08:21] yes, you can make them dimensions too [19:08:32] mforns: and such a mapping would exist per datapoint [19:08:34] mforns: ok [19:09:14] yes, per record, transforms are applied before Druid's rollup [19:15:06] mforns: in your experience: did reindexing data while adding anew dimension worked? [19:16:27] nuria, I remember having succeeded with that at some point [19:16:35] mforns: jaja [19:16:40] hehe [19:16:47] mforns: that sounds like ... ya, maybe ... [19:17:07] I also remember getting errors with doing loading with schema changes [19:17:19] but I think it should work if it is only additions [19:17:32] without type changes or renames [19:31:26] mforns: k, trying [19:33:49] (03CR) 10Framawiki: view.js: Show query run date above outputs section (031 comment) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) (owner: 10Framawiki) [19:33:55] (03PS2) 10Framawiki: view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) [19:34:16] (03CR) 10jerkins-bot: [V: 04-1] view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) (owner: 10Framawiki) [19:36:29] mforns: ok, adding new transformed dimension did work [19:36:37] yay! [19:40:22] (03PS3) 10Framawiki: view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) [19:40:48] (03CR) 10jerkins-bot: [V: 04-1] view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) (owner: 10Framawiki) [19:59:28] 10Quarry, 10Documentation: Example queries for Quarry - https://phabricator.wikimedia.org/T207098 (10Framawiki) 05Stalled→03Invalid There can be no example of use in Quarry access to new data is open, and the use of new servers is made in the tool. I would like to close this stain in this sense, for now. [20:02:36] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikimetrics: Sunset Wikimetrics - https://phabricator.wikimedia.org/T211835 (10mforns) [20:14:22] (03PS4) 10Framawiki: view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) [20:14:44] (03CR) 10jerkins-bot: [V: 04-1] view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) (owner: 10Framawiki) [20:21:31] (03PS5) 10Framawiki: view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) [20:22:11] (03CR) 10jerkins-bot: [V: 04-1] view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) (owner: 10Framawiki) [20:23:29] nuria, I'm trying to access Wikimetric's mysql local database, but can't find how in the docs, and I don't remember... [20:23:35] nuria, do you recall? [20:30:05] (03PS6) 10Framawiki: view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) [20:41:13] (03PS7) 10Framawiki: view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) [20:59:47] (03CR) 10Zhuyifei1999: [C: 03+1] view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) (owner: 10Framawiki) [21:12:41] (03CR) 10Framawiki: [C: 03+2] view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) (owner: 10Framawiki) [21:13:18] (03Merged) 10jenkins-bot: view.js: Show query run date above outputs section [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491284 (https://phabricator.wikimedia.org/T215831) (owner: 10Framawiki) [21:15:26] 10Quarry, 10Patch-For-Review: Show query run date above outputs section - https://phabricator.wikimedia.org/T215831 (10Framawiki) 05Open→03Resolved [21:38:19] fdans: yt? [22:43:03] 10Quarry, 10Patch-For-Review: Show query run date above outputs section - https://phabricator.wikimedia.org/T215831 (10Framawiki) 05Resolved→03Open [22:43:13] 10Quarry: Show query run date above outputs section - https://phabricator.wikimedia.org/T215831 (10Framawiki) [23:57:41] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Performance-Team (Radar): [Bug] Type mismatch between NavigationTiming EL schema and Hive table schema - https://phabricator.wikimedia.org/T214384 (10mforns) Hey all :] Just finished to apply the fix plan. The current status of event.NavigationTiming...