[03:06:21] 10Analytics, 10Cleanup, 10Editing-team: Deletion of limn-edit-data repository - https://phabricator.wikimedia.org/T228982 (10TheSandDoctor) >>! In T228982#5488055, @fdans wrote: > @TheSandDoctor nope. This task is regarding the deletion of https://github.com/wikimedia/analytics-limn-edit-data > > The task y... [05:52:02] morningggg [05:52:03] o/ [05:52:41] first question of the morning - today the SRE team needs to stop mysql on labsdb1012 for a bit to allow rack maintenance [05:52:58] I think it is completely fine since we don't rely on it at this time of the month [05:53:02] but please confirm :) [06:18:03] (03PS1) 10Elukey: Force execution of all the (python) scripts under bin/ with python3 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537255 (https://phabricator.wikimedia.org/T204735) [06:27:35] 10Analytics-Kanban: Test if Hue can run with Python3 - https://phabricator.wikimedia.org/T233073 (10elukey) [06:34:17] 10Analytics-Kanban: Test if Hue can run with Python3 - https://phabricator.wikimedia.org/T233073 (10elukey) [06:37:13] 10Analytics-Kanban: Test if Hue can run with Python3 - https://phabricator.wikimedia.org/T233073 (10elukey) [06:41:22] 10Analytics-Kanban: Test if Hue can run with Python3 - https://phabricator.wikimedia.org/T233073 (10elukey) Opened https://github.com/cloudera/hue/issues/968 [06:41:34] opened a github issue to cloudera, let's see if the answer [06:48:50] (03PS1) 10Elukey: Move reportupdater_test5 to python3 [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/537267 (https://phabricator.wikimedia.org/T204736) [06:54:46] (03PS1) 10Elukey: Move codebase to python3 [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/537268 (https://phabricator.wikimedia.org/T204736) [06:55:11] (03CR) 10jerkins-bot: [V: 04-1] Move codebase to python3 [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/537268 (https://phabricator.wikimedia.org/T204736) (owner: 10Elukey) [06:55:50] ahhaha of course! tox runs python 2.7 [06:55:52] :p [07:01:33] good morning elukey :) [07:01:42] elukey: no prob for me with the mysql box [07:06:08] (03PS2) 10Elukey: [WIP] Move codebase to python3 [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/537268 (https://phabricator.wikimedia.org/T204736) [07:06:13] (03Abandoned) 10Elukey: Move reportupdater_test5 to python3 [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/537267 (https://phabricator.wikimedia.org/T204736) (owner: 10Elukey) [07:07:54] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Move codebase to python3 [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/537268 (https://phabricator.wikimedia.org/T204736) (owner: 10Elukey) [07:17:48] (03CR) 10Joal: "I'll release the artifact manually to archiva" [analytics/ua-parser/uap-java] (wmf) - 10https://gerrit.wikimedia.org/r/536676 (https://phabricator.wikimedia.org/T212854) (owner: 10Joal) [07:20:27] (03CR) 10Joal: [C: 03+1] "Assuming some tests got run it looks good :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537255 (https://phabricator.wikimedia.org/T204735) (owner: 10Elukey) [07:33:50] 10Analytics, 10Patch-For-Review: Move Analytics Report Updater to Python 3 - https://phabricator.wikimedia.org/T204736 (10elukey) This is a bit harder than expected, since tests will need some adjustment to: 1) run tox with python 3.7 in jenkins 2) encode/decode properly strings to avoid failures https://ger... [07:33:56] 10Analytics, 10Patch-For-Review: Move Analytics Report Updater to Python 3 - https://phabricator.wikimedia.org/T204736 (10elukey) p:05Low→03Normal [07:35:50] (03CR) 10Elukey: "I did check the code and ran the 2to3 tool on all files, it seems that the existing code should work with python3 as it is. We might get " [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537255 (https://phabricator.wikimedia.org/T204735) (owner: 10Elukey) [07:42:19] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for release" [analytics/ua-parser/uap-java] (wmf) - 10https://gerrit.wikimedia.org/r/536676 (https://phabricator.wikimedia.org/T212854) (owner: 10Joal) [07:42:24] joal: going to stop superset and reboot the vm [07:42:30] Ack! [07:42:51] (03CR) 10Joal: "This is the only one we have discussed yes :)" [analytics/ua-parser/uap-java] (wmf) - 10https://gerrit.wikimedia.org/r/536677 (https://phabricator.wikimedia.org/T212854) (owner: 10Joal) [07:43:02] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for release" [analytics/ua-parser/uap-java] (wmf) - 10https://gerrit.wikimedia.org/r/536677 (https://phabricator.wikimedia.org/T212854) (owner: 10Joal) [07:43:41] (03CR) 10Joal: "Yes, release by hand :)" [analytics/ua-parser/uap-java] (wmf) - 10https://gerrit.wikimedia.org/r/537168 (https://phabricator.wikimedia.org/T212854) (owner: 10Joal) [07:43:52] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for release" [analytics/ua-parser/uap-java] (wmf) - 10https://gerrit.wikimedia.org/r/537168 (https://phabricator.wikimedia.org/T212854) (owner: 10Joal) [07:50:06] (03PS1) 10Joal: Update ua-parser to new version [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/537317 (https://phabricator.wikimedia.org/T212854) [07:50:47] !log Manually released com.github.ua-parser/uap-java 1.4.4-core0.6.9~1-wmf to archiva [07:50:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:51:19] Need to run errand for ~1h [08:19:39] !log manually decommed analytics1032 for hdfs/yarn on the Hadoop testing cluster - T233080 [08:19:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:19:43] T233080: Decommission analytics1032 - https://phabricator.wikimedia.org/T233080 [08:24:38] Is there a way to distinguish between http and https protocols in the Hadoop webrequest table? [08:34:34] awight: o/ in theory webrequests comes from the varnish frontends that are "proxied" by ngnix that in turn does TLS termination for us [08:34:50] so those should all be HTTPs [08:35:33] the only thing that could be recorded (and not sure if we do) are redirects like [08:35:36] curl -i http://en.wikipedia.org [08:35:39] HTTP/1.1 301 TLS Redirect [08:35:41] Date: Tue, 17 Sep 2019 08:35:05 GMT [08:35:44] Server: Varnish [08:37:02] anything in particular that you are looking for? [08:39:43] elukey: Thanks, that explains some things. I'm trying to tell why a cross-wiki API request is failing with 403, and my current top theory is that I'm making an http request rather than https. I see 301 redirects like you explained above, except for my POST request which fails. [08:41:05] But varnish is logging both http and https, right? It's only guaranteed to be https *after* varnish, when connecting to the nginx proxy if I understand correctly. [08:41:13] (need to run errand for ~1h, will follow up later sorry!) [08:41:21] No worries, thanks for the clues! [08:41:50] yes in theory this is correct, it is the varnish (frontend) that does the redirect, so it should be logged in webrequest [08:41:56] but will need to check in depth :) [08:45:18] Hi awight [08:45:29] By the way, I'm back team [08:50:26] awight: https or not is stored in x_analytics header: spark.sql("select (x_analytics like '%https=1%') as is_https, count(1) as reqs from wmf.webrequest where webrequest_source = 'text' and year =2019 and month = 9 and day = 16 and hour = 17 group by (x_analytics like '%https=1%') ").show(20, false) [08:50:45] +--------+---------+ [08:50:48] |is_https|reqs | [08:50:50] joal: hey! I've been benefitting from all the great Hadoop tables btw, they were indispensible in debugging. [08:50:51] +--------+---------+ [08:50:53] |true |357893783| [08:50:56] |false |8412730 | [08:50:58] +--------+---------+ [08:51:04] I'm glad to hear that :) [08:51:12] What are you currentl building? [08:52:11] Strange--I did "select * from webrequest ..." and all my rows have x_analytics_map=NULL [08:52:28] meh? [08:53:30] The feature is kinda neat, but currently broken ;-) -- https://phabricator.wikimedia.org/T228851 [08:54:09] awight: this query gave me the same exact result as the other one: spark.sql("select (x_analytics_map['https']=1) as is_https, count(1) as reqs from wmf.webrequest where webrequest_source = 'text' and year =2019 and month = 9 and day = 16 and hour = 17 group by (x_analytics_map['https']=1) ").show(20, false) [08:54:15] It's a little flourish where Extension:FileImporter can either edit the original source file to add a template, or delete it. [08:54:19] (i'm kinda reassured) [08:54:44] cool, maybe "select *" causes the map to disappear though? [08:54:51] lemme try your syntax [08:55:59] awight: have you specified webrequest_source = 'text'? [08:56:27] not in this particular query--oh! so there's "union" magic happening, I guess? [08:56:51] awight: It's not a magic union, rather a partition defined field (webrequest_source_ [08:57:10] webrequest_source can be 'upload' or 'text', depending on which caching cluster has been hit [08:57:38] My rows are all from the text cluster, so the extra condition can't hurt. [08:57:49] oh and by the way awight - I tried select * syntax - The response is really not nice due to parquet format not being good at providing all fields [08:58:31] awight: More than that, since webrequest_source is a partition, adding the clause means tghe request won't even read the upload data (less IOs, faster) [08:59:04] Thanks! I have my answer now, and happily it confirms the theory. Somewhere in my code I end up with a relative URL, and the rest is history ;-) [09:00:52] ;) [09:04:54] (03PS2) 10Fdans: Add aggregate mediarequests per referer endpoint [analytics/aqs] - 10https://gerrit.wikimedia.org/r/537114 (https://phabricator.wikimedia.org/T232857) [09:08:27] (03CR) 10Fdans: [V: 03+1] "Job has been successfully tested in beta" [analytics/aqs] - 10https://gerrit.wikimedia.org/r/537114 (https://phabricator.wikimedia.org/T232857) (owner: 10Fdans) [09:14:30] (03CR) 10Joal: [C: 04-1] "Missing grouping set in one hql file - except from that all good." (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/536646 (https://phabricator.wikimedia.org/T232858) (owner: 10Fdans) [09:15:09] joal: lend me your elvic sight [09:16:00] fdans: :) [09:21:34] (03CR) 10Joal: [C: 03+1] "Minor comments, looks ready to go to me!" (032 comments) [analytics/aqs] - 10https://gerrit.wikimedia.org/r/537114 (https://phabricator.wikimedia.org/T232857) (owner: 10Fdans) [09:22:33] (03CR) 10Joal: [C: 03+1] "> I think that this should be enough, but if you people prefer a more in depth set of tests I'll do it :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537255 (https://phabricator.wikimedia.org/T204735) (owner: 10Elukey) [09:27:31] ahhh TIL about the https=1 thing, nice :) [09:29:55] elukey: I knew we had that info, but couldn't remember when before actually looking for it :) [09:49:35] (03PS2) 10Fdans: Add daily and monthly jobs for mediarequests per referer [analytics/refinery] - 10https://gerrit.wikimedia.org/r/536646 (https://phabricator.wikimedia.org/T232858) [09:50:29] (03PS3) 10Fdans: Add daily and monthly jobs for mediarequests per referer [analytics/refinery] - 10https://gerrit.wikimedia.org/r/536646 (https://phabricator.wikimedia.org/T232858) [09:50:36] (03CR) 10Fdans: [V: 03+1] Add daily and monthly jobs for mediarequests per referer (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/536646 (https://phabricator.wikimedia.org/T232858) (owner: 10Fdans) [10:05:12] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for deployment tomorrow" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/536646 (https://phabricator.wikimedia.org/T232858) (owner: 10Fdans) [10:37:25] (03CR) 10Joal: "Comments to be discussed :)" (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/528504 (https://phabricator.wikimedia.org/T208612) (owner: 10Mforns) [10:45:50] 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: eqiad: 1 misc node for the Kerberos KDC service - https://phabricator.wikimedia.org/T227288 (10elukey) @RobH let me know if I can help with the host repurpose (also with the codfw one), I can take care of the DNS/puppet/DHCP/etc.. steps :) [11:00:11] (03CR) 10Joal: "Minimal nits :)" (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/530002 (https://phabricator.wikimedia.org/T208612) (owner: 10Mforns) [11:01:53] (03CR) 10Joal: "Adding a new comment: Should we restructure the data for the dumps? In particular, should we dump user, page and revision data separately?" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/528504 (https://phabricator.wikimedia.org/T208612) (owner: 10Mforns) [11:34:37] 10Analytics, 10Research: Check home leftovers of ISI researchers - https://phabricator.wikimedia.org/T215775 (10elukey) >>! In T215775#5477797, @elukey wrote: > Ping @leila @Isaac :) :) [11:34:54] 10Analytics: Check home leftovers of atgomez - https://phabricator.wikimedia.org/T232821 (10elukey) p:05Triage→03Normal [11:36:59] 10Analytics: Check home leftovers of atgomez - https://phabricator.wikimedia.org/T232821 (10elukey) 05Open→03Resolved Done! [11:57:23] * elukey afk for ~1h for an errand [13:24:29] (03CR) 10Mforns: Add spark job to create mediawiki history dumps (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/528504 (https://phabricator.wikimedia.org/T208612) (owner: 10Mforns) [13:38:16] (03CR) 10Mforns: "> Adding a new comment: Should we restructure the data for the dumps?" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/528504 (https://phabricator.wikimedia.org/T208612) (owner: 10Mforns) [13:42:50] (03CR) 10Joal: "Releasing multiple dumps could make format simpler, and data smaller when only page/user info is interesting. Maybe not worth the work. Le" (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/528504 (https://phabricator.wikimedia.org/T208612) (owner: 10Mforns) [13:53:33] Gone for kids - Back fro standup [13:57:38] 10Analytics, 10Research: Parse wikidumps and extract redirect information for 1 small wiki, romanian - https://phabricator.wikimedia.org/T232123 (10MGerlach) # Language dependent Redirect Codes We can extract the aliases for the redirect code from the corresponding dump in *siteinfo-namespaces.json.gz This c... [14:15:19] (03PS11) 10Mforns: Add Oozie job for mediawiki history dumps [analytics/refinery] - 10https://gerrit.wikimedia.org/r/530002 (https://phabricator.wikimedia.org/T208612) [14:16:26] (03CR) 10Mforns: "Thanks @joal for the review! I applied suggestions." (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/530002 (https://phabricator.wikimedia.org/T208612) (owner: 10Mforns) [14:26:54] PROBLEM - Check if active EventStreams endpoint is delivering messages. on icinga1001 is CRITICAL: CRITICAL: No EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [14:27:34] no bueno [14:27:52] but there is a migration of kafka1003 to kafka-main1003 ongoing [14:39:56] (SRE is handling it, ES should work now) [14:50:48] RECOVERY - Check if active EventStreams endpoint is delivering messages. on icinga1001 is OK: OK: An EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [14:58:07] (03CR) 10Nuria: Force execution of all the (python) scripts under bin/ with python3 (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537255 (https://phabricator.wikimedia.org/T204735) (owner: 10Elukey) [14:59:06] (03CR) 10Elukey: ">" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537255 (https://phabricator.wikimedia.org/T204735) (owner: 10Elukey) [14:59:35] just sent e-scrum but I'll join for staff! [15:01:18] (03CR) 10Nuria: [C: 03+2] Update ua-parser to new version [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/537317 (https://phabricator.wikimedia.org/T212854) (owner: 10Joal) [15:02:10] https://github.com/cloudera/hue/issues/552#issuecomment-532258907 - lovely [15:06:37] elukey: INDEED [15:06:52] (03Merged) 10jenkins-bot: Update ua-parser to new version [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/537317 (https://phabricator.wikimedia.org/T212854) (owner: 10Joal) [15:27:36] fun fact: hive can give you the exact location of a partition with `show table extended like ...`, but it wont be in a row of it's own, it will be mixed into a multi-line formatted string :S [15:30:08] (03PS9) 10Mforns: Add spark job to create mediawiki history dumps [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/528504 (https://phabricator.wikimedia.org/T208612) [15:36:00] (03CR) 10Mforns: ">> Releasing multiple dumps could make format simpler, and data smaller when only page/user info is interesting. Maybe not worth the work." (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/528504 (https://phabricator.wikimedia.org/T208612) (owner: 10Mforns) [15:58:46] a-team will miss standup; in OKR meetingg [16:06:01] elukey: I can't look at you in the eyes. I'm really sorry for the bad delay re the phab task. I'll get back to you about it this afternoon PST or tomorrow. [16:18:13] leila: it is super fine, please don't say that, I simply check once in a while in the phab tasks :) [16:26:15] elukey: thanks! it's unacceptable though. I'll try harder. ;) [17:23:50] * elukey off! [17:33:57] 10Analytics, 10Operations, 10Core Platform Team Legacy (Watching / External), 10Patch-For-Review, and 2 others: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10herron) [17:55:48] 10Analytics, 10Research: Parse wikidumps and extract redirect information for 1 small wiki, romanian - https://phabricator.wikimedia.org/T232123 (10JAllemandou) This is great finding! This file does not contain redirect only, but also every other aliases that might be usefull :) Awesome [18:10:08] 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access to analytics cluster for Martin Gerlach - https://phabricator.wikimedia.org/T232707 (10herron) [18:33:07] 10Analytics, 10Operations, 10User-Elukey: setup/install eqiad kerbos node WMF5173 - https://phabricator.wikimedia.org/T233141 (10RobH) [18:33:33] 10Analytics, 10Operations, 10User-Elukey: setup/install codfw kerbos node WMF6577 - https://phabricator.wikimedia.org/T233142 (10RobH) [18:37:41] 10Analytics, 10CheckUser, 10Core Platform Team: Refactor Comment fields for CheckUser Component - https://phabricator.wikimedia.org/T232531 (10Anomie) [18:37:46] pimg nuria [18:37:50] 10Analytics, 10CheckUser, 10Core Platform Team: Refactor Comment fields for CheckUser Component - https://phabricator.wikimedia.org/T232531 (10Anomie) Looks like T233004 is handling this already. Since there's more discussion on that on, I'm going to close this as the duplicate even though it's older. [18:38:01] joaL; sorry on meeting [18:38:10] no problem - waiting [18:38:23] 10Analytics, 10Operations, 10User-Elukey: setup/install codfw kerbos node WMF6577 - https://phabricator.wikimedia.org/T233142 (10RobH) a:05RobH→03elukey @elukey, Please note that both T233141 (eqiad) and T233142 (codfw) are nearly identical. We need the following info to setup these hosts: * Hostnames... [18:38:28] 10Analytics, 10Operations, 10User-Elukey: setup/install eqiad kerbos node WMF5173 - https://phabricator.wikimedia.org/T233141 (10RobH) a:05RobH→03elukey @elukey, Please note that both T233141 (eqiad) and T233142 (codfw) are nearly identical. We need the following info to setup these hosts: * Hostnames... [18:40:06] (03CR) 10Joal: "Some comments on comments again - Struture is way better IMO :)" (036 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/528504 (https://phabricator.wikimedia.org/T208612) (owner: 10Mforns) [18:40:25] 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: eqiad: 1 misc node for the Kerberos KDC service - https://phabricator.wikimedia.org/T227288 (10RobH) 05Open→03Resolved T233141 created for setup. resolving this request task! [18:40:27] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Make the Kerberos infrastructure production ready - https://phabricator.wikimedia.org/T226089 (10RobH) [18:40:36] 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: codfw: 1 misc node for the Kerberos KDC service - https://phabricator.wikimedia.org/T227425 (10RobH) 05Open→03Resolved a:05RobH→03None T233142 created for setup, resolving this request task! [18:40:38] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Make the Kerberos infrastructure production ready - https://phabricator.wikimedia.org/T226089 (10RobH) [18:43:50] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Ottomata) Just pushed https://github.com/wikimedia/jsonschema-tools/pull/9 for exportable and reusable schema repo... [18:53:14] joal: back [18:53:22] on the call :) [18:55:03] joal: sorry, let's move, the meeting i was on ( a panel) took quite abit longer and i had no way out [18:55:12] no problem nuria [18:55:17] joal: let's move meeting to tomorrow? [18:55:22] nuria: can be done now, or another time :) [18:55:37] joal: tomorrow would be better for me if it works for you [18:55:42] no problem [18:55:45] tomorrow :) [18:56:03] Gone for tonight then :) [18:56:07] See you tema [18:57:36] laters [18:57:38] mgerlach: o/ [18:57:40] you have shell access! :) [19:30:55] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Ottomata) Thoughts: - Is it possible to automate setting up the eslint-plugin-json stuff from jsonschema-tools? S... [19:31:23] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Core Platform Team Workboards (Clinic Duty Team): Develop a library for JSON schema backwards incompatibility detection - https://phabricator.wikimedia.org/T206889 (10Ottomata) I think we can close this... [19:38:37] (03PS10) 10Mforns: Add spark job to create mediawiki history dumps [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/528504 (https://phabricator.wikimedia.org/T208612) [19:39:40] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Pchelolo) > I think we need a way to disable testing of certain schemas and/or of certain tests for schemas. I don... [19:41:24] (03CR) 10Mforns: "Thanks for the CR! Modified for all comments." (036 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/528504 (https://phabricator.wikimedia.org/T208612) (owner: 10Mforns) [19:43:43] 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access to analytics cluster for Martin Gerlach - https://phabricator.wikimedia.org/T232707 (10herron) [19:43:55] (03PS11) 10Mforns: Add spark job to create mediawiki history dumps [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/528504 (https://phabricator.wikimedia.org/T208612) [19:44:33] 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access to analytics cluster for Martin Gerlach - https://phabricator.wikimedia.org/T232707 (10herron) 05Open→03Resolved a:03herron Hi Martin, this access is in place now. If any follow up is needed please don't hesitate to re-open. Thanks! [19:45:29] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Ottomata) The only one we need to disable right now is the monomorphic type test for recentchange, right? [19:48:36] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Pchelolo) >>! In T206814#5500769, @Ottomata wrote: > The only one we need to disable right now is the monomorphic... [20:07:07] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Ottomata) Maybe the right answer is to fix recentchange...investigating. [20:14:30] 10Analytics, 10Fundraising-Backlog, 10Fundraising Sprint S 2019: Identify source of discrepancy between HUE query in Count of event.impression and druid queries via turnilo/superset - https://phabricator.wikimedia.org/T204396 (10DStrine) [20:33:23] mforns: if you are still there take a look at my comment on https://phabricator.wikimedia.org/T226663 and let me know if you think it makes sense [20:39:10] nuria, looking [20:49:07] 10Analytics, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (next): mediawiki/recentchange event should not use fields with polymorphic types - https://phabricator.wikimedia.org/T216567 (10Ottomata) [20:49:25] 10Analytics, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (next): mediawiki/recentchange event should not use fields with polymorphic types - https://phabricator.wikimedia.org/T216567 (10Ottomata) [20:49:30] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Ottomata) [20:50:13] 10Analytics, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (next): mediawiki/recentchange event should not use fields with polymorphic types - https://phabricator.wikimedia.org/T216567 (10Ottomata) a:05Pchelolo→03None [20:51:16] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: CI Support for Schema Registry - https://phabricator.wikimedia.org/T206814 (10Ottomata) I modified the task description of {T216567}. I think we should just fix recentchange and not worry abo... [20:52:01] 10Analytics, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (next): mediawiki/recentchange event should not use fields with polymorphic types - https://phabricator.wikimedia.org/T216567 (10Ottomata) I modified the task description here to convert `log_params` into a map type. Alt... [20:54:12] 10Analytics, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (next): mediawiki/recentchange event should not use fields with polymorphic types - https://phabricator.wikimedia.org/T216567 (10Ottomata) [20:56:06] 10Analytics, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (next): mediawiki/recentchange event should not use fields with polymorphic types - https://phabricator.wikimedia.org/T216567 (10Ottomata) [21:01:29] 10Analytics, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Services (next): mediawiki/recentchange event should not use fields with polymorphic types - https://phabricator.wikimedia.org/T216567 (10Ottomata) [21:03:16] mforns: so i am guessing that according to what joal said this [21:03:25] https://www.irccloud.com/pastebin/1BvTMRpy/ [21:03:40] in turnilo's config is not going to work [21:04:08] yes, I understand that as well [21:04:53] javascript is supposed to execute in druid itself, but if it's restricted, then, it won't work [21:35:50] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10CPT Initiatives (Modern Event Platform (TEC2)), and 2 others: Modern Event Platform (TEC2) - https://phabricator.wikimedia.org/T185233 (10Ottomata) [21:35:59] 10Analytics, 10Better Use Of Data, 10EventBus, 10Product-Analytics, and 2 others: Event Platform Client Libraries - https://phabricator.wikimedia.org/T228175 (10Ottomata) [22:04:56] mforns: i got it to work w/o using js [22:05:14] https://www.irccloud.com/pastebin/By56IA6N/ [22:05:25] mforns: using a plywood lookup [22:11:30] 10Analytics, 10Product-Analytics: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list for Editing - https://phabricator.wikimedia.org/T226855 (10kzimmerman) p:05Normal→03High [23:09:40] 10Analytics, 10Analytics-Kanban: Add more dimensions to netflow's druid ingestion specs - https://phabricator.wikimedia.org/T229682 (10Nuria) ok, i figured out how to replace values using druid lookup transform functions which is a real long way to say "map" ` # Netflow data - name: wmf_netflow title... [23:16:35] 10Analytics, 10Analytics-Kanban: Add more dimensions to netflow's druid ingestion specs - https://phabricator.wikimedia.org/T229682 (10Nuria) ya, flags can be done in the same way , a bunch of them are null though seems like {F30383821} [23:18:45] 10Analytics, 10Analytics-Kanban: Add more dimensions to netflow's druid ingestion specs - https://phabricator.wikimedia.org/T229682 (10Nuria) ` tcp: extractionFn: type: lookup lookup: type: map map: {"32...