[05:28:39] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Marostegui) Thanks @elukey from trying that out. The problems we have with Toku are the following: - Mariadb 10.1 EOL by October 2020 - We are... [05:42:08] if anyone has some time today or tomorrow, it would be great if you could CR/+2 https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/576618/ [05:42:14] shouldn't take too long, mforns already +1'd, thanks! [06:25:51] a-team --^ [06:26:05] (to remember checking lex's code review) [06:38:08] (03PS3) 10Lex Nasser: Configure Oozie job for loading geoeditors data into Cassandra [analytics/refinery] - 10https://gerrit.wikimedia.org/r/582638 (https://phabricator.wikimedia.org/T248289) [07:12:27] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) This review of tables and their size comes at in a good moment since we are figuring out how many hosts Analytics will need next fiscal... [07:50:44] 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban), 10KaiOS-Wikipedia-app (MVP): Capture and send back client-side errors - https://phabricator.wikimedia.org/T248615 (10hueitan) we are in the middle of developing this, let us know when the **new stream is available **so we can add it to our source. [07:56:36] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Marostegui) Thank you. I guess another option (I don't know whether it is doable/supported at the moment) is to archive those big tables into h... [08:26:46] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10jcrespo) There is one other option- I wouldn't be ok with supporting TokuDB due to its huge functionality issues and data loss, but on 10.4 I w... [08:36:20] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Marostegui) It can fit the requirements here, but we are sort in a loop - in order to be able to start a 10.4 host we'd need to convert those t... [10:25:50] (03PS1) 10Gilles: Retain RTT and Downlink effective connection data [analytics/refinery] - 10https://gerrit.wikimedia.org/r/591018 [10:28:01] !log drop /srv/log/mw-log/archive/api from stat1007 (freeing 1.3TB of space!) [10:28:02] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:33:27] /dev/mapper/stat1007--vg-data 7.2T 4.6T 2.3T 68% /srv [10:33:30] \o/ [10:56:05] * elukey lunch! [11:36:33] 10Analytics, 10Operations, 10Traffic: Create replacement for Varnishkafka - https://phabricator.wikimedia.org/T237993 (10ema) >>! In T237993#6066376, @elukey wrote: > There is currently too much data that flows to kafka, for cp3050 we have 36GB * 12 partitions for a single day, definitely too much. How much... [12:14:48] 10Analytics, 10Performance-Team, 10Vue.js: Revise schema and performance dashboards for Vue.js search - https://phabricator.wikimedia.org/T250336 (10Gilles) Since you're working on a specific feature (search result suggestions), it would be ideal to come up with a user-centric performance metric related to t... [12:26:52] 10Analytics, 10Analytics-Kanban, 10Tool-Pageviews, 10Patch-For-Review: Fix double encoding of urls on mediarequests api - https://phabricator.wikimedia.org/T244373 (10fdans) I've documented the few different variations of file paths here: https://wikitech.wikimedia.org/wiki/Analytics/AQS/Mediarequests/Limi... [12:30:18] 10Analytics, 10Performance-Team, 10Vue.js: Revise schema and performance dashboards for Vue.js search - https://phabricator.wikimedia.org/T250336 (10Peter) @Niedzielski I can sync with you when you have time. I've added some measurement for search before in https://grafana.wikimedia.org/d/7zvDe0JZk/user-jour... [12:37:18] 10Analytics, 10Event-Platform, 10Inuka-Team: Create KaiOS error stream - https://phabricator.wikimedia.org/T250177 (10nshahquinn-wmf) @SBisson, the purpose of the "Analytics" column is to track the work meant for me. Obviously, that wasn't clear, so I'm renaming the column to "analyst" and moving this task o... [12:52:38] 10Analytics, 10Event-Platform, 10Inuka-Team: Create KaiOS error stream - https://phabricator.wikimedia.org/T250177 (10SBisson) @nshahquinn-wmf makes sense [12:56:43] 10Analytics, 10Analytics-Kanban, 10Research: covid19 data preservation - https://phabricator.wikimedia.org/T248600 (10Isaac) Thanks for the ping @Nuria @JAllemandou @elukey can you confirm what days would be deleted at this point? I have the data that we need through January 14th and am working on gatheri... [13:08:34] 10Analytics, 10Analytics-Kanban, 10Research: covid19 data preservation - https://phabricator.wikimedia.org/T248600 (10Isaac) Actually, sorry, give me several more hours with the January 9 - 14th data. I just realized that the "Coronavirus" article was not part of our "Covid-19-related articles" dataset, whic... [13:30:34] hey teammm [13:41:28] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: High volume mediawiki analytics events camus import is lagging - https://phabricator.wikimedia.org/T233718 (10Ottomata) Yes, but I don't remember it alarming for e.g. eqiad.mediawiki.revision-create after I separated out the higher volume topics into a dif... [13:43:10] o/ [13:47:05] 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban), 10KaiOS-Wikipedia-app (MVP): Capture and send back client-side errors - https://phabricator.wikimedia.org/T248615 (10Ottomata) What should the stream be named? This will map to Kafka topic names as well as other downstream datastores, e.g. Hive. -... [14:06:59] (03CR) 10Fdans: "@Nuria since this is documented now we can proceed with this change" [analytics/aqs] - 10https://gerrit.wikimedia.org/r/588396 (https://phabricator.wikimedia.org/T244373) (owner: 10Fdans) [14:27:05] yargh milimetric can you get me another example of a kaios user agent string? [14:33:31] elukey ottomata: when you have a chance, could one of you please take a look at https://gerrit.wikimedia.org/r/c/operations/puppet/+/589320? it's got +1s and marcel has looked at it too [14:34:15] bearloga: yep I have it in my backlog, it seems fine to me.. what jobs are going to be migreated to this new user? [14:34:59] also we should set some rules to access that group in my opinion, namely that every user is in privatedata too or similar (otherwise with the sudo rule it is possible to escalate credentials) [14:36:56] elukey: cool, thanks! just wanted to make sure it was on your radar. there's a couple of jupyter reports/dashboards that my team has that are scheduled as crontabs. I have a few things (including google search console data) that I'm planning on productionizing with this. [14:37:25] bearloga: sure sure makes sense [14:37:26] (scheduled as crontabs under their individual staff accounts) [14:37:50] in that case, the analytics-privatedata user should work no bearloga ? [14:38:08] !log restarting eventlogging-processor with updated python3-ua-parser for parsing KaiOS user ageints [14:38:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:39:25] elukey: I think so...although I think mforns imagined a Product Analytics-specific user in https://phabricator.wikimedia.org/T230743 [14:40:06] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Product-Analytics: EventLogging does not properly classify KaiOS user agents - https://phabricator.wikimedia.org/T248560 (10Ottomata) Done. Tested on eventlog1002 with `lang=python from ua_parser import user_agent_parser user_agent_parser.Parse... [14:40:27] (03CR) 10Nuria: [C: 03+2] Retain RTT and Downlink effective connection data [analytics/refinery] - 10https://gerrit.wikimedia.org/r/591018 (owner: 10Gilles) [14:40:33] (03CR) 10Nuria: [V: 03+2 C: 03+2] Retain RTT and Downlink effective connection data [analytics/refinery] - 10https://gerrit.wikimedia.org/r/591018 (owner: 10Gilles) [14:40:39] bearloga: ah it is mforns' fault! Ok now it is more clear :D [14:41:00] ho ho ho [14:41:15] bearloga: jokes aside, it makes perfect sense to me, will also have to create the keytab etc.. I just want to think a little bit about it and add the necessary fences etc.. (like every user in privatedata, etc..) [14:41:21] will comment later on to the patch! [14:41:32] thank you very much elukey! :) [14:42:26] for the record I trust mforns' judgement with these things :D [14:42:49] xD [14:43:03] not sure that'll be good for you [14:43:28] hahaha [14:45:51] 10Analytics: We should get an alarm for partitions that have no data for topics that have data influx at all times, most of the mediawiki.* - https://phabricator.wikimedia.org/T250699 (10Nuria) [14:46:40] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: [Wikistats v2] Default selection for (active) editors is confusing for inexperienced users - https://phabricator.wikimedia.org/T213800 (10fdans) a:03fdans [14:46:41] (03CR) 10Mforns: "Code looks good overall!" (034 comments) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/579022 (https://phabricator.wikimedia.org/T223958) (owner: 10Amire80) [14:47:08] 10Analytics, 10Performance-Team, 10Vue.js: Revise schema and performance dashboards for Vue.js search - https://phabricator.wikimedia.org/T250336 (10Niedzielski) Thank you, @Gilles and @Peter. > Since you're working on a specific feature (search result suggestions), it would be ideal to come up with a user-... [14:47:10] 10Analytics, 10Tool-Pageviews: Image files with quotes do not resolve on the mediarequest API - https://phabricator.wikimedia.org/T247333 (10fdans) a:03fdans [14:47:41] 10Analytics, 10Tool-Pageviews: Image files with quotes do not resolve on the mediarequest API - https://phabricator.wikimedia.org/T247333 (10fdans) All exceptions to url encoding have been documented here: https://wikitech.wikimedia.org/wiki/Analytics/AQS/Mediarequests/Limitations#Issues_with_file_paths [14:48:09] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Product-Analytics: EventLogging does not properly classify KaiOS user agents - https://phabricator.wikimedia.org/T248560 (10Nuria) 05Open→03Resolved [14:48:20] 10Analytics, 10Analytics-Kanban, 10Tool-Pageviews: Image files with quotes do not resolve on the mediarequest API - https://phabricator.wikimedia.org/T247333 (10fdans) [15:00:05] 10Analytics: We should get an alarm for partitions that have no data for topics that have data influx at all times, most of the mediawiki.* - https://phabricator.wikimedia.org/T250699 (10Ottomata) Thanks. It'd be really nice to automate this in some way rather than just setting up specific thresholds for speci... [15:02:12] ping joal [15:05:10] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: High volume mediawiki analytics events camus import is lagging - https://phabricator.wikimedia.org/T233718 (10Nuria) 05Open→03Resolved [15:05:46] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: High volume mediawiki analytics events camus import is lagging - https://phabricator.wikimedia.org/T233718 (10Nuria) Closing parent task focusing on alarming children task [15:10:04] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Import regularly via sqoop mediawiki_imagelinks table - https://phabricator.wikimedia.org/T249113 (10Nuria) 05Open→03Resolved [15:10:07] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Definition of not text content metrics for tunning session (rich media,: images and the linke) - https://phabricator.wikimedia.org/T247417 (10Nuria) [15:22:06] 10Analytics, 10Research: Proposed adjustment to wmf.wikidata_item_page_link to better handle page moves - https://phabricator.wikimedia.org/T249773 (10Nuria) a:03Milimetric [15:23:43] 10Analytics, 10Analytics-Kanban, 10Research: Proposed adjustment to wmf.wikidata_item_page_link to better handle page moves - https://phabricator.wikimedia.org/T249773 (10Nuria) [15:31:14] (03CR) 10Nuria: [C: 03+2] Configure geoeditors monthly public Oozie job to work with geoeditors public monthly table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/576618 (https://phabricator.wikimedia.org/T244597) (owner: 10Lex Nasser) [15:31:17] (03CR) 10Nuria: [V: 03+2 C: 03+2] Configure geoeditors monthly public Oozie job to work with geoeditors public monthly table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/576618 (https://phabricator.wikimedia.org/T244597) (owner: 10Lex Nasser) [15:37:51] 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban), 10KaiOS-Wikipedia-app (MVP): Capture and send back client-side errors - https://phabricator.wikimedia.org/T248615 (10jlinehan) Chatted with @ottomata and we agreed on keeping it simple with a stream called `kaios_app.error`. Will update this ticket when... [15:43:26] a-team: can someone please do the magic hue userlist syncing thing? jmads has production shell access (T249873) but can't access hue [15:43:40] bearloga: on it :) [15:43:49] bearloga: can I ask what is the use case? [15:44:09] because if it is to check tables via the UI I'd suggest to try Superset's sqllab [15:44:15] and see if it is better/worst etc.. [15:44:23] I'd love to eventually deprecate Hue :D [15:44:27] elukey: data analyst for design team [15:45:06] bearloga: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset#SQL_Lab is the starting point [15:45:17] (Basically uses presto behind the scenes to access hive data) [15:46:01] bearloga, elukey: I support elukey's suggestion but will enable hue nonetheless :) jmads: please have a look at supers SQL-labs, using the presto engine :) [15:46:10] 10Analytics: Spike, see how easy/hard is to scoop all tables from Eventlogging log database - https://phabricator.wikimedia.org/T250709 (10Nuria) [15:46:11] thank you! [15:47:22] bearloga: also when you have time - https://phabricator.wikimedia.org/T245179#6054759 [15:47:33] done bearloga [15:47:54] i'll have to try out the superset changes, but last time i tried to use it it was in no way a hql replacement :P [15:48:29] * ebernhardson might have superset related PTSD... [15:49:06] ebernhardson: we recenty added presto support in sql lab, I am pretty sure you haven't tested it :) [15:49:31] indeed i saw some emails about changes but haven't tried :) [15:49:59] it is probably not as complete as hql, but if you need full power then the beeline cli is there for a reason.. Hue is python 2.7 with horrible and stale deps, upstream is still porting to py3 [15:50:20] and it is cloudera specific etc.. so in light of Apache BigTop I'd love just to dump Hue :D [15:50:46] ahh, interesting. Indeed i do almost all exploration from pyspark, i like programming languages, variables, functions, etc :) [15:50:49] it probably depends on your query ebernhardson [15:51:13] also we have notebooks! [15:51:20] * joal has ebernhardson-cluster-bizarre-usage PTSD ;) [15:51:26] joal: indeed it probably does, my last work with superset was druid related, which required defining a bunch of druid related json inside superset by clicking around for hours [15:51:38] I actually mostly do notebooks nowaday [15:52:02] ebernhardson: I also added sqlalchemy bindings for Druid in Superset, no need to weird datasource def anymore in theory [15:52:14] (all accessible via sqllab too) [15:52:23] interesting, that might be significantly better [15:52:25] lexnasser: merged geoeditors 1st patch [15:52:28] hm, more or less - I can feel this --^ backfiring on us :) [15:52:46] nuria: yeah, I saw, thanks!! [15:53:05] joal: sadly it is what superset upstream doesn't consider deprecated, we cannot really continue in the other direction :( [15:53:28] (we also need to upgrade druid) [15:53:38] nuria: Testing loading data from hive into Cassandra - what's the best way to look at Cassandra data/query from Cassandra [15:53:45] ebernhardson: do try presto /sql lab and let us know [15:55:02] lexnasser: i think for that one _since you do not have permits to ssh into cassandra_ there is no way other than pairing up with someone [15:55:39] nuria: got it, thanks! [15:57:43] ebernhardson: I however definitely acknowledge that druid json query language is a pain [15:58:38] joal: not just the query language, although i guess this could be part of it, but defining all of this is basically where my "love" of superset/druid comes frm: https://superset.wikimedia.org/druiddatasourcemodelview/edit/449 [15:59:52] writing a bunch of json based metrics by hand, pasting them into a random web form, loading other pages to see if the definition worked, etc. I want programming languages and test suites :P [16:01:41] ebernhardson: please test the sql datasource and tell me how it works :) [16:01:59] makes sense ebernhardson :) [16:20:26] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add SWAP profile to stat1005 - https://phabricator.wikimedia.org/T245179 (10mpopov) >>! In T245179#6054759, @elukey wrote: > @mpopov hi! Are you using the SparkR kernels that we have on SWAP by any chance? I am asking because on stat100[5,... [16:29:34] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add SWAP profile to stat1005 - https://phabricator.wikimedia.org/T245179 (10elukey) >>! In T245179#6072161, @mpopov wrote: >>>! In T245179#6054759, @elukey wrote: >> @mpopov hi! Are you using the SparkR kernels that we have on SWAP by any... [16:29:41] (03PS1) 10Urbanecm: Add new wiki [analytics/refinery] - 10https://gerrit.wikimedia.org/r/591094 [16:29:43] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add SWAP profile to stat1005 - https://phabricator.wikimedia.org/T245179 (10elukey) [16:32:24] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Druid access for view on event.editeventattempt - https://phabricator.wikimedia.org/T249945 (10kaldari) @fdans - Yes, the 90 day limit works fine for me. @dr0ptp4kt - `webhost` would be very useful for mitigating the `T249944` bug. I can't think of any ot... [17:03:40] ottomata: ok if we enable TLS tomorrow when you join or do you prefer now? [17:03:48] yeah let's do tomorrow [17:05:32] ack :) [17:06:10] ottomata: one qs about stat1007 - is geoip archive needed under /srv for users, or can I migrate it to an-launhcer? [17:06:14] *launhcer [17:06:17] ooofff [17:06:21] launcher :D [17:06:25] hmmm [17:06:29] is there a reason not to keep it? [17:06:33] waitin /srv? [17:06:42] oh archive [17:06:43] ohhhhh [17:06:48] hm yes it is probably needed [17:06:54] well, i dont know if anyone uses it [17:07:02] but the previous use case was to allow for users to do historical comparison between maxmind dbs [17:07:07] not for automated jobs [17:07:31] because my idea would be to move the timer to launcher, and then possibly hdfs-rsync to stat boxes selectively if needed [17:07:55] (since IIRC it pushes to hdfs too) [17:08:25] this is in the context of moving bits of profile::private to something else [17:11:50] hm elukey what does the timer do? just the backup? [17:12:03] that would be fine [17:12:13] maybe we could even put the dbs in hdfs instead? [17:13:58] ottomata: yeah I think it does the backup, and everything is copied to hdfs [17:14:36] wow I just realized that this could be replaced by a hdfs rsync [17:14:47] no? [17:18:59] anyway, going to log off! [17:19:00] o/ [17:19:12] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Druid access for view on event.editeventattempt - https://phabricator.wikimedia.org/T249945 (10Nuria) So, to be clear this dataset will only have the last 90 days of data cc @kaldari @dr0ptp4kt [17:19:42] joal: working on bot s documentation now [17:28:16] yeah it could! [17:28:24] sounds graet then yeah moving to an-launcher sounds good [17:34:28] 10Analytics, 10Operations, 10Wikimedia-Logstash, 10observability: Retire udp2log: onboard its producers and consumers to the logging pipeline - https://phabricator.wikimedia.org/T205856 (10Krinkle) >>! In T126989#5076715, @gerritbot wrote: > Change 498106 **merged** by Filippo Giunchedi: > [mediawiki/core@... [17:35:00] 10Analytics, 10Operations, 10Wikimedia-Logstash, 10observability, 10Performance-Team (Radar): Retire udp2log: onboard its producers and consumers to the logging pipeline - https://phabricator.wikimedia.org/T205856 (10Krinkle) [17:45:22] 10Analytics: clear bot spam-scraping [[en:United States Senate]] not being detected as a bot - https://phabricator.wikimedia.org/T247085 (10Nuria) The dection code running on shadow mode was able to detect this page [17:54:22] 10Analytics: clear bot spam-scraping [[en:United States Senate]] not being detected as a bot - https://phabricator.wikimedia.org/T247085 (10Nuria) {F31769077} [18:15:46] musikanimal: almost there , bot detection is running on shadow mode, documenting effect now: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/BotDetection#Why_do_we_need_more_sophisticated_bot_detection [18:16:10] joal: re. airflow refine: we could have a separate job (i.e. every 15 mins) that just does "hdfs dfs -ls /wmf/data/event" and writes output to a local file. Then the refine DAG code would read that file at DAG-creation time. [18:16:11] very exciting! [18:16:12] musikanimal: I think that we will probably deploy these changes to pageview apis next week [18:16:22] cc joal [18:16:27] \o/ thank you! [18:16:45] ack nuria - have a minute about how you'd like me to update docs? [18:16:51] mforns: hm [18:16:55] joal: I am updating them [18:17:05] mforns: do you want to batcave a bit? [18:17:13] joal: k [18:17:32] joal: sure :] [18:17:33] omw [18:17:53] joal: oops [18:17:57] tardis? [18:18:41] mforns: actually nuria came to discuss bots [18:18:46] will be there in a minute [18:18:52] oh ok, no problemo [18:22:03] mforns: ready :) [18:22:16] ok, batcave or tardis? [18:37:37] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Druid access for view on event.editeventattempt - https://phabricator.wikimedia.org/T249945 (10dr0ptp4kt) Cool. I'll update the patch to add `webhost` and 90 days is good. [19:00:43] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Druid access for view on event.editeventattempt - https://phabricator.wikimedia.org/T249945 (10dr0ptp4kt) Patch is ready for review and deploy. [19:27:40] 10Analytics: Unique devices, retrofit with bot detection code - https://phabricator.wikimedia.org/T250744 (10Nuria) [19:27:51] 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban), 10KaiOS-Wikipedia-app (MVP): Capture and send back client-side errors - https://phabricator.wikimedia.org/T248615 (10jlinehan) [19:27:54] 10Analytics, 10Event-Platform, 10Inuka-Team: Create KaiOS error stream - https://phabricator.wikimedia.org/T250177 (10jlinehan) [19:38:03] 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban), 10KaiOS-Wikipedia-app (MVP): Capture and send back client-side errors - https://phabricator.wikimedia.org/T248615 (10Ottomata) Done! [19:43:38] 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban), 10KaiOS-Wikipedia-app (MVP): Capture and send back client-side errors - https://phabricator.wikimedia.org/T248615 (10SBisson) >>! In T248615#6073135, @Ottomata wrote: > Done! Thanks a lot! [19:45:26] 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban), 10KaiOS-Wikipedia-app (MVP): Capture and send back client-side errors - https://phabricator.wikimedia.org/T248615 (10jlinehan) Yay! So @hueitan the stream name is `kaios_app.error`, so the `meta.stream` field in the JSON that gets sent will need to be s... [21:03:22] 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban), 10KaiOS-Wikipedia-app (MVP): Capture and send back client-side errors - https://phabricator.wikimedia.org/T248615 (10Ottomata) Also, setting `meta.dt` to the ISO-8601 UTC date timestamp of the error is good too! If you don't set it, eventgate will set...