[01:49:14] 10Analytics, 10Event-Platform, 10Multimedia: Mediaviewer views should be reworked to be an eventlogging event - https://phabricator.wikimedia.org/T239630 (10MusikAnimal) Assuming #tool-pageviews was added by mistake [03:41:37] 10Analytics, 10Analytics-Kanban, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services): Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10Nuria) >The deprecated... [09:16:59] * elukey afk! [09:45:00] hello team :) [11:06:31] =ghello  [11:06:36] uff :) [11:06:39] hello fdans :) [11:06:50] just got back home, will take lunch break in a bit [11:07:05] this time it went very smoothly, feel great [11:14:11] Amir1: hi! are you there by any chance? [11:15:10] I'd need to apply some new settings to cron on stat1007 and I see a ton of old processes, some of them from your crontab [11:15:22] ladsgro+ 224678 2.2 0.1 2632388 84328 ? Sl 2019 3053:02 \_ python ./utility extract_scores [11:15:31] for example, this one seems running since ages ago [11:15:35] is it expected? [11:18:43] * elukey sends an email [11:35:17] * elukey lunch break [12:36:14] 10Analytics, 10Analytics-SWAP, 10Discovery-Analysis, 10Product-Analytics: Get 'sparklyr' working on stats1005 - https://phabricator.wikimedia.org/T139487 (10Ottomata) 05Open→03Declined [12:36:18] 10Analytics, 10Analytics-SWAP: Jupyter Notebooks TLC 2018-2019 - https://phabricator.wikimedia.org/T188275 (10Ottomata) [12:36:20] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Newpyter - SWAP Juypter Rewrite - https://phabricator.wikimedia.org/T224658 (10Ottomata) [12:57:22] 10Analytics, 10Operations, 10Research, 10Traffic, and 2 others: Enable layered data-access and sharing for a new form of collaboration - https://phabricator.wikimedia.org/T245833 (10Miriam) THanks @elukey for this summary. There are two macro use-cases for the release/simplified access to article pageview... [12:58:13] 10Analytics, 10Operations, 10Research, 10WMF-Legal, 10User-Elukey: Enable layered data-access and sharing for a new form of collaboration - https://phabricator.wikimedia.org/T245833 (10elukey) [13:49:10] fdans: I have half an hour if you want to meet [13:49:27] elukey: yep! bc? [13:51:12] sure [14:10:06] 10Analytics, 10Analytics-Kanban, 10Release Pipeline, 10Patch-For-Review, and 2 others: Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10akosiaris) >>! In T238658#5993144, @Ottomata wrote: > @akosiaris eventstreams in k8s looks good to me! Nice! Congrats for s... [14:21:56] 10Analytics, 10Analytics-Kanban, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services): Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10hashar) From the call w... [14:34:34] 10Analytics, 10Analytics-Kanban, 10EventStreams: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10stjn) Is it deployed already? Currently (24 March, 14:32 UTC) disconnects at the same frequency, though maybe I need to restart for it... [15:08:18] 10Analytics, 10Growth-Team, 10Product-Analytics, 10Security-Team: Hash edit session ID in EditAttemptStep and VisualEditorFeatureUse whitelisting - https://phabricator.wikimedia.org/T244931 (10Reedy) [15:20:41] 10Analytics, 10Data-release, 10Privacy Engineering, 10Privacy: An expert panel to produce recommendations on open data sharing for public good - https://phabricator.wikimedia.org/T189339 (10JFishback_WMF) [15:22:49] 10Analytics, 10Analytics-Kanban, 10EventStreams: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10Ottomata) p:05Triage→03High Yup it's deployed. I doubt you'd need to if the problem is indeed server side. Interesting, I think... [15:24:32] 10Analytics, 10Event-Platform, 10MediaWiki-Maintenance-scripts, 10WMF-JobQueue, and 3 others: showJobs.php maintenance script useless and misleading in production - https://phabricator.wikimedia.org/T221224 (10Krinkle) [15:46:10] !log restart all cron.service processes on stat/notebook (killing long lingering processes) to move the unit under user.slice [15:46:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:46:14] ottomata: --^ [15:46:20] as FYI [15:46:22] seems to work fine :) [15:49:02] and also cpu limits seems to work on it [15:49:03] https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=stat1007&var-datasource=eqiad%20prometheus%2Fops&var-cluster=analytics&from=now-6h&to=now&fullscreen&panelId=3 [15:59:00] great! [16:00:35] a-team standup! [16:02:59] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade the Hadoop test cluster to BigTop - https://phabricator.wikimedia.org/T244499 (10elukey) Working on the rollback in https://etherpad.wikimedia.org/p/analytics-bigtop [16:09:23] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Inuka-Team (Kanban), 10Patch-For-Review: Set up pageview counting for KaiOS app - https://phabricator.wikimedia.org/T244547 (10Nuria) >All the values of page_id and namespace_id are null. This may be intentional since we already have project and pag... [16:10:02] 10Analytics, 10Analytics-Kanban, 10EventStreams: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10Ottomata) Ah ha, @ema thinks it is https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/records.config.en.html#proxy-config... [16:10:18] 10Analytics, 10Analytics-Kanban, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Infrastructure (phase-out-jessie), and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10hashar) The whole process is extensively described at https://w... [16:14:34] 10Analytics, 10Analytics-Kanban, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Infrastructure (phase-out-jessie), and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10greg) >>! In T210271#5993955, @Nuria wrote: >>The deprecated in... [16:15:26] 10Analytics, 10Analytics-Kanban, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Infrastructure (phase-out-jessie), and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10JAllemandou) One nit: On the screen above, we tick the `Specify... [16:23:30] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog: Produce an instrumentation event stream using new EPC and EventGate from client side browsers - https://phabricator.wikimedia.org/T241241 (10Ottomata) [16:25:24] 10Analytics, 10Analytics-Kanban, 10EventStreams, 10Operations, 10Traffic: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10Ottomata) a:05Ottomata→03ema [16:26:35] 10Analytics, 10Event-Platform, 10Multimedia: Mediaviewer views should be reworked to be an eventlogging event - https://phabricator.wikimedia.org/T239630 (10Nuria) Talked with @jlinehan that (once our phase 1 of client was merged , which just was!) this work would be the next step, so we have some client cod... [16:31:28] ping ottomata retrooo [17:27:29] elukey: hey, sorry I was asleep (RIP my sleeping schedule) [17:27:53] It's actually needed but I agree we should find a better way to run it, maybe pyspark? [17:28:38] Amir1: hi! no problem, I'd say that we could try with systemd timers kicked off periodically [17:28:56] they are way better than crons [17:29:03] it should be once in a month, does it work like that? [17:29:12] cron has once in a month thing [17:29:19] and you can use tools like 'systemctl list-timers' to display all of them, their starting date, etc.. [17:29:33] Amir1: oh yes definitely, timers replace crons [17:30:03] Amir1: if you are curious go to stat1007 and type "systemctl list-timers" [17:30:56] cool I do it, just one thing, is there a puppet case, etc. that I can study so I make the patch? [17:31:00] or documentation [17:32:17] ah yes, I can give you a lot [17:32:25] does it need kerberos auth? (probably not) [17:33:09] what we can do is also to create a task with some details about the command to run, when etc.. and then I'll prep a patch [17:33:23] (where results go, etc..) [17:34:58] elukey: I don't need kerberos [17:38:17] Amir1: super, then in puppet you can find systemd::timer::job [17:38:42] in analytics we use mostly kerberos::systemd_timer, but it is just a wrapper [17:39:00] I was planning also to move other crons that wmde has in puppet (for analytics hosts) to timers [17:39:10] maybe we can open a task to do all [17:49:38] Thanks [17:58:13] (03CR) 10Nuria: "pinging @joal to CR here when he is able, is this job tested?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/576618 (https://phabricator.wikimedia.org/T244597) (owner: 10Lex Nasser) [18:00:36] * elukey off! [18:01:49] (03CR) 10Nuria: [C: 03+2] "Makes sense." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/582500 (https://phabricator.wikimedia.org/T248228) (owner: 10Joal) [18:04:21] (03CR) 10Nuria: "Let's put this change on the back burner until we have the ones it depends on merged as well." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/582638 (https://phabricator.wikimedia.org/T248289) (owner: 10Lex Nasser) [18:08:04] ottomata: in the schema changes: https://gerrit.wikimedia.org/r/#/c/schemas/event/primary/+/582028/ [18:08:27] ya [18:08:30] ottomata: additional properties set to false means that teh schema is more restrictive correct? an [18:08:33] yes [18:08:42] event that sends an additional property will not validate, correct? [18:08:46] yes [18:09:15] soin order to add aproperty (which is a backwards compatible change) [18:09:22] you need to increment the schema version? [18:09:25] yes [18:09:28] ^ ottomata [18:10:26] 10Analytics, 10Analytics-Kanban, 10Event-Platform: Event schemas common schema should set additionalProperties: false - https://phabricator.wikimedia.org/T248173 (10Ottomata) I created a new common schema version 1.1.0. Only mediawiki/client/error uses this ATM. If/when we create new versions for other sch... [18:14:08] nuria: ya, makes sense yes? [18:14:21] unfortunetely, it only applies for the root of the schema [18:14:35] but putting it on the root is better than nothing! [18:16:27] ottomata: k [18:16:39] ottomata: ya, makes sense, just wanted to understand the convention [18:16:42] coo [18:19:38] nuria: if you have a moment could you review https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/582131 ? [18:19:56] ottomata: yes, will do today [18:20:16] ty! [18:26:10] nuria qq am thinking about https://phabricator.wikimedia.org/T238138 [18:26:34] and where to fit schemas that we migrate from meta [18:26:50] do you think it would be better to make a schema hierachy just for the migration? [18:26:54] eventlogging_legacy/... [18:26:55] or somoething? [18:26:58] or [18:27:17] should we try to fit them into a semantic hierachy somewhere [18:27:27] mediawiki/content/page/view [18:27:30] or something? [18:27:42] hmm, maybe for the port from EL it would be best to isolate those [18:27:54] and hope we can get folks to switch to a new nice hierarchy for new stuff or later [19:09:15] (03CR) 10Joal: "Minor details, overall looks good :)" (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/582131 (https://phabricator.wikimedia.org/T240985) (owner: 10Ottomata) [19:25:52] (03CR) 10Joal: "Missing data dependency for project_namespace_map, and minor things." (035 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/570681 (https://phabricator.wikimedia.org/T243090) (owner: 10Mforns) [19:31:22] 10Analytics, 10Analytics-EventLogging, 10Wikimedia-production-error: Uncaught ExtensionDependencyError: EventLogging requires EventStreamConfig to be installed. in /srv/mediawiki/php-1.35.0-wmf.25/includes/registration/ExtensionRegistry.php:398 - https://phabricator.wikimedia.org/T248409 (10mmodell) [19:33:19] 10Analytics, 10Analytics-EventLogging, 10Wikimedia-production-error: Uncaught ExtensionDependencyError: EventLogging requires EventStreamConfig to be installed. in /srv/mediawiki/php-1.35.0-wmf.25/includes/registration/ExtensionRegistry.php:398 - https://phabricator.wikimedia.org/T248409 (10mmodell) [19:33:27] 10Analytics, 10Analytics-EventLogging, 10Wikimedia-production-error: Uncaught ExtensionDependencyError: EventLogging requires EventStreamConfig to be installed. in /srv/mediawiki/php-1.35.0-wmf.25/includes/registration/ExtensionRegistry.php:398 - https://phabricator.wikimedia.org/T248409 (10mmodell) p:05Tri... [19:35:18] 10Analytics, 10Analytics-EventLogging, 10Wikimedia-production-error: Uncaught ExtensionDependencyError: EventLogging requires EventStreamConfig to be installed. in /srv/mediawiki/php-1.35.0-wmf.25/includes/registration/ExtensionRegistry.php:398 - https://phabricator.wikimedia.org/T248409 (10DannyS712) Depend... [19:46:39] nuria: is it ok if we postpone out bot-talk to tomorrow? [19:53:16] 10Analytics, 10Analytics-Kanban, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Infrastructure (phase-out-jessie), and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10hashar) >>! In T210271#5995465, @JAllemandou wrote: > One nit:... [19:55:06] 10Analytics, 10Analytics-Kanban, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Infrastructure (phase-out-jessie), and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10JAllemandou) > Looks like we should be able to save you that ma... [19:59:26] 10Analytics, 10Analytics-Kanban, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Infrastructure (phase-out-jessie), and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10hashar) Bah forget me, the refinery `pom.xml` already has the p... [20:21:49] 10Analytics, 10Analytics-EventLogging, 10Wikimedia-production-error: Uncaught ExtensionDependencyError: EventLogging requires EventStreamConfig to be installed. in /srv/mediawiki/php-1.35.0-wmf.25/includes/registration/ExtensionRegistry.php:398 - https://phabricator.wikimedia.org/T248409 (10DannyS712) [20:25:39] 10Analytics, 10Analytics-EventLogging, 10Wikimedia-production-error: Uncaught ExtensionDependencyError: EventLogging requires EventStreamConfig to be installed. in /srv/mediawiki/php-1.35.0-wmf.25/includes/registration/ExtensionRegistry.php:398 - https://phabricator.wikimedia.org/T248409 (10Ottomata) AH! Th... [20:33:00] 10Analytics, 10Operations, 10ops-eqiad: analytics1044 hardware failure - https://phabricator.wikimedia.org/T248413 (10Volans) p:05Triage→03Medium [20:36:47] 10Analytics, 10ChangeProp, 10Core Platform Team, 10Discovery-Search, and 2 others: Better way to pause writes on elasticsearch - https://phabricator.wikimedia.org/T230730 (10Pchelolo) [20:37:21] 10Analytics, 10Analytics-EventLogging, 10Wikimedia-production-error: Uncaught ExtensionDependencyError: EventLogging requires EventStreamConfig to be installed. in /srv/mediawiki/php-1.35.0-wmf.25/includes/registration/ExtensionRegistry.php:398 - https://phabricator.wikimedia.org/T248409 (10mmodell) 05Open... [20:37:34] 10Analytics, 10ChangeProp, 10Discovery-Search, 10Event-Platform, and 2 others: Better way to pause writes on elasticsearch - https://phabricator.wikimedia.org/T230730 (10Pchelolo) [20:39:05] 10Analytics, 10Operations, 10ops-eqiad: analytics1044 hardware failure - https://phabricator.wikimedia.org/T248413 (10Ottomata) I say proceed! This should be one of the Hadoop data disks, which just uses JBOD. Hadoop is configured to tolerate 1 (or some?) disk failures (the data on the failed disk should b... [20:42:44] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10CPT Initiatives (Modern Event Platform (TEC2)), 10MW-1.34-notes (1.34.0-wmf.20; 2019-08-27): Refactor EventBus mediawiki configuration - https://phabricator.wikimedia.org/T229863 (10WDoranWMF) [20:44:36] 10Analytics, 10Event-Platform, 10MediaWiki-Maintenance-scripts, 10WMF-JobQueue, and 3 others: showJobs.php maintenance script useless and misleading in production - https://phabricator.wikimedia.org/T221224 (10Anomie) p:05Triage→03Low [20:45:31] 10Analytics, 10Operations, 10ops-eqiad: analytics1044 hardware failure - https://phabricator.wikimedia.org/T248413 (10Volans) Ack, the host led me into the config anyway, even pressing any key. The configuration was already skipping VD3, and confirmed that the RAID1 where the OS resides is in optimal state.... [20:49:08] 10Analytics, 10Operations, 10ops-eqiad: analytics1044 hardware failure - https://phabricator.wikimedia.org/T248413 (10Volans) Ok, now the message is more readable: ` There are offline or missing virtual drives with preserved cache. Please check the cables and ensure that all drives are present. Press any key... [21:05:14] (03PS4) 10Ottomata: Support multiple possible schema base URIs in EventSchemaLoader [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/582131 (https://phabricator.wikimedia.org/T240985) [21:05:16] (03CR) 10Ottomata: Support multiple possible schema base URIs in EventSchemaLoader (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/582131 (https://phabricator.wikimedia.org/T240985) (owner: 10Ottomata) [21:05:43] joal: ty for review, just updated ^ [21:07:19] 10Analytics, 10Analytics-EventLogging, 10Wikimedia-production-error: EventLogging-based extensions cause errors on test2.wikipedia.org - https://phabricator.wikimedia.org/T196309 (10Krinkle) @Ottomata As far as I'm concerned, yes it could. The use case for the current configuration is that test2wiki is its o... [21:07:52] (03CR) 10jerkins-bot: [V: 04-1] Support multiple possible schema base URIs in EventSchemaLoader [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/582131 (https://phabricator.wikimedia.org/T240985) (owner: 10Ottomata) [21:11:53] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230 (10Ottomata) > meta.wikimedia.org schemas ported to draft 7 JSONSchema in a git schema repo with common schema included via $ref. These shoul... [21:17:22] (03PS5) 10Ottomata: Support multiple possible schema base URIs in EventSchemaLoader [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/582131 (https://phabricator.wikimedia.org/T240985) [21:41:22] 10Analytics, 10Operations, 10ops-eqiad: analytics1044 hardware failure - https://phabricator.wikimedia.org/T248413 (10Volans) This is related to T245910, the same disk failed then and from that moment it started logging into racadm some failure that I think led the controller fail today. I've cleared the for... [21:47:00] 10Analytics, 10Product-Analytics (Kanban), 10Readers-Web-Backlog (Needs Product Owner Decisions), 10covid-19: Weekly updates on readers - https://phabricator.wikimedia.org/T248426 (10kzimmerman) [21:47:25] 10Analytics, 10Product-Analytics (Kanban), 10Readers-Web-Backlog (Needs Product Owner Decisions), 10covid-19: Weekly updates on readers - https://phabricator.wikimedia.org/T248426 (10kzimmerman) p:05Triage→03High [21:53:38] 10Analytics, 10Product-Analytics (Kanban), 10Readers-Web-Backlog (Needs Product Owner Decisions), 10covid-19: Weekly updates on editors - https://phabricator.wikimedia.org/T248427 (10kzimmerman) [21:53:54] 10Analytics, 10Product-Analytics (Kanban), 10Readers-Web-Backlog (Needs Product Owner Decisions), 10covid-19: Weekly updates on editors - https://phabricator.wikimedia.org/T248427 (10kzimmerman) p:05Triage→03High [22:00:16] 10Analytics, 10Cite, 10Reference Previews: Remove or simplify tracking metrics - https://phabricator.wikimedia.org/T242127 (10CCicalese_WMF) [22:00:48] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10Event-Platform, and 4 others: eventgate-wikimedia should support using remote stream configuration - https://phabricator.wikimedia.org/T238657 (10CCicalese_WMF) [22:01:04] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10Event-Platform, 10Services (watching): Eventlogging Client Side can use the stream config module to dynamically adjust sampling rates - https://phabricator.wikimedia.org/T234594 (10CCicalese_WMF) [22:06:30] 10Analytics: Ingest api data (for posts) into druid - https://phabricator.wikimedia.org/T218348 (10CCicalese_WMF) [22:07:17] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Goal, 10Services (watching): Modern Event Platform: Stream Connectors - https://phabricator.wikimedia.org/T214430 (10CCicalese_WMF) [22:08:39] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10CCicalese_WMF) [22:09:23] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Goal, 10Services (watching): Modern Event Platform: Stream Configuration - https://phabricator.wikimedia.org/T205319 (10CCicalese_WMF) [22:12:26] 10Analytics, 10Operations, 10ops-eqiad: analytics1044 hardware failure - https://phabricator.wikimedia.org/T248413 (10Ottomata) Looks ok from here! Both hadoop-hdfs-datanode and hadoop-yarn-nodemanager seem ok. [22:15:29] 10Analytics, 10Event-Platform, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Wikimedia-production-error: EventBus rejecting events because of malformed characters in the comment - https://phabricator.wikimedia.org/T184698 (10CCicalese_WMF) @Pchelolo Is this still a problem? If not, can you please res... [22:16:22] 10Analytics, 10Event-Platform, 10Multi-Content-Revisions, 10CPT Initiatives (MCR): Redesign revision-related event schemas for MCR - https://phabricator.wikimedia.org/T186371 (10CCicalese_WMF) [22:17:38] 10Analytics, 10MediaWiki-API, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Patch-For-Review, 10User-Addshore: Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables - https://phabricator.wikimedia.org/T137321 (10CCicalese_WMF) @eprodromou Are you still planning on working on this? [22:18:38] 10Analytics, 10Event-Platform, 10CPT Initiatives (Modern Event Platform (TEC2)), 10Wikimedia-production-error: EventBus rejecting events because of malformed characters in the comment - https://phabricator.wikimedia.org/T184698 (10Pchelolo) 05Open→03Resolved a:03Pchelolo We haven't done anything to f... [22:27:31] inspired by nuria, just used presto for the first time to check on a few examples of a field from webrequest. so fast. :happy sigh: [22:29:06] 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, 10CPT Initiatives (Modern Event Platform (TEC2)), and 3 others: Modern Event Platform (TEC2) - https://phabricator.wikimedia.org/T185233 (10CCicalese_WMF)