[06:05:29] 10Analytics, 10Patch-For-Review, 10Performance-Team (Radar): Eventlogging processors are frequently failing heartbeats causing consumer group rebalances - https://phabricator.wikimedia.org/T222941 (10elukey) >>! In T222941#5492448, @Krenair wrote: > Is puppet on deployment-eventlog05.deployment-prep.eqiad.wm... [06:13:41] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` clo... [06:13:45] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cloudvirtan1001.eqiad.wmnet'] ` Of which those **F... [06:15:27] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` clo... [06:15:31] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cloudvirtan1001.eqiad.wmnet'] ` Of which those **F... [06:15:38] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` clo... [06:24:55] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['an-presto1001.eqiad.wmnet'] ` Of which those **FAI... [07:11:41] Hi team [07:12:44] o/ [07:23:54] elukey: With your approval I'll delete matomo failure - URL is: http://Wikipedia/screen/Explore [07:24:46] +1 [07:25:01] !log Delete matomo error with URL http://Wikipedia/screen/Explore [07:25:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:25:06] Thanks elukey [08:57:24] love some oozie shenanigans to start the week right [09:06:45] fdans: your karma will be balanced (one day) [09:06:48] keep fighting :D [09:07:08] elukey: job hasn't failed yet, this could be a good monday [09:30:32] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10elukey) * an-presto1001 ` NIC in Slot 4 Port 1: QLogic 577xx/578xx 10 Gb Ethernet BCM57810 - F4:E9:D4:... [09:30:44] this was my morning --^ [09:30:56] definitely not a good monday :D [09:35:57] elukey: from the task above, looks like we have 5 presto nodes ? really ? [09:36:01] MAAAAAAN [09:37:52] yes in theory eventually we should have 5 :D [09:38:11] those were the dedicated cloud nodes [09:38:27] that we are reclying into an-presto100[1-5] [09:38:44] (03CR) 10Fdans: [V: 03+1] "Job tested successfully for both monthly and daily granularities" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/536646 (https://phabricator.wikimedia.org/T232858) (owner: 10Fdans) [09:38:55] not sure exactly what setup Andrew has in mind though :) [09:41:56] elukey: I don't know either, but I hope we'll have some fun :D [09:56:15] 10Analytics: Install Debian Buster on Hadoop - https://phabricator.wikimedia.org/T231067 (10elukey) >>! In T231067#5487635, @MoritzMuehlenhoff wrote: > So, let me summarise to make I got this correctly. We have the following two options: > > 1. Upgrade to CDH 6.3 on Stretch which provides Hadoop and Scala suppo... [09:58:31] it is also very nice to read that all our users seem to use python 3 [10:04:01] !log disable puppet on an-coord1001 and manually forcing python3 for camus - T204735 [10:04:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:04:04] T204735: Move the Analytics Refinery to Python 3 - https://phabricator.wikimedia.org/T204735 [10:07:46] already one job started, all good [10:10:00] 10Analytics: Move the Analytics Refinery to Python 3 - https://phabricator.wikimedia.org/T204735 (10elukey) [10:11:26] (03PS1) 10Elukey: camus: switch to python3 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537055 (https://phabricator.wikimedia.org/T204735) [10:15:13] (03PS1) 10Elukey: is-yarn-app-running: move to python3 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537056 (https://phabricator.wikimedia.org/T204735) [10:16:01] 10Analytics, 10Patch-For-Review: Move the Analytics Refinery to Python 3 - https://phabricator.wikimedia.org/T204735 (10elukey) [10:16:11] 10Analytics, 10Patch-For-Review: Move the Analytics Refinery to Python 3 - https://phabricator.wikimedia.org/T204735 (10elukey) p:05Low→03Normal [10:23:36] 10Analytics, 10Patch-For-Review: Move the Analytics Refinery to Python 3 - https://phabricator.wikimedia.org/T204735 (10elukey) [10:23:58] (03PS1) 10Elukey: refinery-oozie-rerun: move to python3 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537057 (https://phabricator.wikimedia.org/T204735) [10:24:53] ok first code reviews out :) [10:26:22] 10Analytics, 10Patch-For-Review: Move the Analytics Refinery to Python 3 - https://phabricator.wikimedia.org/T204735 (10elukey) @mforns can you check in the task's description the `seems unused in puppet`? IIUC those got replaced by `refinery-drop-older-than`, so we could possibly delete some of those? Also, c... [10:29:54] * elukey lunch! [11:15:15] 10Analytics, 10CheckUser, 10Core Platform Team: Refactor Comment fields for CheckUser Component - https://phabricator.wikimedia.org/T232531 (10Rxy) LGTM `lang=SQL CREATE ALGORITHM=UNDEFINED DEFINER=`viewmaster`@`%` SQL SECURITY DEFINER VIEW `comment` AS select `testwiki`.`comment`.`comment_id` AS `comment_... [12:41:00] !log rebooting the hadoop test cluster with the new spicerack cookbook as test [12:41:02] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:43:12] 10Analytics: Verify what Python 2 packages deployed to Analytics hosts are needed - https://phabricator.wikimedia.org/T204737 (10elukey) From the response in the mailing list it seems that all our users have already migrated to python 3 a while ago. Let's keep this task open for a bit more to allow people to chi... [13:19:38] 10Analytics, 10Operations, 10Core Platform Team Legacy (Watching / External), 10Patch-For-Review, and 2 others: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10Ottomata) @herron FYI last week we decommissioned eventlogging-s... [13:20:45] 10Analytics, 10Operations, 10Core Platform Team Legacy (Watching / External), 10Patch-For-Review, and 2 others: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] - https://phabricator.wikimedia.org/T225005 (10herron) @Ottomata excellent thx for the heads up! [13:25:52] ottomata: o/ [13:25:53] morningggg [13:26:11] about --^ - is it ok to clean up the leftovers of profile::eventbus on role kafka main? [13:31:58] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10Ottomata) Thanks for doing this Luca! BTW we don't want buster...I mean I guess we could try it but I'd exp... [13:32:08] elukey: yes please! I jus thaven't done it yet [13:32:33] ack [13:32:54] Hi ottomata - Let me know when you have 5 minutes for me to explain the ua-parser python findings I have (5 minutes really, small :) [13:33:07] ottomata: And good morning, and please :) [13:41:27] goodmorning [13:41:33] joal checking email etc. you want IRC or batcave? [13:41:44] ottomata: batcave please, a lot easier [13:41:48] I have time though :) [13:41:55] can be post-standup as well [13:42:33] (03CR) 10Ottomata: [C: 03+1] "yeehaw" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537055 (https://phabricator.wikimedia.org/T204735) (owner: 10Elukey) [13:42:47] (03CR) 10Ottomata: [C: 03+1] "Super yeehaw" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537056 (https://phabricator.wikimedia.org/T204735) (owner: 10Elukey) [13:42:56] (03CR) 10Ottomata: [C: 03+1] refinery-oozie-rerun: move to python3 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537057 (https://phabricator.wikimedia.org/T204735) (owner: 10Elukey) [13:44:53] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10Ottomata) Ah, Luca just noted that we will have a Java 8 package for buster. I'm ok with Buster then! [13:45:04] joal: email done let's bc [13:45:28] (03CR) 10Elukey: [V: 03+2 C: 03+2] camus: switch to python3 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537055 (https://phabricator.wikimedia.org/T204735) (owner: 10Elukey) [13:45:50] (03CR) 10Elukey: [V: 03+2 C: 03+2] is-yarn-app-running: move to python3 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537056 (https://phabricator.wikimedia.org/T204735) (owner: 10Elukey) [13:45:58] (03CR) 10Elukey: [V: 03+2 C: 03+2] refinery-oozie-rerun: move to python3 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537057 (https://phabricator.wikimedia.org/T204735) (owner: 10Elukey) [13:48:02] joal: ? :) [13:48:17] oh excuse me missed the previous [13:48:19] joining [13:57:01] (03PS1) 10Fdans: Add aggregate mediarequests per referer endpoint [analytics/aqs] - 10https://gerrit.wikimedia.org/r/537114 (https://phabricator.wikimedia.org/T232857) [14:06:58] heya tem [14:07:00] team [14:07:26] hello! [14:07:39] joal: i was going to try to include the uap-core version number in the python-ua-parser version [14:07:58] but, it seems uap-python doesn't necessarily update the submodule to a tag? [14:08:00] ottomata: not released, but 0.6.9 [14:08:21] hmm ok yea [14:08:38] i can use that...maybe the unofficial 0.6.9 in some way? [14:10:38] it is ottomata - I decided to go with that cause 0.6.8 is broken for java code :( [14:11:44] joal: https://github.com/ua-parser/uap-python/issues/75 :) [14:12:03] Thanks ottomata :) [14:12:26] joal i'm going to go with [14:12:27] 10Analytics: Move the Analytics Refinery to Python 3 - https://phabricator.wikimedia.org/T204735 (10mforns) @elukey The following scripts should be removable after migration to drop-older-than: ` refinery-drop-banner-activity-partitions:#!/usr/bin/env python --> seems unused in puppet refinery-drop-eventlogging... [14:12:28] 0.8.0~uap0.6.9-1 [14:12:29] not sure we'll get an answer, but that's good (i'm not used to send issues - I should do !) [14:12:37] sound ok? [14:12:53] sounds great ottomata - Should I match the naming convention for java? [14:13:10] not a bad idea! unless java actually updates their versions properly when the submodule changese? [14:13:28] but i guess it might be nice to have the uap-core version explicit in the jar version too eh? [14:13:48] Yes, I think so - Will provide a CR [14:13:57] s/provide/update [14:14:13] joal is this really uap-coree 0.6.9 [14:14:14] ? [14:14:21] perhaps we should indicate that it is prerelease? [14:14:39] i'm going to try [14:14:39] ottomata: how would we? [14:14:40] 0.8.0~uap0.6.9~1-1 [14:14:41] k [14:14:46] Ahhhh [14:14:47] not sure if deb will like it [14:14:48] ok [14:17:55] (03PS10) 10Mforns: Add Oozie job for mediawiki history dumps [analytics/refinery] - 10https://gerrit.wikimedia.org/r/530002 (https://phabricator.wikimedia.org/T208612) [14:19:16] 10Analytics: Move the Analytics Refinery to Python 3 - https://phabricator.wikimedia.org/T204735 (10elukey) @mforns checked puppet without a simple grep, and: * refinery-drop-hive-partitions ` kerberos::systemd_timer { 'refinery-drop-query-clicks': ensure => absent, descr... [14:19:22] (03PS8) 10Mforns: Add spark job to create mediawiki history dumps [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/528504 (https://phabricator.wikimedia.org/T208612) [14:22:33] 10Analytics: Move the Analytics Refinery to Python 3 - https://phabricator.wikimedia.org/T204735 (10mforns) @elukey OK thanks! Will add those as a TODO for me. [14:38:37] joal, actually version will be [14:38:37] 0.8.0+uap0.6.9~1-1 [14:38:46] using + makes it sort after the last 0.8.0 version [14:38:51] so will be an upgrade [14:39:37] seems to work joal: https://gerrit.wikimedia.org/r/c/operations/debs/python-ua-parser/+/537123 [14:42:47] joal: ya works fine, can add to our apt whenever you are ready [14:57:13] ottomata: sorry went for kids - Trying to update my CR accordingly for java [15:02:00] nuria: standup? [15:02:08] ottomata: yesss [15:03:53] 10Analytics, 10Analytics-Kanban, 10Services (watching): Add mediarequests per referer endpoint to AQS - https://phabricator.wikimedia.org/T232857 (10fdans) [15:04:04] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Services (watching): Add cassandra loading job for mediarequests per referer - https://phabricator.wikimedia.org/T232858 (10fdans) [15:17:03] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10Services (watching): Add cassandra loading job for mediarequests per referer - https://phabricator.wikimedia.org/T232858 (10fdans) p:05Triage→03High [15:30:41] 10Analytics: Release wikimedia history dumps sorted and partitioned by user ID and page ID - https://phabricator.wikimedia.org/T232844 (10fdans) p:05Triage→03Low [15:30:48] 10Analytics: Release wikimedia history dumps sorted by user ID and page ID - https://phabricator.wikimedia.org/T232844 (10fdans) [15:31:25] 10Analytics: Release wikimedia history dumps sorted by user ID and page ID - https://phabricator.wikimedia.org/T232844 (10Nuria) This sorting makes total sense to facilitate user-focused research [15:41:56] 10Analytics: Check home leftovers of atgomez - https://phabricator.wikimedia.org/T232821 (10fdans) a:03Nuria [16:24:52] Gone for diner - Will be back after [16:25:04] 10Analytics: Check home leftovers of atgomez - https://phabricator.wikimedia.org/T232821 (10Nuria) Ya, +1 to removing all [16:25:31] 10Analytics: Check home leftovers of atgomez - https://phabricator.wikimedia.org/T232821 (10Nuria) a:05Nuria→03elukey [16:54:37] (03CR) 10Nuria: [C: 03+2] "If we have throughly tested job, let's have @joal give it a fast CR and we can merge." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/536646 (https://phabricator.wikimedia.org/T232858) (owner: 10Fdans) [17:11:10] * elukey off! [17:39:36] 10Analytics, 10Operations, 10Traffic: We are not capturing IPs of original requests for proxied requests from operamini and googleweblight. x-forwarded-for is null and client-ip is the same as IP on Webrequest data - https://phabricator.wikimedia.org/T232795 (10Nuria) @bblack Let me add more contex here, w... [17:45:17] 10Analytics: Can we add ORES data so it can be easily retrieved per revision present on mediawiki history? - https://phabricator.wikimedia.org/T232843 (10Nuria) [17:47:24] 10Analytics: Can we add ORES data so it can be easily retrieved per revision present on mediawiki history? - https://phabricator.wikimedia.org/T232843 (10Nuria) [17:51:27] (03PS1) 10Joal: Update pom.xml to release v1.4.4-core0.6.9~1-wmf [analytics/ua-parser/uap-java] - 10https://gerrit.wikimedia.org/r/537167 (https://phabricator.wikimedia.org/T212854) [17:52:25] (03Abandoned) 10Joal: Update pom.xml to release v1.4.4-core0.6.9~1-wmf [analytics/ua-parser/uap-java] - 10https://gerrit.wikimedia.org/r/537167 (https://phabricator.wikimedia.org/T212854) (owner: 10Joal) [17:53:46] (03Abandoned) 10Joal: Update pom.xml file to release v1.4.4-wmf [analytics/ua-parser/uap-java] (wmf) - 10https://gerrit.wikimedia.org/r/536678 (https://phabricator.wikimedia.org/T212854) (owner: 10Joal) [17:54:03] (03PS1) 10Joal: Update pom.xml to release v1.4.4-core0.6.9~1-wmf [analytics/ua-parser/uap-java] (wmf) - 10https://gerrit.wikimedia.org/r/537168 (https://phabricator.wikimedia.org/T212854) [17:54:30] ottomata: I have updated uap-java xml to try to match my comments in python --^ [17:55:40] hmm [17:55:51] i'm ok with that [17:56:04] joal +1 iwll change [17:56:30] \o/ [17:56:42] ottomata: deploy for wednesday evening is ok for you? [17:56:49] sure [17:59:01] nuria: I have 3 CRs I'd like to merge for wednesday deploy: https://gerrit.wikimedia.org/r/#/c/analytics/ua-parser/uap-java/+/536676/, [17:59:12] https://gerrit.wikimedia.org/r/#/c/analytics/ua-parser/uap-java/+/536677/ [17:59:34] and https://gerrit.wikimedia.org/r/#/c/analytics/ua-parser/uap-java/+/537168/ [17:59:58] I also have the refinery-source that goes with those, but will wait for artifact to be released [18:32:32] joal, I've seen that there are 2 types of event_timestamps that are weird: 1) the ones that are null, and 2) the ones that have crazy dates like from 1984 [18:33:05] right now I've put them all into a dump file called unknown.tsv.bz2 [18:33:07] mforns: There are null indeed - We should decide whether we keep them or not [18:33:20] is that OK to you? ^ [18:34:06] mforns: About crazy dates, can you please confirm they all come from user-data, event-type being on blocks? [18:34:22] joal, will do [18:34:23] it is mforns - I wonder if we shouldn't explicitely remove them [18:34:31] instead of keeping them weird [18:34:37] from dumps or from hive? [18:34:42] from dumps [18:34:57] from hive there is a task about weird dates, but there is nothing we can do about nulls [18:35:06] we can do that, it's a 1-liner change. Or else leave it very clear in the docs [18:35:25] yea, maybe it's less confusing to remove them [18:35:43] maybe 2-3 liner change :] [18:36:19] mforns: something else to check about the nulls, is if there is a pattern - If there are many, it could be related to pageCreation not being informed, and we should use pageFirstTimestamp instead (can't recall the choixe we've made in page table :( Need to read the docs) [18:36:44] there's very few [18:41:22] sounds good then :) [18:42:29] for enwiki, there's 49MB of weird-date events overall. compared to the 112GB of the total [18:43:28] it's 0.04% [18:44:11] 1 of each 2500 is weird [18:45:40] joal, and for the wiki I tested, all weird events are page creates [18:46:07] mforns: this is the pattern I was thinking of - Not good [18:46:13] :/ [18:46:31] 49M is significant when thinking in number of pages [18:46:48] 49MB not 49M [18:46:54] in size [18:46:59] Ah! [18:47:08] hm [18:47:19] the count is... [18:48:35] 3.1M [18:48:40] for enwiki [18:49:31] mforns: From https://phabricator.wikimedia.org/T220507, you should use page first edit timestamp, not page creation timestamp [18:49:56] Oh but wait, this should be done I think [18:50:11] I'm mistaken - please excuse me mforns [18:50:18] no prob [18:50:38] You use event_timestamp, and should have a correct value, except for page-creation events we don't know of [18:50:41] makes sense [18:50:53] Absolutely ok - We should drop those events [18:50:53] ok, :D [18:51:01] sorry for the mislead :) [18:51:09] and the ones with dates older than 2001? [18:51:46] same, from my previous ananlyses, they all come from us generating unblocks in the past due to wrong manual dates/rnages [18:51:50] we should discard them [18:52:06] joal, ok, will modify code accordingly [18:52:12] thanks!! [18:52:14] Thanks you :) [19:03:20] Gone for tonight team - see you tomorrow [19:04:26] laters! [19:24:54] 10Analytics, 10Analytics-Cluster, 10DC-Ops, 10Operations, 10ops-eqiad: analytics1045 - RAID failure and /var/lib/hadoop/data/j can't be mounted - https://phabricator.wikimedia.org/T232069 (10wiki_willy) a:03Cmjohnson [19:28:39] 10Analytics, 10Analytics-Cluster, 10DC-Ops, 10Operations, 10ops-eqiad: analytics1045 - RAID failure and /var/lib/hadoop/data/j can't be mounted - https://phabricator.wikimedia.org/T232069 (10wiki_willy) Hi @Dzahn @jbond - looks like this host is out of warranty, and about 3/4 of a year away from a hardwa... [19:29:42] 10Analytics, 10Analytics-Cluster, 10DC-Ops, 10Operations, 10ops-eqiad: analytics1045 - RAID failure and /var/lib/hadoop/data/j can't be mounted - https://phabricator.wikimedia.org/T232069 (10Dzahn) That's a question for the analytics team, please. [20:40:12] 10Analytics, 10Analytics-Cluster, 10DC-Ops, 10Operations, 10ops-eqiad: analytics1045 - RAID failure and /var/lib/hadoop/data/j can't be mounted - https://phabricator.wikimedia.org/T232069 (10Ottomata) Hello! In our FY2019-2020 hardware budgeting, we had planned to replace these nodes in Q4, when they ac... [20:44:52] mforns: i think we need to update this wikitech also to meantion we have decided to release data as a dataset rather than presto: https://wikitech.wikimedia.org/wiki/Analytics/Projects/Public_Data_Lake [20:46:01] nuria, ok will do [20:46:13] (03CR) 10Nuria: "How are we releasing this change to archiva?" [analytics/ua-parser/uap-java] (wmf) - 10https://gerrit.wikimedia.org/r/536676 (https://phabricator.wikimedia.org/T212854) (owner: 10Joal) [20:48:46] (03CR) 10Nuria: [C: 03+1] "Looks good, this is the only change we want in the wmf branch?" [analytics/ua-parser/uap-java] (wmf) - 10https://gerrit.wikimedia.org/r/536677 (https://phabricator.wikimedia.org/T212854) (owner: 10Joal) [20:54:54] (03CR) 10Nuria: [C: 03+1] "Looks good, do we upload this to archiva by hand?" [analytics/ua-parser/uap-java] (wmf) - 10https://gerrit.wikimedia.org/r/537168 (https://phabricator.wikimedia.org/T212854) (owner: 10Joal) [21:38:31] 10Analytics, 10EventBus: Clean up descriptions of fields in included common schemas in mediawiki/event-schemas repository - https://phabricator.wikimedia.org/T233057 (10Ottomata)