[21:00:15] #startmeeting RFC meeting [21:00:15] Meeting started Wed Aug 1 21:00:15 2018 UTC and is due to finish in 60 minutes. The chair is kchapman. Information about MeetBot at http://wiki.debian.org/MeetBot. [21:00:15] Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. [21:00:15] The meeting name has been set to 'rfc_meeting' [21:00:27] #topic RFC: Modern Event Platform - Choose Schema Tech [21:00:43] #link https://phabricator.wikimedia.org/T198256 [21:00:56] who is here for this discussion today? [21:00:57] o/ [21:01:00] o/ [21:01:11] hi all [21:01:21] hello everyone [21:02:29] ottomata do you want to kick things off with a brief summary? [21:02:42] sure! [21:02:55] can I assume that everyone has at least skimmed https://phabricator.wikimedia.org/T198256 ? [21:03:18] probably, but give us the tl;dr ;) [21:03:29] alllright [21:04:18] the Modern Event Platform program is about unifying our production event systems to make it easier for tech, product and analytics to all build event based systems [21:04:58] This has been done at WMF so far with EventLogging, and more recently with the EventBus service (which is a part of EventLogging software) [21:05:37] before proceeding with more Modern Event Platform (MEP? ok MEP.) planning, we need to choose a schema tech [21:05:52] this choice informs a lot of the work we will have to do going forward [21:06:20] as summarized in T198256, the choices we are looking at at JSONSchema and Avro [21:06:21] T198256: RFC: Modern Event Platform - Choose Schema Tech - https://phabricator.wikimedia.org/T198256 [21:06:28] Obvious question: Do EventBus and EventLogging use the same schema tech currently? [21:06:33] yes and no [21:06:39] they both use JSONSchema [21:06:47] so from that perspective yse [21:06:59] for the use cases I've encountered, JSON one looks more attractive. It's much easier to produce/consume by every platform (everything knows JSON now) and for human comprehension [21:07:15] but, EventLogging uses on wiki Draft 3 JSONSchema, and events neeed to be stitched together into an EventCapsule [21:07:34] of course there's performance question, but I don't know how important that is. [21:07:37] EventBus uses Draft 4 JSONSchemas (stored in a git repo), and event schemas are all self contained [21:08:11] SMalyshev: yeah, I waffled about this choice for a really long time, and only in the last month or so started being sure that I preferred JSONSchema. [21:08:53] one of the main arguments for Avro is that it is widely used in the event+stream processing world. There are a bunch of open source tools and library that already exist for it. [21:09:12] we'd have to build some of those tools ourselves for JSONSchema [21:09:36] One of the tools I want to be able to use is Kafka Connect [21:09:57] its hard to use JSON with Kafka Connect (and many other systems) because JSON isn't strongly typed, even if it does have a JSONSchema [21:10:04] but, it is possible! [21:10:05] kafka connect allows you to supply it custom (de)serialisers, so that should be a non-issue wrt picking JSON [21:10:13] sort of [21:10:16] but yes [21:10:20] i've got a poc implementation of that [21:10:29] https://github.com/ottomata/kafka-connect-jsonschema [21:10:39] there you go :) [21:10:57] ya, understanding how that all worked was one of the things that tipped me onto the JSONschema side [21:10:58] all the way [21:12:44] it works for most things, but there are edge cases where it won't [21:12:49] doesn't look outrageously complex... so the questions is between "more usability and more work on our side" vs "less friendly format but more existing tools for it" [21:12:50] the main one i'm thinking of is JSON's number type [21:13:01] pretty much SMalyshev [21:13:22] there might even be 'more work on our side' if we choose Avro, in that we'd have to eventualy port all the existing JSONSchema usages over [21:13:51] as a user, I'd be on the "more usability" side of course but I think ultimately it's on people that are going to implement the missing parts to say if it's ok with them [21:14:03] SMalyshev: yeah, Avro is a pain sometimes [21:14:20] and performance... do we suffer anything noticeable from going JSON and having all that parsing? [21:14:22] since you need the schema to read it, you'd always need some special library to get the schema for you when you are reading data [21:14:41] noticeble, sure. if we were to compare performance (i haven't) json would almost certainly lose [21:14:53] but, the goal of the MEP program isn't to improve performance [21:15:04] we're using json for webrequest logs [21:15:16] and are doing ok [21:15:25] its unlikely we'll need to scale beyond that for other events [21:15:30] ok then I think if it's good enough for that firehose it's probably good enough for the rest of us [21:15:35] and they probably make up 90+% of all events [21:15:55] ya [21:17:55] hm, any more questions I can help with? [21:17:59] the task has more detail on pros/cons [21:18:02] the extra benefit of using json is that even if we realise that we need more performance, we can always start using something else internally in the event data platform itself, but do the conversion between it and the clients [21:18:22] <_joe_> +1 [21:18:51] that's a good point [21:19:00] aye like if there was a really complicated series of stream jobs that had to do a lot of deserialization and reserialization? [21:19:24] <_joe_> I am concerned about what will happen if this platform starts getting adopted widely for asyncronous messaging between applications, in terms of performance/size, but avro is hardly the best choice if we want to optimize for performance anyways [21:19:44] <_joe_> (I would pick thrift probably) [21:20:13] <_joe_> so I second json as a user-friendlier public-facing interface, and I don't think we need to optimize early [21:20:28] <_joe_> changing protocols later is going to be painful though [21:20:42] aye. i suppose if an app needed something like thrift, they wouldn't need the whole event platform infrastructure? [21:20:46] e.g. a schema repo, etc. [21:20:46] ? [21:21:20] <_joe_> Well for thrift you *need* a schema repo of sorts [21:21:30] <_joe_> but we don't really need to consider it now [21:21:38] <_joe_> as you said, it's unlikely we need it [21:21:40] yes, but if there is some high perf app that uses thrift for message passing...they'd just have the thrift schemas in code? [21:21:49] <_joe_> yes [21:21:50] _joe_: i don't think the idea is for this data platform to be used as a service bus, and i would personally discourage such a paradigm [21:21:51] aye [21:22:06] <_joe_> mobrovac: +1 of course [21:22:59] <_joe_> ottomata: schema migrations can be painful with jsonschema, we need schema best practices to make sure we don't shoot ourselves in the foot with those [21:23:03] listening to and reacting to events is one thing, but expecting certain events (or msgss) to appear should be outside the scope of this [21:23:11] indeed. the rule so far has been: you are only allowed to add optional fields [21:23:12] <_joe_> do you have a good solution for that? [21:23:35] anything else isn't allowed [21:24:09] <_joe_> ottomata: ok, well, I think you can even be smarter than that (you can add required fields in the new schema, as long as the old one has additionalProperties: true) [21:24:15] MEP has in it a schema(+metadata?) repo/registry component, which hopefully can enforce that [21:24:20] <_joe_> anyways, I see you considered the problem [21:24:27] additionalProperties is kidna bad for us though [21:24:34] we need the schema to be predictable [21:24:45] for job queue stuff we had to allow it for backwards compatibility reasons [21:24:49] <_joe_> well, we're going a bit offtopic [21:25:03] but for any future schemas we'd have stricter rules [21:25:09] <_joe_> but one thing we should do is to have precise policies, I guess :) [21:25:11] ya [21:25:13] for sure [21:25:34] i would be +1 on constraining the things we can and/or cannot use from json schema [21:25:43] <_joe_> I would assume writing those is part of the project - and pretty related to this rfc too [21:25:44] ya [21:25:50] hm, aye [21:26:12] the RFC mentions things like additionalProperties [21:26:31] but doesn't nail down the policy 100% [21:26:42] that is part of the schema guildelines component of MEP [21:27:32] strong-typed schemae are easier to get right and easier to enforce and more predictable [21:28:43] yeah, that's why folks more often choose avro (or thrift or whatever) for this kinda stuff. But I think JSON + JSONSchema with strict policies will work for us [21:29:08] <_joe_> it's also easier to consume for casual clients that are more liberal in what they accept [21:29:16] ya [21:29:47] maybe we can formalize the stricter policies and enforce them automatically when people commit schemas to the repository [21:29:53] yeah [21:30:11] i think that will be done by the schema registry service we build [21:30:24] maybe on wiki, maybe not [21:30:42] we are about half way through the meeting. Don't forget to make use of the /#info tag to note important details or conclusions in the minutes [21:31:06] ah right, good point [21:31:26] so ya, any other questions? it sounds like we are in agreement here, yes? [21:31:46] #info broad consensus about going with JSON / JSONSchema with strict policies [21:32:16] We'll want the schema registry to be easy for non-programmers to follow. But that's independent of the format of the messages. JsonSchema seems reasonable. [21:32:18] yup [21:33:14] dr0ptp4kt: whatever it is will be better than the schemas on meta now [21:33:51] hopefully anyway [21:33:52] :) [21:35:46] great broad consensus! :p [21:35:48] what now? [21:36:36] i think we may be done here :) [21:36:50] are there any details that need to be discussed now? _joe_ mobrovac anything else you think TechCom needs to know? [21:37:20] (it is also true we don't have to use a full hour just because) [21:37:50] +1 for not using metawiki anymore. I've always been concerned there that making a revision there makes it valid acceptable in prod even if the consumer didn't want to. [21:38:15] One can be strict about the 'revision' property right now in consumers, but that's not often done, and even then, would still leak it into the 'valid' topic. [21:38:16] yup, it gets so weird with taht EventCapsule too [21:38:18] ottomata: given that quite a part of the conversation was about the schemae and their policies, it sounds like it would be good to prepare another RfC around that [21:38:19] Preventing that would be desirabe. [21:38:24] <_joe_> kchapman: nope, I think we'd be interested in a further discussion of the architecture of the platform imaybe [21:38:42] <_joe_> but not now ofc [21:38:43] ok cool, those can come in future RFCs or regular meetings about MEP [21:38:56] getting this decision made lets us start planning those [21:39:17] oh look even sounds like we have consensus to close the meeting :) [21:39:57] ok great! Should I note the decision on the task, and link to the RFC meeting notes when they are up? [21:40:19] that would be great. They will be up in seconds when I end it :) [21:40:24] #endmeeting [21:40:33] Meeting ended Wed Aug 1 21:40:24 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) [21:40:33] Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2018/wikimedia-office.2018-08-01-21.00.html [21:40:33] Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2018/wikimedia-office.2018-08-01-21.00.txt [21:40:33] Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2018/wikimedia-office.2018-08-01-21.00.wiki [21:40:33] Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2018/wikimedia-office.2018-08-01-21.00.log.html [21:40:34] coool, thanks yall [21:42:22] off for the eve, ttyl