[20:56:59] robla: well, that took what, about a minute? I love it when shit like this is reproducible :D [20:57:57] router of doom :D [20:58:07] fun! [20:59:39] anomie, you ready for https://phabricator.wikimedia.org/E237 ? [20:59:55] robla: Yes [21:00:39] #startmeeting ArchCom 2016W30: authenticated key-value store [21:00:40] Meeting started Wed Jul 27 21:00:40 2016 UTC and is due to finish in 60 minutes. The chair is robla. Information about MeetBot at http://wiki.debian.org/MeetBot. [21:00:40] Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. [21:00:40] The meeting name has been set to 'archcom_2016w30__authenticated_key_value_store' [21:01:01] #topic Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/ [21:01:25] tgr: When you say "large amounts of global data which are needed infrequently", what do you mean specifically? [21:01:38] #link https://phabricator.wikimedia.org/E237 Phab event link [21:02:09] #link https://phabricator.wikimedia.org/T128602 authenticated key-value store RFC [21:02:13] the specific use case that resulted in this RfC was reading lists [21:02:42] ie lists of favorite articles which are synchronized across devices [21:03:03] > While "reading lists" is one of the reasons the app teams want this, this isn't an implementation of page lists. [21:03:05] dbrant and anomie: thanks for your work on this! [21:03:07] This task makes no sense to me. [21:03:28] I think maybe we can start with the list of questions in the task [21:03:37] If you want a private list of favorite articles, that sounds like a watchlist. [21:03:41] question: does the store itself needs to be authenticated? I mean, we store sessions relying just on long random ID to be secure. Can't we use another long random ID to secure prefs? [21:04:06] Hi All! [21:04:16] o/ [21:04:18] SMalyshev: by default you'd do it behind the API which is authenticated [21:04:23] SMalyshev: Sessions are that way because of the bootstrapping problem: you have to start *somewhere*. [21:04:32] No need to invent new auth methods [21:04:47] alright, I guess we'll start with SMalyshev 's question, and then move to "Should this be implemented as a MediaWiki action API endpoint or a restbase service?" [21:04:51] SMalyshev: at least for the apps' use case, the reading lists should be private and therefore we'd need authentication [21:04:54] hi all, btw [21:04:55] SMalyshev: not sure what you mean by the store itself being authenticated [21:05:06] brion: well, yes, if it's mediawiki API then sure. but understand it's one of the options? [21:05:10] robla: I think focusing on implementation before defining use-cases is silly. [21:05:10] the API should authenticate, just like sessions do [21:05:38] #info Question discussed: does the store need to be authenticated? [21:05:56] Marybelle: the implementation of the store? Or the client side app feature that uses it? [21:06:06] If it's in the action API, it'll be authenticated just like anything else. If it's in restbase, I think that has support for the same sort of thing (but I don't know details). [21:06:16] how that's represented in the backend is an implementation details that is IMO low relevance compared to other factors for choosing backends [21:06:20] The app is a thing that already exists and already has a reading list feature. [21:06:23] brion: The use-cases of having another authenticated store. [21:06:43] If this is for private lists of articles, that sounds like a watchlist. [21:07:02] Marybelle ok. The use case is to store per user private data on the server and be able to retrieve and update it, with no versioning. [21:07:23] We do that already in several places. :-) [21:07:47] I'm not sure making yet another private place is needed. [21:07:55] Marybelle: yes, in ways that don't match this use case as already replied on the bug. [21:08:08] Okay. I guess I'm the only person not getting it. [21:08:09] Either we can change one of those methods to match, or make a new one. [21:08:13] note that the apps already store global authenticate private data, but they are currently using user_props for that, which is not terrible but suboptimal [21:08:13] Marybelle: this is a sort of generalization of watchlists, where the user can have multiple lists, each with its own attributes (name, description, whether it's saved for offline reading, etc) [21:08:26] so reading lists are not in fact the only current use case [21:08:47] #info conversation turned to discussion of use cases for Mobile App [21:08:56] tgr: we currently use user prefs to store theme info. we'd probably migrate that to this new store [21:09:02] brion: Right. I'd really like to avoid yet another special thing to maintain indefinitely. [21:09:05] (just as an example) [21:09:21] *nod* keep em simple when possible :) [21:09:22] reading https://phabricator.wikimedia.org/T128602#2499662 it looks like the delta vs. user_properties is mostly size/performance concerns? [21:09:30] niedzielski: You think MediaWiki's database should store a user's app theme preference? [21:09:31] dbrant: do you see this to eventually covering collection (book) functionality as well? [21:10:22] SMalyshev: and keeping complexity down [21:10:36] user prefs need to be loaded as a bundle, for perf reasons [21:10:50] Marybelle: what i mean to say is that we currently use user options to store a theme preference. if we had a general purpose key / value store, that would be a more appropriate spot [21:10:52] reading lists would have to be loaded one by one, again for perf reasons [21:11:01] SMalyshev: On the back end, anyway, although there's a question of whether it should be cross-wiki or if users should just pick one "central" wiki. On the front end it'll support fetching individual keys instead of having to fetch the whole thing at once, and possibly some other bits. [21:11:14] niedzielski: I'm not sure I get why a mobile app/client gets to use MediaWiki's database to store its preferences. [21:11:18] mixing the two would probably result in more awkward code then having different services for them [21:11:25] That seems weird. [21:11:28] from the RFC, it seems to me that one premise was "re-doing watchlists is hard, let's just a do a key/value store, that's easy". [21:11:30] But perhaps it's not that easy to do right, and so we should perhaps re-do watchlists instead? After all, we discussed global watchlists last week! Perhaps we'll be doing that anyway... [21:11:52] Marybelle: do you mean why store preferences on a server vs on device? or mediawiki specifically? [21:11:57] Marybelle: the MediaWiki db already stores plenty of user settings. Whether these settings apply to the desktop browsing experience or the app experience shouldn't matter. [21:12:00] Marybelle: because it's an app for the site which you log in with your site credentials, and it's that or invent a new storage service ? [21:12:15] We have several needs that drive changes to watchlists: global watchlists, multiple watchlists, automatic expiry... [21:12:25] DanielK_WMDE__: Part of the idea was prototyping their reading lists on top of a basic key-value store, instead of designing something then having to redo it when they find out they need different behavior. [21:13:01] DanielK_WMDE__: I think it's bigger than just watchlists, watchlists would be one usecase for this? [21:13:24] DanielK_WMDE__: actually the original proposal was to the k-v story as a temporary solution that makes sense on its own as well and then eventually migrate to a dedicated lists API based on some sort of lists concept in core [21:13:35] Right thisll be used for the other user data that's currently shoehorned into userjs prefs as I understand? [21:13:36] we might have given up on that by now, not sure [21:13:39] brion: I guess I think about non-Wikimedia Foundation apps. Would those clients also be using MediaWiki's/Wikimedia's database to store their user preferences and data? [21:13:52] anomie: having a flexible solution sounds really nice. what if we find out it didn't work the way we wanted it to? How does this not become tech debt? [21:13:52] gwicke: i wouldn't see why not; do you mean something like "turn my reading list into a pdf"? [21:13:55] Marybelle: sure, why not? [21:14:02] DanielK_WMDE__: As for watchlists in particular, the apps want the ability to have multiple lists that each aren't limited to a single wiki, and extra metadata, but not (I think?) actually the recentchanges-filtering functionality watchlists have. [21:14:04] SMalyshev: the driving use cases is (named) reading lists (aka bookmarks). which is very similar to watchlists. a k/v store could be used to cover this to some degree, as long as the lists don't become very large. [21:14:13] dbrant: yes [21:14:13] It seems outside the scope and responsibility of MediaWiki a bit. [21:14:57] Marybelle: no more than watchlist and user prefs for the web UI surely [21:15:00] robla: Are you asking about the key-value store, or a specialized reading-list service? If the former, that's the nice thing about a simple, generic key-value service. [21:15:04] anomie: cross-wiki watchlists are a lot of fun, as discussed here last week (or was it the week before)? [21:15:28] brion: MediaWiki the application using the MediaWiki database isn't so crazy. Any random client application using the MediaWiki database seems a lot zanier. [21:15:29] DanielK_WMDE__: apart from lists in core being a big and long project, I think it would be much more sane to go into it *after* we have a good understanding of the use cases, and a key-value store is great for prototyping [21:15:32] Those can all be implemented separately and could invent separate places to store their data, but I think it would not be super practical [21:16:03] Depends how narrowly or broadly you view MediaWiki IMO [21:16:15] i see two questions here, and we should perhaps pick one to discuss. a) do we want/need a generic k/v store, what needs does it address, what features does it need? and b) how do we best implement (or prototype) reading lists for mobile? [21:16:20] And how narrowly or broadly you view Wikipedia as a site or product or place [21:16:22] which of the two should we discuss? [21:16:27] Gather tried to build its own API from the start and maintain it across use-case changes, and pretty much ended up with a key-value store (JSON blobs in an SQL table) with lots of cruft around it [21:16:33] anomie: more the latter. I'll simplify to 3 options: 1) wild success 2) questionable success 3) obvious failure. outcome 2 is where tech debt accrues [21:16:41] one issue with a generic key-value store without schema enforcement is that any client side app could write any kind of blob to any key [21:16:45] And really, one of the open questions here is whether the key-value store should actually be in MediaWiki (action API) or should be a separate service for WMF app use (restbase). [21:17:04] this would put the burden of schema checking / validation squarely on the client [21:17:12] brion: Do other big sites let client applications use their databases for arbitrary private data? Like Twitter and Facebook and friends? [21:17:51] gwicke: You say that like it's a disadvantage. It could as well be an advantage. [21:17:56] Marybelle: are they platforms for sharing free knowledge? [21:18:01] :) [21:18:11] anomie: it's a trade-off [21:18:21] Anyway, we already can store tons of arbitrary data as you point out Marybelle [21:18:37] The question is can we do it in a way that's efficient and meets the needs of users [21:18:40] I mean, if I were making a regular bookmark application, I wouldn't expect MediaWiki to be my back-end off-hand. [21:18:45] there is the related issue of schema migrations [21:18:58] robla: Yeah, a specialized reading-list service would certainly have the danger of falling into #2. That's why I personally don't want to build one, at least not without decent planning to make it more likely to hit #1. [21:18:59] given that you can just create a wiki page and dump the data there I don't think it changes a lot [21:19:06] and format versioning [21:19:20] Schemas are out of scope for now, imo [21:19:32] (gwicke: can you please clarify what "collection (book) functionality" is? Thanks) [21:19:54] Scott_WUaS: I'm guessing https://www.mediawiki.org/wiki/Extension:Collection [21:19:58] brion: they are necessarily in scope, the question is just where you handle them [21:20:03] Marybelle: maybe, or you might store it in one of the several places in the application servers user database that it makes available for that sort of thing [21:20:04] brion: This use-case seems a bit against sharing free knowledge, if these are per-user and private, FWIW. [21:20:15] anomie: thnx [21:20:20] if you say it's out of scope on the server, then that implicitly means that they will need to be handled on the client [21:20:32] Or a separate server. [21:20:34] would it be an option to just expose an existing K/V system to the public (with the necesssary auth in place)? [21:20:39] gwicke: yes it's explicitly in the clients sphere of responsibility [21:21:00] DanielK_WMDE__: Existing like Redis or something? [21:21:05] gwicke: Once you start shoving schemas and stuff into it, it's no longer a generic key-value store. The client is free to implement a schema on top of a generic key-value store if it wants, which makes the store itself more flexible. [21:21:15] with multiple clients, this might be tricky to support [21:21:19] DanielK_WMDE__: if we can query individual items and not send them to every view, user props would work. [21:21:26] DanielK_WMDE__: I think that would be one of the solutions. If we have a suitable one [21:21:27] Marybelle: yes. though redis explicitly says it's designed to be accessed by trusted clients only (i just checked) http://redis.io/topics/security [21:21:31] That's basically the difference [21:21:46] Marybelle is not likely to be convinced about use cases this hour, but other folks seem more interested in talking implementation, so let's focus on implementation [21:21:52] :) [21:21:58] DanielK_WMDE__: it doesn't have to be redis directly, can be redis (or other non-auth k/v) behind Mediawiki API front [21:22:03] anomie: I guess something similar to how EventLogging schemas are handled could be done, I doubt it's worth the effort though [21:22:05] I asked on the task about just using a separate key/namespace in user_props and just filtering. [21:22:14] You don't have to send every user option on every page load. [21:22:17] I'm not sure why we do. [21:22:17] brion: ok, new plan: hack user props that keys starting with an underscore will be skipped when writing props into jsconfig. [21:22:20] DanielK_WMDE__: which supports only API like "give me my data", not "give me her data" [21:22:21] Marybelle might work [21:22:29] DanielK_WMDE__: +1 [21:22:47] :) [21:23:05] DanielK_WMDE__: Disadvantage: every existing user of user_props has to be updated to deal with the filtering, and we have to make sure the additional data doesn't have negative performance impact on the existing uses. [21:23:10] filtering user_props by prefix seems the simples solution by far... [21:23:21] Are there any low level probs with how that's stored? [21:23:21] would user options be able to hold thousands of arbitrary keys ok? [21:23:30] anomie: yes. existing uses need to be surveyed [21:23:35] see https://phabricator.wikimedia.org/T128602#2499662 for a list of problems with user_props [21:23:39] DanielK_WMDE__: if there's separate keys for up_property that would work I think [21:24:01] setting up key-value storage is fairly easy, I would say quite a bit easier than handling format versioning & migrations correctly [21:24:08] Also, jcrespo didn't like the idea of putting it in the main database much in https://phabricator.wikimedia.org/T128602#2476545 [21:24:10] Ok there's a byte limit I think on those, is that a problem? [21:24:16] Could be lifted with a schema tweak [21:24:17] anomie: does existing API right now just dumps all keys for the user, or it chooses specific ones? [21:24:32] Ah yes, and we did have req to move to separate db cluster [21:24:36] #info Discussion turned to authentication possibilities, and then to using user_props [21:24:37] Which is easy to do per table iirc [21:24:45] tgr: First three bullets seem trivially solvable. [21:24:46] SMalyshev: The existing API query only supports fetching all data for the user, unless I'm completely mistaken. [21:25:01] hm, i just found this: https://remotestorage.io/ [21:25:02] brion: what's the limit, roughly? [21:25:03] Yeah API needs enhancement for query [21:25:07] maybe it would be easier to drop a new generic key value store if the feature is unpopular than to clear user options [21:25:15] no idea if it's good, but it seems worth a look. [21:25:16] dbrant: 65534 bytes iirc [21:25:17] anomie: right, but that's only one API. I don't think it should be too hard to make this API skip certain keys in DB? [21:25:28] Should be a matter of changing column type [21:25:54] niedzielski: if there are clear patterns in how keys are structured, or there is only a single use case using this service, yes [21:25:56] anomie: also, if main DB is bad, we could make it two-stage - store opaque id in main db, store actual data in better storage [21:26:03] https://www.mediawiki.org/wiki/Manual:User_properties_table [21:26:11] SMalyshev: Then you're **really** making things complicated. [21:26:30] it's not *that* complicated I think... just one more call [21:26:31] At that point, why not just use the actual-data storage? [21:26:33] How is a filter complicated? [21:26:35] directly [21:26:41] anomie: because auth, etc. [21:26:54] again, if the goal is keeping things simple then having two separate systems with a very simple mode of operation seems better than having one that tries to be the mix of the two [21:26:56] I think maintaining two auth systems in sync is worse [21:26:58] SMalyshev: No, not direct access to the backend. [21:27:00] Huh. interesting. https://remotestorage.io/ is sponsored by the Wau Holland Stiftung? that indicates it's not industry bullshit. doesn't ell me if it's any good for our use case, but [21:27:10] #link https://remotestorage.io/ [21:27:22] Ooh that's neat [21:27:30] SMalyshev: But the frontend directly accessing the better-storage backend instead of looking up in user_properties then in the better-storage. [21:27:32] * brion bookmarks for later [21:27:33] anomie: that means two APIs instead of one, otherwise the same. I'd rather have users learn one API :) [21:28:04] SMalyshev: Ah, the Gather approach? [21:28:19] my vote is to just add a table [21:28:35] data like reading lists would likely be interesting to multiple applications; I would be surprised if the app would end up being the only user [21:28:46] avoid joins so that you can hack up some cross-server thing later if need be, query groups or something [21:29:01] * anomie is actually more interested in "BagOStuff vs some custom abstraction" over "where exactly do we hide the database table" [21:29:07] this whole project seems like something that could be done in hardly any more time than it takes to have this meeting [21:29:16] Hehe [21:29:31] simple key-value starage, sure [21:29:34] defining what to do is often longer than doing it... it's normal :) [21:29:42] but that's hardly a solution [21:29:46] +1 to gwicke -- i can easily see mobile web and/or desktop surfacing reading lists for logged-in users. [21:29:50] anomie: should we try to answer that question now? [21:29:52] :) [21:30:25] Is this spec in line with what we need? https://datatracker.ietf.org/doc/draft-dejong-remotestorage/ [21:30:29] That might be a more interesting question for the watchlist cross wiki magic future discussion :) [21:30:39] robla: Wouldn't hurt. Although "action API vs restbase" is an even more interesting question, since it's the difference between me writing it and me not ;) [21:30:49] if so, perhaps we can rely on some ready-to-go solution. [21:31:07] I just don't want mobile apps to get bogged down in the meantime while we bikeshed reading lists [21:31:17] dbrant: once we have multiple clients, it might make more sense to provide an API that gives a bit more guarantees than just "it's a blob of bytes" [21:31:25] DanielK_WMDE__: does it have TLDR description? [21:31:47] SMalyshev: https://remotestorage.io/ [21:32:48] updating the schema when all your schema handling is implemented in $n clients sounds hard [21:32:58] DanielK_WMDE__: this seems to be client-side storage? [21:33:26] DanielK_WMDE__: how would auth work? [21:33:30] or the picture is misleading [21:33:41] SMalyshev: no, it's not. [21:33:50] DanielK_WMDE__: "If is a URL, the user can supply their credentials for accessing the account (how, is out of scope) [21:33:58] - not so promising [21:33:59] SMalyshev: the client picks it's storage provider. local is one option. all providers implement the same protocol [21:34:00] DanielK_WMDE__: At a glance, that looks like it would serve the purpose. It wouldn't be able to be done in the context of the action API, but that's not a requirement (assuming it's not me writing it, anyway). [21:34:00] Is 65,535 bytes really insufficient? [21:34:24] Long names add up and it's hard to make guarantees :) [21:34:25] tgr: they seem to use oauth [21:34:29] That sounds like a lot of page IDs. [21:34:43] 64k would likely be enough in practice until it breaks on some extreme case [21:34:43] SMalyshev: it looks like the storage is OAuth protected server storage (that basically provides similar constraints to most client storage solutions) [21:35:00] forcing all apps to go through an extra oauth authorization screen might be suboptimal [21:35:14] using an actual standard to implement this would be nice. [21:35:27] forhtering the idea of "clients pick where they store their data" is even nicer [21:35:29] DanielK_WMDE__ agreed [21:35:32] anyway, that protocol seems like it might be a better solution but evaluating it within this IRC meeting is not realistic [21:35:49] Well ideally you'd probably want the same login for login and storage in a case like this [21:35:51] even if we implement this ourselves, perhaps we should implement the protocol defined there. [21:36:20] tgr: i agree. i'll link to it from the ticket [21:36:23] I am curious about it and it smells useful for unofficial apps and such, do bring it up in future :) [21:36:24] and worth mentioning again that building a k-v store that relies on the action API is super simple [21:36:27] Marybelle: brion: for reference, the current record holder for the most pages in reading lists in our app is over 7000. [21:36:49] bulding a new API that should match a draft protocol with OAuth 2 and whatnot is no [21:36:52] i don't think 64k would be too small personally. it would encourage favoring distinct keys instead of json blobs [21:36:53] *not* [21:37:08] #info DanielK_WMDE__ makes the case for using remoteStorage IETF draft implementation; discussion about that will likely continue in the RFC [21:37:12] 7000 dang! Might fit in int IDs bit maybe not titles yeah :) [21:37:38] Anyway that bits changeable if we need it [21:38:01] Easy to bump the column type up one [21:38:32] I understand that if we use IETF one we'd need a client for remoteStorage protocol and a backend. which should rely on some kind of k'v storage inside mediawiki [21:39:07] so we're kind of back to sq. 1? maybe with more standard client API [21:39:20] so...my understanding is that the Mobile Apps team plans to prototype *something* this quarter; should they use one of our existing tech, or should they explore something new? [21:39:38] So we have some interest in that protocol; and some talk about tweaking user props with a modified MW API method, and some talk about just adding a table still. Any other current main alternatives? [21:39:56] :) [21:40:11] restbase I suppose? [21:40:15] * robla notes that the "IETF one" is a draft that hasn't made it to "Proposed Standard", and that submitting an IETF draft is trivial [21:40:27] tgr: does restbase authenticate? [21:41:35] SMalyshev: restbase can authenticate [21:42:02] Ok so that's another possibility yes :) [21:42:24] Though I think gwicke would prefer restbase services to be better specified for their data schemas ? [21:42:41] And this is still a young feature likely to change in details [21:42:46] dbrant: How common is a reading list of 7000 pages? Just one user? A dozen users? [21:42:52] robla: to be clear, this isn't a blocker for our current goals this quarter. it would simply make our reading lists become a "complete" feature, the way we intended it to be (technically, two quarters ago :) ). [21:42:54] Eventually hopefully merging into fancy watchlists [21:43:25] dbrant: thanks for the clarification! [21:43:34] That helps us set timelines :) [21:43:38] brion: I get pretty frustrated at how bad watchlists are, especially when I see people basically working around them instead of investing resources to fix them. :-/ [21:43:51] Understandable! [21:44:12] I think we need to bring some more focus on that I agree [21:44:46] added a comment re versioning & API stability at https://phabricator.wikimedia.org/T128602#2500338 [21:44:48] For the general idea of private reading lists, it's kind of maddening that watchlists won't work. [21:44:54] Sigh. [21:45:06] * anomie wouldn't mind working on watchlists, but there are currently enough other cooks in that kitchen and that's not the problem at hand here. [21:45:09] * robla tries to remember what ori and Steven Walling were pushing for a few years ago on the watchlist front [21:45:14] SMalyshev: the point was that client libraries and backend implementations exist, we wouldn't have to write them (if the current ones are good - which i don't know) [21:45:39] DanielK_WMDE__: I don't think backend which we need (i.e. with mediawiki auth) exists? [21:45:42] (sorry for jumping back to this, don't let me distract you9 [21:45:53] dbrant: does the enhanced user pref model sound like a good short term for you as a sync mechanism? I think we all still agree fancier watchlist integration would be great for future [21:45:55] SMalyshev: backends with oath exist [21:46:25] with a page lists API it really helps if you have a good idea what features you'll need exactly [21:46:37] Marybelle: there are over 100 users with at least 1000 pages in lists. Over 4000 users with at least 100 pages, etc... [21:46:42] that's one place where Gather dug itself into the ground [21:46:54] a k-v store is ideal for prototyping [21:47:21] DanielK_WMDE__: yeah but oauth against what? we need something on mediawiki side to do actual r/w... even if oauth plugs seamlessly there. Maybe I just don't understand yet what that API does :) [21:47:24] for reading lists, and for a number of (non-list-related) future features I'd imagine [21:47:39] brion: i think that can definitely work, as long as it can handle a large number of keys. [21:48:07] Large number of keys should work with that model yes. You'd need a bulk lookup thiugh too? [21:48:12] DanielK_WMDE__: note that this seems to be using OAuth 2 which MediaWiki does not support [21:48:18] dbrant, brion: "as long as it can handle a large number of keys" is a good question. Again, https://phabricator.wikimedia.org/T128602#2476545 [21:48:21] tgr: hm, right... [21:48:24] Heh [21:48:37] probably not a huge undertaking to fix but way larger than the one proposed in this RfC [21:48:44] robla: https://www.mediawiki.org/wiki/Requests_for_comment/Support_for_user-specific_page_lists_in_core [21:49:18] ah, that's the one, thanks Marybelle [21:49:25] brion: right, lookup too. [21:50:16] but then, if we're talking about a short term solution, we can limit things on the client end, too. [21:50:32] (anomie: just noticed that you're mentioned in this BBC article "Meet the 'bots' that edit Wikipedia" - http://www.bbc.com/news/magazine-18892510 :) [21:51:26] one thing that hasn't been discussed is how much effort it would take to prevent abuse / how afraid we are it would happen [21:51:38] dbrant: ok let's maybe model a couple variants. Large blob, vs row per title? Then confirm whether they make sense on a tweaked user props table, and decide whether to look more at the alternatives? [21:51:53] pirates using it for movie distribution or whatnot [21:52:22] a quota can take care of that [21:52:36] brion: +1 niedzielski: what do you think of that? & [21:52:45] tgr: good question. There's little we can do to prevent use of user props as a file sharing or DoS space usage against us, other than "it's inconvenient and there are probably easier ways to abuse the system" [21:53:01] brion dbrant: one of the problems we consider with row vs blob were race conditions between clients. user options didn't seem well designed to handle that [21:53:07] tgr: the security considerations section of https://datatracker.ietf.org/doc/draft-dejong-remotestorage/?include_text=1 looks like a good start on a list [21:53:12] Changing API migh make it a bit easier to abuse buy not much [21:53:48] brion dbrant: for example in the blob scenario, if two clients try to update the same list, the last client wins. there's also bandwidth concerns for the 7000 title person [21:53:53] niedzielski: yeah, you'd have to detect conflicts through some other means like a signal value [21:54:10] Goes smoother with smaller bits, but that complicates the filtering [21:54:15] brion: we could do all kinds of usage tracking, user agent filtering etc [21:54:18] conflict resolution is somewhat orthogonal from storage strategy [21:54:30] but building it is little effort and those things probably aren't [21:54:33] Ok were low on time :) [21:54:48] robla: shall we plan next steps? [21:55:06] * robla ponders what that would be [21:55:25] overall, I'm honestly sceptical about the value of using a generic key-blob storage service for use cases like reading lists [21:55:39] I think we want to bump pruoity on the more watchlist specific rfc! [21:55:44] if the use case is so well defined, then I think it deserves a real API [21:55:46] Priority [21:56:00] gwicke: "if" [21:56:02] gwicke: agreed, medium to long term [21:56:18] #action ArchCom needs to bump the priority on a watchlist specific RFC [21:56:30] "if" [21:56:35] brion dbrant: IIRC, we also had concerns with a list of page IDs vs a page ID with a list of lists. i don't think we came up with a great way to handle that and had to use the list title as the ID [21:56:58] Short term: I'll follow up with dbrant and niedzielski on using user prefs modified and see if that still makes sense [21:57:29] And anyone else want to do more research on general user data storage with that protocol? [21:57:45] It sounds potentially very useful for unofficial third-party tools and such [21:57:52] brion: sounds good [21:57:55] brion: I'd suggest to run it by jcrespo too [21:58:02] Ah yes good [21:58:21] brion: I should probably take some of those action items from you, but yes, this all looks good [21:58:27] Hehe ok [21:58:30] (thank you for spelling this out!) [21:58:45] :) [21:59:09] anomie: dbrant - any last comments questions before we close this out? [21:59:18] Good discussion folks! [21:59:28] Yes! [21:59:28] robla: nope, really glad to see this moving forward [21:59:29] * anomie has no more comments at the moment [21:59:40] great discussion indeed....thanks everyone! [21:59:43] \o [21:59:45] Woohoo [21:59:48] o/ [21:59:56] Ok I gotta run, catch y'all later [22:00:02] #endmeeting [22:00:03] Meeting ended Wed Jul 27 22:00:02 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) [22:00:03] Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-07-27-21.00.html [22:00:03] Minutes (text): https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-07-27-21.00.txt [22:00:03] Minutes (wiki): https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-07-27-21.00.wiki [22:00:03] Log: https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-07-27-21.00.log.html