[00:09:47] dbrant: the dashboard is at https://grafana.wikimedia.org/dashboard/db/reading-list-service [00:10:05] wow, that's an ugly error rate [00:15:21] most of the errors on the MediaWiki side are internal ones during delete, I'll look into that [00:15:50] oh, that's cool [00:15:58] tgr: sounds good, thanks! [00:16:15] most of the errors on the RESTBase side seem to be 404s, those are probably malformed URLs, the service does not use that response code [00:17:37] I'll see if there is more detailed data about that [00:17:53] ok, I'd be curious to know. [00:18:42] so then, we should probably wait to roll out further until these items are ironed out? [00:19:32] yeah, also I should probably find someone to review the grafana code [00:19:52] it claims 200000 deletes in a two-minute span, which seems... unlikely [00:20:28] I might be summing response times instead of counts or something [00:20:41] lol yes, i see [00:24:44] 200 reads and 500 writes per hour, does that sound plausible? [00:36:16] tgr: hmm, that sounds a bit high... Is there a way to filter the counts by whether they're coming from the Beta app (vs. the developers' or QA's installs)? [00:36:52] not in graphite, I don't think so [00:37:06] More writes than reads seems fishy. Maybe an artifact of testing? Seems like the apps assume too much about the state on the server side. [00:37:25] but the way those numbers behave is completely crazy, they sometimes get smaller for larger time spans [00:37:43] Or maybe it just takes more batches to write things. Oh well. [00:38:06] which is par for the course for graphite, I just don't know it well enough to be able to work around all its weirdnesses [00:38:30] if most people only have one device more writes than reads is not that strange [00:38:39] where is ui for reading lists? [00:38:42] especially early on when a lot of syncing is happening [00:38:46] i thought there's a special page or something [00:38:57] the UI is in the Android app [00:39:09] can i have it for desktop please as well? :) [00:39:16] also, batch endpoints are not used yet, right? [00:39:42] Sveta: I think that's planned towards the middle of the year [00:39:47] tgr: batch endpoints are indeed used. (should they not be?) [00:39:54] ok. phab task, tgr? :) [00:39:59] i'll subscribe [00:40:20] dbrant: no, that's good, I just thought that's what makes the write query count high [00:40:36] for delete that's surely the case, you can see some huge spikes [00:42:45] according to client-side event logging, users are adding/removing between 10-20 pages per hour on the latest beta. [00:44:03] Sveta: T188830 apparently [00:44:03] T188830: Adding a list system for the desktop version - https://phabricator.wikimedia.org/T188830 [00:44:28] thanks tgr :-) [00:45:56] (but of course client-side eventlogging doesn't account for first-time sync that a newly-upgraded app performs) [00:49:13] hm logstash says there are somethoing like 30 deletion errors per hour for the last 24 hours [00:49:25] I don't trust grafana but logstash is pretty reliable [00:49:38] could the apps be retrying on error? [00:55:33] I can look into that... [00:55:38] what are the errors, though? [01:11:31] dbrant: T189272, T189273 [01:11:32] T189272: Error: Deleted row returned in non-changes mode - https://phabricator.wikimedia.org/T189272 [01:11:32] T189273: LogicException from line 517 of /srv/mediawiki/php-1.31.0-wmf.23/extensions/ReadingLists/src/ReadingListRepository.php: deleteListEntry failed for unknown reason - https://phabricator.wikimedia.org/T189273 [01:12:27] the error code generated from that is internal_api_error_LogicException [01:13:07] not sure if anything else gets through RESTBase [19:30:59] bearND: https://ifttt.com/applets/MizSfWT6-if-an-article-is-trending-on-wikipedia-then-send-a-rich-notification-from-the-ifttt-app :) [19:32:33] jdlrobson: nice. What's in a "rich" notification? [19:32:51] photo, title, link, editors, edits [19:34:03] pretty cool [20:13:34] bearND: hey so the reading list feature.. has there been any public announcements? [20:13:43] it's complete right? [20:13:45] https://www.mediawiki.org/wiki/Wikimedia_Apps/Synced_Reading_Lists < is this the best place to link to? [20:23:55] jdlrobson: I think tgr or coreyfloyd would know about this better than me. I seem to remember that the initial round is for apps, desktop is slated for later. [20:24:33] well, right now it's breaking all over the place... [20:24:43] but in theory, complete [20:25:16] Android is doing public beta, I don't think there have been any announcements [20:26:23] there is https://www.mediawiki.org/wiki/Reading/Reading_List_Service although the Android page is probably more informative [20:26:37] jdlrobson: ^ [20:37:05] tgr: breaking? how? [20:37:34] https://phabricator.wikimedia.org/T189272 mainly