[00:13:14] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/450782 (owner: 10L10n-bot) [00:59:09] o/ [01:18:10] harej: My favorite type of herring! [01:35:55] (03CR) 10jenkins-bot: Maintenance script to backfill scores in PageTriage queue [extensions/ORES] - 10https://gerrit.wikimedia.org/r/449475 (https://phabricator.wikimedia.org/T198982) (owner: 10Sbisson) [12:57:51] 10Scoring-platform-team, 10ORES, 10Wikimedia-log-errors: ORES Storage::SqlScoreStorage exception every 2-3 minutes - https://phabricator.wikimedia.org/T201412 (10jcrespo) [12:59:44] 10Scoring-platform-team, 10ORES, 10Wikimedia-log-errors: ORES Storage::SqlScoreStorage exception every 2-3 minutes: Model contains an error for [id]: TimeoutError - https://phabricator.wikimedia.org/T201412 (10jcrespo) [14:00:52] o/ [14:32:43] * halfak joins the docs meeting [14:32:50] srrodlund & awight: ^ [14:45:13] * halfak works on the docs rather than having a meeting [16:01:41] harej, got some quick links to the etherpads you have been working on re. JADE? [16:01:58] https://etherpad.wikimedia.org/p/JADE_deployment [16:02:06] https://etherpad.wikimedia.org/p/JADE_integrations [16:02:39] also https://etherpad.wikimedia.org/p/JADE_multiple_judgments which has been on hold (but is still an open tab) [16:03:50] Thanks! [16:34:40] 10Scoring-platform-team, 10ORES: "could not decode as JSON" error during autolabeling - https://phabricator.wikimedia.org/T201434 (10awight) [16:37:07] rats, an overnight error might block me from deploying, again. [16:38:05] On the bright side, we’re at srwiki so only 8 models remaining. [16:45:18] 10Scoring-platform-team (Current): [Discuss] Random sampling by PAWS vs API requests - https://phabricator.wikimedia.org/T193789 (10Halfak) It seems to me that we want to have a long-term snapshot of input data. This is helpful when we discover some type of statistical anomaly, it'll be nice to check the *exact... [16:52:16] halfak: I’ll ask you to look over the editquality model_info diffs later today…. Some of the models looks better, some worse. [16:52:32] I don’t know how much variation to expect by random chance. [16:52:40] Anything look substantial? We should have a utility to help with this. [16:52:42] There are also some changes in training data, I’m wondering what that’s about. [16:52:44] +1 [16:53:02] Or at least a rubric I can learn. [17:15:09] 10Scoring-platform-team (Current), 10DBA, 10JADE, 10Operations, 10User-Joe: Extension:JADE scalability concerns - https://phabricator.wikimedia.org/T196547 (10Halfak) [17:21:14] 10Scoring-platform-team (Current), 10DBA, 10JADE, 10Operations, 10User-Joe: Extension:JADE scalability concerns - https://phabricator.wikimedia.org/T196547 (10awight) [17:27:58] halfak: This probably exists somewhere, but I’d like to know the standard deviation of how much fluctuation I should see in AUC etc. stats after retraining. [17:28:12] -> utility like you said. [17:29:08] Also: I love typing “make” and nothing else. [17:29:18] Now *that’s* living. [17:58:50] This might be a good outreach and learning event, https://aspirationtech.org/events/devsummit18 [17:59:02] Maybe we’ll have some experience with JADE to share :-/ [18:01:51] 10Scoring-platform-team (Current), 10ORES, 10User-Ladsgroup: Rewrite ORES "reference" UI using React - https://phabricator.wikimedia.org/T195274 (10awight) @Ladsgroup Kicking this back to "Active" since it sounds like you're doing a round of coding? [18:08:36] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10User-Ladsgroup: Use rc_timestamp index when joining to ores_classification - https://phabricator.wikimedia.org/T138444 (10awight) @Ladsgroup It looks like this task should be renamed or even deleted, now that the solution is to use the `rc_thi... [18:11:59] 10Scoring-platform-team (Current), 10Wikilabels, 10User-Ladsgroup: Write deployment guide for Wikilabels - https://phabricator.wikimedia.org/T194809 (10awight) Looks good. I'd like to see some explanation about how to sample revisions for labeling, and where to document that sample, but it can wait for futu... [18:14:32] 10Scoring-platform-team (Current), 10ORES, 10Documentation: Draft of ORES threshold optimization documentation - https://phabricator.wikimedia.org/T198232 (10awight) One more important detail to document is to explain why different threshold optimization functions are used in various use cases, aside from ju... [18:21:50] 10Scoring-platform-team (Current), 10DBA, 10JADE, 10Operations, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) [18:34:26] 10Scoring-platform-team (Current), 10DBA, 10JADE, 10Operations, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Nemo_bis) > To me it still seems the easiest solution would be to put this on a separate wiki. This was... [18:44:18] * awight counts to 36,000 before responding… [18:46:03] 10Scoring-platform-team (Current), 10Documentation, 10Easy, 10MW-1.32-release-notes (WMF-deploy-2018-07-17 (1.32.0-wmf.13)): Document JADE judgment structure - https://phabricator.wikimedia.org/T179301 (10awight) 05Open>03Resolved [18:52:50] 10Scoring-platform-team (Current), 10ORES: Experiment with LIME integration for ORES, providing explanations for its predictions - https://phabricator.wikimedia.org/T196475 (10awight) a:03awight [18:54:48] 10Scoring-platform-team (Current), 10DBA, 10JADE, 10Operations, 10User-Joe: Extension:JADE scalability concerns - https://phabricator.wikimedia.org/T196547 (10awight) a:03awight [18:55:03] 10Scoring-platform-team (Current), 10DBA, 10JADE, 10Operations, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) a:03awight [18:55:05] 10Scoring-platform-team (Current), 10ORES, 10editquality-modeling, 10artificial-intelligence: Duplicated feature name in editquality - https://phabricator.wikimedia.org/T197679 (10awight) a:03awight [18:55:44] 10Scoring-platform-team (Current): Write annual plan proposal for Scoring Platform FY19 - https://phabricator.wikimedia.org/T183203 (10awight) a:03Halfak [18:56:15] 10Scoring-platform-team (Current), 10JADE, 10I18n, 10MW-1.32-release-notes (WMF-deploy-2018-07-10 (1.32.0-wmf.12)), 10Patch-For-Review: Copy Wiki Labels translations over to Extension:JADE - https://phabricator.wikimedia.org/T198302 (10awight) a:03awight [18:58:08] 10Scoring-platform-team (Current), 10DBA, 10JADE, 10Operations, 10TechCom-RFC: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Halfak) @Nemo_bis, thanks for chiming in. There are a lot of concerns I have about a central wiki from a... [18:58:14] awight, ^ [18:58:24] Anything I missed? [18:59:44] ah maybe dump those good thoughts into the evaluation of alternatives that harej has been working on? [18:59:55] I’ll read through in a minute and comment here. [19:08:47] taking a late lunch [19:08:51] halfak: great point re. i18n [19:08:56] :D [19:09:28] Ooh and on the complexity of getting cross-wiki buy-in. [19:10:23] Thanks for fielding that. [19:19:10] FWIW, harej’s doc has additional arguments against the central wiki, * technical efforts to integrate across wikis, * Turbulence and possible failure of wishing a new community into existence. * Trust and social integration between central and target wikis. [19:24:00] I’ve copied your arguments to harej’s doc. [19:51:38] lunching. [19:54:40] awesome [20:19:15] Is there a single argument in *favor* of a central wiki? Other than making ops happy? [20:19:31] Nemo adds some [20:19:47] I don't think they stand up, but they are certainly valid arguments. [20:19:57] That's what I thought; I'll give another look though [20:20:03] "Do it in your own wiki. We're not sure we want JADE yet." [20:22:30] I think he made his post in good faith, but I am not sure he thought through the problem fully, which is totally his prerogative but I think it doesn't help his argument. [20:23:22] If you think of the problem as purely "using the revision table breaks it" then a central wiki seems like a great idea. [20:24:22] If you ask the harder question "how do we best accomplish these particular business goals" then you have to factor more considerations than just technical feasibility [20:25:35] I’m not impressed by the difficulty of consensus argument, because we’ll have almost an identical discussion in either case. [20:26:03] I think it's mooted by the fact that JADE is only going to operate in places where there's consensus anyway [20:26:13] awight, right. I totally agree. We shouldn't be doing something that affects a wiki community deeply without a consensus discussion either way. [20:26:26] +1 +1 [20:26:35] By the way, with a central wiki, people would be able to write judgments for any wiki right away. [20:26:55] harej: Sure, but halfak brought up the point that we might need to restrict that technically. [20:27:13] Otherwise, we’re targeting wikis without their consensus and probably causing social friction. [20:27:34] err I guess I added the technical part, but halfak pointed out the problematic aspect. [20:27:42] Also, people are totally going to vandalize and harass people in JADE [20:27:51] Who is going to patrol that for the unrepresented wikis!? [20:28:44] In any case, I'm just about ready to declare the central wiki idea dead [20:29:04] It introduces challenges that we do not have the resources for. [20:29:17] Challenges that would cost millions of dollars to address. [20:29:22] It’ll probably have a higher proportion of patrollers, in its favor, but they will be annoyed at extra workload on the very service meant to help with all that. [20:29:43] harej: which challenges specifically are that expensive? [20:30:08] I just see an enormous time sink, much higher risk with no benefit, and an annoying few months of development. [20:30:24] I'm drawing from the experience of Wikidata causing friction with wiki communities that don't trust it [20:30:32] kk that sounds valid [20:30:52] And the product development and software engineering efforts to make Wikidata better integrated with the projects [20:32:28] When Ami.r1 is back to scoring Platform work tomorrow, we could ask him to lay out what would need to be done to mimic Wikidata integration with client wikis. [20:33:58] You do have a fair (implied) point that we can copy from the experiences/lessons of Wikidata for free [20:36:47] But even despite those, I worry there are fundamental incompatibilities that we can't just paper over so easily. [20:37:44] Like the whole idea of an itwiki editor patrolling edits from zhwiki without being a part of that community [20:38:30] And we'd have to get into messy political battles of who is allowed to touch what, and have this play out on a central platform rather than let the community handle their own project [20:39:36] * halfak turns on fast paced music and works on something technical for a little while. [20:41:30] lol [20:42:05] BTW, awight I finally took a pass on Threshold Optimizations docs. [20:42:24] https://www.mediawiki.org/wiki/ORES/Thresholds [20:42:31] halfak: I saw, it looks much better! I left some comments on the task for the next round of improvements I’d like to make before calling it a draft. [20:42:52] Cool :) [20:44:32] harej: I want to nudge T200297, and ask TechCom to schedule us into the IRC meeting whenever possible, but currently thinking that we should wait until your alternatives doc is ready, and use that as the fulcrum to push this off the cliff… If that sounds good to you? [20:44:32] T200297: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 [20:45:19] awight: i think the document is basically ready; do you have more specific criteria in mind? [20:45:37] lemme see... [20:46:06] Good enough, we’ll say :-) [20:46:11] Want to move to a wiki? [20:47:51] JADE/Implementations/Alternatives? [20:49:53] I want to combine it with the FAQ to create a contingency planning doc [20:50:08] I will work on that [20:54:18] OK, let me know how I can help. Just use the FAQ as raw material to draw on, no need to preserve it as-is. [20:56:31] halfak: Whenever you feel like CR’ing model_info, https://github.com/wiki-ai/editquality/pull/169/files [21:00:14] wiki-ai/editquality#364 (T197679_new_feature - cc63a37 : Adam Wight): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/413307859 [21:00:27] awight eswikiquote goodfaith model looks weird. [21:00:32] Lots of new false-positives. [21:00:45] PR AUC drops substantially. [21:01:11] Might be worth re-training from scratch to see if something clogged up the pipeline. [21:01:17] I wish we kept old data in git... [21:01:39] Observations didn't change it seems [21:02:09] * halfak mostly looks at ROC-AUC and PR-AUC [21:02:12] And Counts [21:02:15] halfak: I don’t see an eswikiquote goodfaith, only reverted [21:02:33] Ah eswikibooks.goodfaith [21:02:41] eswikibooks.goodfaith.md [21:02:45] Woops [21:06:11] halfak: This is strange, {"review_reason": null, "needs_review": true} [21:06:15] Or is that expected? [21:06:31] Expected. [21:06:49] kk [21:06:49] Sometimes no reason for exclusion or inclusion applies and that defaults to inclusion. [21:07:22] Wasn't reverted. User wasn't blocked. Not a trusted user. Just an edit by some ~newcomer. [21:11:07] awight, eswikibooks.goodfaith is the only one with the issue. [21:11:12] Just noted that in the PR [21:11:30] Great, I’ll fix in a followup patch [21:32:26] awight: "JADE/Contingency planning" okay for a page name? [21:33:46] +1, but see halfak’s note above, we already have some alternative technical implementations documented in [[mw:JADE/Implementations]] so that would be a reasonable page to take over or nest under [21:33:46] 10[5] 10https://www.mediawiki.org/wiki/JADE/Implementations [21:45:17] 10Scoring-platform-team, 10ORES, 10Wikimedia-log-errors: ORES Storage::SqlScoreStorage exception every 2-3 minutes: Model contains an error for [id]: TimeoutError - https://phabricator.wikimedia.org/T201412 (10awight) We're retrying revisions many dozens of times, it would be better to follow the upstream OR... [21:49:13] halfak: Rebuilding eswikibooks data and models fixed the glitch [21:49:21] awight, great! [21:49:28] I'm glad it wasn't something deeper. [22:10:47] awight, looks like this issue is not on us. I'm going to step back and head out for the day. [22:13:18] right o [22:14:34] o/ [22:14:38] 10Scoring-platform-team, 10ORES, 10Wikimedia-log-errors: ORES Storage::SqlScoreStorage exception every 2-3 minutes: Model contains an error for [id]: TimeoutError - https://phabricator.wikimedia.org/T201412 (10awight) There's some MediaWiki API unhealthiness corresponding to the beginning of our trouble, so... [22:14:41] have a good one! [22:14:57] 10Scoring-platform-team, 10ORES, 10Performance-Team, 10Wikimedia-log-errors: ORES Storage::SqlScoreStorage exception every 2-3 minutes: Model contains an error for [id]: TimeoutError - https://phabricator.wikimedia.org/T201412 (10awight) [22:45:25] awight: how does this look? https://www.mediawiki.org/wiki/JADE/Implementations#Implementation_strategies [22:45:32] Or [22:45:34] That page as a whole [22:46:19] Tried to remove some duplicative copy in the process [22:52:03] sorry, on a call for another few [23:29:19] harej: I think you hit the nail on the head. Good structure, solid arguments, nice formatting. [23:30:28] Excellent, thank you. [23:31:27] harej: Care to ping the RFC task with that link? [23:33:25] I'll take care of that [23:43:03] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/451147 (owner: 10L10n-bot) [23:47:21] 10Scoring-platform-team, 10Multi-Content-Revisions, 10ORES, 10Epic: MCR support in ORES - https://phabricator.wikimedia.org/T195779 (10awight) [23:48:09] 10Scoring-platform-team, 10Multi-Content-Revisions, 10ORES, 10Epic: MCR support in ORES - https://phabricator.wikimedia.org/T195779 (10awight) [23:48:12] 10Scoring-platform-team, 10ORES: ORES feature extraction triggers new MCR-related deprecation warning - https://phabricator.wikimedia.org/T201332 (10awight) [23:49:25] 10Scoring-platform-team, 10ORES: ORES feature extraction triggers new MCR-related deprecation warning - https://phabricator.wikimedia.org/T201332 (10awight) These changes update us to the new API, but we only process main slot content: https://github.com/wiki-ai/revscoring/pull/409 [23:49:39] 10Scoring-platform-team (Current), 10ORES: ORES feature extraction triggers new MCR-related deprecation warning - https://phabricator.wikimedia.org/T201332 (10awight) [23:52:39] wiki-ai/revscoring#1520 (mcr_api - 087dd52 : Adam Wight): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/413363614