[00:26:46] I realized that we don’t need the Kafka feed to populate the ORES mirror of JADE data. [00:32:14] Another surprise twist is that suppressions never affect the latest judgment, so we don’t need to do anything special in ORES unless we’re storing historical data. Which we aren’t in this next phase. [00:32:19] halfak: ^ [03:16:38] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [03:22:38] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [03:41:39] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [03:51:48] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [04:13:48] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [04:20:48] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [04:30:58] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [04:32:58] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [05:03:59] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [05:14:08] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [05:18:29] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikipedia - https://phabricator.wikimedia.org/T174878#3970670 (10Bodhisattwa) @Halfak , the Bengali translation for "Edit quality (5k revisions)" is "গুণম... [05:34:24] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikisource - https://phabricator.wikimedia.org/T186711#3970691 (10Bodhisattwa) The Bengali translation for "Edit quality (5k revisions)" is "গুণমান সম্পাদনা (৫০০০ স... [05:34:40] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikisource - https://phabricator.wikimedia.org/T186711#3970692 (10Bodhisattwa) List of trusted user groups - sysops [05:34:59] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikisource - https://phabricator.wikimedia.org/T186711#3970693 (10Bodhisattwa) [07:15:34] PROBLEM - ping4 on ORES-worker04.experimental is CRITICAL: PING CRITICAL - Packet loss = 100% [07:15:35] PROBLEM - ssh on ORES-lb02.Experimental is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:15:57] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [07:16:06] PROBLEM - ping4 on ORES-lb02.Experimental is CRITICAL: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [07:16:14] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [07:17:18] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:17:49] PROBLEM - ORES home page on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:17:58] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:17:58] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:56:02] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [07:56:17] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [08:31:28] 10Scoring-platform-team (Current), 10ORES: Preliminary deployment of ORES to new cluster - https://phabricator.wikimedia.org/T185901#3970942 (10akosiaris) [08:31:30] 10Scoring-platform-team, 10ORES: Switch ORES to dedicated cluster - https://phabricator.wikimedia.org/T168073#3970943 (10akosiaris) [08:31:33] 10Scoring-platform-team, 10ORES, 10Operations, 10Patch-For-Review: rack/setup/install ores2001-2009 - https://phabricator.wikimedia.org/T165170#3970944 (10akosiaris) [08:31:35] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3970939 (10akosiaris) 05Open>03Resolved a:03akosiaris Yes that's right. Resolving [08:31:48] 10Scoring-platform-team, 10ORES: Rebuild ORES models on Stretch - https://phabricator.wikimedia.org/T184072#3871494 (10akosiaris) This is done, right ? [08:31:55] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Rebuild ORES wheels on Stretch - https://phabricator.wikimedia.org/T184135#3970947 (10akosiaris) I am guessing we can call this resolved with the `stretch_conversion` branch is merged into `master` ? [08:32:15] 10Scoring-platform-team, 10ORES: Switch ORES to dedicated cluster - https://phabricator.wikimedia.org/T168073#3355113 (10akosiaris) [08:32:17] 10Scoring-platform-team (Current), 10ORES: Preliminary deployment of ORES to new cluster - https://phabricator.wikimedia.org/T185901#3928152 (10akosiaris) 05Open>03Resolved Yes, this is done [08:33:46] (03PS4) 10Alexandros Kosiaris: Remove the cluster server group and related stuff [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/409932 (https://phabricator.wikimedia.org/T171851) [08:34:37] 10Scoring-platform-team, 10ORES, 10Operations, 10Scap: Use external dsh group to list pooled ORES nodes - https://phabricator.wikimedia.org/T179501#3970955 (10akosiaris) And scap configuration updated in https://gerrit.wikimedia.org/r/#/c/409932/. When that one is merged this can be called done as well [08:36:02] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [08:36:17] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [08:46:13] 10Scoring-platform-team, 10ORES: Switch ORES to dedicated cluster - https://phabricator.wikimedia.org/T168073#3970976 (10akosiaris) [08:46:53] 10Scoring-platform-team, 10ORES: Switch ORES to dedicated cluster - https://phabricator.wikimedia.org/T168073#3355113 (10akosiaris) And https://gerrit.wikimedia.org/r/#/c/410398/ finishes the migration. All that is left is to disable ORES on scb* boxes, which I find prudent to stall for a bit (couple of days a... [09:16:03] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [09:16:17] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [09:56:02] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [09:56:17] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [10:36:02] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [10:36:17] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [11:16:07] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [11:16:22] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [11:56:12] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [11:56:27] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [11:59:56] awight i think that ^^ may be related to https://phabricator.wikimedia.org/T187292 [12:02:41] paladox: nice, thanks for the connection [12:02:52] Your welcome :) [12:03:12] ive reported it in -cloud to make sure :) [12:08:19] youch, 127C is smoking hot [12:36:12] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [12:36:27] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [13:16:12] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [13:16:32] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [13:56:17] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [13:56:37] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [14:36:22] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [14:36:42] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [14:44:36] halfak: o/ [14:44:55] Hey dude. [14:44:59] I sent some feverish thoughts last night, a pretty big epiphany nonetheless. [14:45:33] We only need the rev_id of create_revision and suppression events. [14:45:45] So we don’t need to listen to Kafka, simple change prop will do. [14:53:31] awight, was thinking the same thing about changeprop :) [14:54:15] I’m trying to set this up in mw-vagrant, but need to design extensibility for the changeprop config file. [15:07:46] akosiaris: Web worker count graph looks strange, do you know what’s happening there? [15:08:16] Does this mean that the ores* web servers have been pooled successfully? [15:08:39] awight: define "strange" [15:08:48] but yes all ores* hosts are fully pooled [15:08:54] I think I’m less concerned now [15:08:56] and all scb hosts fully depooled [15:09:00] scb1002 had a max of 15 workers [15:09:03] aha! [15:09:06] Congrats!! [15:09:06] web wise that is [15:09:10] celery still works [15:10:44] akosiaris: Is it possible that scb1002 is still pooled? [15:12:04] awight: it's changeprop. It's the only client that has a really weird network path to ores and will effectively be done when ores is removed from scb [15:12:14] aha ty [15:12:28] That’s a major client, of course. [15:12:48] I think 95% or so of our work comes through changeprop [15:14:00] We’re adding another changeprop endpoint for another data stream fwiw, it would be nice if the internal networking were sane. Please lmk if you see things to fix there. [15:14:18] halfak also fyi if you see "PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs)" [15:14:25] that's due to https://phabricator.wikimedia.org/T187292 [15:14:30] it's on the list there [15:16:22] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [15:16:42] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [15:28:05] akosiaris: changeprop uses URLs like https://ores.wikimedia.org/v2/scores/arwiki/, which is surprising, I would have expected the internal domain name. Do you know why it hits scb1002 specifically? [15:30:44] awight: it uses http://ores.svc.eqiad.wmnet:8081/v2/scores/arwiki, not ores.wikimedia.org. And the reason is that the active changeprop currently runs on scb1002 so it hits the local ores instance [15:31:04] oh haha that’s wacky. [15:31:16] it's one of the reasons we push forward for the kubernetes stuff [15:31:17] I guess it doesn’t matter what the URL is, then. [15:31:23] it solves that weird problem [15:31:27] no the URL does matter [15:31:36] I’m looking at wikimedia/change-propagation/config.example.wikimedia.yaml [15:31:38] if it was ores.wikimedia.org we would not have that [15:31:44] but we don't want it to be that [15:31:51] cause it makes no sense to go through varnish [15:32:01] Is the real config somewhere else? [15:32:13] puppet [15:32:39] profile::changeprop::ores_uris: in hieradata/role/common/scb.yaml [15:56:22] PROBLEM - Host ORES-worker04.experimental is DOWN: CRITICAL - Host Unreachable (ores-worker-04.ores.eqiad.wmflabs) [15:56:42] PROBLEM - Host ORES-lb02.Experimental is DOWN: CRITICAL - Host Unreachable (ores-lb-02.ores.eqiad.wmflabs) [16:14:18] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES: ORES extension highlights edits that are patrolled - https://phabricator.wikimedia.org/T187337#3972193 (10Ladsgroup) [16:15:33] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 442 bytes in 2.266 second response time [16:15:50] RECOVERY - Host ORES-lb02.Experimental is UP: PING OK - Packet loss = 0%, RTA = 2.09 ms [16:15:54] RECOVERY - Host ORES-worker04.experimental is UP: PING OK - Packet loss = 0%, RTA = 6.49 ms [16:15:55] RECOVERY - ping4 on ORES-worker04.experimental is OK: PING OK - Packet loss = 0%, RTA = 3.92 ms [16:16:02] RECOVERY - ORES home page on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 420 bytes in 0.004 second response time [16:16:02] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 457 bytes in 0.556 second response time [16:16:22] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 457 bytes in 0.544 second response time [16:20:32] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikipedia - https://phabricator.wikimedia.org/T174878#3972253 (10Halfak) {{done}} :) [16:20:32] How efficient, wikibugs! [16:20:54] lolol [16:20:58] battle bots [16:22:05] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Complete edit quality campaign for Bengali Wikipedia - https://phabricator.wikimedia.org/T187339#3972257 (10Halfak) [16:22:18] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikipedia - https://phabricator.wikimedia.org/T174878#3575726 (10Halfak) [16:22:20] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Complete edit quality campaign for Bengali Wikipedia - https://phabricator.wikimedia.org/T187339#3972268 (10Halfak) [16:23:02] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Train damaging/goodfaith models for Bengali Wikipedia - https://phabricator.wikimedia.org/T187340#3972271 (10Halfak) [16:23:14] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Complete edit quality campaign for Bengali Wikipedia - https://phabricator.wikimedia.org/T187339#3972257 (10Halfak) [16:23:16] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Train damaging/goodfaith models for Bengali Wikipedia - https://phabricator.wikimedia.org/T187340#3972281 (10Halfak) [16:24:03] 10Scoring-platform-team, 10Documentation, 10Easy: Mock JADE discussion page - https://phabricator.wikimedia.org/T179301#3972285 (10Halfak) @awight, do you think we still need to do this? [16:24:20] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikipedia - https://phabricator.wikimedia.org/T174878#3972287 (10Halfak) [16:24:57] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for Bosnian - https://phabricator.wikimedia.org/T174784#3972294 (10Halfak) a:03Halfak [16:25:02] 10Scoring-platform-team, 10Documentation, 10Easy: Mock JADE discussion page - https://phabricator.wikimedia.org/T179301#3972296 (10awight) I think we should still document what's happening in a JADE judgment, lemme rewrite the task. [16:25:48] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10Hindi-Sites, 10artificial-intelligence: Add language support for Hindi - https://phabricator.wikimedia.org/T173122#3972297 (10Halfak) Checking in on this. Any progress? [16:25:59] 10Scoring-platform-team, 10Documentation, 10Easy: Document JADE judgment structure - https://phabricator.wikimedia.org/T179301#3972298 (10awight) [16:30:14] Ewww https://phabricator.wikimedia.org/T186557 [16:30:18] I'm looking into this now [16:31:13] oh dear. You must really want to get into that text editor. [16:32:43] It's very grounding and makes me happy :) [16:37:44] halfak: Here’s the company I was mumbling about, Riot Games: https://www.wired.com/2014/05/fighting-online-harassment/ [16:38:05] They seem to have not done the thing I imagined, but the article details a few initiatives that actually did happen. [16:41:00] Right on. A buddy of mine is in the middle of researching this at Riot. [16:41:15] He comes to CSCW. Hopefully I'll be able to make an introduction in november. [16:42:49] They had some tremendous successes, according to their stats! [16:43:55] Right. It would be cool if we could pull that sort of thing in. We have a different sort of community, but I'm sure that a bit of this is applicable. [16:44:02] E.g. highlighing and rewarding the good. [16:44:12] The insight that everyone is at least a little trollish. [16:46:53] hell yeah [16:47:21] wooo I fixed it! [16:49:12] 10Scoring-platform-team, 10Wikilabels: WikiLabels OAuth handshake doesn't work with HTTPS - https://phabricator.wikimedia.org/T186557#3972422 (10Halfak) https://github.com/wiki-ai/wikilabels/pull/226 [16:49:24] 10Scoring-platform-team (Current), 10Wikilabels: WikiLabels OAuth handshake doesn't work with HTTPS - https://phabricator.wikimedia.org/T186557#3972425 (10Halfak) [16:50:45] 10Scoring-platform-team (Current), 10ORES: Rebuild ORES models on Stretch - https://phabricator.wikimedia.org/T184072#3972431 (10Halfak) [16:50:58] 10Scoring-platform-team (Current), 10ORES: Rebuild ORES models on Stretch - https://phabricator.wikimedia.org/T184072#3871494 (10Halfak) 05Open>03Resolved [16:51:01] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Make sure ORES is compatible with stretch - https://phabricator.wikimedia.org/T182799#3972438 (10Halfak) [16:52:09] wiki-ai/wikilabels#309 (fix_https_auth_redirect - 14007fc : halfak): The build failed. https://travis-ci.org/wiki-ai/wikilabels/builds/341524122 [16:52:18] boo [16:53:24] halfak: Are “blueprints” enabled? Or did you smoke test locally? [16:54:03] Oh I smoke tested. Thanks for asking. [16:54:18] Just fixed the flake8 issue [16:54:57] OK, I think I see where Blueprint is registered [16:55:05] Only asking cos I’m too lazy to do so :p [16:55:25] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES: ORES extension highlights edits that are patrolled - https://phabricator.wikimedia.org/T187337#3972193 (10Halfak) Is this a #edit-review-improvements-integrated-filters problem or the old-school ORES interface problem? [16:56:09] codezee, is that oneVsRest stuff ready for re-review? [16:56:59] 10Scoring-platform-team (Current), 10editquality-modeling, 10User-Ladsgroup, 10User-Tgr, 10artificial-intelligence: Train/test damaging and goodfaith model for Hungarian Wikipedia - https://phabricator.wikimedia.org/T185903#3972478 (10Halfak) Hey! Checking in here. How's progress? If you're stalled, I... [16:57:29] wiki-ai/wikilabels#311 (fix_https_auth_redirect - 43803fc : halfak): The build was fixed. https://travis-ci.org/wiki-ai/wikilabels/builds/341526257 [16:57:42] 10Scoring-platform-team (Current), 10JADE, 10Epic: Deploy JADE prototype in Cloud VPS - https://phabricator.wikimedia.org/T176333#3972484 (10Halfak) [16:57:59] awight, when you get a chance, could you adjust https://phabricator.wikimedia.org/T176333 to match the deployment plan you laid out? [16:58:14] good call [16:58:45] 10Scoring-platform-team, 10ORES: Meta ORES: API data storage and querying - https://phabricator.wikimedia.org/T153145#3972501 (10Halfak) [16:58:47] 10Scoring-platform-team (Current), 10ORES: Design JADE data storage schema - https://phabricator.wikimedia.org/T153152#3972496 (10Halfak) 05Open>03Resolved a:03Halfak [16:58:49] 10Scoring-platform-team, 10MediaWiki-extensions-CentralNotice, 10ORES: Target CN banners at users based on recent ORES scores - https://phabricator.wikimedia.org/T187346#3972502 (10Jseddon) [17:00:11] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikisource - https://phabricator.wikimedia.org/T186711#3972538 (10Halfak) Thanks! Working on this now. [17:00:23] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikisource - https://phabricator.wikimedia.org/T186711#3972539 (10Halfak) a:03Halfak [17:04:44] 10Scoring-platform-team (Current), 10JADE: Deploy JADE prototype in Beta Cluster - https://phabricator.wikimedia.org/T176333#3972568 (10awight) [17:05:20] 10Scoring-platform-team (Current), 10JADE: Deploy JADE prototype in Beta Cluster - https://phabricator.wikimedia.org/T176333#3621812 (10awight) [17:05:22] 10Scoring-platform-team (Current), 10JADE: Build prototype JADE extension - https://phabricator.wikimedia.org/T187216#3972578 (10awight) [17:16:41] 10Scoring-platform-team (Current), 10Wikilabels: WikiLabels OAuth handshake doesn't work with HTTPS - https://phabricator.wikimedia.org/T186557#3947294 (10Halfak) a:03Halfak [17:22:23] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for Bosnian - https://phabricator.wikimedia.org/T174784#3972653 (10Halfak) Ping @Srdjan_m, it looks like we need some translations. I'll be getting this deployed, but you'll likely see... [17:26:24] awight, do you know the command to build the new, automated makefile? [17:26:37] yes, one moment [17:26:46] Oh! "generate make" [17:26:48] :D [17:27:25] editquality generate_make [17:27:50] Should DTRT with no options [17:28:14] I suggest running it normally, then --diff-only “svwikibooks,enwiki” etc. [17:28:18] Oh my. it looks like defaults were implemented manually [17:28:20] ewww. Fixing. [17:28:28] hrm [17:28:44] It's nice to have explicit defaults as part of docopt. [17:29:01] Yeah I think I wrote package in 45 minutes [17:29:10] just cos, fun [17:29:31] heh. it works :) [17:29:43] ah they were left out of docopt because they need to be relative to the root dir [17:29:47] please do fix :) [17:30:08] also, I lied about the —diff-only syntax, as you can see. [17:31:56] I'm going to make stdout the default output, OK? [17:32:18] so ./utility generate_make > Makefile [17:32:35] or ./utility generate_make --output=Makefile [17:34:02] ^ awight [17:34:17] meh why? [17:34:21] * halfak standardified the arguments. [17:34:29] because that's how all of our other utilities work [17:34:35] We’ll eventually be writing multiple files. [17:35:13] Oh? [17:35:25] IMO yeah it’s much nicer to have a makefile for each wiki [17:35:41] OK well we'd need to change the args anyway for that. [17:35:58] When files are being generated, I don't really care if its one big file or not. [17:35:59] I don’t think the stdout thing is necessarily consistent, because this tool is closer to autoconf than the data-munging scripts. [17:36:19] But it does produce a single output. [17:36:26] hehe [17:36:50] You can still provide the argument. [17:37:02] If you like doing things that way. [17:37:56] It's nicely unixy to read from stdin and writing to stdout. [17:38:40] Then you can do things like generate_make | grep "####" to see all the wikis. [17:38:58] IMO, the most unixy way to do things would be that “make” generates its own makefile [17:39:13] e.g. Makefile is a stub which will run the code generation if necessary [17:39:29] sure. That'd be fine. but for a CLI utility... [17:39:44] anyway, I won’t block this small change, please go ahead with whatever is inspiring! [17:40:22] It doesn’t conceptually work for me, because it’s not a pipe, it has a known input, but I’m fine with a few extra chars [17:40:48] automake/autoconf are generally reviled and are no role models to follow [17:41:25] "it's not a pipe" what do you mean? [17:43:24] It’s rendering templates, so pulls data from multiple sources. It’s not a goes-inna-goes-outta [17:43:50] The fact that it produces a single output file is a temporary condition, and not its… essence [17:44:14] that’s why it takes —input-dir and —config args, cos “<“ would be inappropriate [17:44:39] awight, also true of our model trainers. [17:44:52] Just because it originates from multiple sources doesn't mean we shouldn't write stdout. [17:45:15] okay! [17:45:26] definitely an interesting point [17:45:38] We'll see. I'll have it be a separate PR. [17:45:39] makes me want to reconsider the model trainers, though :) [17:46:13] model training feels like “gcc” [17:46:59] … which of course is inconsistent [17:47:15] I guess we give gcc -o arguments [17:48:00] Glad you’re digging into the codegen, though! I think it’s a fun package, and should be extracted into its own thing so we can reuse. [17:54:07] +1 [17:56:33] I did a codegen thing for a side job, which combined multiple configuration files, multiple templates, and read DDL from a database to create multiple outputs. [17:56:52] I’ve been idly thinking about a generalized multi-file templating instruction file. [18:00:02] Argh, calendar fail. halfak: Could you tell me the date of the invite I just sent? [18:00:05] it’s… gone [18:00:12] not on the day I intended, I guess [18:01:00] lol [18:01:10] nvm it is on the day [18:01:12] but rejected [18:01:36] Feb 21 [18:01:38] ? [18:02:39] yup I’ve moved already [18:02:42] disastrous [18:03:03] 8:30am is a good way to get everyone’s attention, though ;-) [18:10:17] lol [18:20:38] awight, looking at differ.compare() [18:20:45] It seems to output all lines [18:20:57] And only provide annotations for the lines that changed. [18:21:10] Is that the intended behavior or should we limit it to minimal context [18:21:13] ? [18:21:16] I think full context is better [18:21:27] OK I'm fine with that. Just harder to work with, I think [18:21:52] cos eyeballing that comparison involves a lot of checking whether unchanged rules existed in the first place [18:22:03] full context lets you search [18:22:47] fwiw, the differ can go away once we’re done converting, it has no place in regular codegen AFAIK [18:23:28] I think it's a nice output for checking your work. But maybe then we can only have minimal context :)( [18:23:34] :O [18:26:25] ooh I just noticed the spec endpoint, cool! https://ores.wikimedia.org/v3/spec/ [18:27:40] :D [18:28:27] halfak: Here’s an example where that would have been annoying: https://gist.github.com/Ladsgroup/f1abb376b5ad329a68845e954ffebb8f [18:29:03] eh that one doesn’t demonstrate well [18:29:33] I think that's a fair point. [18:29:55] In that case, I'd rather use `diff` directly. [18:30:16] here’s an example, https://gist.github.com/Ladsgroup/ad77bf59273263134e4c4dc5a1cc0309#file-templating_last_batch-patch [18:30:53] enwiki.autolabeled_revisions.w_cache.20k_2015.json gets added as a dependency, and it doesn’t have a rule [18:31:19] not possible to tell whether that rule is missing, without the full context of the wiki’s rules [18:31:41] How do you suggest we use diff directly? Entire Makefile vs. Makefile.automated? [18:32:13] Those files are supposed to be disjoint with the exception of the bit we’re converting, so the diff is huge. [18:32:14] Right. [18:32:35] Ahh. So you want to compare a single section at a time but have the full diff. [18:32:45] full section diff [18:33:01] That gist I linked is an example of what we were inspecting [18:33:22] rules for each wiki are encapsulated, so they can be reviewed in isolation [18:34:49] :) [18:35:23] still a lot of boiler plate, but it’s nice, right? [18:35:54] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10Bengali-Sites, 10artificial-intelligence: Edit quality campaign for Bengali Wikisource - https://phabricator.wikimedia.org/T186711#3972965 (10Halfak) https://github.com/wiki-ai/editquality/pull/126 [18:36:06] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for Bosnian - https://phabricator.wikimedia.org/T174784#3972966 (10Halfak) https://github.com/wiki-ai/editquality/pull/126 [18:38:22] wiki-ai/editquality#111 (bawiki_bnwikisource - 7142834 : halfak): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/341566806 [18:38:48] OK off to lunch [18:38:50] back in an hour [18:38:51] o/ [18:39:10] wiki-ai/editquality#112 (generate_make - b694797 : halfak): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/341567304 [19:48:45] lunching [19:51:42] wiki-ai/ores#922 (jade_data - 756b192 : Adam Wight): The build failed. https://travis-ci.org/wiki-ai/ores/builds/341594988 [20:00:21] was in meetings. [20:00:33] Actually I'm going to step out again for a few minutes. [21:07:05] I now have rough vagrant and ores code to [21:07:06] to [21:07:07] to [21:07:14] to [21:07:15] wire changeprop into ores [21:07:17] lol [21:07:19] :D [21:07:22] fail [21:08:06] I guess I should stick with the ORES architecture, where requests create a *Request object, which is processed in Celery? [21:08:20] It’s all IO work, fwiw [21:08:52] one MW API call. [21:08:56] awight, that doesn't sound crazy. [21:09:02] MWAPI call to get the JSON? [21:09:13] Exactly. [21:10:02] Roger that. [21:10:12] I mentioned this to ottomata and he said, < 2MB or so is reasonable for Kafka packets, so in theory we could send new events with the entire revision content [21:10:13] but [21:10:20] I think we should have a separate storage strategy for JADE data. [21:10:23] I decided not to, thus far [21:10:28] oh? [21:10:36] The ORES cache is an LRU [21:11:24] so we don’t want jade inserts resetting the timer? [21:11:27] Hmm... It would be good if we could tell the difference between "this revision has no judgements" and "this revision has judgements but we deleted them from the LRU" [21:11:52] Good point. [21:11:56] We don't want to have to make an extra call to MW every time that a new revision is saved. [21:12:24] No, that’s not the strategy so far [21:12:35] Let's say that a judgement cycles out of the LRU [21:12:39] I’m only making a MW API call to get Jade: content, so it’s only on JudgmentContent revision-create [21:12:52] But where to store it? [21:13:08] So far, I’ve been imagining we could jam it into the same cell as the ORES scores [21:13:15] make that a list or hash [21:13:21] hash is better [21:13:41] Ahh. I see. So any time we generate a score, we'll make an extra call to check if there's judgement data in MW? [21:14:12] No, the only time we call MW is when we receive a changeprop notification about a specific Jade: page [21:14:22] awight, but the scores are stored in an LRU [21:14:25] I hadn’t thought about cache expiration, yeat [21:14:30] ahh yeah. [21:14:32] yeah got that, it’s a good point [21:14:52] We can make the Jade: MW API request for old scores [21:14:57] for new scores it’s never necessary [21:15:13] That depends on how new. [21:15:13] the judgments will be up-to-date as long as the score is in Redis [21:15:18] If it's changeprop, we're probably safe [21:15:23] But anything after that isn't [21:15:36] What do you mean by “after”? [21:15:45] After changeprop happens [21:15:49] For the edit being scored [21:16:02] "precaching" [21:16:03] changeprop will continue to update judgments for that revision [21:16:12] oh—precaching is independent of the JADE changeprop [21:16:19] two different streams. [21:16:31] precaching are Main: notifications, jade changeprop are only for Jade: pages [21:16:38] Right. So I guess we'd have a race, but it would work most of the time. [21:16:52] I don’t see a race condition [21:16:58] Now then, how old is old enough that we should make a MW call? [21:17:21] Either Redis operation (insert ORES score or updade JADE judgment) will be atomic [21:17:40] Old enough == if the score isn’t found in the LRU cache, I guess. [21:17:40] Oh if there's a minor delay in changeprop while someone requests a score, they might get the score after a judgement has been saved, but before it propagated. [21:18:06] IMO that’s fine, as long as our store is eventually consistent. [21:18:08] awight, right, but for most scores that are not in the cache, there will be no judgement and trying to find one is a waste of time. [21:18:20] The bad race condition would be if we failed to insert either the score or judgment. [21:18:24] halfak: +1 [21:18:25] Maybe not that bad of a waste of time. [21:18:39] Especially because we can hit varnish with our JADE queries [21:18:47] Assuming we request them a wiki-artifact at a time. [21:18:50] lessee—for any score not in the cash, we’re already making a MW API request to get page content. [21:18:53] err cache [21:19:04] $$$ [21:19:07] lol [21:19:10] :S [21:19:12] *:D [21:19:15] making it reign [21:19:21] lololol [21:19:57] so if we make the Jade: and Main: content requests in parallel, it’s heating up a MW API server, but won’t affect our performance. [21:20:15] Main: content is almost always going to be bigger and take longer to serve. [21:22:44] Agreed. [21:22:50] JADE should be very small [21:23:08] The other road we could take is to cache JADE and ORES stuff separately, like you were suggesting earlier. If there were a way to make the Redis requests in parallel.. that would be great. [21:23:39] That new python coroutine sugar would make it a lightweight twist. [21:23:54] Oh good point. I forgot about the parallelization problem. [21:24:00] A celery job to fetch from Redis seems too heavy. [21:24:04] Also the LRU. It's nice to have score tied to judgements [21:24:12] Well it doesn’t have to be parallel, I just want to be efficient. [21:24:43] Of course, I think we’re trading off a small write inefficiency for read efficiency. [21:24:50] And cached reads are already hella fast. [21:28:02] ooh [21:28:07] actually there’s no write inefficiency [21:28:21] Redis has an update-key-within-cell operation. [21:29:09] https://redis.io/commands/hset [21:29:44] weird. getting all the values is O(N) ugh, https://redis.io/commands/hvals [21:29:48] whatever though [21:30:03] That’s all internal to Redis, no network latency. [21:31:18] eh we can also use MGET and MSET, and store the ORES and JADE values in separate cells [21:31:38] Looks like the same deal to an amateur. [21:32:39] Sounds good enough for now. We can run some performance tests later. I really don't expect us to take a big hit if the only difference is internal to redis. [21:33:11] hmm, https://redis.io/topics/memory-optimization#use-hashes-when-possible [21:33:26] +1 I’m hacking through this as you can see… [21:33:40] Just taking care around the parts which will bite us later. [21:36:25] Anyway, I like that the score and judgment would expire together. [21:37:26] Agreed. [21:42:15] We keep 21MM scores in the cache. Happen to know what that means in terms of the oldest entry? [21:42:58] Wondering about how long we would have to keep a GET-then-HGET migration for. [21:44:05] awight, we can just blow out the cache. We recover pretty quickly. [21:44:10] kk. [21:45:03] Recovering is going to be interesting when we're adding another query to each scoring job. That shouldn't be too bad though. [21:45:11] Arg... ORES has no notion of wiki entity. [21:45:17] revision = revision [21:45:20] And that's probably good. [21:45:23] hmm [21:45:39] It’s good enuf for revision and diff, since that’s how they’re queried now. [21:45:49] But yes. drafttopic will throw a wrench at that assumption. [21:45:54] Right. [21:46:02] Actually drafttopic is revision-based too [21:46:05] ?? [21:46:09] uh, oh. [21:46:15] It's trained on the first version of an article. [21:46:30] What if the first version is a stub... [21:46:41] That's the problem we set out to deal with [21:47:02] We train it on the first version of pages and test it against the first version of pages. [21:47:20] Stubs contain a lot of useful signal. Even one sentence can work pretty well apparently. [21:48:29] I’ve made a different assumption for JADE, should I rework?https://github.com/adamwight/mw-ext-JADE/blob/test_operations/jsonschema/judgment/v1.json#L63 [21:48:56] It’s actually just configuration that ties the drafttopic schema to page, come to think of it. [21:49:09] https://github.com/adamwight/mw-ext-JADE/blob/test_operations/extension.json#L48 [21:49:50] I think that, in some cases, we will want to make predictions about a page. [21:50:12] But yeah. not in this case. [21:52:00] IMO it would be interesting to repeat the drafttopic scoring a few days after a new article is created, to compare the prediction confidence, but that’s also an argument for being revision-based. [21:52:49] Maybe the catch is that we can query drafttopic by page_id, and it searches for the latest scored revision? [21:54:33] Boo [21:54:53] I've pushed back against that feature request and I think it's been a good call. [21:55:08] If there's going to be something that looks up a rev_id, it should be something other than ORES. [21:55:55] hehe okay that’s fine. The data should be copied back to the mw db anyway, to make it available in work queue filters. [21:56:03] It’s easy to join with page there. [21:59:45] +1 [22:11:57] Is “revids” an undocumented param of query-revisions? [22:13:46] K that’s weird. revscoring/extractors/api/extractor.py uses a “revids” param, which doesn’t seem to be handled by ApiQueryRevisions.php [22:14:33] ookay I get it [22:14:41] revids is handled by a base API thing [22:15:12] right [22:15:13] yeah [22:15:18] revids, titles, etc. [22:29:11] awight, https://github.com/wiki-ai/editquality/pull/128 [22:29:15] Should be really easy :D [22:31:18] I discovered that there’s another schema we need to define: judgment as recorded in Jade:, plus the metadata contained in the revision. [22:31:38] wiki-ai/editquality#117 (bnwikisource_fix - 4f514fc : Aaron Halfaker): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/341654849 [22:31:50] {“meta”: rev_doc} perhaps? [22:32:12] I'm confused. [22:32:21] I’m getting sloppy just to have the framework in place, we can give this more time tomorrow. [22:32:25] kk [22:32:40] This would include user, user_id, timestamp, and comment [22:33:07] stuff that we pull out of the MW revision, and don’t duplicate in Jade: pages. [22:33:16] Oh gotcha [22:33:42] For right now, let's skip the comment because that'll require suppression-fu [22:33:51] Or maybe it won't be that hard. I dunno [22:34:08] Suppression foo is here to stay, either way :) [22:34:13] username needs suppression, too. [22:34:21] (also content!) [22:34:49] ah right—in the ORES use case, content doesn’t require suppression [22:34:54] but comment and username might. [22:35:28] Arg! [22:35:52] I got all the way to loading data in wikilabels before I figure out that a tiny fraction of bnwikisource edits "need review" so I need to re-do the sample [22:35:54] >:( [22:36:19] Same for bawiki [22:37:34] Out of curiosity, the fraction was too small to make it worth importing? [22:37:43] Right. [22:37:44] Those samples should be 100% needs review? [22:38:04] Right yeah. bnwikisource only had 368 edits that needed review. [22:38:07] We aim for 5k [22:38:22] So I'm boosting the sample by an order of magnitude. [22:38:30] 20k edits --> 200k edits [22:38:38] Probably because of a lot of bot activities. [22:38:56] bawiki needed a boost too from 20k edits to 60k edits. [22:39:06] * awight hopes this is mentioned in a newbie howto :) [22:39:37] lol nope. This is no newbie work. But still it would be nice to have it discussed. [22:56:48] halfak: oh. Scores are already split out by model name? [22:56:59] in the cache) [22:59:01] That suggests I should just stuff the judgment into wiki:judgment:123:1 [22:59:06] gtg [23:00:21] o/ [23:00:35] Not sure what you mean, but I think we'll need to so a bit of splitting, yeah [23:01:21] By “split”, I mean that we’re storing the ORES scores under a different key for each model [23:01:50] so retrieving all ORES scores already involves Redis requests for each possibly populated value [23:02:25] and we can just have “judgment” behave like a new model [23:04:21] o/! [23:04:43] oh interesting [23:04:44] hmm [23:29:50] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:30:49] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 443 bytes in 0.559 second response time