[08:36:09] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10User-Ladsgroup, 10artificial-intelligence: Add language support for Swahili (sw) - https://phabricator.wikimedia.org/T162271#3739929 (10Baba_Tabita) Hi @Halfak . Thanks for the link. And sorry for the delay. It's still on my to-do lis... [09:30:26] 10Scoring-platform-team (Current), 10ORES: Deploy ORES early Nov 2017 - https://phabricator.wikimedia.org/T179837#3737792 (10elukey) @Halfak hi! Was the fix for the 500s deployed yesterday? We are still seeing a ton of them in logstash.. [11:50:11] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:18:13] PROBLEM - puppet on ores-web-03 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[/var/lib/prometheus/node.d] [12:20:41] RECOVERY - puppet on ores-worker-07 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [12:25:13] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[/var/lib/prometheus/node.d] [12:47:43] RECOVERY - puppet on ores-web-03 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [12:53:43] RECOVERY - puppet on ores-lb-02 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [14:27:00] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review, and 2 others: Stress/capacity test new ores* cluster - https://phabricator.wikimedia.org/T169246#3741136 (10awight) Turns out we can't use the other concurrency pools for Celery. The other options, gevent and eventlet, are both s... [15:38:55] halfak: T179874 [15:38:55] T179874: Specter of revscoring 1 haunting ORES - https://phabricator.wikimedia.org/T179874 [15:38:58] Good times. [15:39:04] Back to the bad old days. [15:39:14] It’s just that commit… WHY [15:39:44] Does “scorer_models.defaults.class” actually do anything? Cos that is still pointing to a revscoring 1 thing. [15:40:14] What still points to a revscoring 1 thing? [15:40:16] What specifically [15:41:38] Oh! The config. [15:41:39] looking at that [15:45:51] awight, confirmed that the problem does not exist with revscoring 2.0.8 [15:46:17] +1, I rolled back one revision on beta and everything was fine. [15:46:31] I can believe I made a bad wheel. still, : [15:46:55] (03PS1) 10Awight: Correct default model class [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/389726 [15:47:40] (03PS2) 10Awight: Correct default model class [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/389726 [15:50:07] I don't know if your wheel that is bad. But when I fix the reference in the config (which is totally wrong) I get a new error in wikiclass. [15:50:10] Looking into that,. [15:51:26] halfak: You’re not gonna like this: revscoring 2.0.8 still has that class. [15:51:31] at least, the wheel does. [15:53:33] Right. I think that was probably due to git being weird. [15:53:48] Still trying to find the bottom of this rabbit hole. [15:54:03] Internet here is terrible :( [15:54:48] Here’s the screwy part: I can import the editquality model manually, using the revscoring 2.0.9 wheel. [15:54:55] That’s why I was suspecting config tidbits. [15:55:28] Agreed [15:55:28] Is config tidbit [15:55:43] replace with 'revscoring.Model' [15:55:52] Can you paste the wikiclass error? [15:58:22] File "./wikiclass/utilities/score.py", line 20, in [15:58:32] from revscoring.scorer_models import MLScorerModel [15:58:33] ImportError: No module named 'revscoring.scorer_models' [15:58:40] Not sure why the score utility is getting imported [16:00:09] Was going to look into that but I can't git pull our repos [16:00:15] lol [16:00:15] too big [16:00:24] baahaha [16:00:25] brb [16:03:11] (03PS3) 10Awight: Correct default model class [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/389726 (https://phabricator.wikimedia.org/T179874) [16:04:12] (03CR) 10Ladsgroup: [V: 032 C: 032] Correct default model class [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/389726 (https://phabricator.wikimedia.org/T179874) (owner: 10Awight) [16:04:44] awight: are you deploying to beta? [16:04:56] today is wikidata day, can't help much :( [16:05:04] Amir1: I probably should, but hadn’t thought about it yet. [16:05:12] ok good to know. Thanks for the stealth reviews! [16:07:26] Amir1: ^ one more if you would [16:07:38] definitely [16:08:14] https://github.com/search?q=org%3Awiki-ai+MLScorerModel&type=Code [16:08:22] awight: There is one instance in docs too [16:08:28] (in revscoring) [16:09:52] wiki-ai/wikiclass#11 (model2 - d3c9510 : Adam Roses Wight): The build passed. https://travis-ci.org/wiki-ai/wikiclass/builds/298623322 [16:13:05] I got the one in wikiclass/doc, but didn’t see the one you’re talking about in revscoring. [16:15:43] How does our github->phabricator mirror work? [16:15:53] Wondering if I have to intervene. [16:16:42] up to five hours [16:16:52] haarrrrr [16:16:56] /o\ [16:17:05] I know aaron has rights to force it [16:17:07] well I guess it’s not a rush [16:17:15] awight: where is the repo in phab? [16:17:15] I might be able to help [16:17:26] meh it really isn’t a rush, not worth wasting your time on [16:21:35] awight, just scheduled an update for phab's wikiclass. [16:21:38] Anything else? [16:22:05] Nothing! [16:22:31] I’m trying to prioritize what to do next. [16:22:47] Deploying to ores* is all jacked up, was going to drift towards that perhaps [16:23:18] I also saw a *lot* of that “KeyError: features” response, which I’m guessing is another error being masked. [16:25:51] halfak: Not a great thing for weird hours and internet outages, but I’d like to talk about the ores-wmflabs vs ores-prod workflow soon. [16:26:14] I <3 branches and would lobby for using that instead of separate repos. [16:41:04] awight, that error is because of overlap in task names. [16:41:27] awight, we can't do that. Code needs to come from phab [16:41:30] The features vs non-features thing? A patch for that is included. [16:41:35] Right [16:41:47] It’s patched, and still failing. Maybe my patch doesn’t work. [16:43:11] halfak: https://github.com/wiki-ai/ores/commit/a79d09e958bd28369d6040d4f07021392ff94810 [17:05:54] awign, what do you mean "doesn't work"? [17:06:00] *awight [17:53:54] (03PS1) 10Awight: Bump wikiclass [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/389756 (https://phabricator.wikimedia.org/T179874) [17:56:32] (03CR) 10Awight: [V: 032 C: 032] "Self-merging library bump" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/389756 (https://phabricator.wikimedia.org/T179874) (owner: 10Awight) [18:24:37] 10Scoring-platform-team, 10Scap: Scap deploying arbitrary revision, incorrectly reporting correct revision - https://phabricator.wikimedia.org/T179965#3742204 (10awight) [18:25:26] 10Scoring-platform-team, 10Scap: Scap deploying arbitrary revision, incorrectly reporting correct revision - https://phabricator.wikimedia.org/T179965#3742217 (10awight) [20:25:15] 10Scoring-platform-team (Current), 10Community-Tech, 10Wikimedia-Site-requests, 10Patch-For-Review, 10User-Ladsgroup: Enable draftquality model in ORES extension for enwiki - https://phabricator.wikimedia.org/T179596#3742620 (10DannyH) [21:02:36] Halfak awight o/ [21:37:51] 10Scoring-platform-team, 10Scap: Deployment fails with something gitty - https://phabricator.wikimedia.org/T179987#3742873 (10awight) [21:38:04] ^ 10/10 phab title [22:09:47] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter): UX check RC Filters in beta (revscoring 2.0/thresholds release) - https://phabricator.wikimedia.org/T178395#3742940 (10Etonkovidova) Just general observations: checking ORES filters in wmf.6 (plwiki... [22:15:06] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Deploy ORES (revscoring 2.0) - https://phabricator.wikimedia.org/T175180#3742964 (10Etonkovidova) [22:15:08] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter): UX check RC Filters in beta (revscoring 2.0/thresholds release) - https://phabricator.wikimedia.org/T178395#3742963 (10Etonkovidova) 05Open>03Resolved