[00:01:11] * awight curses up and down about having to rebuild puppet certs [00:02:07] RoanKattouw: Looks like the failure is our fault, I'll write a followup that you can squash in [00:07:44] Cool OK [00:08:00] Meanwhile, ORES has launched successfully on arwiki, cawiki, lvwiki and huwiki [00:11:45] Right on, thanks! [00:12:14] I'm curious what you live-hacked to make that happen... [00:12:59] Just the patch I sent you [00:13:19] Plus an embarrassing number of typo fixes for my config patches [00:13:31] prevision instead of precision, => instead of >= (twice), and a missing 'max' => 1 [00:13:38] hehehe [00:13:45] Still though, I hate a pedantic lint [00:13:50] For each wiki I had to run createExtensionTables.php and PopulateDatabase.php [00:14:07] The latter I aborted after it did 30-60 rows, I just needed it because the ores_model entries etc are lazy-created [00:14:28] dang, we need to clean up our act. [00:14:59] I swear this was working at one point, but it must have been before the 'enabled' config change [00:15:18] As soon as the extension was enabled, the first edit would lazy-create ores_model entries. [00:15:30] Presumably it'll work that way again once we've merged your fix. [00:16:09] 10Scoring-platform-team (Current), 10ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Patch-For-Review: Deploy ORES advanced editquality models to cawiki - https://phabricator.wikimedia.org/T192501#4195924 (10Catrope) 05Open>03Resolved a:03Catrope [00:16:29] 10Scoring-platform-team (Current), 10ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Patch-For-Review: Deploy ORES advanced editquality models to lvwiki - https://phabricator.wikimedia.org/T192499#4195926 (10Catrope) 05Open>03Resolved a:03Catrope [00:16:34] 10Scoring-platform-team (Current), 10ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Patch-For-Review: Deploy ORES advanced editquality models to huwiki - https://phabricator.wikimedia.org/T192496#4195928 (10Catrope) 05Open>03Resolved a:03Catrope [00:17:05] 10Scoring-platform-team (Current), 10ORES, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10Patch-For-Review: Deploy ORES advanced editquality models to arwiki - https://phabricator.wikimedia.org/T192498#4195930 (10Catrope) Damaging is deployed, but goodfaith isn't, per my earlier comment. Should... [00:32:29] RoanKattouw: yuck, so the $models parameter is expected to have a different format than $wgOresModels [00:45:19] 10Scoring-platform-team: deployment-ores01 should get more puppet roles by default - https://phabricator.wikimedia.org/T194315#4195978 (10awight) [00:46:14] oh, that's not the issue actually. Probably better that you don't get the livetweet from inside my head right now. [01:18:17] 10Scoring-platform-team: deployment-ores01 needs more disk space - https://phabricator.wikimedia.org/T194315#4196028 (10awight) [01:18:26] 10Scoring-platform-team: deployment-ores01 needs more disk space - https://phabricator.wikimedia.org/T194315#4195429 (10awight) 05Open>03Resolved [01:19:39] 10Scoring-platform-team (Current), 10ORES, 10drafttopic-modeling, 10Patch-For-Review, 10artificial-intelligence: Deploy drafttopic model to ORES - https://phabricator.wikimedia.org/T176336#4196032 (10awight) [01:19:41] 10Scoring-platform-team (Current), 10drafttopic-modeling: Drafttopic breaks ?features parameter - https://phabricator.wikimedia.org/T194322#4196031 (10awight) [01:38:04] (03PS3) 10Awight: ScoreFetcher: Fix determination of enabled models [extensions/ORES] - 10https://gerrit.wikimedia.org/r/432327 (owner: 10Catrope) [01:38:20] (03CR) 10Awight: [C: 032] ScoreFetcher: Fix determination of enabled models [extensions/ORES] - 10https://gerrit.wikimedia.org/r/432327 (owner: 10Catrope) [01:42:22] (03Merged) 10jenkins-bot: ScoreFetcher: Fix determination of enabled models [extensions/ORES] - 10https://gerrit.wikimedia.org/r/432327 (owner: 10Catrope) [01:43:53] (03CR) 10jenkins-bot: ScoreFetcher: Fix determination of enabled models [extensions/ORES] - 10https://gerrit.wikimedia.org/r/432327 (owner: 10Catrope) [04:39:27] (03CR) 10jenkins-bot: ScoreFetcher: Fix determination of enabled models [extensions/ORES] - 10https://gerrit.wikimedia.org/r/432327 (owner: 10Catrope) [07:47:33] PROBLEM - puppet on ORES-web02.Experimental is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 1 minute ago with 3 failures. Failed resources (up to 3 shown): Service[puppet],Service[uwsgi],Service[uwsgi-ores] [08:16:03] RECOVERY - puppet on ORES-web02.Experimental is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [14:21:39] Woops. Forgot to change that [15:10:00] 10Scoring-platform-team, 10Analytics, 10EventBus, 10ORES, and 3 others: Emit revision-score event to EventBus and expose in EventStreams - https://phabricator.wikimedia.org/T167180#4197451 (10Ottomata) Ok its about time to resurrect this! @Pchelolo, from at https://github.com/wikimedia/change-propagation/... [15:42:59] o/ awight [15:43:09] Just got to your email re. beta breakage. [15:43:11] What's the status? [16:57:03] 10Scoring-platform-team, 10Analytics, 10EventBus, 10ORES, and 3 others: Emit revision-score event to EventBus and expose in EventStreams - https://phabricator.wikimedia.org/T167180#4197940 (10Pchelolo) @Ottomata, when we send the `revision-create` event to ORES, precache endpoint we get the scores as a res... [17:04:09] AFK for lunch [18:39:31] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for Bosnian - https://phabricator.wikimedia.org/T174784#4198261 (10Srdjan_m) @Halfak: Whoops, I totally forgot about this, and I didn't get that ping for some reason... I just finished translati... [18:40:13] Forgot about a meeting I had. Back soon [19:11:29] Hmm... I'm going to self merge my change to wmflabs deploy [19:33:00] http://ores-staging.wmflabs.org/v3/scores/enwiki/12345678\ [19:33:02] http://ores-staging.wmflabs.org/v3/scores/enwiki/12345678 [19:33:05] It's alive! [19:38:26] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for Bosnian - https://phabricator.wikimedia.org/T174784#4198401 (10Halfak) That should be the campaign for Bashkir Wikipedia. My untrained eye doesn't see something clearly wrong. If you give... [19:51:06] 10Scoring-platform-team (Current), 10ORES, 10drafttopic-modeling, 10artificial-intelligence: Check drafttopic model memory usage - https://phabricator.wikimedia.org/T192293#4198424 (10Halfak) https://graphite-labs.wikimedia.org/graphlot/?width=792&height=372&_salt=1525981244.19&target=ores-staging.ores-sta... [19:51:36] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for Bosnian - https://phabricator.wikimedia.org/T174784#4198425 (10Srdjan_m) Well, this task is for the [[ https://bs.wikipedia.org/ | Bosnian Wikipedia ]], not the [[ https://ba.wikipedia.org/ |... [19:51:36] 10[1] 04https://meta.wikimedia.org/wiki/https://bs.wikipedia.org/ [19:55:34] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for Bosnian - https://phabricator.wikimedia.org/T174784#4198435 (10Halfak) OH! My mistake! I've restarted work for the right wiki. https://quarry.wmflabs.org/query/26936. I'll get data loaded... [19:58:12] PROBLEM - ORES web node labs ores-web-01 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 234 bytes in 0.020 second response time [19:58:26] Doing a deployment. Might be nothing. [19:58:51] Still waiting for it to finish [19:59:42] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 234 bytes in 0.018 second response time [20:00:22] PROBLEM - ORES web node labs ores-web-02 on ores.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 234 bytes in 0.015 second response time [20:04:40] Looks like LFS is failing. [20:04:53] It's just pulling reference file. [20:05:10] Why did this work in staging? [20:05:19] Maybe someone was experimenting on staging... [20:06:18] "git: 'lfs' is not a git command. See 'git --help'." [20:06:25] hmm... but recognized on staging. [20:06:29] What's up with that? [20:41:53] RECOVERY - ORES web node labs ores-web-02 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 456 bytes in 3.179 second response time [20:42:12] RECOVERY - ORES web node labs ores-web-01 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 442 bytes in 0.555 second response time [20:42:32] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 457 bytes in 0.541 second response time [20:44:34] So, the moral of the story is that our version of git doesn't play nice with LFS and you need to run an explicit LFS command to actually pull down the files :) [20:47:21] wiki-ai/editquality#326 (bswiki - 799fbb8 : halfak): The build passed. https://travis-ci.org/wiki-ai/editquality/builds/377456736 [20:52:19] 10Scoring-platform-team, 10Analytics, 10Analytics-Wikistats, 10ORES: Discuss Wikistats integration for ORES - https://phabricator.wikimedia.org/T184479#4198576 (10Milimetric) @awight. Those look great and in my opinion they fit right in for the most part. We organize metrics into three groups: content:... [20:55:54] And we're back online [20:56:11] (Actually were before this. I'd just only confirmed it now.) [20:58:30] Looks like we should cut down parallelization on the workers a little bit [20:58:36] They have maybe 100MB free [20:58:55] Looks like we're running 16 workers per node. [20:59:03] Cutting that down to 14 [21:02:22] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for Bosnian - https://phabricator.wikimedia.org/T174784#4198647 (10Halfak) https://github.com/wiki-ai/editquality/pull/159 [21:02:52] 10Scoring-platform-team (Current), 10ORES, 10drafttopic-modeling, 10artificial-intelligence: Check drafttopic model memory usage - https://phabricator.wikimedia.org/T192293#4198659 (10Halfak) We're running really low on memory for the celery workers so I'm reducing concurrency from 16 to 14. [21:27:11] wikilabels is broken in such a strange way! [21:27:16] WTF [21:27:39] All of the version limits don't appear in requirements.txt when I deploy to wmflabs [21:27:43] But they appear elsewhere [21:27:50] version mismatches are causing a bunch of issues [21:27:51] hmmmm [21:28:14] same commit [21:29:40] I clearly see different sha1s for the file in question [21:30:26] SOMEONE OVERWROTE THE FILE AND LEFT IT LIKE THAT [21:30:28] >:( [21:30:52] Seems like our reset should be recursive [21:33:05] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Edit quality campaign for Bosnian - https://phabricator.wikimedia.org/T174784#4198762 (10Halfak) OK the new campaign is live at http://labels.wmflabs.org/ui/bswiki/ I did a deployment, but I see the UI translations h... [21:34:20] 10Scoring-platform-team (Current), 10ORES, 10drafttopic-modeling, 10artificial-intelligence: Check drafttopic model memory usage - https://phabricator.wikimedia.org/T192293#4198764 (10Halfak) OK everything looks good. We're running nicely on ores.wmflabs.org. See https://ores.wmflabs.org/v3/scores/enwiki... [21:34:25] * halfak runs away