[06:47:40] PROBLEM - puppet on ORES-web02.Experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:57:24] PROBLEM - puppet on ORES-web01.Experimental is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle. [07:17:07] RECOVERY - puppet on ORES-web02.Experimental is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:26:43] RECOVERY - puppet on ORES-web01.Experimental is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [09:54:14] 10Jade, 10Scoring-platform-team (Current), 10DBA, 10Operations, and 2 others: Review Jade data storage and architecture proposal [RFC] - https://phabricator.wikimedia.org/T200297 (10awight) Congratulations, looking forward to seeing this deployed! [12:13:37] PROBLEM - puppet on ORES-redis02.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:42:45] RECOVERY - puppet on ORES-redis02.experimental is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [14:03:09] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @Lucas_WMDE & @cscott - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:31:30] 10Scoring-platform-team, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Add accraze to deployment and deploy-service groups. - https://phabricator.wikimedia.org/T228191 (10fsero) done. @Halfak thanks for the patch [14:31:50] 10Scoring-platform-team, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Add accraze to deployment and deploy-service groups. - https://phabricator.wikimedia.org/T228191 (10fsero) 05Open→03Resolved p:05Triage→03Normal [14:31:52] 10Scoring-platform-team: Onboard Andy Craze -- Accounts and access - https://phabricator.wikimedia.org/T226416 (10fsero) [14:52:41] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @Lucas_WMDE & @cscott - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [15:06:51] 10Scoring-platform-team (Research), 10Machine vision, 10artificial-intelligence: Implement NSFW image classifier using Open NSFW - https://phabricator.wikimedia.org/T214201 (10Tgr) Scoring revisions of the file page instead of revisions of the image is a somewhat dirty hack but doesn't really have disadvanta... [15:56:04] 10Scoring-platform-team: Implement hunspell dictionary for euwiki article quality model - https://phabricator.wikimedia.org/T223788 (10Halfak) Just for the record, there is a campaign with more labels for this dataset up here: https://labels.wmflabs.org/stats/euwiki/ See T215351 completing that. For this task... [15:57:51] 10Scoring-platform-team: Implement hunspell dictionary for euwiki article quality model - https://phabricator.wikimedia.org/T223788 (10Halfak) I've already got a work in progress PR for adding hunspell dictionary support for basque. See https://github.com/wikimedia/revscoring/pull/400 [16:05:27] Special async day because of cultural orientation. [16:06:02] Yesterday I finished up some start of FY stuff. Reviewed actuals and got our goals in the right format. [16:06:34] I also have been talking with Kashik about projects. The ROC AUC one was a little deep, but I have a new and better task, I think. It is adding dictionary based features for euwiki's article quality model. [16:07:00] ROC AUC? [16:07:19] Area under the Curve of the Receiver Operating Characteristic. [16:07:24] It's a metric of model fitness. [16:07:27] ah [16:07:46] https://en.wikipedia.org/wiki/Receiver_operating_characteristic [16:08:45] Basically, it's the probability that any two observations are sorted correctly. E.g. A truly vandalism edit is flagged as more likely to be damaging than a non-vandalism edit. [16:09:09] 10Scoring-platform-team, 10CommRel-Specialists-Support (Jul-Sep-2019), 10User-Johan: Community Relations Specialist support for Scoring Platform - https://phabricator.wikimedia.org/T217232 (10Elitre) a:05Johan→03Keegan [16:09:11] halfak: thats what i was thinking [16:09:31] * halfak could give an offhand lecture on classifier evaluation [16:13:57] 10Scoring-platform-team, 10CommRel-Specialists-Support (Jul-Sep-2019), 10User-Johan: Community Relations Specialist support for Scoring Platform - https://phabricator.wikimedia.org/T217232 (10Halfak) \o/ @Groceryheist is back from travels on July 19th. I'll set up a kickoff meeting after that. So, probabl... [16:37:04] 10Scoring-platform-team, 10CommRel-Specialists-Support (Jul-Sep-2019), 10User-Johan: Community Relations Specialist support for Scoring Platform - https://phabricator.wikimedia.org/T217232 (10Keegan) >>! In T217232#5341746, @Halfak wrote: > \o/ @Groceryheist is back from travels on July 19th. I'll set up a... [16:46:25] 10Scoring-platform-team, 10CommRel-Specialists-Support (Jul-Sep-2019), 10User-Johan: Community Relations Specialist support for Scoring Platform - https://phabricator.wikimedia.org/T217232 (10Halfak) OK sounds good. [17:44:56] my async update -- [17:45:15] Y: another attempt to deploy ORES but didn't have access to deploy groups, also started updating the config to retrain the huwiki model [17:45:25] T: machine learning meeting w/ search team, cultural orientation, another ORES deployment attempt (third times a charm...?) and maybe get around to retraining the huwiki model. [18:31:48] Research showcase! https://www.youtube.com/watch?v=i9vvwV5KfW4 [18:31:54] Ewhit is presenting today [18:31:57] accraze, ^ [18:32:11] awesome! checkin it out [18:32:11] Join #wikimedia-research to see the backchannel discussion and to ask questions. [20:05:06] halfak, i think its deploy time again, just confirmed I have access now...is it ok to give it another try? [20:05:26] Yes. Let's do it. [20:05:29] Call when ready [20:09:58] 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Build revert model for glwiki - https://phabricator.wikimedia.org/T228331 (10Halfak) [20:09:59] accraze, T228331 [20:10:00] T228331: Build revert model for glwiki - https://phabricator.wikimedia.org/T228331 [20:10:01] 10Scoring-platform-team (Current), 10editquality-modeling, 10artificial-intelligence: Build revert model for glwiki - https://phabricator.wikimedia.org/T228331 (10Halfak) https://github.com/wikimedia/editquality/pull/204 [20:26:27] 10ORES, 10Scoring-platform-team: Canary node not fully useful when deploying new model - https://phabricator.wikimedia.org/T228334 (10Halfak) [20:27:20] 10ORES, 10Scoring-platform-team: Canary node not fully useful when deploying new model - https://phabricator.wikimedia.org/T228334 (10Halfak) ` halfak@ores1001:~$ curl localhost:8081/v3/scores/glwiki/12412 { "glwiki": { "models": { "articlequality": { "version": "0.8.0" }, "reve... [20:28:27] 10ORES, 10Scoring-platform-team: Canary node not fully useful when deploying new model - https://phabricator.wikimedia.org/T228334 (10Halfak) I figure it out! The celery system caches the key error. You can circumvent the cache by including "?features" in the request. [20:48:16] 10ORES, 10Scoring-platform-team (Current): Canary node not fully useful when deploying new model - https://phabricator.wikimedia.org/T228334 (10Halfak) 05Open→03Resolved [21:02:25] hey halfak, got a quick question re: re-training the huwiki model. I'm getting an error when trying to merge the new labeled revisions with the original labeled revisions due to the original revisions having `"autolabel":{}` [21:02:33] any ideas? [21:02:47] What's the exact error? [21:02:51] (in meeting) [21:04:18] https://phabricator.wikimedia.org/P8768 [21:33:13] That's weird! Can you figure out what row it is erroring on and get a snippet of the input that errors our? [21:33:14] *out [21:33:26] I need to run now, but I can look at this in the AM or maybe when I get back tonight. [21:33:32] Sorry to leave you hanging [21:33:35] accraze_brb, ^ [21:58:05] no worries halfak, i'll keep digging around and will get a snippet w/ input etc.