[05:57:47] good morning! [06:38:04] kevinbazira: o/ good morning [06:38:15] when you have a moment we can deploy ORES if you want [06:38:27] elukey: good morning [06:39:39] (the viwiki thing) [06:41:27] yep, we can deploy. I've sent you a meeting on you calendar. [06:45:19] perfect, I'll join in a minute :) [08:10:42] elukey: has anything in the logs or metrics jumped at you yet? [08:11:38] nope, all good! [08:12:09] https://ores.wikimedia.org/v3/scores/viwiki/123125/articletopic works fine [08:12:09] All seems fine to me too. Going to declare victory. [08:12:12] +1 [08:12:42] Victory! ORES deploy looks good thanks to elukey! [08:13:09] \o/ [08:13:31] and Kevin :) [08:13:54] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10articlequality-modeling, and 2 others: ORES deployment - Spring 2021 - https://phabricator.wikimedia.org/T278723 (10kevinbazira) The ORES deployment with rebuilt viwiki models has been completed. Thanks to @elukey! In case there are any issues... [08:18:12] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10articlequality-modeling, and 2 others: ORES deployment - Spring 2021 - https://phabricator.wikimedia.org/T278723 (10elukey) 05Open→03Resolved Metrics are good, no weird ERROR registered in logstash afaics, https://ores.wikimedia.org/v3/score... [08:25:19] elukey: Do you have +2 rights on this https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/690010 [08:31:54] nop [08:31:56] nope [08:32:44] Ok ... I guess Andy set it up that way for everyone. Will follow up with him. Thank you for confirming. [08:35:25] It may be due to the repo's default settings [08:41:30] Yep [10:55:30] * elukey lunch [12:08:03] back! [12:08:28] so the istio images are also built with docker-pkg, and their size is inline with the upstream ones on dockerhub (not the distroless ones) [12:09:01] the main issue for the moment is that we cannot call them "istio/somename", but "istio-somename" [12:09:38] this is not a problem for helm, but istioctl IIUC doesn't work with the - format [12:12:07] (we can specify a "hub" parameter, like docker-registry.wikimedia.org/istio, that works as prefix) [12:34:03] ahh lovely [12:34:04] https://github.com/istio/istio/commit/dfa95ce5b04812d9ec8ddff5316a07f19d4209c7 [12:34:20] this was merged in istio 1.7, istioctl verify-install on 1.6 fails [12:39:49] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Install Istio on ml-serve cluster - https://phabricator.wikimedia.org/T278192 (10elukey) https://github.com/istio/istio/commit/dfa95ce5b04812d9ec8ddff5316a07f19d4209c7 is included in 1.7.0+, without it `istioctl verify-install` doesn't work.. [13:15:07] * elukey bbiab [16:21:49] kevinbazira: ah yeah sorry about the repo confusion, will fix it [16:22:12] do i need to add us all to a group on that repo? [16:23:13] hey Andy [16:23:29] there may already be a group for ML, if not we can create one [16:23:44] ideally, IIRC, the parent repo contains base settings so every time a child repo is created it inherits [16:24:36] (brb) [16:57:14] requested gerrit priveleges for everyone: https://phabricator.wikimedia.org/T282797 [17:05:26] also a timely discussion about git lfs from yesterday: https://news.ycombinator.com/item?id=27134972 [17:09:12] ahahahah [17:10:34] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10articlequality-modeling, and 2 others: ORES deployment - Spring 2021 - https://phabricator.wikimedia.org/T278723 (10Halfak) Nice work! Thanks folks. [17:26:27] lovely, I get something like [17:26:39] (missing) models/trwiki.damaging.gradient_boosting.model (64f23a3aaefc59f3808a6d168f0376fcdfc6a4efcaa7d90f912278c3d5a8fd15) [17:26:42] (missing) models/trwiki.goodfaith.gradient_boosting.model (b8579942781c8d97a1291f171a726ce9fbb286dd0cd74e79098a3eee752c68c2) [17:26:45] hint: Your push was rejected due to missing or corrupt local objects. [17:26:54] when pushing to gerrit [17:27:05] I am running git lfs fetch --all origin master on the github repo [17:27:13] super slow but let's see if it works [17:37:32] 10Lift-Wing, 10Machine-Learning-Team, 10revscoring: Implement model storage for enwiki-goodfaith inference service - https://phabricator.wikimedia.org/T282802 (10ACraze) [17:38:23] 10Lift-Wing, 10Machine-Learning-Team, 10revscoring: Implement model storage for enwiki-goodfaith inference service - https://phabricator.wikimedia.org/T282802 (10ACraze) [17:38:25] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Find a way to store models for Kubeflow - https://phabricator.wikimedia.org/T280025 (10ACraze) [17:50:31] 10Lift-Wing, 10Machine-Learning-Team, 10artificial-intelligence, 10revscoring: Implement model storage for enwiki-goodfaith inference service - https://phabricator.wikimedia.org/T282802 (10ACraze) [18:01:07] Downloading LFS objects: 51% 4.6GB [18:01:12] * elukey cries in a corner [18:17:55] is that for the deploy repo? [18:19:38] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Install Istio on ml-serve cluster - https://phabricator.wikimedia.org/T278192 (10elukey) https://gerrit.wikimedia.org/r/688211 reached a good stage, but there are still some unclear points about to deploy istio. IIUC there are two ways: * pure helm c... [18:20:10] accraze: nono editquality [18:20:32] I tried to push the git lfs stuff to gerrit but failed with the errors above [18:20:40] so I am re-fetching all to start clean [18:21:05] but of course I didn't imagine a 10GB repository :D [18:21:36] going afk for the moment, I added some info about my discoveries today for istio [18:22:01] still very confused about it, but we should have the first draft of the docker images to build/push to our registry :) [18:22:04] * elukey afk! [18:22:13] awesome! have a good evening [21:05:33] 10Lift-Wing, 10Machine-Learning-Team, 10artificial-intelligence, 10revscoring: Implement model storage for enwiki-goodfaith inference service - https://phabricator.wikimedia.org/T282802 (10ACraze) [22:22:45] 10Lift-Wing, 10Machine-Learning-Team, 10artificial-intelligence, 10revscoring: Implement model storage for enwiki-goodfaith inference service - https://phabricator.wikimedia.org/T282802 (10ACraze) @elukey or @klausman: whenever you have time, can you please upload the `enwiki-goodfaith` model to Thanos Swi...