[06:17:36] 10Lift-Wing, 10Machine-Learning-Team, 10artificial-intelligence, 10revscoring: Implement model storage for enwiki-goodfaith inference service - https://phabricator.wikimedia.org/T282802 (10elukey) @ACraze I added the following to all our home directories on ml-serve1001: ` elukey@ml-serve1001:~$ ls -l /ho... [06:20:11] good morning [06:21:08] for some reason the scoring editquality repo seems to be not mirrored [06:21:14] I don't see the recent git commits [06:37:25] ah yes phabricator is failing to push to gerrit [06:37:27] * elukey sigh [06:42:12] https://phabricator.wikimedia.org/T224996 [06:42:22] there was already a task opened but very stale [06:42:37] 10Machine-Learning-Team, 10artificial-intelligence, 10Diffusion, 10editquality-modeling, 10Release-Engineering-Team (Seen): Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10elukey) Hi everybody, I found this task while investigating on the gerrit editquali... [07:07:48] 10Machine-Learning-Team, 10artificial-intelligence, 10editquality-modeling, 10Turkish-Sites: Update Turkish Wikipedia's labeling campaign for 2020 - https://phabricator.wikimedia.org/T257359 (10elukey) This is currently blocked by T224996 [07:10:03] 10Lift-Wing, 10Machine-Learning-Team: Upgrade Istio & Knative on sandbox cluster - https://phabricator.wikimedia.org/T282752 (10elukey) Since we are building our own docker images (that will be publicly exposed via our registry) I think it would be great if we could use them (once ready) to have a quick feedba... [08:15:08] 10Lift-Wing, 10Machine-Learning-Team, 10artificial-intelligence, 10revscoring: Implement model storage for enwiki-goodfaith inference service - https://phabricator.wikimedia.org/T282802 (10klausman) @ACraze ` $ ./model_upload.sh CHECKING FOR MODEL_BUCKET Bucket 's3://wmf-ml-models/' created UPLOADING enw... [08:30:28] https://github.com/istio/istio/blob/master/operator/ARCHITECTURE.md#manifest-creation [08:30:39] this looks more digestible --^ [08:30:48] even if it is not so straightfoward [08:30:52] *forward [08:30:56] so my understanding: [08:31:37] 1) istioctl and istio operator (the binary working on the controlplane) get compiled with the istio repositories helm charts [08:31:44] in which there are some profiles, etc.. [08:32:36] 2) when istioctl gets an istio manifest, it translates it to heml and adds the overrides to the common profiles etc.. provided [08:32:51] 3) it ships the config to the istio operator that applies it [08:34:46] still not entirely sure who does what, namely what is the boundary between istioctl and istio operator [08:34:59] but upstream seems to embed/bypass helm [09:36:38] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Install Istio on ml-serve cluster - https://phabricator.wikimedia.org/T278192 (10elukey) https://github.com/istio/istio/blob/master/operator/ARCHITECTURE.md#manifest-creation is very informative. To refine what I wrote above: * istioctl and istio ope... [10:15:15] * elukey early lunch [10:15:33] elukey: will give that ARCH doc a read over lunch, see if I can spot anything [11:15:47] klausman: ack! I think it gives a better picture, but no idea what it is best for us.. I also asked to Joe/Alex opinions, but they are not sure as well [11:16:03] on one side, we could just add istio's helm charts in our repos [11:16:12] passing the right parameters etc.. [11:16:32] on the other we could use the istio manifests and let istio manage the complexity [11:17:02] fine tuning will be needed, and I am not sure how flexible a istio manifest is when applying helm overrides [11:17:18] we'll need to be familiar with istio helm charts for sure [11:17:28] (and they change across versions etc..) [11:43:59] I think going with what Isitio provides, while a larger amount to learn, is the better first approach. I wager if we ty to force its approach to this functionality it will be more headache in the long run. Plus, learning how they work may prove to be a prerequisite for replacing them with something that fits us better. [11:46:33] the only doubt that I have is related to how many things we'll have to deploy and manage for a specific stack [11:47:04] for example, if we use istioctl + manifests we may need another repo outside deployment-charts, in which we have base kubernetes setup [11:47:38] Mh. good point [11:48:01] then there is knative, so far I didn't see helm support from upstream but only kubectl apply -f something.yaml [11:48:05] and finally kubeflow :D [11:48:17] So many moving parts! [11:48:32] yeah :( [11:51:02] also knative/serving doesn't provide dockerfiles afaics sigh [11:54:38] ah it may use https://github.com/google/ko [11:55:44] Love that logo :D [11:58:41] I am loving this stack less and less the more I look at it :D [12:12:23] 10Machine-Learning-Team, 10Diffusion, 10Wikimedia-GitHub, 10Release-Engineering-Team (Seen): LFS objects are not mirroring from Github through Phab to Gerrit consistently - https://phabricator.wikimedia.org/T212818 (10elukey) 05Open→03Declined This is now documented on Wikitech, and there seems to be n... [18:28:28] 10Machine-Learning-Team, 10artificial-intelligence, 10Diffusion, 10editquality-modeling, 10Release-Engineering-Team (Seen): Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10thcipriani) FWIW, I can confirm from the Gerrit side that the project config and gro... [19:46:53] 10Machine-Learning-Team, 10artificial-intelligence, 10Diffusion, 10editquality-modeling, 10Release-Engineering-Team (Seen): Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10mmodell) @thcipriani: I suspect you are right, broadly speaking. We have been movin... [20:23:10] 10Machine-Learning-Team, 10artificial-intelligence, 10Diffusion, 10editquality-modeling, 10Release-Engineering-Team (Seen): Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10mmodell) ok it seems to be mirroring now. [20:24:16] 10Machine-Learning-Team, 10artificial-intelligence, 10Diffusion, 10editquality-modeling, 10Release-Engineering-Team (Seen): Gerrit repo scoring/ores/editquality not mirroing - https://phabricator.wikimedia.org/T224996 (10mmodell) I had to grant the delete references privilege in gerrit for the phabricato... [20:35:22] 10Machine-Learning-Team, 10artificial-intelligence, 10Diffusion, 10editquality-modeling, 10Release-Engineering-Team (Seen): Gerrit repo scoring/ores/editquality not mirroring - https://phabricator.wikimedia.org/T224996 (10Aklapper) [21:38:57] 10Machine-Learning-Team, 10ORES, 10SRE, 10Release Pipeline (Blubber): Build blubber file for ORES - https://phabricator.wikimedia.org/T210268 (10thcipriani) [21:39:01] 10Machine-Learning-Team, 10ORES, 10Release Pipeline, 10SRE, 10Release-Engineering-Team (Seen): Execution of the deployment pipeline should be configurable via .pipeline/config.yaml - https://phabricator.wikimedia.org/T210267 (10thcipriani) 05Open→03Resolved a:03dduvall There are now many services t...