[05:07:49] 10Machine-Learning-Team, 10Wikilabels: Translations updates are blocked - https://phabricator.wikimedia.org/T282449 (10Ladsgroup) I added translatewiki to list of exceptions for wikilabels, is there any other ML repos using trnalsatewiki. [05:27:24] 10Machine-Learning-Team, 10Wikilabels: Translations updates are blocked - https://phabricator.wikimedia.org/T282449 (10elukey) Thanks a lot @Ladsgroup and @RhinosF1! Can somebody add a TL;DR about this use case? +1 for the exception if needed (sorry for the trouble), but we'll migrate to the new pipeline durin... [05:29:56] 10Machine-Learning-Team, 10Wikilabels: Translations updates are blocked - https://phabricator.wikimedia.org/T282449 (10Ladsgroup) It's about i18n of interface of https://labels.wmflabs.org/ An example commit would be https://github.com/wikimedia/wikilabels/commit/ef085313dbbfa5ff009164c07d2873202ef0c852 I don... [06:30:10] good morning folks, I'll be afk for ~2h, ttl! [06:50:48] 10Lift-Wing, 10Machine-Learning-Team, 10Outreach-Programs-Projects, 10Google-Summer-of-Code (2021): Retraining models from ORES to be deployable on Lift Wing - https://phabricator.wikimedia.org/T278261 (10Gopavasanth) 05Open→03Declined @Chtnnh We are sorry to say that we could not allocate a slot for y... [06:51:32] 10Lift-Wing, 10Machine-Learning-Team, 10Outreach-Programs-Projects, 10Google-Summer-of-Code (2021): Retraining models from ORES to be deployable on Lift Wing - https://phabricator.wikimedia.org/T278261 (10Gopavasanth) 05Declined→03Open [07:01:18] 10Lift-Wing, 10Machine-Learning-Team, 10Outreach-Programs-Projects, 10Google-Summer-of-Code (2021): Retraining models from ORES to be deployable on Lift Wing - https://phabricator.wikimedia.org/T278261 (10Gopavasanth) [07:14:01] 10Lift-Wing, 10Machine-Learning-Team, 10Outreach-Programs-Projects, 10Google-Summer-of-Code (2021): Retraining models from ORES to be deployable on Lift Wing - https://phabricator.wikimedia.org/T278261 (10Gopavasanth) [07:16:11] 10Lift-Wing, 10Machine-Learning-Team, 10Outreach-Programs-Projects, 10Google-Summer-of-Code (2021): Retraining models from ORES to be deployable on Lift Wing - https://phabricator.wikimedia.org/T278261 (10Gopavasanth) [07:16:34] 10Lift-Wing, 10Machine-Learning-Team, 10Outreach-Programs-Projects, 10Google-Summer-of-Code (2021): Retraining models from ORES to be deployable on Lift Wing - https://phabricator.wikimedia.org/T278261 (10Gopavasanth) [09:35:15] I am understanding the helm charts a little better, and there seems to be the way to have a custom image name policy for proxyv2 and pilot (good) but not for operator (bad) [09:35:18] https://github.com/istio/istio/blob/master/manifests/charts/istio-operator/templates/deployment.yaml#L19 [09:35:21] * elukey sigh [10:47:44] * elukey lunch [15:27:09] 10Machine-Learning-Team, 10Wikilabels: Translations updates are blocked - https://phabricator.wikimedia.org/T282449 (10calbon) +1 an exception. Sorry all for the blockage. [15:34:56] knative has another way of building things, and also requires golang 1.14 to build (and we have 1.13 on our docker registry) [15:35:27] will work with SRE to figure out what to do (probably add a bullseye base image) [15:37:43] 10Machine-Learning-Team, 10artificial-intelligence, 10editquality-modeling, 10Patch-For-Review, 10Turkish-Sites: Update Turkish Wikipedia's labeling campaign for 2020 - https://phabricator.wikimedia.org/T257359 (10calbon) Thanks all for getting this deployment out! [15:40:07] 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10editquality-modeling: Complete viwiki edit quality campaign - https://phabricator.wikimedia.org/T130273 (10calbon) @Halfak knows far more than me about the implications of using old datasets for training, but +1 to building a model and see... [16:05:49] elukey: I plan to do more testing today with the target versions of istio and knative. I expect the targets will be fine, although I'm running into alot of issues with the versions after the target that may require us to use a different API for the inference services. [16:06:10] hey accraze [16:06:12] morning :) [16:06:53] the targets are very different from what it was suggested to us though, but we may be able to use istio 1.9.x on k8s 1.16? [16:07:31] also, what issues are you talking about? Related to kfserving or on other layers? [16:08:02] yeah, it's mostly KFServing [16:09:47] still don't understand everything that changes in the newest release, but it looks like there are big changes with KFServing & Istio [16:10:27] it does seem like it should run on k8s 1.16 though, so that could be good [16:12:48] i'll confirm this stuff and share my notes later today [16:14:25] ack [16:15:59] accraze: IIRC with knative > 0.18 the istio cluster local gw was replaced with knative local gw, but not much else [16:16:08] what version of kfserving/kubeflow are you testing? [16:16:22] just to make sure that it doesn't require some specific version of knative etc.. [16:17:43] 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10editquality-modeling: Complete viwiki edit quality campaign - https://phabricator.wikimedia.org/T130273 (10MikePlantilla) Thanks @Halfak, @calbon. Since some bots have flood flag (not bot flag), the old dataset had their edits. Bots on vi... [16:18:41] i accidentally installed kubeflow v1.3 (newest release) on a new vm. it has istio 1.9.0, knative v0.14.3 and kfserving v0.5.1 and then i realized none of the inference services worked the same [16:19:47] the big thing is kfserving moves from the v1alpha2 api to the v1beta1 api [16:20:25] accraze: is kubeflow 1.3 installing those versions of istio/knative by itself (maybe as part of an install script etc..?) [16:20:48] yep that's correct it's all packaged together in the minikf distribution [16:21:37] ok so then the suggestion about using 1.6.2 with 0.18 made in the task may be revised [16:21:50] something like istio 1.9.x + knative 0.18 could be good [16:22:47] yeah i'm starting to think so [16:26:17] I am currently trying to create the knative 0.18 docker images for our registry [16:26:35] for istio I targeted 1.6.2 for the moment, but we can add 1.9 surely later on [16:27:38] ah accraze today I had a chat with Alex about the ingress gw replication, and IIUC we should make sure that its pods are replicated, it doesn't matter a lot to what worker, since kube-proxy will take care of re-routing if needed [16:27:55] so the idea of using a LVS endpoint in front of our serving cluster is viable [16:27:55] nice! [16:28:05] that's good to know [16:31:59] so then traffic flow would be: wimikedia api -> LVS endpoint -> ingress gw -> predictor ? [16:47:13] accraze: yes in theory this should be the flow [16:47:25] unrelated: i think the permissions issue was fixed for the inference-services repo. kevinbazira is able to +2 now we should be good....gerrit can be tricky :) [16:47:27] the ingress gw will run envoy, so I hope that we'll get metrics out of it [16:48:39] oh nice, yeah i was wondering where we'll tap in for metrics [16:50:16] kfserving seems to give us access to granular metrics that we didn't have for the ores models [16:50:43] very nice [16:50:48] we can even scale-to-zero models that aren't used often [16:51:03] but then that opens up the question of when do we EOL a model [16:51:56] I am still a little worried about how knative and istio will work in production, I don't usually like to run stuff that are in alpha/beta stage :D [16:53:22] yeah for sure, things seem to moving fast on their end though so i think the software is becoming more mature [16:58:40] I'll feel better when we'll be on k8s 1.20 :D [17:03:35] going afk, ttl! [17:05:00] see ya elukey [20:49:38] just did something weird to the sandbox and now a bunch of the pods are crash-looping [20:54:09] i think the container images are having problems with auto-scaling since the model binary is packaged inside the container and not being streamed from a storage uri