[11:00:52] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Create ml-serve k8s cluster - https://phabricator.wikimedia.org/T272918 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` ['ml-serve2002.codfw.wmnet'] ` The log can be found in `/var/log/wmf-au... [12:18:14] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Create ml-serve k8s cluster - https://phabricator.wikimedia.org/T272918 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by klausman on cumin2001.codfw.wmnet for hosts: ` ['ml-serve2002.codfw.wmnet'] ` The log can be found in `/var/log/wmf-... [12:45:00] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Create ml-serve k8s cluster - https://phabricator.wikimedia.org/T272918 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['ml-serve2002.codfw.wmnet'] ` and were **ALL** successful. [12:54:33] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Create ml-serve k8s cluster - https://phabricator.wikimedia.org/T272918 (10klausman) All worker nodes are now up and visible in both DCs: ` ml-serve-ctrl1001:~$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION IN... [12:54:52] chrisalbon accraze kevinbazira ^^^ Latest comment on the ticket \o/ [13:33:48] 10artificial-intelligence, 10SRE, 10Services, 10Service-deployment-requests: New Service Request 'open_nsfw' - https://phabricator.wikimedia.org/T250110 (10akosiaris) Hello, >>! In T250110#6924592, @Chtnnh wrote: > Hello! > > Yes, we would love to have this service deployed. Although, over the course of... [14:03:25] 10artificial-intelligence, 10SRE, 10Services, 10Service-deployment-requests: New Service Request 'open_nsfw' - https://phabricator.wikimedia.org/T250110 (10Chtnnh) I understand @akosiaris ! Is it possible to deploy to production as volunteers? As in, is it possible for long time volunteers to have deploy... [14:34:58] 10artificial-intelligence, 10SRE, 10Services, 10Service-deployment-requests: New Service Request 'open_nsfw' - https://phabricator.wikimedia.org/T250110 (10akosiaris) >>! In T250110#6928585, @Chtnnh wrote: > I understand @akosiaris ! > > Is it possible to deploy to production as volunteers? As in, is it... [14:38:23] 10artificial-intelligence, 10SRE, 10Services, 10Service-deployment-requests: New Service Request 'open_nsfw' - https://phabricator.wikimedia.org/T250110 (10Chtnnh) I see. I think the team (@Harshineesriram, @Abbasidaniyal and I) will have to put some thought into that. As far as the timeline is concerned... [14:48:42] 10Machine-Learning-Team, 10Analytics-Radar, 10SRE: Kubeflow on stat machines - https://phabricator.wikimedia.org/T275551 (10akosiaris) Just a few clarifications and answers. > cloud vps is a kubernetes cluster It's toolforge that's half powered by a kubernetes cluster. The other half is powered by son of g... [16:00:18] klausman: wow! that's exciting! [16:00:25] nice one! [16:01:11] So I talked with Luca, and next week he's going to do some "hello world" testing, and set up some more bits like Monitoring. And if all goes swell, maybe start looking at how we might set up Istio [16:01:23] But that's all very dependent on each other. [16:03:09] agreed, that sounds like a good plan [17:02:04] is prometheus still what's being used for monitoring? promql is pretty nice [22:45:52] 10Jade, 10Voice & Tone: Address Voice and Tone issues in Jade - https://phabricator.wikimedia.org/T277946 (10Reedy)