[06:51:50] 10Machine-Learning-Team, 10artificial-intelligence, 10editquality-modeling, 10Turkish-Sites: Update Turkish Wikipedia's labeling campaign for 2020 - https://phabricator.wikimedia.org/T257359 (10kevinbazira) I seem not to have access to `ores-misc01.eqiad.wmflabs` This is what I run: ` kbazira@kbazira:~$... [08:26:27] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Naming convention for the model storage structure - https://phabricator.wikimedia.org/T280467 (10klausman) > Oh yes, I'd say this is a best practice I recommend. It comes from tensorflow-model-server that reads all subdirectories in the specified model direc... [08:34:41] Amir1: o/ - when you have a moment, could you add Tobias, Andy, Kevin, Chris and me to the ores wmcs project as admins? [08:35:11] (assuming you are an admin but I guess so) [09:38:14] elukey: morning, sure. Do you have their LDAP usernames [09:38:22] I can look it up if you're busy [09:40:25] Andy and Chris are already there [09:41:45] Amir1: sure! [09:41:55] so there is this elukey person that you might already know [09:42:01] lol [09:42:15] I already added you, Kevin, Tobias [09:42:18] highly untrustable I know but I can vouch for him [09:42:20] https://github.com/wikimedia/puppet/blob/cf4f712cdccab08293181189fb98f79ddeae70d7/modules/admin/data/data.yaml [09:42:35] 🤬 [09:42:39] kevinbazira [09:43:00] accraze [09:43:09] klausman [09:43:14] klausman [09:43:34] https://github.com/wikimedia/puppet/blob/cf4f712cdccab08293181189fb98f79ddeae70d7/modules/admin/data/data.yaml#L622 I got the most from here [09:43:43] so ores project should be fine [09:43:46] now ores-staging [09:44:00] ah there is also ores staging [09:45:47] ores staging is should be fine too now [09:45:53] <3 [09:45:55] thanks a lot! [09:45:57] I added everyone's that's missing [09:46:20] no worries. Let me know if I can be of any service [09:48:12] Kevin needed ssh access to ores-misc01, I think that once puppet runs we should be good [09:48:15] kevinbazira: --^ [10:32:43] Thank you so much Amir1 and elukey, I will confirm soon as I can access ores-misc01.eqiad.wmflabs [10:33:12] \o/ [13:18:25] klausman: o/ I added my thoughts/research about the feature store to your doc [13:18:30] lemme know if it is understandable [13:18:44] kevinbazira: did you manage to ssh? [13:19:01] ah snap I can't, I guess the same for you [13:20:17] Can't seem to get in. [13:20:49] maybe puppet didn't run on the host for some reason mmm [13:21:01] in theory my assumption was that being project admin would have granted us ssh access [13:26:53] and I can't find a way to have a tty via horizon for the instance, it used to be possible IIRC [13:27:50] let's see if a soft reboot helps :D [13:32:53] Amir1: sorry to ping you again, but if you have a moment I'd ask for a quick check on ores-misc01.eqiad.wmflabs [13:32:59] to see if puppet is broken or not [13:33:12] sure [13:33:58] the brand new hostname should be ores-misc01.eqiad1.wikimedia.cloud [13:34:25] I think it should be either ores-misc01.ores.eqiad1.wikimedia.cloud or ores-misc01.ores-staging.eqiad1.wikimedia.cloud [13:34:56] ah okok, I thought the project name was not required [13:35:21] I honestly don't know. Might be, might not be [13:35:25] it doesn't let me in [13:36:35] lovely, I tried to reboot the instance but no luck [13:37:23] I'm in [13:37:30] it looks okay so far [13:37:56] hmm, the process can't load up [13:38:01] they fail [13:38:14] elukey: added some comments to your additions [13:41:03] Amir1: do you see my user by any chance? [13:41:13] maybe puppet is broken [13:41:32] I don't see yours [13:41:40] one thing, this is really old [13:42:12] this VM should be basically completely torn apart and rebuilt. Has quite long list of security issues [13:53:00] elukey: updated the section, resolved comments. [14:09:27] Amir1: lovely, but does puppet run? [14:09:45] puppet works [14:09:48] with access we could unblock the current testing, then possibly re-do all those machines [14:10:01] then I have no idea how accounts are pushed to the vms [15:40:42] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10drafttopic-modeling, 10Machine-Learning-Team (Active Tasks): ORES deployment - Spring 2021 - https://phabricator.wikimedia.org/T278723 (10Halfak) We do our production test deployments in deployment-prep (ores-beta.wmflabs.org). ores.wmflabs... [16:28:40] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10drafttopic-modeling, 10Machine-Learning-Team (Active Tasks): ORES deployment - Spring 2021 - https://phabricator.wikimedia.org/T278723 (10elukey) @Halfak good to know, is there any documentation about the whole deployment process somewhere? [16:50:01] kevinbazira: I am able to ssh to the instance now, I had the wrong name though, missing '-' : ores-staging-01 vs ores-staging01 [16:58:22] elukey: are you able to get into ssh ores-misc01.eqiad.wmflabs? [16:59:01] kevinbazira: yes try twith ores-misc-01.etc.. [16:59:06] it fooled me as well [16:59:17] or the fancy new hostname, ores-misc-01.ores-staging.eqiad1.wikimedia.cloud [17:05:05] Thanks a lot elukey, I can now confirm that I have access to ssh ores-misc-01.ores-staging.eqiad1.wikimedia.cloud and ssh ores-misc-01.eqiad.wmflabs [17:06:05] \o/ [18:00:47] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10drafttopic-modeling, 10Machine-Learning-Team (Active Tasks): ORES deployment - Spring 2021 - https://phabricator.wikimedia.org/T278723 (10Halfak) https://wikitech.wikimedia.org/wiki/Ores/Deployment is the key reference [18:04:34] * elukey afk! [21:34:30] taking kevinbazira's revscoring image out for a test run on our Kubeflow sandbox this afternoon [22:37:32] ugh build times for scipy take foreveeerrr