[14:49:15] o/ lzia [14:49:26] I have a lead on someone who would like to work with our GPU [14:49:35] And do some neural network building with it. :D [14:50:34] Lucie is working on a neural network for generating https://www.mediawiki.org/wiki/Extension:ArticlePlaceholder content from Wikidata items and her team has made a lot of progress. Is that something you're interest in? [14:50:50] *Lucie-Aimee Kaffee [14:57:24] o/ halfak [14:57:45] Sorry to message bomb you before 8AM :S [14:57:49] * leila reads and check the thread on GPU [14:58:16] haha. no worries, halfak. Actually, let me grab breakfast and answer a few non-work email and will get back to you. :D [14:58:28] Cool :) Sounds good. [14:58:59] BTW, for reading back and thinking, Lucie is already on an NDA so that part is easy. She's been working with WMDE for a while. [19:44:20] halfak: sorry my breakfast took longer than expected. [19:44:31] No worries! [19:44:48] halfak: I reviewed the task we had with Andrew for GPU testing, and I think the points raised in that task are still valid. [19:45:11] Oh. Can you link? [19:45:26] check the last few comments, halfak: https://phabricator.wikimedia.org/T148843 [19:46:43] basically, halfak: the GPU testing will need to be done, and someone should lead that effort. There is no lead on our end (ORES/Research). [19:50:05] OK gotcha. I was thinking that maybe you our now that I think of it, dsaez, might be interested in working with some already-wiki-competent researchers on GPU testing. I've warned them that the GPU is not known to work yet and they were still excited by the idea of eventually getting a beefy GPU to support their (very wikimedia relevant) work. :) [19:50:27] But yeah, time is always an issue. I don't think they are blocked by this. Just seemed like a good opportunity. [19:52:24] I think they are doing something outside of tensor flow too, so it might be handy to have someone test the GPU without worrying about getting the tensor flow --> GPU pipeline running. [19:52:57] Not 100% sure how they are making use of a GPU now. [19:54:16] * lzia reads [19:55:59] yeah, halfak. a chat with dsaez can be a good idea, if he has the expertise, interest, and time. Given that he's in the onboarding process these days, I'd give him some time ;) but that's a decision between the two of you and DarTar. :) [19:56:23] Gotcha! Thanks for your consideration. :) [19:56:28] * dsaez reading the ticket [19:56:38] np, halfak. :) [19:57:32] * dsaez is trying to undestand the problem [19:59:06] halfak: btw, I'm slowly working with the tensorflow (and spark) people in bringing first class support to JupyterHub :) [19:59:09] unrelated to your current stuff ofc [19:59:37] That's great. I'll be related soon enough. I think that people have been running spark stuff from internal paws. [19:59:55] yuvipand: hi, do we have JupyterHub ?! [20:00:05] Probably j o a l (who is sleeping and doesn't need a ping) [20:00:12] on the stats machines? [20:01:32] dsaez: https://wikitech.wikimedia.org/wiki/SWAP [20:01:53] not well supported [20:03:09] wow [20:03:16] amazing [20:04:19] ottomata: the user/pass corresponds to...? [20:04:33] sorry [20:04:39] found it [20:07:26] madhuvishy: you may be interested in a new user for SWAP. ^ ;) [20:08:05] dsaez: hey :) yeah let me know if you run into anything! [20:09:02] sure, jupyter rulez [20:12:16] ottomata: could be possible that I don't have a wikitech account? I'm trying to reset the password, but not succeding :( [20:13:07] dsaez: you should definitely have a wikitech account if you got as far as ssh access [20:13:14] dsaez: didn't you have two? one staff and one volunteer? I'm trying to reconstruct conversations with robh at the time they were giving you access. [20:13:27] yeah, what madhuvishy says. [20:14:00] oooh [20:14:15] got it, is associated to my alias not, my email [20:14:25] dsaez: for SWAP, make sure to use the lower case version of your username, because users have run into issues with that (just in case this is a thing for you) [20:17:00] madhuvishy: Invalid username or password [20:17:14] dsaez: what's your shellname? [20:17:23] diego [20:18:36] shout out to madhuvishy giving SWAP love (and that name SWAP) [20:19:06] dsaez: it thinks invalid password [20:19:32] I've just reseted my password in wikitech [20:23:07] madhuvishy: Maybe the LDAP needs time for refresh.. [20:23:43] SWAP relies on LDAP for auth yes [20:23:47] possibly [20:23:54] btw, how do you install python packages there? I imagine that virtualenvs wont work [20:23:57] dsaez: can you get into say https://yarn.wikimedia.org/ [20:24:15] dsaez: all users have a venv created when you log in [20:24:35] you can just do !pip install and you'll automatically be installing things in your venv [20:24:36] madhuvishy: no, I can't [20:24:54] yeah I think then ldap needs a prod [20:25:41] ok. [20:26:39] great! [20:28:52] dsaez: you can also pip install from the terminal. it seems like ldap changes should be instant, can you login to wikitech from a different browser/incognito with your new password? [20:29:43] hey lzia, dsaez: I was out for lunch with Abbey and I’m a bit lost about the context of the ping [20:29:53] yes, I can [20:30:50] madhuvishy: I'm logging to wikitech without any problem :\ [20:31:26] hall1467, give me a couple minutes to join up. [20:31:32] Just came out of another meeting. [20:31:49] DarTar: I'm also not understanding exactly [20:32:03] halfak: Sure! [20:32:12] * lzia works on a summary for DarTar and dsaez [20:33:30] lzia: thanks, appreciated [20:33:52] :) [20:34:01] we had a task a while back: T148843 for GPU testing. The task is at the stage where if someone in Research/ORES leads the testing, can get done, otherwise not. On a separate note, halfak knows a researchers in WMDE who is interested to do neural net using the GPU. So, the question is, if I understand it correctly: do we want to pick up that task now that dsaez is in, if he's interested to work on it and have the time for it. [20:34:02] T148843: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 [20:34:03] dsaez: so your wikitech name is dsaez? [20:34:07] for this account? [20:34:09] no, diego [20:34:13] DarTar: ^ [20:34:51] DarTar: my gut feeling is that it's a bit early for dsaez to diversify even more. :D but I left it to you, him, and halfak to decide. :) [20:36:48] * dsaez is going for pizza, coming back soon [20:37:16] dsaez: right, I think your usernames may be mixed up for shell/wikitech [20:37:55] can you log in to ldap with your shell name [20:38:04] which I think is dsaez [20:39:04] * lzia wonders about the joys of living in Europe, going to grab Pizza at 22:40! :D [20:45:56] lzia: thanks, I’m a bit worried of asking diego to pick up stuff outside of the onboarding ideas we drafted TBH [20:46:16] unless there’s a strong reason for us to do so [20:55:35] yup, DarTar. I see it the same. Let's remember that ticket and come back to it when we have more time. [21:32:25] * dsaez is back [21:33:23] leila: don't you have pizza at 10pm ? [21:33:43] kitchens close at 9:30pm, pretty much without exception, dsaez. ;) [21:34:40] madhuvishy: nop, at least wikitech doesn't recoginizes dsaez as valid username [21:35:25] leila: wow, even in my small home town in Chile you can eat until 1am [21:35:40] dsaez: you really don't have to bring it up like that. ;p [21:36:51] DarTar, leila, I still don't know what exactly need to be done about that ticket. [21:37:25] dsaez: I suggest you don't take action until DarTar gets back to you. [21:37:52] ok [21:38:08] dsaez: uhhh and ldap? [21:38:28] nop [21:39:26] bd808: are you here? [21:39:37] o/ [21:40:04] dsaez here is dealing with the ldap issues [21:41:38] dsaez: if you are trying to log into LDAP auth things there are confusingly 2 different user names for the same account: the "wiki" user that you login to Wikitech with, and the "shell" user that you login to VMs with. [21:42:20] Your "wiki" username is "Diego" and your "shell" username is "dsaez" [21:42:37] bd808: in both cases I think is diego, at least this what my prompt says when logging in the stats machines [21:43:00] your LDAP account says different [21:43:13] I see [21:43:24] the shell user diego has been around since 2014 [21:43:25] so, how can I reset the LDAP password? [21:43:52] bd808: yes, that's also myself [21:44:10] oh.. but 2 different accounts [21:45:05] the "Diego Queiroz"/"diego" user is showing LDAP auth failures. The "Diego"/"dsaez" user is not [21:45:07] diego@stat1005:~$ echo $USER [21:45:09] diego [21:45:18] that is not controlled by LDAP [21:45:30] ok [21:45:54] what UID do you have there? Type `id` to find out [21:47:01] having the unx shell name not match between prod and LDAP/Cloud Services is unfortunate but may be causing confusion here [21:49:02] 17677 [21:49:56] Ok. That matches the cn=Diego / uid=dsaez LDAP account. [21:50:20] so in theory that is the LDAP account that you should be using as well. [21:50:34] ok, but I don't know the password for that username or how to reset it [21:50:47] cn == common name == wiki user; uid == user identifier == shell user [21:51:58] dsaez: go to https://wikitech.wikimedia.org/wiki/Special:PasswordReset and give username "Diego" and email "diego@wikimedia.org" [21:52:45] I think I've already have done, but I'm going to again [21:54:51] dsaez: The auth failures I see on the uid=diego account make me think you are trying to use the "diego" shell name somewhere with the uid=dsaez account's password [21:55:21] ok, I can change, but still I'm not recognized SWAP [21:56:07] nor yarn.wikimedia.org [21:57:19] for yarn you would use username "Diego" and the password that you reset via wikitech. That also assumes that your LDAP account is in the correct user group (not sure what one is used there) [21:57:36] and diego for SWAP [21:57:53] doesn't work, none of them [21:58:00] madhuvishy: does swap use uid? [21:58:25] bd808: yeah [21:58:31] oh [21:58:49] then he needs to use dsaez [21:58:57] right sorry [21:59:04] this is a bit of a mess :/ [21:59:11] verifying again [21:59:37] nor dsaez, diego, Diego works neither yarn or swap [22:00:03] :( [22:00:15] let me lookup the group that is guarding yarn... [22:01:12] yarn is behind cn=wmf,ou=groups,dc=wikimedia,dc=org and cn=nda,ou=groups,dc=wikimedia,dc=org [22:01:50] and this is SWAP https://github.com/wikimedia/puppet/blob/production/modules/jupyterhub/files/jupyterhub_config.py#L124 [22:02:05] hey dsaez, are you still around for a quick chat or superbusy with this thread? [22:02:28] uid=dsaez is not in the cn=wmf user group [22:02:33] DarTar: Let's talk [22:02:36] I can email you otherwise, I had a great chat with halfak about user-interactions and topical span of reverts [22:02:39] huh [22:03:11] so that explains why he can't log into either service [22:03:16] right [22:03:26] dsaez: oh darn, I actually have to leave now to pick up my little one. I might call you onm a hangout from the bus in 10 [22:03:41] ok. [22:04:02] brb dsaez! [22:04:23] any root can add dsaez to the cn=wmf group [22:04:43] assuming dsaez is actually staff :) [22:04:44] bd808: I'm happy to [22:04:47] he is [22:05:00] madhuvishy: have you extracted the requisite sacrifices? [22:05:09] he has prod shell access [22:05:29] to all of stat [22:05:58] yes yuvipanda I think enough sacrifices have been made [22:06:22] which kind of sacrifices? [22:07:29] dsaez: sorry, it's just a running joke between yuvipanda and I that all access related things in the WMF need sacrifices to the gods before you get it ;) [22:08:11] bd808: I don't think I know how to add someone though :) [22:08:38] interesting [22:08:40] ooh i found https://wikitech.wikimedia.org/wiki/LDAP#Add_a_user_to_a_group [22:09:56] ldapvi: the grossest way to use LDAP ever but our recommended tool. [22:10:28] pfft, the previous shell scripts were more gross :P [22:10:34] and had a lot of root exploits in them. [22:11:03] ldapmodify and LDIF files :) That's the right way to mange LDAP from a CLI [22:11:14] * bd808 is old and thinks like an old person [22:12:06] while sacrificies does not include using windows or iOS ... all the rest is doable [22:12:24] bd808: those instructions dont actually work lol [22:13:24] oh need to try from terbium [22:15:26] bd808: better now? [22:15:50] it is terrifying to edit that file! [22:17:22] dsaez: try now with username dsaez for SWAP? [22:21:41] not working :( [22:22:12] I don't see a login attempt here [22:22:36] notebook1002.eqiad.wmnet [22:22:43] let me try with 1001 [22:22:49] dsaez: aah, yes please use 1001 [22:23:12] failed :( [22:23:16] Sep 06 22:23:02 notebook1001 jupyterhub[3835]: [W 2017-09-06 22:23:02.612 JupyterHub jupyterhub_config:117] User dsaez not in allowed groups (researchers, statistics-privatedata-users, analytics-privatedata-users, ops) [22:23:25] is weird [22:23:27] looking [22:23:35] but atleast, not invalid password [22:23:39] so progress! [22:25:07] should i try with random password just to double check? [22:25:56] sure [22:26:18] madhuvishy: I think the next level of check is against admin.yaml group membership [22:26:40] I know I can't use SWAP because I'm not in any of those groups [22:26:40] bd808: yeah i think this may be some inadvertent bug on my end [22:26:49] he is on the right groups [22:26:54] but with shellname diego [22:27:21] https://github.com/wikimedia/puppet/blob/production/modules/jupyterhub/files/jupyterhub_config.py#L109 checks if the username is in the groups [22:27:22] the admin.yaml name and the LDAP name not matching seems highly problematic [22:27:35] but yes because of that [22:27:38] it doesn't match [22:27:39] so [22:27:44] one check or the other fails [22:28:08] his prod shell name should be changed to dsaez honestly [22:28:32] yeah I don't think we've encountered this before [22:28:37] but that's going to cause other problems on any host where he has an existing $HOME [22:28:47] ugh [22:28:58] it was a mistake of the initial addition to admin.yaml [22:29:29] i think there was a long thread about this [22:29:34] in the glorious future™ admin.yaml will be driven from LDAP and then his account will break [22:29:51] https://phabricator.wikimedia.org/T172891 [22:31:58] I wold have given a different answer than rob did in https://phabricator.wikimedia.org/T172891#3513095. I would have told them to abandon the new Wikitech account. There is no mandate from Legal or SuSa to have separate "staff" and "volunteer" Wikitech/LDAP personas [22:32:03] yeah [22:33:02] well okay, should we make a ticket then to change it? [22:33:22] either that or a ticket to swap the uids in LDAP [22:33:44] that seems easy? [22:33:57] I haven't looked to see if that has gross implications for Toolforge or Cloud VPS [22:34:30] uid=diego is a Toolforge member [22:34:34] so yeah... [22:34:42] jsut as gross as the prod change [22:35:09] * bd808 votes for prod pain over Toolforge pain [22:35:26] uid diego is the personal account? [22:43:20] madhuvishy: yeah [22:46:43] wait [22:47:04] you said that "diego" is associated to "Diego Queiroz" ? this is not me. [22:47:20] I use to have an account in the tool-labs, i think it was diego [22:47:28] but honestly I don't remeber [23:18:04] dsaez: oh! [23:25:35] bd808: seems like dsaez doesn't really own the other ldap account [23:26:09] ha! so the only fix is to fix prod's uid [23:27:54] there are a lot of hits for "diego" on https://tools.wmflabs.org/contact/ [23:35:09] bd808, madhuvishy, I found my old personal account, is not diego [23:36:43] bd808: we need prod's shell name to be dsaez, right? [23:36:52] and afaik no one else has that [23:38:08] dsaez: can you make a ticket to change your prod uid to be dsaez, so it can match with the ldap uid? [23:39:05] eeh... yes, if you think this is the problem [23:40:03] yeah [23:40:23] where/how? phabricator? [23:40:33] dsaez: yup, tag Operations [23:41:41] please ping me or bd808 on the task if you need help responding to folks, we'll convince them :) [23:42:02] let me see how to do it.. is just creat a task? [23:42:05] yeah [23:42:16] https://phabricator.wikimedia.org/maniphest/task/edit/form/1/ [23:42:43] and add tag "Operations" [23:43:24] assigned to... ? [23:44:40] no one [23:44:53] whoever picks it up in Ops will assign it to themselves [23:45:53] i'm off to the gym, you can subscribe me to the task, I'll keep an eye out for it [23:46:01] ok. thanks [23:46:06] cya all research folks!