[15:05:37] halfak: o/ tell me if you're around [17:55:41] o/ Amir1 [17:55:51] hey! [17:55:55] I was waiting for you [17:56:05] Sorry. What's up? [17:56:26] last night I tried to bypass the git_fat issue by building a .deb file and installing it [17:56:43] no success, even though I learned about debian packages [17:56:59] Why would that help us with git fat? [17:57:11] git-(fat|annex|lfs) [17:57:30] because we need that and it's not released [17:57:40] so I tried to get the current version and make .deb file [17:57:56] at a moment of despair I learned that scap is python and can simply be installed by "sudo python setup.py install" [17:58:02] after feeling sooo dumb [17:58:05] I did that [17:58:13] and I got the same error [17:58:35] it turns out scap does a great job at hiding real reasons of failure [17:58:52] Damn it. It seems like that would be the first thing you'd consider. [17:58:53] and I realized how I can fix it [17:59:18] I think I did but it gave me an error before and I didn't check again [17:59:19] E.g fabric gives you all stderr and stdout by default, so if the terminal can communicate the error to you, you'll see it in fab. [18:00:04] * halfak does $ git checkout -b swagger [18:00:10] Let's do this [18:00:11] I know who scap works exactly (I read most of their codes) I even have several suggestion to improve their system [18:00:26] *how [18:00:45] halfak: Right now we can simply fix scap and merge it [18:00:45] Amir1, that's great. We'll need to prioritize a bit. [18:00:57] But let's look at fixing scap if you think it won't take too long. [18:01:03] and it's easy [18:01:03] In the end, we'll need to use scap anyway. [18:01:08] it's very easy [18:01:09] :) [18:01:11] Great! [18:01:20] the only thing I need is an instance [18:01:33] that has public proxy [18:01:54] strangely I can't add DNS proxy in https://wikitech.wikimedia.org/wiki/Special:NovaProxy [18:02:09] we need this instance like tin.eqiad.wmnet [18:03:00] halfak: can you create an instance in ores-staging [18:03:18] called "ores-deploy" [18:04:04] and create a DNS proxy for a folder in /srv/ [18:04:29] Amir1, I'm guessing that releng is going to have some strong opinions on running our own git-(lfs|fat|annex) server, but I'm a fan of experiments, so I'll get you set up. [18:04:31] so https://ores-deploy.wmflabs.org would points out to /srv/deployment in that server [18:04:40] Just know that we probably need a blessing before doing anything interesting :) [18:04:42] exactly [18:04:54] It's for test [18:04:56] You should have the power in ores-staging [18:05:03] not even for real deployment [18:05:04] I'm checking on that now [18:05:15] I can create that instance [18:05:23] Yeah. You are a proper admin. [18:05:33] but I can't change DNS procies [18:05:35] *proxies [18:05:45] https://wikitech.wikimedia.org/wiki/Help:Proxy [18:05:55] ores-blobs.eqiad.wmflabs? [18:05:59] I read this but not very helpful [18:06:06] ores-blobs.wmflabs.org [18:06:12] ^ ? [18:06:14] great [18:06:25] What machine should I point it to? [18:06:52] make a machine in ores-staging [18:06:59] kk [18:07:11] thanks [18:07:17] small OK? 20GB [18:07:23] 1 CPU [18:07:29] 2GB RAM [18:07:42] Maybe we want medium [18:07:45] Yeah [18:07:47] 40GB [18:07:56] yeah [18:07:59] medium [18:08:11] Does this need to be web-facing? [18:08:26] Or can it only be visible within labs? [18:08:42] visible within labs [18:08:45] We'll only build blobs within labs. [18:08:49] No DNS necessary! [18:09:00] Are you sure [18:09:04] Yeah! [18:09:14] ssh ores-compute-01.eqiad.wmflabs [18:09:17] does work [18:09:33] but http://ores-compute-01.eqiad.wmflabs does not [18:09:43] From within labs? [18:09:47] that's why I want to make a DNS proxt [18:09:49] You need http? [18:09:50] *proxy [18:10:01] that's how scap works.... [18:10:13] Amir1, try https://ores-web-01:8080 [18:10:25] Woops! [18:10:28] That won't work [18:10:39] http://ores-web-03 [18:11:06] I don't want to use a ores web server to deploy [18:11:16] You won't need to. [18:11:25] I just wanted to demonstrate that http works internally. [18:11:26] because I need to do git clone inside ores-web [18:11:45] cool [18:11:49] what about ores-compute [18:12:11] https://www.irccloud.com/pastebin/z2DNJHFZ/ [18:12:16] halfak: ^ [18:12:25] connection refused [18:12:47] I even tried the private IP [18:13:18] https://gist.github.com/halfak/650db5a987d541ef2f2a [18:14:55] ok [18:14:58] I get it now [18:15:02] it's the port [18:15:07] that's why It's not working [18:15:10] let me check [18:17:32] I just started up ores-blobs-01 in ores-staging. [18:17:57] Regretfully, I don't have a good way to give it a basic configuration (installing python, setting up virtualenv, etc.) [18:18:00] halfak: thanks! [18:18:03] But maybe you don't want that anyway. [18:18:24] I have a patchset for adding a simple "compute" role so that we don't have to configure compute nodes. [18:18:28] ^manually [18:18:41] https://gerrit.wikimedia.org/r/#/c/278455/ [18:22:36] I can't access it [18:22:42] using http [18:22:46] but let's do it later [18:22:50] to the swagger [18:23:13] Amir1, you'll need to run a server over http to access it. [18:23:19] But yeah. To the swagger. [18:23:34] I think I have a good method for delivering this. [18:23:53] We'll have a swagger at the top of each "scores/" path [18:24:35] v1 and v2 [18:24:41] Yup [18:24:55] This corresponds pretty closely to a scoring context. [18:25:10] So a scoring context with define it's own sub-paths. [18:25:20] It has direct access to the models. [18:25:31] So each model will fill in details for its path. [18:26:01] +1 [18:27:19] so do you want to work on v1 or v2? [18:28:11] I think we'll have to do them together to find out how it works. [18:28:31] I want to get a minimal together and run it through the HTML generator. [18:28:55] If you hack on the spec, I'll work on the utilities. [18:30:48] halfak: https://etherpad.wikimedia.org/p/ores_swagger ? [18:31:37] Yeah [18:31:44] kk [18:31:52] I'm looking at the python utilities for swagger now. [18:34:54] Looks like we can embed swagger yaml in our docstrings and have the system read from that. [18:35:14] Not sure I want to do that though since we'll need logic to generate docstring matter for each model. [18:38:50] let's stick with swagger.yaml [18:39:36] ? [18:41:51] I think we'll at least want to generate our documentation from our config and that won't work well with one giant yaml file. [18:43:57] Looks like `flasgger` isn' [18:43:58] t bad [18:46:15] that would work [18:46:18] :) [18:46:30] halfak: Are you editing the swagger etherpad? [18:46:35] nope. [18:46:42] Still looking at potential libraries we can use. [18:46:56] Trying to figure out which one will be good to invest in. [18:47:13] e.g. pyswagger looks to be too low-level [18:47:20] There's some nice flask integration options. [18:50:46] OK. Looks like flaskswagger is the winner. [18:51:12] Now to work out how we can manually give it pre-generated pyJSON [18:51:36] Editing in etherpad now. [18:51:47] I want to mock up a single model's documentation. [18:52:04] I guess it will be a set of paths. [19:10:57] Starting to consider having minimal swagger specs in the short term [19:11:12] And having the sec doc contain information about each model being a future feature. [19:30:59] halfak: I made an http-server using node js [19:31:02] it bloody works [19:31:06] \o/ [19:54:46] lol swagger doesn't support some behaviors of the ORES API because they don't want to encourage that kind of behavior. [19:54:47] Well. [19:54:58] If the API already behaves that way and we'd like to document it... [19:55:15] I guess I can always make a hyper-flexible json schema [20:03:35] Or that doesn't work either. [20:18:38] I think I found a way to hack it with url params [20:36:31] I think there is a very stange bug in scap3 regarding config [20:36:44] when you run a command directly it works fine [20:37:08] but when you do it via ssh (to do it in a target instance) it doesn't work properly [20:37:33] the config values that being over written is not very important for devs in prod, but it'll block us [20:59:03] What is the command? [21:06:38] "/usr/bin/ssh -oBatchMode=yes -oSetupTimeout=10 -F/dev/null -oUser=ladsgroup ores-compute-01.eqiad.wmflabs /srv/deployment/scap/scap/bin/deploy-local -v --repo ores-wikimedia-config -g worker fetch" [21:07:06] the main one is "/srv/deployment/scap/scap/bin/deploy-local -v --repo ores-wikimedia-config -g worker fetch" [21:07:14] the latter works [21:07:23] the former works but without proper config [21:07:42] I'm not even sure of that now :(( [21:07:48] I need a break [21:09:43] afk for an hour [21:09:58] kk [21:09:59] halfak: ping me if you need me on swagger [21:10:13] Na. I'm doing good. Just finished v1. [21:10:16] Starting on v2. [21:10:21] \o/ [21:46:40] Amir1: !log from which bot? :P [21:47:22] hey addshore, wm-bot I guess [21:47:31] *looks at remembering how to do things* [21:47:39] I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 2.8.0.0 [libirc v. 1.0.3] my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features [21:47:39] @help [21:47:39] Asimov v. 2, By jem (IRC) / -jem- (Wikimedia), 2010-16 - Bot IRC de apoyo a los proyectos y al movimiento Wikimedia programado en PHP - Las órdenes deben escribirse precedidas de alguno de los prefijos admitidos (@!-=) - 13Lista de órdenes: @ord - 13Problemas o sugerencias: @sug - 13Ayuda: @? 15,02orden / @ic - 10http://wikimedia.es/asimov?uselang=en [21:48:02] you just want channle logs right? [21:48:04] or something else? :D [21:49:40] addshore: I want "!log" command, adds to ORES SAL [21:49:49] ahhhh, thats a different bot ;) [21:50:11] * Amir1 keeps forgetting about IRC bots [21:51:25] Amir1: http://tools.wmflabs.org/sal/ores [21:51:59] hey yuvipanda ! :D [21:52:07] hey [21:52:09] hey, yuvipanda in order to add log to ores SAL we need to go to #wikimedia-labs and do "!log ores etc." [21:52:11] (still on vacation :P) [21:52:24] but I want to do it in this channel using "!log etc." [21:52:29] Amir1: I think there is something in git somewhere to make it join more channels? [21:52:30] you can bug bd808 to add it here if you wish. [21:52:51] it's already here [21:52:52] !log [21:52:52] No hay log abierto en #wikimedia-ai - log on para abrirlo, log list para listar los logs disponibles. [21:52:55] look [21:53:07] (I don't know why it talks Spanish here though) [21:53:08] well, doing !log here doesn't add it to 'ores' [21:53:12] well [21:53:16] sal is from stashbot not AsimovBot [21:53:29] hmm [21:53:35] stashbot: hey! [21:53:39] :D [21:54:00] it's here but not working on this [21:54:04] hmm [21:55:01] yuvipanda: btw. scap3 is much more complicated than what I thought [21:55:23] I'm still wrestling with it [21:55:38] probably I'll need your help [21:55:54] you need releng's help :) [21:56:04] I have no idea of scap3 [21:56:09] and hopefully never have to :) [21:56:40] these are very ops related stuff [21:56:43] puppet, etc. [21:57:21] speaking of which, halfak made tons of puppet patches in gerrit for ores, please take a look at them as soon as possible, most of them are pretty straighforward [21:58:05] well, I'll try but no guarantees. [21:58:13] awesome [21:58:14] and *do* talk to releng about scap3 issues. [21:58:14] thanks [21:58:37] We've been working with releng on a daily basis recently [21:58:45] cool! [21:58:47] Puppet patches can wait until vacation is done :) [21:58:55] Back on Monday? [21:59:00] indeed, but if they're related to scap3 I won't be able to do those either (mostly?) [21:59:25] but I guess this might fall through the cracks in which case it'll land on me again, but I guess I'm talking now from a complete ignorance perspective :D [21:59:30] with no idea what y'all are talkinga boutt [21:59:36] so I'll just stop saying things now [22:02:27] yuvipanda, halfak I just finished most of issues related to scap3 but several issues remained unresolved [22:02:37] 1- we need to build a git_fat store [22:02:46] I think yuvipanda can help with that [22:02:57] ask releng :) [22:03:01] Amir1, BTW, I'm much less excited about git-fat than git-lfs [22:03:13] if they tell you to ask me, *then* ask me :) [22:03:28] I'll chime in on the wheels ticket tomorrow tho [22:03:41] wheels are done :) [22:03:46] yuvipanda, kk. No rush there. It seems the issue may have been settled. [22:03:48] we already deployed using wheels [22:03:54] oh [22:03:56] nice [22:04:11] BTW, I had a good conversation with csteipp about wheels and Amir1's security for commits/merges/deploys. [22:04:15] halfak: I'm not a fan of these things in general [22:04:16] So we have approval and plans [22:04:22] nice [22:04:28] (I meant git-fat) [22:04:36] Amir1, have you tried using git-lfs? [22:04:37] It' [22:04:40] s kinda nice [22:04:49] It *just works* and I don't have to think about it very hard. [22:04:51] let me run a test and if I saw we don't need it at all, we don't [22:04:59] halfak: scap3 doesn't support git_lfs [22:05:04] git-lfs only 'just works' on github :D [22:05:12] yeah. That is a problem. [22:05:15] Both of those things. [22:05:58] another thing is the strange bug I told you [22:06:07] I investigate it in more depth soon [22:06:16] but first let me fix checks [22:13:03] halfak: it seems we don't need git-fat at all [22:13:07] it's fast enough [22:13:43] Agreed. [22:13:49] For the time being. [22:14:01] At some point we might find that we have HUGE repos and need to solve the problem [22:14:03] Not there yet [22:15:36] https://usercontent.irccloud-cdn.com/file/Ah6QEGvJ/ [22:15:49] halfak: "worker deploy successful" ^ [22:16:10] \o/ [22:22:42] halfak: one thing is that scap would deploy it to /srv/ores/ores-wikimedia-config instead of /srv/ores/config [22:22:51] I'm trying to find a solution for that [22:23:19] but since what I see, it's not possible [22:23:25] let me check and be sure [22:27:24] can't you make /srv/ores/config a symlink to that? [22:29:58] Platonides: that's possible but i wanted to make it "robust" [22:30:35] final solution: It's implemented but very incomplete [22:32:39] doesn't worth the headache [22:37:13] Amir1, https://github.com/wiki-ai/ores/pull/131 [22:38:52] Seems to work pretty well. It's definitely just a start though. [22:39:01] nice [22:39:06] looking [22:39:13] Right now, it's just generated from some yaml files. [22:39:31] But in the future, I think we'll want to generate paths for each model. [22:40:41] halfak: is it possible to use CDN instead of getting the whole swagger UI? [22:41:42] Amir1, Regretfully, it looks like there is not. [22:41:50] This is the recommended mode of deployment [22:41:59] I included the LICENSE and README.md explicitly [22:42:09] Also, it's apache 2.0, so we're fully compatible. [22:42:49] OK. I'm outta here. [22:42:54] Have a good day folks! [22:44:51] kk [22:44:56] just wanted to make sure [23:19:32] Amir1: halfak am writing a response on https://phabricator.wikimedia.org/T130205 now, fyi [23:20:18] thank you [23:20:25] :) [23:32:32] Amir1: halfak done [23:35:01] yuvipanda: great