[10:35:21] (03CR) 10Thiemo Mättig (WMDE): [C: 031] Let user make hidenondamaging default (032 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/278839 (https://phabricator.wikimedia.org/T130560) (owner: 10Ladsgroup) [13:12:59] halfak: is it ok that I merge this ? https://gerrit.wikimedia.org/r/#/c/278836/ ? It will bring some changes around and I 'll have to do some manual action to start redis with the correct db file. I expect a redis downtime of around 5 mins. [13:14:44] akosiaris, want to do it right now? [13:15:16] Any chance we could bundle it with other changes that will need some downtime? [13:15:23] Or is this blocking other changes? [13:23:03] halfak: it is blocking setting up ores redis in production [13:24:00] akosiaris, I understand that it is important. I'm thinking about timing. [13:24:18] Any change we can bundle this change with any others that would take the service down? [13:26:12] oh, timing is up to you. Hence me asking. I will anyway work towards the rest of ORES deployment issues while waiting. If you have other changes that would need to take the service down, yes we can bundle them . But are there any such changes in the queue ? [13:27:53] akosiaris, was wondering from your side. We just did a deployment over the weekend that cause some unexpected downtime. I hadn't planned any maintenance that would take the service down this week. [13:27:57] How blocking is this? [13:28:42] err, on a scale of 1=>10 ? at the moment a 3-4 I think. [13:28:42] Would it (1) not matter and it's easy to continue, (2) be slightly annoying to continue working without getting this change deployed or (3) block something important in a bad way? [13:28:51] OK. So more like (2) [13:28:53] Let's merge [13:29:04] I'll send out an announcement. [13:29:07] something between 1 and 2 [13:29:12] tell you what [13:29:20] I can live with it for another day I think [13:29:23] You expect 5-6 minutes? This is just going to reboot the redis host, right? [13:29:25] at the least [13:29:55] so lemme see if I will come up with other breaking changes in the next day and see what we do by then [13:30:20] but in the general scheme of things, I 'd like to merge it by Monday [13:30:32] Monday -- a week from now? [13:30:40] at the latest [13:30:52] well, it's 5 days more or less yes [13:31:45] akosiaris, I'm thinking that I'm OK with doing this now if it's going to disrupt your process. [13:31:52] o/ schana [13:31:56] hey halfak [13:32:02] Have you met akosiaris ? [13:32:09] schana: hey [13:32:12] He's helping us get the production machines set up. [13:32:19] hi akosiaris [13:32:28] schana is the newish software engineer on the research team. [13:32:40] :-) [13:33:31] ok. wanna schedule it then for say tomorrow some European morning time ? Thursday or Friday will work as well for me btw [13:33:45] the European morning time is less negotiable :P [13:34:20] How's 1400 UTC tomorrow? [13:34:42] sounds fine to me [13:35:09] it's a European afternoon time but I am ok with it :-) [13:37:19] akosiaris, sorry. That's the start of my day. [13:37:22] Thanks :) [13:40:14] thanks as well [13:54:52] o/ Amir1 [14:08:53] o/ akosiaris [14:09:01] Since you're already waist deep in ores. [14:09:05] Mind taking a look at https://gerrit.wikimedia.org/r/#/c/278455/? [14:09:18] Should be super easy [14:09:34] it has a -1 from me already [14:09:38] Oh! Saw that now [14:10:02] tl;dr, actually put something in that class, otherwise, why have it ? [14:10:22] akosiaris, so, I had assumed that it would install all of ores requirements -- e.g. languages for myspell/aspell. [14:10:35] Which are part of base ores [14:10:43] Really, a compute node is base ores. [14:10:52] The rest of work is done manually -- at least for now. [14:10:57] it will, but then why not include ores::base on the compute node ? [14:11:10] why go through an abstraction layer that does nothing ? [14:11:22] akosiaris, I guess because it is a machine type [14:11:35] We may want to do more configuration later. [14:11:41] ah, then that file does not belong in the module [14:11:42] Nice to have a role match an abstract role [14:11:47] it belongs in the roles [14:11:53] Oh! [14:12:21] roles get applied to machines, not modules [14:13:15] on the other hand, if that class actually gets filled with some useful stuff that does not belong in the roles, like package installation, file configuration and so on [14:13:20] I would be fine with it [14:13:49] akosiaris, fine with doing roles. I think this is evidence that I don't full grok puppet yet [14:14:10] Where are "roles"? [14:14:25] manifests/role? [14:14:59] Looks like ores doesn't have an entry there. [14:15:02] that's the old location. we are migrating it piece by piece to modules/role/ (confusing I know) [14:15:05] it kind of does [14:15:19] lol [14:15:21] modules/role/manifests/labs/ores/ [14:15:29] notice the "labs" part there [14:15:49] * halfak feels less bad for not finding this [14:16:04] I am adding the normal hierarchy here https://gerrit.wikimedia.org/r/#/c/278758/ [14:16:10] starting with redis [14:16:22] which has me thinking. [14:16:34] yuvipanda: around ? [14:17:17] yuvipanda: I don't think we will be needing https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/role/manifests/labs/ores/lb.pp in production, right ? [14:17:27] I see no reason for an nginx caching layer [14:17:41] we will be doing this with the varnish entrypoint [14:17:55] unless I hugely misunderstand something here ... [14:18:14] akosiaris, I don't think we're doing anything intersting with nginx. [14:18:38] My only interaction with the lb server is https://wikitech.wikimedia.org/wiki/Hiera:Ores [14:18:47] Note how the web nodes are defined. [14:20:03] akosiaris, have another look? https://gerrit.wikimedia.org/r/#/c/278455/ [14:29:07] akosiaris: yeah, no need for nginx in prod [14:29:20] akosiaris: it is just doing the role of lvs here since we've no lvs in labs [14:57:03] halfak: o/ [14:57:05] ^ [14:57:25] and I just saw yuvipanda merged your patches in puppet o/ [14:57:57] akosiaris merged most :) [14:59:24] still some of his works need merge: https://gerrit.wikimedia.org/r/#/q/owner:%22Alexandros+Kosiaris+%253Cakosiaris%2540wikimedia.org%253E%22,n,z [14:59:28] :) [15:01:16] o/ Amir1 [15:01:20] Just starting the skype call [15:01:39] it's in one hour [15:02:07] halfak: did you have daylight change? [15:02:47] A week ago, yeah. [15:02:50] Woops [15:02:57] So can you make it now, Amir1 [15:03:09] I can [15:03:18] jsut give me about two minutes [15:03:23] if that's okay [15:04:27] it will be [15:55:26] halfak: I can stay more to chat if we need anything to discuss regarding prod. etc. [15:55:56] oh I wanted to talk about the HPI hackathon [16:02:04] Amir1, sorry had to let dog out [16:02:07] What's up? [16:02:23] oh I heard that :) [16:02:59] I'm saying if we need to chat about anything we need to do, I can stay over time [16:03:33] and there is one thing I want to discuss, I guess you saw Lydia's email [16:03:57] I did. [16:03:59] :) [16:04:04] BRB [16:04:13] back [16:06:56] halfak: I might not get visa but it's unlikely, I will act from the German embassy which is much better than the Italian embassy [16:12:37] * halfak crosses fingers [16:23:43] thanks [16:29:55] o/ awight [16:31:56] halloo! [16:32:17] Amazing push by Amir1, the extension is looking great. [16:32:32] :) We're getting pretty close. [16:32:47] Just wanted to confirm that it was cool that I reassigned one of your tasks to Amir1 :) [16:33:03] any time :) [16:33:09] awight: o/ [16:33:12] I would be honored [16:34:06] I would be more than happy if you need anything done for the extension :) [16:34:20] fwiw, there are two mw-core patches remaining, https://gerrit.wikimedia.org/r/#/c/247249/ and https://gerrit.wikimedia.org/r/#/c/266670/ [16:34:30] They look like they'll need minimal nursing [16:35:33] yeah, I finish them if that's okay with you [16:35:52] I'm also brought back the reverted patchset with fixes included [16:36:53] Ooh, thanks for noticing that [17:19:03] Amir1, do you know what we need to do if we change any of our message labels? [17:19:09] I'm about to make a change to edit_type. [17:19:13] form in wikilabels [17:19:38] you just need to change the en one [17:19:54] OK. [17:20:05] and other messages will be marked as "outdated" so it shows up again in translatation list [17:20:42] Wholy moley! We have edit type translated into like 10 languages! [17:24:26] \o/ [17:25:53] we have to wait until Thursday and then you can see th real magic (these translations are for the first two or three days not first week) [17:26:13] Amir1, could you review this one carefully? https://github.com/wiki-ai/wikilabels-wikimedia-config/pull/15 [17:26:24] I can confirm that the form works. [17:26:48] brb lunch! [17:29:59] halAFK: LGTM, I can merge it, one note I should say about ordering: translatewiki told us that these translation appears in the order we define in the json file so it's not needed to make them sorted alphabetically. They changed some of orderings already [17:31:08] hmmm, /srv/ores/data/nltk ... what is this used for ? I see no use of it anywhere [17:31:22] I 'll upload a change to ores::base to remove it and take it from there [17:32:38] akosiaris: hey, it's being used [17:32:45] let me get you the link [17:33:53] it's empty on the web servers for sure [17:34:05] lemme check the workers as well [17:34:10] akosiaris: https://github.com/wiki-ai/ores-wikimedia-config/blob/master/config/00-main.yaml [17:34:51] I don't see a ref to /srv/ores/data/nltk there ... [17:35:05] there is one to /srv/ores/config/submodules/wheels/nltk/ [17:35:13] which is a different path altogether [17:35:35] oh I see [17:36:00] It was /srv/ores/data/nltk but I changed it to /srv/ores/config/submodules/wheels/nltk/ during the wheels migrations [17:36:19] also ores::base clones straight from github, which is a no go in production [17:36:29] probably it's left out [17:36:39] as I guessed [17:36:55] can we make a mirror in diffusion? [17:37:41] I am still looking at what that repo has inside it, and whether it makes sense to ship that content via a git::clone or via the deployment system [17:38:16] scap3 that is, cause we have to go scap3 these days [17:38:20] trebuchet is deprecated [17:39:30] we support scap3! [17:39:55] some modifications will be needed when we are moving to prod [17:41:15] (03CR) 10Ladsgroup: Let user make hidenondamaging default (032 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/278839 (https://phabricator.wikimedia.org/T130560) (owner: 10Ladsgroup) [17:41:27] yeah I noticed. But I do expect changes to be required [17:41:35] as you already pointed out [17:41:53] like list of targets, etc. [17:42:11] that's the easy things [17:42:30] the fact this will be the first python app to be deployed by scap3 makes me more afraid [17:43:09] hmm, I think for checks.yaml we need some adjustments in puppet (as I was told by releng) [17:43:33] I was able to deploy it (with lots of hacks) inside labs [17:43:46] locks of hacks I hear [17:43:50] lots* [17:44:08] not sure how to perceive that other than worrysome... [17:44:27] anyway scap3 written by us and used by us, so I can't worry too much [17:44:42] the hacks I did was to make the envirment more like tin.eqiad.wmnet [17:45:05] I would love to help if needed [17:46:02] oh you are helping already. Just asking questions is a great help. For example [17:46:14] what is the idea behind https://github.com/wiki-ai/ores-wikimedia-config ? [17:46:33] I see python files, requirements for pip, submodules, scap and a Makefile [17:46:38] * yuvipanda can provide background info if halAFK or Amir1 are unable to [17:46:45] and a fabfile !! [17:46:54] I assume the fabfile is the "old" way of deploying [17:47:01] yuvipanda: please do [17:47:06] * schana listens intently [17:47:17] I am wondering what that repo is for and how to ship it to production [17:47:21] akosiaris: yes, that's how we deploy/deployed [17:47:42] akosiaris: I think I have something you might liek [17:47:44] *like [17:47:56] https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/Deployment [17:48:07] akosiaris: yeah, fabfile is for labs mostly. because my other option was trebuchet [17:48:38] akosiaris: so the -deploy repository just is meta repo. has config and dependencies [17:49:05] akosiaris: it represents the current deployment in production, including a complete requirements.txt file (From which wheels are generated) and pins for submodules [17:49:32] link to the -deploy repo ? [17:49:39] or did you mean -config ? [17:49:44] akosiaris: -config, sorry [17:49:50] ok [17:50:15] so that will e the one repo we will be shipping to production [17:50:39] yuvipanda: hey, one thing: we generate wheels from requirements.txt but complete list of them (recursively) is in requirements-frozen.txt [17:50:51] akosiaris: yes [17:51:00] yuvipanda: I wanted to ask if that's okay [17:51:15] akosiaris: yup. [17:51:36] ok, in that case that git::clone in ores::base needs to go [17:51:50] and be replaced by scap3 [17:52:43] and now I have to dig a bit into scap3 documentation to see how to do that [17:52:51] yay, scap3! [17:53:38] akosiaris: I can point you to the phab cards we discussed with releng [17:53:47] Amir1: that would be helpful [17:53:59] and appreciated greatly :-) [17:54:33] https://phabricator.wikimedia.org/T128670 [17:54:42] https://phabricator.wikimedia.org/T130515 [17:55:22] https://phabricator.wikimedia.org/T130205 [17:56:31] also there is discussion regarding using git_fat. We haven't used git_fat because even though it's a little bit slow but not too slow. [17:56:43] I tested it in several places [17:57:19] so we are planning to use git_fat but not now [17:57:28] but if akosiaris or yuvipanda think we must do it before moving to prod, I start doing it :) [17:58:42] git_fat ? it might very well not be needed for now. We 've mostly used it to avoid storing giant .jar dependencies when deploying some (much loathed) java apps [17:59:08] I 've reserve judgement but my hope is we won't currently need it [17:59:12] we have pickle files sized around 30 MBs [17:59:19] sigh [17:59:23] overall about 200MB [18:00:23] we can reduce the number by using other kind of classifiers (SVC instead of RF) but halAFK should decide on that [18:00:46] I'm so excited \o/ [18:11:07] (03PS2) 10Ladsgroup: Let user make hidenondamaging default [extensions/ORES] - 10https://gerrit.wikimedia.org/r/278839 (https://phabricator.wikimedia.org/T130560) [18:21:20] I missed a lively conversation [18:21:40] :D [20:17:40] Wikilabels day! [20:17:52] All the forms. Get the data. Much new machine learning models. [20:24:30] halfak: http://ores.wmflabs.org/ui/ [20:24:33] 404 [20:24:35] :( [20:24:55] Yeah. Been working on some weird redirecting behavior. [20:25:01] That's actually been broken for a while. [20:25:05] I thinK i have a card for it. [20:25:15] We should be able to do a deploy today that'll fix it. [20:25:46] \o/ [20:25:57] tell me if I can do anything [21:14:02] (03PS5) 10Krinkle: Integrate with Special:Contributions [extensions/ORES] - 10https://gerrit.wikimedia.org/r/264608 (https://phabricator.wikimedia.org/T122537) (owner: 10Awight) [21:14:17] (03CR) 10Krinkle: "Removed "@" symbol so that hash is linked." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/264608 (https://phabricator.wikimedia.org/T122537) (owner: 10Awight) [21:15:10] (03CR) 10jenkins-bot: [V: 04-1] Integrate with Special:Contributions [extensions/ORES] - 10https://gerrit.wikimedia.org/r/264608 (https://phabricator.wikimedia.org/T122537) (owner: 10Awight) [21:22:29] halfak: Update on nlwiki anomaly? [21:22:52] Krinkle, yeah. Check the task! We deployed a new model that seems to behave better by the stats. [21:23:32] https://phabricator.wikimedia.org/T129624 [21:25:30] Hey Krinkle, it would be great if you check out this, if you have some time: https://gerrit.wikimedia.org/r/#/c/278841/ It was already merged but got reverted and I put your suggestions in it [21:45:15] o/ yuvipanda [21:45:23] We're getting some performance issues in wikilabels [21:45:29] What's a good way to check on the postegres DB? [22:02:14] Yeah... having persistent problems now. [22:02:18] * halfak gumbles [23:12:29] halfak: depends on what you mean by 'check' [23:12:42] halfak: your account has access to it, and outside of that, I just file bugs and bug akosiaris [23:50:47] yuvipanda, resource usage. Sorry for not being clear!