[01:01:32] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 07Browser-Support-Google-Chrome, 07JavaScript: When Chrome is preventing the pop-out window , the code is stuck ! - https://phabricator.wikimedia.org/T131667#2347853 (10Danny_B) [08:37:57] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review, 07Puppet: ORES-staging is broken due to service::uwsgi mandatory scap::target invoke - https://phabricator.wikimedia.org/T136488#2336803 (10akosiaris) Changed merged, with the hope that the parameter will be removed soon per the comment. I 've a... [09:46:05] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2348513 (10schana) The stub status page only provides info like the following: ``` Active connections: 12 server accepts handled requests 155124 15... [10:04:55] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 07JavaScript: When "Prevent this website from pop-out again" is checked. - https://phabricator.wikimedia.org/T131667#2348572 (10Ghassanmas) [10:06:18] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 07JavaScript: When "Prevent this website from pop-out again" is checked. - https://phabricator.wikimedia.org/T131667#2174235 (10Ghassanmas) @ Danny_B the bug reproduced in all browsers, thus it's irreverent to add it "Browser-Support-Google-Chrome". th... [10:13:25] Amir1: o/ [10:14:11] akosiaris: hey [10:14:19] Thanks for merging [10:14:31] so ores is deployed on the hosts [10:14:39] \o/ [10:14:41] \o/ [10:14:43] but I got a couple of problems that stop me from enabling lvs [10:14:44] yess [10:14:47] ImportError: Could not load stopwords for revscoring.languages.english. You may need to install the nltk 'stopwords' corpora. See http://www.nltk.org/data.html [10:14:49] this is one [10:14:57] hmm [10:15:06] which is... only showing up on 2 of the 4 hosts [10:15:12] akosiaris: it seems the configs to load nltk are not right [10:15:21] let me show it to you [10:15:53] argh indeed [10:15:57] it is missing the config file [10:16:02] wat ? [10:16:10] https://github.com/wikimedia/operations-puppet/blob/production/modules/ores/manifests/web.pp#L27 [10:16:11] er, that's weird... [10:16:26] that part should be written in the yaml file [10:16:37] ah, I know what happened [10:16:46] interesting [10:17:03] oh, what a nice race condition [10:17:17] so, the config file lives in the directory deployed by scap [10:17:32] but on every deploy, that directory effectively changes due to scap's design [10:17:55] scap, trying to be as atomic as possible, deploys the code everytime in a new directory [10:18:03] yeah [10:18:03] and then just changes a symlink [10:18:27] I'm a little worried about storage issues though [10:18:43] but when the symlink has changed, the directory that the symlink points to does not yet have the config file and needs puppet to run to populate it [10:18:44] lol [10:19:00] Amir1: about how scap works ? [10:19:01] I am too [10:19:11] yeah [10:19:32] sudo du -hsc deploy-cache/ [10:19:32] 3.0G deploy-cache/ [10:19:32] 3.0G total [10:19:44] and that's with 2 deploys [10:19:59] wohaa [10:20:09] it does do shallow copies from what I remember, so that's not really true [10:20:40] as in it uses hard links, so in fact the actual taken space is not so much [10:20:46] but still, I am a bit worried [10:21:02] maybe submodules are not shallow (a bug?) [10:21:30] that is quite possible [10:21:36] we 'll have to ask releng [10:21:43] anyway, that is not something urgent [10:21:48] yup [10:22:05] the urgent 2 things are: fixing sudo rules, fixing the race condition with that config file [10:22:05] I added it to my to-do list and try to fix it ASAP [10:22:29] the sudo rules fix is practically in https://gerrit.wikimedia.org/r/#/c/291751/ [10:22:46] which halfak does not seem to like. Yuvi, if done correctly is more receptive [10:23:02] but the config file fix, I can not really help much there [10:23:17] akosiaris: even with it, it doesn't solve worker restart [10:23:28] so we have two restart issues [10:23:46] Amir1: yeah but that can be fixed with extra sudo rules, I 'll take a look later on [10:23:56] at least I hope [10:24:06] but I 'd like some help on the config file issue [10:24:09] exactly [10:24:28] the thing with adding sudo rules is that we can fix the uwsgi restart as well (and very simply) [10:24:32] so, in order for what we got to work, we need ores to be able to load a config file that is not in the deploy [10:24:42] (until we get that patch merged) [10:24:42] is that possible ? [10:25:10] if you add a paramtere to service::uwsgi called sudo rules and pass it on to scap::target [10:25:25] we can add both uwsgi-ores and celery-ores-worker [10:25:32] Amir1: please, let's focus on the config file problem, leave the sudo rules to me [10:25:39] sure [10:25:41] sorry [10:26:00] hmm [10:26:15] for the config file, it is a little bit simple [10:26:35] we can have a branch called "prod" or "deployment" (not "deploy") [10:26:58] and add these configs to the yaml files directly [10:27:36] akosiaris: it's not very robust and we need a more long term solution [10:27:37] remember that that config file contains the redis password [10:27:51] so it can not be in a repo :-) [10:27:57] it needs to be provisioned by puppet [10:28:09] we can do put some configs not all of them [10:28:19] the ores merges them together [10:28:24] https://github.com/wiki-ai/ores-wikimedia-config/tree/master/config [10:29:13] the only thing here is that we need to remove the nltk config from puppet [10:29:44] (because that would overrides the repo config) [10:29:44] that's not enough [10:29:57] there still is the issue of redis host and password [10:30:43] I think I understand you now, the address for the yaml file changes every time [10:30:50] (in every deploy) [10:30:53] effectively yes [10:31:11] so after every deploy /srv/deployment/ores/deploy/config/99-main.yaml does not exist [10:31:18] we need to wait for puppet to run and populate it [10:32:38] one very simple solution would be that we trigger a puppet agent after deploys [10:32:45] (via checks.yaml) [10:33:06] (so we need sudo rule for puppet agent for deploy-service too) [10:33:32] yeah, but I do not like the idea of software deployment trigger puppet runs. We can do it temporarily to solve the immediate problem, but in the long run it is not nice [10:33:46] yeah, I thought so [10:33:57] what we need is ores being able to read from a config file outside it's repo. say /etc/ores/config.yaml [10:34:31] is that possible ? [10:34:44] I'm pretty sure it's able to read another directory if permissions allow [10:35:00] but in that case we can't read it's own config dir [10:35:13] with some changes I might be able to get worked out [10:35:39] it won't be really hard but you need to give me about one hour or so [10:36:45] akosiaris: do you want to continue on lvs and I get this done for you? [10:37:33] Amir1: ok deal [10:37:42] nice [10:37:48] I am moving on with the rest, lemme know when you got someting [10:37:50] something [10:37:55] sure [10:38:00] oh and another thing. I 've had to create multiple repos in gerrit [10:38:11] to host the deploy dir and the submodules [10:38:27] cause scap does not currently support anything but gerrit :-( [10:38:40] I thought it also supported phab, but no dice [10:38:40] facepalm [10:39:31] https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/scap/manifests/source.pp;114c13f0d9993b799d4b8b33173fa5bde716c21b$102 [10:39:49] I suppose we should work with releng on that at some point [10:40:26] yeah, it seems it's pretty easy to solve [10:40:55] anyway, I 've created https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/ores https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/ores/deploy https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/ores/editquality https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/ores/wikiclass [10:41:27] just to get this moving. I 've pushed all the contents from the respective wiki-ai repos [10:41:46] and amended the ores/deploy one to use the git submodules from the others [10:41:54] you should have privileges to push there already [10:41:58] both you and halfak [10:42:21] akosiaris: is it a mirror (I mean is it being updated automatically) [10:42:32] thanks! [10:42:40] which is a mirror ? [10:42:56] well, which is a mirror of which more precisely [10:42:58] these repos. Are they mirror of the github repos? [10:43:06] unfortunately not [10:43:19] not sure that's even possible with gerrit [10:43:34] I 'll ask releng [10:43:44] awesome, thanks [10:43:49] please let me know [11:15:31] akosiaris: this will enable you to have more configs https://github.com/wiki-ai/ores-wikimedia-config/pull/60/files [11:16:33] If it's possible, let's wait until halfak wakes up and merge it, if it's super urgent. i can merge it (I tested it and worked prefectly fine) [11:18:57] I really need to sleep, please ping me if you need this merged and if I'm awake, I hear it and I do it [13:12:49] Just finished my test of celery with dependent sub-tasks. [13:12:56] See https://github.com/halfak/dependent_task_testing [13:13:10] It seems to work OK, but could result in deadlocks depending on how task ordering happens. [13:13:18] So I'm looking into workflows in celery to resolve this. [13:25:23] Regretfully, there doesn't seem to be a workflow designed for this. [13:25:26] Hmm. [13:32:32] I just ran a test with 6 requesters in parallel and all worked as expected. Now, I'm going to try to fill the queue up with junk and see what happens. [14:10:39] OK. now running 4 requesters that behave like precached and 3 that randomly request ids. [14:27:03] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2349237 (10Halfak) > do you have any more specific information about where/when there's missing info? Sorry, I'm confused. The scenario is that a... [14:34:51] I'm going to declare victory here. The test environment seems to be sound. [14:38:48] o/ schana [14:38:59] hey halfak [14:39:00] I've completed a test of dependent tasks using celery/redis [14:39:02] See https://github.com/halfak/dependent_task_testing [14:39:10] It works better than expected. [14:39:48] Celery warns us not to call "get()" within a task's execution, but it doesn't seem to provide us with reasonable alternatives in it's primitives (chain, group, map, etc.) [14:40:32] We can use (1) ordering and (2) priority to make sure that tasks get executed as planned. [14:40:59] I've been able to run many parallel requesters with different dynamics and they all seem to work as intended. [14:46:01] 10Revision-Scoring-As-A-Service-Backlog, 10bwds, 10revscoring: Language assets for Norwegian - https://phabricator.wikimedia.org/T131855#2180701 (10Johan) As someone who can read Norwegian, it looks finished. (I'd possibly move "cool" to from "bad words" to "informal".) [14:51:36] I was able to get the task queue filled up pretty well and didn't get any deadlocking because of priority. :) [15:32:24] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 10rsaas-editquality: Complete eswiki edit quality campaign - https://phabricator.wikimedia.org/T131963#2349460 (10Oscar) We definitely need more advertising for this [[phab:T131963|task]], we're only three or four volunteers active atm (https://es.wikipe... [15:32:24] 10[1] 04https://meta.wikimedia.org/wiki/phab:T131963 [15:32:24] T131963: Complete eswiki edit quality campaign - https://phabricator.wikimedia.org/T131963 [15:38:02] 10Revision-Scoring-As-A-Service-Backlog, 10bwds, 10revscoring: Language assets for Norwegian - https://phabricator.wikimedia.org/T131855#2349479 (10Nettrom) I went through and copied some commonly used insults into the bad words list. Noticed that the generated list contains quite a lot of variations, will y... [16:46:05] so much backscroll [17:30:21] halfak: o/ around? [17:35:37] 10Revision-Scoring-As-A-Service-Backlog, 10bwds, 10revscoring: Language assets for Norwegian - https://phabricator.wikimedia.org/T131855#2349957 (10Ladsgroup) >>! In T131855#2349479, @Nettrom wrote: > I went through and copied some commonly used insults into the bad words list. Noticed that the generated lis... [17:36:04] 06Revision-Scoring-As-A-Service, 10bwds, 10revscoring: Language assets for Norwegian - https://phabricator.wikimedia.org/T131855#2349961 (10Ladsgroup) a:03Ladsgroup [17:39:57] akosiaris: thanks for the notes, I will fix it [17:55:20] 10Revision-Scoring-As-A-Service-Backlog, 10Wikimania-Hackathon-2016: A training session on supporting ores in more languages in Wikimania Hackathon - https://phabricator.wikimedia.org/T134628#2350111 (10Johan) What can I do to support you, @Ladsgroup? [18:02:55] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Setup ores on scb cluster - https://phabricator.wikimedia.org/T124201#2350159 (10akosiaris) 05Open>03Resolved Finally done. ORES is deployed on the SCB cluster. [18:02:57] 06Revision-Scoring-As-A-Service, 06Operations, 06Research-and-Data-Backlog, 10Research-management, and 3 others: [Epic] Deploy Revscoring/ORES service in Prod - https://phabricator.wikimedia.org/T106867#2350161 (10akosiaris) [18:03:57] \o/ [18:04:39] akosiaris: shall we move on to the LVS or we need to finish this pull requests before? [18:05:18] also thanks for sudo rules patch :) [18:06:33] Amir1: I was about to ask that, that request looks good to me, feel free to move with it. I suppose we will have to amend a bit the uwsgi::app but it should work [18:06:52] as far as LVS goes, I am moving onwards independently anyway [18:07:00] yesss [18:07:34] one thing, what is your suggested path? /etc/ores/? [18:10:21] yup [18:11:29] kk [18:11:39] probably make a patch there soon [18:15:30] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels: Edit quality campaign for Swedish Wikipedia - https://phabricator.wikimedia.org/T131451#2350236 (10Johan) I have completed the translations. [18:22:10] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels: Edit quality campaign for Swedish Wikipedia - https://phabricator.wikimedia.org/T131451#2350286 (10Ladsgroup) @Johan Awesome thanks. Now we need a landing page. If you translate [[https://en.wikipedia.org/wiki/Wikipedia:Labels|This page]] to Swedish. I fin... [18:22:10] 10[2] 04https://meta.wikimedia.org/wiki/https://en.wikipedia.org/wiki/Wikipedia:Labels [18:22:58] 10Revision-Scoring-As-A-Service-Backlog, 10Wikimania-Hackathon-2016: A training session on supporting ores in more languages in Wikimania Hackathon - https://phabricator.wikimedia.org/T134628#2350292 (10Ladsgroup) Thanks for showing interest. I will let you know soon. I don't have anything particular in my min... [18:41:06] halfak: tell me when you are around. Lots of news to share [18:41:15] Just got back [18:41:25] yess [18:41:30] one sec [18:41:56] halfak: first, the ores is deployed in scb cluster now [18:42:16] Awesome. I imagine that it is generally inaccessible, right? [18:42:25] we are working on the public endpoints (LVS as LB and varnish) [18:42:57] it's accessible but not to the outside world [18:43:10] you can imagine it's not very hard [18:43:19] the hardest part is done now [18:44:00] Indeed. :) Great news. [18:44:07] before the next deployment we need to finish some stuff, it's a little bit long but tell me when you want to (and have some time) talk about it [18:44:28] Right now, i have an hour. [18:44:59] 1- first we need to move puppet-related yaml files to another directory (/etc/ores) [18:45:09] Why? [18:45:10] WTF [18:45:18] so the ores should be able to read several directories [18:45:33] scap3 [18:45:52] https://github.com/wiki-ai/ores-wikimedia-config/pull/60 [18:45:59] so I put up this PR [18:46:01] I'm getting very tired of changing something that is simple and works nicely to something more complicated. [18:46:44] it's complicated but they had a good reason for it [18:46:44] Amir1, was looking at that. We don't usually run the ores_wsgi.py as a script. [18:46:49] uwsgi runs it as a module. [18:47:25] hmm, I thought uwsgi can pass arguments to it [18:47:56] I don't think so. [18:48:11] I don't think it can run it as a module we don't have main() entry point [18:48:42] if __name__ == '__main__': ... [18:48:45] At the bottom of the file [18:49:11] We don't *run* it, uwsgi imports it as a module and then accesses the "application" variable. [18:50:05] do you have anything in your mind to bypass that? [18:50:25] 1st, it would be good to know why we need to have multiple directories. [18:50:36] I think it's because redis needs to use passwords [18:50:40] and these passwords come from puppet [18:50:45] YuviPanda, we already have a solution for that [18:50:52] Multiple configs are OK [18:51:00] Multiple directories of configs -- that's just weird. [18:51:10] Let's use symlinks or something if the files *must* be elsewhere. [18:51:12] ah, I see. I guess you could just add a private config [18:51:18] and deal with the fact that it'll dirty the git tree [18:51:21] by adding a .gitignore [18:51:28] puppet can put the file anywhere [18:51:33] because scap3 actually deploys in a new directory [18:51:35] every time [18:51:45] Amir1, can scap3 make symlinks? [18:51:56] lol wat [18:51:56] and it makes [18:51:58] really?! [18:52:24] BEST SCOPED SYSTEM EVER DESIGNED [18:52:46] because in case of failure it can rollback [18:52:49] * YuviPanda pulls halfak out of rabbit hole again [18:53:01] (one of the reasons, I guess) [18:53:33] (we have storage issues that we need to take care, I need to talk to releng about it) [18:53:53] So, right now, I like symlinks as a solution better than two config directories. [18:54:22] But if we must have two config directories, let's just hard-code them in ores_wsgi.py and ores_celery.py. [18:54:43] Those files are installation specific configuration anyway. [18:54:53] https://usercontent.irccloud-cdn.com/file/u3zCWR0K/ [18:55:24] then it makes symlink from /srv/ores/deploy-cache/rev/#hash/ to /srv/deployment/ores/deploy [18:55:41] so all of our stuff are in /srv/ores/deploy-cache/rev/#hash/ [18:56:02] OK. So can we have a symlink in the /rev/#hash/config/ for the system specific configs? [18:56:32] the thing is we need to wait for puppet after every deploy to populate the new dir [18:57:39] I can't believe I'm going to suggest this but [18:57:45] you can check in a symlink into the git repo [18:58:15] halfak: I don't think that would be possible. We don't know the hash [18:58:34] Amir1, but after the symlink is created, we don't need to know it. [18:58:44] Also, per Yuvi's proposal, If we're doing that, we might as well hard-code the directory in the celery/uwsgi modules. [18:59:00] I'm assuming beta will look the same. Is that right? [18:59:06] yup [18:59:25] I think YuviPanda's suggestion sounds really good [18:59:36] Symlinks in github? [18:59:41] yup [18:59:42] OK. That can work for me. [19:00:14] remember to document it extensively for the next person who encounters a dangling symlink in a repo! [19:00:17] but how can we read from symlink [19:00:33] symlink looks just like any other FS node [19:01:25] we might need to tell uwsgi to read the config dir recursively [19:01:39] You can symlink the files directly [19:01:54] That's how you do these things in apache's config pattern [19:02:04] sites-enabled --> sites-available [19:02:22] yeah, that would be easy [19:02:44] I thought, like wikilables, we might need several config files from puppet [19:04:13] Amir1, sure. We'd link each one explicitly. [19:04:33] yeah, you're right [19:05:18] okay, the first issue is resolved. We have another one [19:05:56] it seems scap doesn't support anything except gerrit (even differntial) for submodules [19:06:17] Why does scap support gerrit and not git? [19:06:31] so Alex made several respos in gerrit and before every deploy we need to update those because scap will actually reads from gerrit [19:06:46] gerrit uses git too [19:06:56] I mean gerrit.wikimedia.org [19:07:23] https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/scap/manifests/source.pp;114c13f0d9993b799d4b8b33173fa5bde716c21b$102 [19:07:46] These are the repos: [19:07:47] Can we have gerrit.wikimedia.org auto-replicate our github repos? [19:07:47] https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/ores https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/ores/deploy https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/ores/editquality https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/ores/wikiclass [19:08:03] halfak: I asked, no [19:08:04] "mediawiki/services/" [19:08:07] :( [19:08:08] Well... [19:08:28] Why is the answer "no"? [19:08:35] Because obviously it is technically possible [19:08:46] E.g. part of our deploy process could be a pull/merge to the gerrit repos [19:09:11] gerrit can't do it on it's own [19:09:28] I might need to write a bot to deal with the API and does it for us [19:09:51] API == just git, right? [19:10:09] gerrit's replication plugin is outgoing only unfortunately [19:10:10] I suppose the bot will need rights to push to gerrit. [19:10:12] and "git review", yeah I think so [19:10:27] hey akosiaris [19:10:50] hey [19:11:53] is it okay for you to have symlinks for configs? [19:12:05] ? [19:12:14] link 99-config.yaml -> /etc/ores/config.yaml ? [19:12:20] yup [19:12:20] s/link/like/ [19:12:42] akosiaris, like apache configs [19:12:54] sites-enabled --> sites-available [19:12:54] not in the long run, no. [19:13:11] but for the short term, yup [19:13:18] What's wrong with the long term? [19:13:31] you are provisioning a dangling symlink [19:13:42] not exactly the sanest thing to do [19:14:06] akosiaris, but it's apache's normal config pattern. [19:14:11] Not saying apache is sane [19:14:17] But it's at least not unheard of. [19:14:18] apache does not provision dangling symlinks [19:14:27] Apache does not provision [19:14:33] it creates symlinks after making sure the sites exist [19:14:37] But it does use symlinks to bring config into the config dir. [19:14:59] Apache doesn't create the symlinks, the operator does when configuring the service. [19:15:18] yeah, by using a2ensite, which checks that you don't create a dangling symlink [19:15:38] But you still *can* create a dangling symlink :/ [19:16:03] and you can shoot your foot with a shotgun, but if you do it, that's your problem [19:16:20] Either way, I think it's a fine option to hard-code the location of the /etc/ores/ dir in the .py files just like the local "config" folder is now hard-coded. [19:16:33] akosiaris, yes, let us not have shotguns? [19:16:40] if you are fine with that, I am too [19:16:59] deal! [19:17:31] Maybe I should have said "let us not have feet" [19:17:33] ;) [19:17:41] hehe [19:17:43] * YuviPanda makes the case for the second git --amend [19:18:18] (not actually! still...) [19:20:32] Amir1, when we think about hard-coding this in the ores_*.py files, I think we should also consider if there is any functionality we want to move up to "ores" [19:20:48] Or yamlconf [19:21:36] heh, icinga green finally. deploying the LVS change [19:21:47] hmm, we have two things that puppet writes for us: nltk data path, redis settings [19:21:55] \o/ [19:22:43] nltk data can be specified in our config [19:22:53] We could even have /etc/ores/ be specified in our config [19:23:04] It would be a little bit weird to read the config in order to modify it though. [19:23:31] hmm, yeah [19:27:20] Amir1: btw, re https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/scap/manifests/source.pp;114c13f0d9993b799d4b8b33173fa5bde716c21b$100 - note that the FIXME there is now fixed, and git::clone accepts the default_source param, which can be set to gerrit or phabricator [19:27:44] so if you can configure git mirroring in diffusion, you can use that [19:29:39] akosiaris: so, varnish and public DNS are left? [19:29:40] akosiaris: ^ [19:30:49] YuviPanda: oooh, that's nice to know [19:31:13] Amir1: the public DNS is easy, varnish, I 'll have brandon verify I am not doing something stupid before merging [19:31:52] nice thanks :) [19:35:01] 06Revision-Scoring-As-A-Service, 10ORES: [Spike] Implement & test dependent tasks in Celery - https://phabricator.wikimedia.org/T136875#2350625 (10Halfak) [19:36:25] akosiaris@neon:~$ curl -I ores.svc.eqiad.wmnet:8081/ [19:36:25] HTTP/1.1 200 OK [19:36:25] Content-Type: text/html; charset=utf-8 [19:36:25] Content-Length: 2720 [19:36:27] :-) [19:36:43] * Amir1 goes for dancing [19:36:47] 06Revision-Scoring-As-A-Service, 10ORES: [Spike] Implement & test dependent tasks in Celery - https://phabricator.wikimedia.org/T136875#2350642 (10Halfak) See repo here: https://github.com/halfak/dependent_task_testing I created some test output here that demonstrates the whole system works together: https://... [19:36:49] and champagne [19:37:16] 06Revision-Scoring-As-A-Service, 10ORES: [Spike] Implement & test dependent tasks in Celery - https://phabricator.wikimedia.org/T136875#2350643 (10Halfak) @schana, any notes on the implementation? [19:37:35] so, there's a small snafu, we are also deployed on codfw, but there is no redis server over there [19:38:04] but that's solveable [19:38:24] we will just not be enabled over there for a while. But codfw is anyway the backup DC [19:40:26] cool [19:50:29] icinga is green, and with that I am off. c ya [19:50:39] o/ [20:33:06] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: Deploy ORES extension in fawiki - https://phabricator.wikimedia.org/T130211#2350829 (10Ladsgroup) a:03Ladsgroup [20:33:32] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 10Wikidata: Deploy ORES extension in wikidatawiki - https://phabricator.wikimedia.org/T130212#2350845 (10Ladsgroup) a:03Ladsgroup [21:20:40] 06Revision-Scoring-As-A-Service, 10Wikilabels: Edit quality campaign for Swedish Wikipedia - https://phabricator.wikimedia.org/T131451#2351062 (10Ladsgroup) [21:48:07] /38 [22:40:00] OK. I think I am getting close to the end of this refactor. The lines of code are starting to go down substantially. [22:40:04] Which is usually a good sign. [22:40:52] Have a good one folks! [22:40:53] o/