[09:47:39] akosiaris: o/ [09:47:47] hey, tell me when you have some time [11:27:45] 06Revision-Scoring-As-A-Service, 10Wikilabels: Wikilabels is down - https://phabricator.wikimedia.org/T136520#2337544 (10Ladsgroup) >>! In T136520#2337562, @Halfak wrote: > The CORS regex doesn't match wikidata.org. This is unrelated as I have been testing on en.wikipedia.org > > ``` > (3.4)[halfak@graphite:... [12:51:42] 06Revision-Scoring-As-A-Service, 10Wikilabels: Write post-mortem for wikilabels downtime (2016-05-29) - https://phabricator.wikimedia.org/T136523#2338624 (10Ladsgroup) https://meta.wikimedia.org/wiki/Research_talk:Revision_scoring_as_a_service/Work_log/2016-05-30 [12:52:58] akosiaris: it would be great if you take a look at https://gerrit.wikimedia.org/r/291527 and https://gerrit.wikimedia.org/r/291572 [12:53:00] thanks [13:19:25] afk for a while [13:24:05] Amir1: o/ [13:24:17] sorry for not responding earlier today, family issues [13:43:25] akosiaris: no worries at all [13:43:49] tell me when you have some time [13:47:00] akosiaris: wrt https://gerrit.wikimedia.org/r/#/c/291527/ [13:47:17] E.g. fabric in our setup in labs [13:47:36] (right now our staging setup is broken because it can't connect to tin.eqiad.wmnet) [13:50:49] About this: https://gerrit.wikimedia.org/r/#/c/291527/ I must add, I think you had this in your mind while implementing it. Just look at the documentations. "Only applicable when # The user that will own the service code. Only applicable when $deployment ='scap3'" [13:52:15] sigh, copy paste error. What I 've had in my mind what that scap3 would be the only deployment method (at least for a while) [13:52:37] so, the problem is the scap deploy-local thing failing, right ? [13:52:44] yup [13:53:10] I tried everything, even I built a tin instance in the ores-staging project [13:54:14] it would be enough to add an /etc/hosts entry pointing tin.eqiad.wmnet to the IP of deployment-tin.deployment-prep.eqiad.wmflabs [13:54:20] scap3 is not an easy deployment system for test purposes [13:54:52] indeed [13:54:56] akosiaris: but the deployment-tin in beta cluster is not accessible for other projects [13:55:05] or maybe I'm wrong [13:55:53] it is [13:56:10] okay [13:56:13] let me try that [13:57:09] I think we can start working on deployment of ores in prod [13:57:45] I don't have access otherwise I would help much more [14:09:30] akosiaris: "[sudo] password for deploy-service:" it seems we haven't granted sudo rights for restarting ores services to deploy-service user via scap::target [14:09:46] let me find my settings [14:10:46] (line 14 https://gerrit.wikimedia.org/r/#/c/280403/47/modules/ores/manifests/scapdeploy.pp) [14:11:08] (right now, trying to find a fix) [14:12:36] akosiaris: sorry to over ping, you haven't defined sudo rules to pass to scap::taregt in https://github.com/wikimedia/operations-puppet/blob/production/modules/service/manifests/uwsgi.pp. Tell me if you think we should have it and I make the patch [14:15:57] yes, that is on purpose. those are now defined in scap::target [14:16:44] well, at least one of them for what I see. The worker part will not be autogenerated [14:17:24] It got error for both of them [14:19:21] ? [14:21:15] akosiaris: I mean, per what you say, it should have rights to restart at least the web service, but it couldn't do it [14:30:59] hmm, indeed. it sets deploy-service ALL=(root) NOPASSWD: /usr/sbin/service ores * [14:31:06] which is not what we want per se... [14:31:30] damn uwsgi::app is to blame here, lemme see if it is fixable there [14:41:18] kk [14:49:03] akosiaris: this fixes some issues (I need to do some stuff with our code again) https://gerrit.wikimedia.org/r/#/c/291751/1 Can you tell me when do you want to merge it? [14:49:21] do you want to merge it right now or wait for a while [14:58:00] I 'd like YuviPanda and godog to have a look first [14:58:16] otherwise, ASAP [15:08:05] akosiaris: awesome, after that we need to grant rights for restarting the celery worker service, what is your suggestion? letting service::uwsgi append to scap::target sudo rules or we should do it explicitly in ores::web. The latter sounds harder to me [15:12:32] going to sleep, have a very important meeting tomorrow morning [15:12:38] o/ [15:18:37] o/