[07:51:13] o/ [07:51:22] I just got connected to the wifi at the WikiCite event [07:51:40] Will be hacking on scripts for converting a giant list of DOIs into metadata [07:55:03] Luckily, I got some sleep on the flight, but that means I didn't get more work done on the ORES refactor. If I find some downtime in the next couple of days, I'll be updating the etherpad with links to code bits. [07:55:16] https://etherpad.wikimedia.org/p/ores_refactor [17:39:51] akosiaris: hey, if you are around. Can you give me an eta for merging this https://gerrit.wikimedia.org/r/#/c/288618/ (the patch for using the new uwsgi class for ores) I want to deploy new version of ores and I'm kinda blocked on this [17:45:59] Amir1: I am on meeting right now, wanna do it right after ? [17:46:05] like in 30 mins ? [17:46:23] akosiaris: yup why not [17:46:31] thanks and sorry for taking your time [18:23:18] Amir1: ok, are you ready ? I 'll disable puppet across the ores project, merge and run on only one host first to make sure we don't bring the entire thing down [18:23:34] akosiaris: okay [18:23:37] \o/ [18:24:33] Amir1: btw, in case you haven't noticed, the scap3 keyholder status is finally being sorted out and is on a good track. Which means we can pick up https://gerrit.wikimedia.org/r/#/c/280403/ again and move on with it [18:24:53] awesome [18:25:06] akosiaris: and one thing: I finally got my NDA [18:25:16] \o/ [18:25:20] https://phabricator.wikimedia.org/T134651 [18:25:32] great! [18:32:39] Amir1: merged. running puppet on ores-web-03 [18:32:48] yess [18:33:05] let me check the whole ores [18:35:32] invalid byte sequence in US-ASCII at /etc/puppet/modules/service/manifests/deploy/scap.pp:1 on node ores-web-03.ores.eqiad.wmflabs [18:35:33] akosiaris: I just did a curl 0.0.0.0:8080 in ores-web-03 and it worked just fine [18:35:35] wat ? [18:35:47] Amir1: yeah, I 've disabled puppet everywhere in the ores project [18:35:52] the change has not propagated yet [18:35:59] oh okay [18:36:01] and then, there is that weird error... [18:36:25] akosiaris: git blame? [18:36:54] yeah, but funny thing is .. this file has not been touched since Mar 14 [18:37:10] o.O [18:38:00] hmm [18:38:05] puppet—it [18:38:12] that dash there... it's not a dash ... [18:38:21] it's definitely not ASCII [18:40:35] was puppet even running in ores, this is getting more and more weird [18:42:53] akosiaris: everything in ores depends on puppet, it was definitely being ran [18:43:24] maybe we are introducing dependency to scap somehow? [18:43:57] yeah we are, but funny thing is that had not triggered anywhere else [18:44:46] anyway, it's being applied now on ores-web-03 [18:44:51] let's see what we broke :P [18:44:59] akosiaris: I saw such errors before [18:45:07] it's 400. I think connection was bad [18:45:13] on different files, etc. [18:45:23] no I fixed it in https://gerrit.wikimedia.org/r/#/c/290728/ [18:45:33] it was a genuine non ASCII char [18:45:36] puppet does not love that [18:45:53] :D [18:45:54] okay [18:49:31] akosiaris: we need to kill off the uwsgi-ores-web service manually (I think) [18:52:56] Amir1: yeah did that already [18:53:39] awesome [18:58:58] Amir1: ImportError: No module named site [18:59:16] super strange [18:59:19] encountered that before ? I seem to remember something [18:59:27] can't find it [18:59:27] akosiaris: I'm looking at logs [19:01:54] heh, an import site works in /srv/ores/venv [19:02:02] ok this is getting weird [19:02:15] it seems it doesn't load venv properly [19:02:27] (I'm still checking logs and stuff) [19:02:46] it does say Set PythonHome to /srv/ores/venv [19:02:47] oh wait [19:02:52] Python version: 2.7.9 (default, Mar 1 2015, 13:01:26) [GCC 4.9.2] [19:02:54] argh [19:02:58] that is my fault [19:03:19] -p python 3 [19:03:19] 04Error: Command “p” not recognized. Please review and correct what you’ve written. [19:04:41] Amir1: ok I think we are going to be ok, it's now running [19:04:46] lemme fix the puppet config [19:04:56] awesome [19:04:59] thanks akosiaris [19:05:16] I see requests comming in /var/log/ores/main.log [19:05:17] yay! [19:05:35] no more journalctl? [19:05:58] akosiaris: I will kill uwsgi-ores-web in other nodes [19:06:08] tell me once the puppet patch is there [19:12:29] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 10Research-management, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2327592 (10Ladsgroup) >>! In T134651#2324874, @Wwes wrote: > Approved Thanks :) [19:14:11] akosiaris: I have this patch for watchdog in precaching that I need to be merged after I deployed the new version. Tell me when it would be okay for you or YuviPanda to check this out. https://gerrit.wikimedia.org/r/#/c/290733/1 [19:14:13] thanks [19:14:45] Amir1: I 'll take it [19:14:52] so I am enabling puppet all over [19:14:55] awesome [19:14:59] yess [19:15:11] I see ores-web-03 working just fine [19:15:18] so let's get this everywhere [19:15:33] yeah, you just enable it and I do the puppet agent and killing off [19:15:44] niah don't worry I got it [19:16:51] Could not find node 'ores-worker-10.ores.eqiad' [19:16:59] there's something fishy on this box [19:17:23] thanks [19:17:31] arning: Unable to fetch my node definition, but the agent run will continue: [19:17:31] Warning: Find /production/node/ores-worker-10.ores.eqiad?transaction_uuid=d79ef341-a614-4a8b-afab-a1903016fc19&... resulted in 404 with the message: Not Found: Could not find node ores-worker-10.ores.eqiad [19:17:32] lol [19:17:44] ok but that needs to be fixed but it is not a problem right now [19:17:51] this probably is around for a long time [19:18:20] lol [19:18:40] ok I see the pip install going on in 4 and 5 [19:18:55] so the scap cmd.sh deployment cmd is working there as well fine [19:19:44] akosiaris: in ores setup in labs we use fabricator [19:19:53] in beta we use scap [19:19:57] fabric you mean [19:20:20] yeah, feel free to do so in the future as well [19:21:02] yeah [19:21:03] :D [19:24:16] Amir1: ok, I declare it done [19:24:21] I think we are good to go [19:24:32] so, I 'll take a quick look into worker-10 now [19:24:40] yess [19:24:40] yes [19:24:51] yes [19:25:42] so /etc/resolv.conf on ores-worker-10 was corrupted [19:25:51] as in .wmflabs was missing [19:26:00] domain ores.eqiad. [19:26:00] search ores.eqiad. eqiad. [19:26:01] fixed [19:26:11] but I know not how and why this happened [19:26:26] Amir1: so, one last thing, wanna do a deploy ? [19:26:41] yup [19:26:44] I'm doing it [19:26:50] it's a little bit complicated :D [19:27:00] hehe [19:29:30] https://gerrit.wikimedia.org/r/#/c/290733/1 +1ed, tell me when you want me to merge that [19:29:40] sure [19:29:46] just in one or two minutes [19:32:29] akosiaris: please merge [19:33:03] done [19:33:09] awesome thanks [19:35:02] Amir1: I was thinking we probably have to adapt a bit the fabfile to have it use the new path now [19:35:33] config_dir = '/srv/deployment/ores/deploy' that is. The rest probably needs no change [19:35:46] akosiaris: in the ores setup in labs I use /srv/ores/config [19:35:56] I changed in hiera for the setup [19:36:07] https://wikitech.wikimedia.org/wiki/Hiera:Ores [19:36:28] I'm okay with going to the new patch [19:36:30] *path [19:38:19] Amir1: er, "ores::base::config_path": /srv/ores/config does not specify the deploy repo's path, just the .yaml file's position [19:39:50] which btw can be anywhere you want in labs, but in production we 'll have to put it either under scap3's control (not sure about it yet, I see that support just arrived yesterday) or under /etc/ores/ and managed by puppet [19:40:30] * YuviPanda would highly prefer that be controlled by scap3 and not puppet [19:40:39] that makes 2 of us [19:40:51] hence decreed! [19:41:06] Amir1: sigh, no I was wrong, ignore what I said above [19:41:09] I'm reading codes [19:41:15] about "ores::base::config_path": /srv/ores/config that is [19:41:17] you are correct [19:41:54] It took some days of mine to finally make it configurable for both setups [19:42:05] :-) [19:42:22] (in all of scap configs, including beta it's /srv/deployment/ores/deploy) [19:42:48] so, unless I am needed for anything else, I 'll be going. Lets get https://gerrit.wikimedia.org/r/#/c/280403/ merged tomorrow finally [19:45:57] akosiaris: one thing [19:46:17] I made a patch [19:46:39] in puppet, but my internet connection is not good [19:46:46] https://gerrit.wikimedia.org/r/290740 [19:47:07] can you check this out akosiaris? I promise I won't bother you again for today [19:47:08] :D [19:47:45] merged [19:47:51] thanks [19:47:52] yess [19:47:56] :) [21:29:47] 06Revision-Scoring-As-A-Service, 10Wikimania-Hackathon-2016, 10bwds: Generate bad words for all languages more than 100K articles - https://phabricator.wikimedia.org/T134629#2328251 (10Ladsgroup) >>! In T134629#2322707, @Psychoslave wrote: > Well, I didn't know ORES, I will look at that when I have some time... [21:55:13] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: [research] Why is the japanese 'reverted' model so bad? - https://phabricator.wikimedia.org/T133405#2328382 (10Ladsgroup) Thanks @He7d3r ! [21:55:24] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: [research] Why is the japanese 'reverted' model so bad? - https://phabricator.wikimedia.org/T133405#2328384 (10Ladsgroup) a:05Ladsgroup>03None [22:20:52] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure: ores-beta is down - https://phabricator.wikimedia.org/T135677#2328519 (10Ladsgroup) With [[https://github.com/wiki-ai/ores-wikimedia-config/commit/b633f8700c2a87b720f7176e1fb1648c4a657484|b633f87]], we moved forward but still #scap3 doesn't unders... [22:20:53] 10[1] 04https://meta.wikimedia.org/wiki/https://github.com/wiki%2Dai/ores%2Dwikimedia%2Dconfig/commit/b633f8700c2a87b720f7176e1fb1648c4a657484 [22:22:28] 06Revision-Scoring-As-A-Service, 03Scap3: Scap3 checks should act based on alphabetical name of checks - https://phabricator.wikimedia.org/T136253#2328523 (10Ladsgroup)