[15:02:28] <halfak>	 o/ Amir1 
[15:02:30] <halfak>	 How's it going? 
[15:02:42] * halfak is working from bd808's living room
[15:02:43] <halfak>	 :) 
[15:09:15] <Amir1>	 halfak: hey, I just come back from a break
[15:09:28] <Amir1>	 we had a huge presentation about ORES today
[15:09:50] <Amir1>	 it's a long story I prefer to say it than to write it
[15:09:57] <Amir1>	 let's do it in the staff
[15:10:02] <halfak>	 OK sounds good. 
[15:10:08] <halfak>	 I'm very curious about that :) 
[15:51:54] <halfak>	 \o/
[16:54:54] <awight>	 halfak_: you’re frozen
[17:28:17] <awight>	 halfak: When do you want to chat about your Celery tweaks?
[17:28:42] <awight>	 &, I think I’ll skip Scrum of Scrums today due to lack of things to report, and time.
[17:29:05] <halfak>	 Oh yes.  One minute. 
[17:30:10] <awight>	 wow—this was all?  “disable_sync_subtasks=False”
[17:31:05] <halfak>	 Right -- mostly yeah :) 
[17:32:40] <awight>	 The documentation for that param is pretty much unparseable.
[17:33:02] <awight>	 Default value is true, yet “Disable tasks to wait for sub tasks this is the default configuration. CAUTION do not enable this unless you must."
[17:33:40] <awight>	 O_o
[17:35:05] <awight>	 ah.  so by “enable” they mean, set the double negative.
[17:35:13] <halfak>	 right :) 
[17:35:27] <halfak>	 Looks like it should be easy to test.
[17:35:56] <awight>	 I think I agree with your plan here: leave our sync subtasks in “inadvisable” mode, but benefit from new Celery poll() fix
[17:37:06] <halfak>	 Awesome :) 
[17:38:25] <halfak>	 Been thinking more about advisable strategies that we can test out too. 
[17:38:41] <awight>	 There should be no reason we have to do this wait subtask thing
[17:38:44] <halfak>	 check out https://www.mediawiki.org/wiki/ORES/Multiscore_problem
[17:38:57] <awight>	 But let’s leave it for a separate investigation.
[17:39:07] <halfak>	 Trying to get good at talking about it so we can get good at talking about fixing it. 
[17:39:09] <awight>	 :D
[17:39:19] <awight>	 Thanks for illustrating it, that helps me a lot
[17:42:26] <awight>	 Perhaps the solution is to flatten the subtasks up front?
[17:43:47] <halfak>	 Right, I have more to write on that regarding de-duping related requests. 
[17:43:57] <halfak>	 The "D" part of that graph is super common. 
[17:44:24] <halfak>	 One option we have is to *always* score for every model when a rev_id is requested. 
[17:44:45] <halfak>	 That would work out.  It would tie our hands for what every model can do and how long evaluation takes place. 
[17:45:54] <awight>	 Why would scoring evey model help?
[17:47:35] <awight>	 I’m guessing that’s to help track and dedupe unique jobs, but I must be missing something, cos we can just identify jobs as (rev_id, model) which I think we already do?
[17:55:02] <awight>	 halfak: So, how to proceed?  Shall I fire this up and mess with it locally, or are we ready to deploy to ores.wmflabs?
[17:55:55] <halfak>	 awight, do you think a local test can help us figure out if we'll crash on the ORES cluster?
[17:56:10] <halfak>	 I think we might need to do another stress test with this version of ORES. 
[17:56:10] <awight>	 perhaps.  Lemme start with that.
[17:56:13] <halfak>	 kk
[17:56:22] * halfak finishes up blog post. 
[17:56:29] <awight>	 I’ll double-check that I can crash the old code first :)
[17:56:43] <halfak>	 https://phabricator.wikimedia.org/phame/post/view/77/status_update_october_6_2017/ 
[17:56:53] <halfak>	 I took some liberties with merging sections and adding some discussion
[17:58:03] <travis-ci>	 wiki-ai/revscoring#1262 (pytest - 8d15b20 : Amir Sarabadani): The build failed. https://travis-ci.org/wiki-ai/revscoring/builds/289621277
[18:04:08] <Amir1>	 halfak: nice job on the status update, thank you
[18:04:20] <halfak>	 awight, ^ 
[18:04:22] <halfak>	 :) 
[18:04:22] <awight>	 I was just about to say, LGTM, especially the new sections
[18:04:37] <awight>	 Large file support could use a supporting sentence, if you want to add that.
[18:04:40] <halfak>	 Great!  One thing I really like about using the blog is that we can make edits as necessary. 
[18:04:49] <halfak>	 Much easier than giant emails :D 
[18:04:55] <awight>	 ++
[18:51:30] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2017): UX check RC Filters in beta (revscoring 2.0/thresholds release) - https://phabricator.wikimedia.org/T178395#3694788 (10Catrope)
[18:51:41] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2017): UX check RC Filters in beta (revscoring 2.0/thresholds release) - https://phabricator.wikimedia.org/T178395#3690748 (10Catrope) a:03Etonkovidova
[18:55:58] <awight>	 File "/vagrant/srv/ores/lib/python3.4/site-packages/billiard/connection.py", line 571, in Pipe
[18:55:59] <awight>	 fd1, fd2 = os.pipe()
[18:55:59] <awight>	 OSError: [Errno 24] Too many open files
[18:56:12] <awight>	 So… different stack trace than before
[18:56:30] <awight>	 I’ll see if there’s an easy limit to raise
[19:12:02] <wikibugs>	 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10artificial-intelligence: Add language support for Icelandic - https://phabricator.wikimedia.org/T178524#3694816 (10Snaevar)
[19:17:03] <halfak>	 awight, oh!  We'll need to update the wheels -- unless you've done that manually. 
[19:22:57] <awight>	 On vagrant, that’s manual
[19:23:10] <awight>	 I’ll do the wheels before the labs test, though.
[19:39:50] <halfak>	 o/ quiddity 
[19:39:57] <halfak>	 Thanks for your edit to the blog post :) 
[19:40:23] * halfak plays with using JSON web tokens to allow MediaWiki to safely talk directly to JADE 
[19:40:39] <quiddity>	 :)  Good luck with your presentation.
[19:43:31] <awight>	 halfak: well, I got it to work at least, and it crashes in a promisingly different way
[19:43:44] <halfak>	 thanks quiddity :D 
[19:43:51] <halfak>	 \o/  awight 
[19:44:13] <awight>	 however, I’m blocked from further vagrant testing.  There’s a bizarrely low limit on no_files, which I can’t seem to override.
[19:44:25] <awight>	 So I’m thinking of moving on to stress testing on labs.
[19:49:04] <halfak>	 +1
[19:49:08] <halfak>	 wait...on labs?
[19:49:14] <halfak>	 what does that mean?
[19:49:16] <awight>	 hmm
[19:49:30] <awight>	 I was going to try this on ores.wmflabs.org
[19:49:39] <halfak>	 awight, I wouldn't use that. 
[19:49:51] <halfak>	 Experimentation for our users -- not us ;) 
[19:49:56] <awight>	 k.  lol /me consults FAQ
[19:50:28] <awight>	 ores-staging.wmflabs, then?
[19:51:22] <awight>	 I suppose beta just makes me nervous cos then we have experimental code merged to master.
[19:52:07] <awight>	 But come to think of it, the code is already merged.  this is just a wheels update and a deploy.
[20:00:29] <halfak>	 Right.  I think this should go to the new cluster. 
[20:00:43] <halfak>	 ores-staging would be OK, but it's not really intended for this.  ores-misc would be the best place in labs. 
[20:00:56] <awight>	 new cluster.  right thx
[20:04:04] <awight>	 Can you remind me how to set the HTTPS proxy so I’m able to clone from git-ssh.wmo from labs?
[20:06:24] <halfak>	 use phab
[20:06:43] <halfak>	 Or wait... not sure I've done this before. 
[20:07:02] <halfak>	 Why do you need a proxy? 
[20:09:22] <awight>	 Maybe I don’t.  This is actually a permissions denied thing rather than timeout connecting.
[20:09:39] <awight>	 har.  ok so I have to edit .gitmodules?
[20:09:50] * awight curses own notes for not having these steps
[20:10:24] <awight>	 I’ll just build the wheels in vagrant, NBD
[20:32:01] <awight>	 halfak: nvm the above, I hadn’t uploaded my ores-misc-01 public key into Phabricator.  Now git is able to pull the submodules.
[20:32:10] <halfak>	 Nice
[20:33:33] <halfak>	 https://www.mediawiki.org/w/index.php?title=Topic:U0080oecglhdiqry&topic_showPostId=u09gqla48p6jjtf8#flow-post-u09gqla48p6jjtf8
[20:35:49] <awight>	 That’s a little scary.  So we’re performing wiki actions on behalf of the user, but skipping OAuth by performing all API calls from the browser...
[20:44:16] <awight|brb>	 Now that my git submodules work on ores-misc-01: fatal: write error: No space left on deviceiB | 14.68 MiB/s     
[20:44:18] <awight|brb>	 lol
[20:44:59] * awight|brb ruthlessly rm’s editquality repo
[20:50:14] <awight>	 halfak: Question about rebuilding wheels.  Are you using ores/requirements.txt as the source for -r ?
[20:50:26] <halfak>	 awight, yup
[20:50:33] <awight>	 ty
[21:15:11] <awight>	 halfak: Any tips on how to push submodules from ores-misc for review?
[21:15:40] <awight>	 I guess I add the gerrit+ssh remote and add the key to my gerrit prefs?
[21:15:57] <halfak>	 Oh... are you working from prod config on ores-misc? 
[21:16:01] <halfak>	 Maybe you should try labs config ;) 
[21:16:25] <halfak>	 Or is it wmflabs config but the wheels repo?
[21:16:40] <awight>	 I’m in the prod repo
[21:17:19] <awight>	 I don’t think it would be easy to deploy the labs repo to beta, but I’m unclear on what you’re suggesting
[21:22:30] <awight>	 halfak: What exactly are you suggesting?
[21:22:42] <halfak>	 beta?
[21:22:51] <awight>	 yes
[21:22:54] <halfak>	 Aren't you working on ores-meta?
[21:22:56] <awight>	 or
[21:22:58] <awight>	 urgh
[21:23:02] <halfak>	 We should not be deploying this to beta
[21:23:09] <awight>	 ok
[21:23:16] <awight>	 *sigh*
[21:23:23] <awight>	 I’m really wasting a lot of time here
[21:23:31] <halfak>	 Or we could deploy to the ORES* cluster
[21:23:53] <awight>	 I was planning to deploy to the new cluster
[21:24:03] <halfak>	 Oh yes.  I think that *is* a good idea. 
[21:24:06] <awight>	 so need to put the changes into the prod-deploy repo
[21:24:27] <awight>	 git submodules as usual are screwing me up
[21:27:29] <awight>	 I thought there was some business about changing submodule remotes so I can submit to gerrit, from an ores-deploy checkout?
[21:41:37] <halfak>	 awight, I usually don't work from the submodules. 
[21:42:38] <awight>	 I… don’t think I am
[21:48:02] <wikibugs>	 (03PS1) 10Awight: Update to Celery 4.x [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/385102
[21:48:40] <awight>	 aha, I see what you mean.
[21:49:22] <awight>	 yeah I’m working from the mediawiki/services/ores/deploy repo, but was doing -w submodules/wheels rather than using a separate checkout
[21:56:58] <halfak>	 awight, looks like it is missing some of the sub-dependencies from wikiclass and draftquality
[21:57:13] <halfak>	 check out the Makefile line for frozen-requirments.txt
[21:57:20] <halfak>	 I'll submit a PR to update the file. 
[21:57:26] <halfak>	 And then you can work from that. 
[21:59:11] <halfak>	 awight, https://github.com/wiki-ai/ores-wmflabs-deploy/pull/90
[21:59:38] <halfak>	 Check that out.  You can just copy-paste that frozen-requirements.txt file and do that. 
[21:59:51] <halfak>	 Note how a bunch of wheels are excluded. 
[22:02:02] <awight>	 <scratching head>
[22:03:09] <awight>	 Give me an example of an excluded wheel?
[22:04:10] <awight>	 I’m thinking I can get the same effect by doing pip wheel -w submodules/wheels -r editquality/requirements.txt -r wikiclass/requirements.txt etc...
[22:05:04] <awight>	 nope, that doesn’t work.
[22:07:57] <awight>	 oof, pip freeze | grep -v setuptools > requirements-frozen.txt
[22:12:38] <wikibugs>	 (03PS2) 10Awight: Update to Celery 4.x [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/385102