[10:28:21] 06Revision-Scoring-As-A-Service, 10Mobile-Content-Service, 10ORES, 10RESTBase-API, and 2 others: Add ORES WP10 data to summaries? - https://phabricator.wikimedia.org/T157132#2996522 (10Tgr) [10:28:23] 06Revision-Scoring-As-A-Service, 10ORES, 06Reading-Infrastructure-Team: [Spike] Review ORES architecture for Reading Product plans - https://phabricator.wikimedia.org/T153321#3074936 (10Tgr) 05Open>03Resolved This has been scoped down to reviewing architecture needs for T157132 and T145829 specifically,... [10:28:25] 06Revision-Scoring-As-A-Service, 10Mobile-Content-Service, 10ORES, 10Reading-Web-Trending-Service, 06Services (designing): Trending API should consult ORES - https://phabricator.wikimedia.org/T145829#2642399 (10Tgr) [10:33:50] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 07Epic: Estimate ORES capex for FY2017-18 - https://phabricator.wikimedia.org/T157222#2999472 (10Tgr) >>! In T157222#3072376, @Halfak wrote: > * kubernetes servers have 16 cores and 64GB. We can fit 32 web and 32 celery workers in that. Based on that es... [12:58:09] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 07Epic: Estimate ORES capex for FY2017-18 - https://phabricator.wikimedia.org/T157222#3075280 (10akosiaris) >> On each SCB node, we use ~80GB of memory and 5% of 24 CPU cores. > scb1001 and scb1002 each have total of 32 GB memory so we definitely are not... [15:01:45] That's me deleting old branches :D [15:40:22] :D [17:49:40] halfak: re: vagrant, I have some questions related to pip and flask [17:49:58] 1) I had to modify the venv module to do editable package installs: https://gerrit.wikimedia.org/r/#/c/341294/1/puppet/modules/virtualenv/manifests/package.pp [17:50:25] I am wondering if there is a better way, or that way is wrong somehow [17:50:59] specifically, when I use it for ores, the dependencies defined by ores get installed but the dependencies defined by revscoring do not [17:51:03] 06Revision-Scoring-As-A-Service, 10rsaas-articlequality : [Discuss] Hosting the monthly article quality dataset on labsDB - https://phabricator.wikimedia.org/T146718#3076328 (10yuvipanda) CSCW is now over, and I think I can just delete the database now. [17:52:08] (this is on trusty so rather old pip and everything - maybe the problem will go away as we witch vagrant to jessie) [17:53:05] for now I have this horrible workaround: https://gerrit.wikimedia.org/r/#/c/341294/1/puppet/modules/ores/manifests/init.pp line 42-48 [17:53:48] also, pylru and redis seem to be undeclared dependencies (for revscoring and ores respectively) [17:54:58] the other thing is just to make sure I am configuring/running the services in a sane way - the readmes were a bit outdated [17:55:56] https://gerrit.wikimedia.org/r/#/c/341294/1/puppet/modules/ores/templates/upstart-ores-celery.erb [17:56:00] https://gerrit.wikimedia.org/r/#/c/341294/1/puppet/modules/ores/templates/upstart-ores-wsgi.erb [17:56:03] https://gerrit.wikimedia.org/r/#/c/341294/1/puppet/modules/ores/templates/upstart-wikilabels.erb [17:56:07] are the relevant files [17:56:50] (also is there a way to move the port configuration into the yaml files?) [18:13:41] o/ tgr [18:13:49] Sorry to step away for a minute and not say so [18:13:53] Reading scrollback [18:14:39] no rush [18:14:49] For the "editable" install, I'm not sure what's happening here. [18:14:56] are there packages with data files missing? [18:20:42] halfak: the goal is to be able to test ORES/Wikilabels patches in Vagrant so they need to be editable [18:20:59] (now that I think about it probably revscoring should be too) [18:21:30] some necessary files are also missing when installed as non-editable, e.g. the config directory [18:21:46] not sure if that's what you were asking [18:22:57] Oh! I see. This makes sense. [18:23:29] Usually we'd do something like this with sub-modules, but obviously that wouldn't make sense here. [18:23:37] I don't have a better solution than yours handy. [18:23:41] OK looking next messages. [18:24:59] Exec['pip_install_revscoring_dependencies_hack'] ? [18:25:14] yeah [18:25:18] Oh my. [18:25:24] OK. I think I see what's happening here. [18:25:26] in theory that should not be needed [18:25:37] Essentially, I think we need to have a "deploy repo" for vagrant. [18:25:43] I've been thinking about this. [18:25:53] It would have revscoring, ores and all other dependencies stated. [18:26:18] E.g. we'd have something like https://github.com/wiki-ai/ores-wmflabs-deploy [18:26:43] AIUI pip install is recursive so if I say pip install ores, ores depends on revscoring and revscoring depends on pyzt then pip should pull pyzt in [18:26:44] Note that we include the optional dependency celery in https://github.com/wiki-ai/ores-wmflabs-deploy/blob/master/requirements.txt [18:26:57] Oh! It's not even doing that!? [18:26:58] in practice that does not seem to be working right [18:27:17] to some extent it does, but the versions are wrong [18:27:24] weird! [18:27:37] What version of pip is in vagrant by default? [18:28:04] I suppose the version of python3 would be good to know too. [18:28:16] the virtualenv has pip 9.0.1 [18:28:49] the default one is 1.5.4 but unless I messed up the virtualenv puppetization that should not be used [18:29:27] Python is 3.4.3 [18:30:03] Cool. That should be OK. [18:30:08] * halfak thinks. [18:30:18] vagrant will soon switch from trusty to jessie so versions might increase but then the virtualenv already makes them mostly independent from the OS I think [18:30:40] If we switch to jessie, we can use our wheels repository to install dependencies. [18:30:53] https://github.com/wikimedia/research-ores-wheels [18:31:27] ores is not present because we import that from a submodule. [18:31:34] In our deploy [18:31:39] But this would get all of the other dependencies. [18:32:14] I don't know much about wheels, can just ice this issue until then if you think that will be easier [18:32:23] the current puppet code is ugly but does work [18:33:00] tgr, let's stick with your ugly puppet code for now [18:33:51] Can you file a to-do for cleaning it up and list out the concerns you have here? [18:34:05] (forgot to say that you are starting the services in a sane way too) :D [18:34:20] ack [18:38:35] 06Revision-Scoring-As-A-Service, 10MediaWiki-Vagrant, 10ORES, 10Wikilabels: ORES services should have vagrant roles - https://phabricator.wikimedia.org/T159105#3076618 (10Tgr) [18:42:30] 06Revision-Scoring-As-A-Service, 10MediaWiki-Vagrant, 10ORES, 10Wikilabels: ORES services should have vagrant roles - https://phabricator.wikimedia.org/T159105#3076639 (10Tgr) [18:43:23] 06Revision-Scoring-As-A-Service, 10MediaWiki-Vagrant, 10ORES, 10Wikilabels: ORES services should have vagrant roles - https://phabricator.wikimedia.org/T159105#3056869 (10Tgr) a:03Tgr [18:48:51] halfak: one more thing: when I visit e.g. http://wikilabels.local.wmftest.net:10010/campaigns/ I get an error about the campaign table not existing [18:48:58] did I miss some setup step? [18:49:08] I ran both SQLs from https://github.com/wiki-ai/wikilabels/tree/master/wikilabels/database [18:49:57] tgr, good q. Maybe it's not connecting to the right DB for the setup? [18:50:25] tgr, there's a utility for this [18:50:35] See https://github.com/wiki-ai/wikilabels/blob/master/wikilabels/utilities/load_schema.py [18:50:43] or "./utility load_schema -h" [18:50:53] It'll use the wikilabels config directly. [18:53:30] thanks, that'll be simpler [18:54:28] :) [18:54:43] If that doesn't solve the "campaigns" table issue, let me know. [20:19:19] PROBLEM - ORES web node labs ores-web-03 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:25:18] PROBLEM - ORES worker labs on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:25:44] ARG [20:27:08] RECOVERY - ORES web node labs ores-web-03 on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 457 bytes in 0.121 second response time [20:27:09] RECOVERY - ORES worker labs on ores.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 457 bytes in 0.544 second response time [21:02:56] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3077417 (10jcrespo) [21:16:54] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3077417 (10Halfak) @Catrope, @Mooeypoo, @jmatazzoni, @Ladsgroup, @tgr FYI ^ [21:22:21] TIL to not use [] or {} as a default parameter value in an object’s init method [21:22:23] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3077558 (10jcrespo) As a clarification, **yes, there are other larger tables**, and I am also complaining about them, and there are plans to... [21:44:15] \o/ I got bit by that a few years ago and did some evangelizing. [21:44:22] Sorry I didn't talk to you about it Nettrom [21:57:20] 10Revision-Scoring-As-A-Service-Backlog, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017): Recent changes page: ORES scored entries are not highlighted when 'Enhanced recent changes' option is used - https://phabricator.wikimedia.org/T146291#3077689 (10jm... [21:58:45] I'm curious what breaks when you do that? [22:01:20] Does the object instance not own its variable or something? [22:02:53] yep, I see. Thanks for the tip! [22:04:01] oh hey, same deal if you pass a list or dict literal to a normal class method as well. [22:04:02] halfak: no worries, it was easy to find SO posts about it and understand why I happened [22:06:16] 10Revision-Scoring-As-A-Service-Backlog, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017): Recent changes page: ORES scored entries are not highlighted when 'Enhanced recent changes' option is used - https://phabricator.wikimedia.org/T146291#3077710 (10jm... [22:07:16] halfak: o/ were you saying last week that the glaciers of annual planning might be shifting again? [22:13:51] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Concerns about ores_classification table size on enwiki - https://phabricator.wikimedia.org/T159753#3077780 (10Tgr) There are two way that table can grow: by automatically processing recent changes (which should be largely linear over time)... [22:30:58] 06Revision-Scoring-As-A-Service, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017): Add feature flag to enable parts of ORES extension by default - https://phabricator.wikimedia.org/T159763#3077869 (10Catrope) [22:42:39] 06Revision-Scoring-As-A-Service, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017): Enable the ORES good faith and damaging UI by default, on wikis that have these ORES models available (instead of behind ... - https://phabricator.wikimedia.org/T158225#3077929 [22:57:40] 10Revision-Scoring-As-A-Service-Backlog, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017): Recent changes page: ORES scored entries are not highlighted when 'Enhanced recent changes' option is used - https://phabricator.wikimedia.org/T146291#3077974 (10jm... [23:08:03] o/ awight [23:08:10] Sorry to miss your ping. Still around? [23:15:49] hey! [23:16:14] Under a small pile of meetings, but everything went well [23:16:51] halfak: happy to chat any time [23:17:09] We're in the annual planning phase where finance reviews funding proposals. [23:17:45] Basically, the next step is finance producing a report about how terrible our over-budgeting is so that we can go back to the drawing board. I think that'll happen in a week. [23:18:04] oho perfect [23:19:22] A battle of subjectivities [23:20:51] \o/ [23:20:55] What a poetic way to put it.