[00:10:09] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Deploy ores precaching in production - https://phabricator.wikimedia.org/T137370#2366612 (10yuvipanda) A couple of questions:  1. When the MW Extension gets deployed, from my understanding the job queue will hit ORES for each revision. Is that accurate? If so, is that g...
[00:13:23] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Deploy ores precaching in production - https://phabricator.wikimedia.org/T137370#2366332 (10Legoktm) >>! In T137370#2366612, @yuvipanda wrote: > 1. When the MW Extension gets deployed, from my understanding the job queue will hit ORES for each revision. Is that accurate...
[00:14:35] <legoktm>	  scb100[12]
[00:14:35] <AsimovBot>	 El búfer 12 está vacío.
[00:14:39] * legoktm pets AsimovBot
[00:15:32] <legoktm>	 https://tools.wmflabs.org/bash/quip/AVUyg4Bm1EdYBLP_78xA
[04:02:48] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Deploy ores precaching in production - https://phabricator.wikimedia.org/T137370#2366916 (10Ladsgroup) >>! In T137370#2366617, @yuvipanda wrote: > @legoktm do you know the the expected latency between the edit being saved > and the job being run?  They made no latency p...
[04:14:04] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Deploy ores precaching in production - https://phabricator.wikimedia.org/T137370#2366952 (10Ladsgroup) >>! In T137370#2366612, @yuvipanda wrote: > A couple of questions: >  > 1. When the MW Extension gets deployed, from my understanding the job queue will hit ORES for e...
[08:01:25] <Amir1>	 akosiaris: hey, do you have some time to checkout some puppet patches?
[08:01:47] <Amir1>	 it would be awesome, I'm finishing setting up a grafana dashboard, etc.
[08:24:23] <akosiaris>	 Amir1: I am around now
[08:24:27] <akosiaris>	 what can I help with ?
[08:24:58] <Amir1>	 akosiaris: https://gerrit.wikimedia.org/r/293429
[08:25:07] <Amir1>	 https://gerrit.wikimedia.org/r/293442
[08:25:10] <Amir1>	 these two
[08:25:13] <Amir1>	 thanks :)
[08:25:51] <Amir1>	 Also, if you have about ten minutes, I can make the patch for syslink and deploy it to fix the race condition today
[08:26:18] <akosiaris>	 syslink ?
[08:26:25] <akosiaris>	 not sure I follow...
[08:27:18] <Amir1>	 https://gerrit.wikimedia.org/r/292516
[08:27:35] <Amir1>	 akosiaris: we need to deploy a change in ores-config before merging this one
[08:27:58] <Amir1>	 so i.e. I need to know when this will gets merged
[08:29:10] <akosiaris>	 I can merge this now if you can merge the code that makes ores lookup that file
[08:29:16] <akosiaris>	 well, more like dir I 'd say
[08:29:30] <akosiaris>	 so, ores now will supports /etc/ores as well ? 
[08:29:42] <akosiaris>	 aside from the config/ dir I mean
[08:30:34] <Amir1>	 akosiaris: not yet, but fixing it would take about ten minutes
[08:30:48] <Amir1>	 I wanted to be sure of this and then make the patch
[08:31:40] <akosiaris>	 this looks fine to me
[08:31:48] <Amir1>	 awesome
[08:31:54] <Amir1>	 let me fix the graphite patch
[08:35:06] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-articlequality : [Explore] Spam and Vandalism new page creation - https://phabricator.wikimedia.org/T135644#2367226 (10Soni) Copying over content from my personal documentation....   Overall plan -      - Build an ML model based on ORES to go through new page creations...
[08:36:03] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-articlequality : [Explore] Spam and Vandalism new page creation - https://phabricator.wikimedia.org/T135644#2367227 (10Soni) Current Progress -  - We will be querying the following tables on enwiki_p database - logging (for deletions), page, revision (for good dataset)...
[08:52:08] <Amir1>	 akosiaris: I amended it https://gerrit.wikimedia.org/r/#/c/293442/
[08:52:15] <Amir1>	 thanks
[09:22:36] <Amir1>	 akosiaris: ready to merge: https://github.com/wiki-ai/ores-wikimedia-config/pull/62
[09:22:45] <Amir1>	 tell me when do you have time to merge these two
[09:25:33] <Amir1>	 afk for ten min
[09:53:11] <Amir1>	 back
[10:25:45] <Amir1>	 akosiaris: I added hiera for this too: https://gerrit.wikimedia.org/r/#/c/293442/
[11:14:48] <Amir1>	 akosiaris: It's mentioned graphite1001 in wikitech help page: https://wikitech.wikimedia.org/wiki/Graphite#Service
[11:15:03] <Amir1>	 should I change the manual or I misunderstood anything
[11:16:55] <akosiaris>	 Amir1: Well, where the graphite server is hosted is not very related to what the inbound endpoint should be. 
[11:17:12] <akosiaris>	 but I 'll add a paragraph in that wikitech page explaining it and why it's like that
[11:17:32] <Amir1>	 okay
[11:19:42] <Amir1>	 akosiaris: I amended the patch now 
[11:22:44] <Amir1>	 akosiaris: and about using sysink vs. multiple config dirs, I have another PR ready for using multiple directories, but in order to merge that I need to talk to halfak (which he probably asleep right now) and it might take a while. but per what I know about our earlier discussion, we agreed on dangling syslink as a short-term solution and multiple
[11:22:44] <Amir1>	 directories as the long-term solution. Do you think we should straight go the long-term solution?
[11:24:17] <akosiaris>	 well, how much time will it take to get the long term merged ?
[11:24:44] <akosiaris>	 for me short term in that context meant "hours/days".. not much more btw 
[11:25:18] <Amir1>	 what about a week?
[11:25:51] <Amir1>	 akosiaris: ^
[11:26:32] <akosiaris>	 tops
[11:26:45] <Amir1>	 sure
[11:29:01] <Amir1>	 the graphite is ready btw
[11:29:32] <Amir1>	 awesome thanks :)
[12:22:43] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Create grafana dashboard for ores.wikimedia.org - https://phabricator.wikimedia.org/T137367#2367599 (10Ladsgroup) https://grafana-admin.wikimedia.org/dashboard/db/ores  Still needs some love (and I ran a precaching script so we can get some data to...
[12:39:12] <Amir1>	 XMLHttpRequest cannot load https://ores.wikimedia.org/scores/fawiki/. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://fa.wikipedia.org' is therefore not allowed access.
[12:39:20] <Amir1>	 CORS is not enabled for prod
[12:39:25] <Amir1>	 :((((
[12:39:35] <Amir1>	 I will try to find a solution ASAP
[12:39:45] <Amir1>	 but right now I need to pack up
[12:39:49] <Amir1>	 afk for hours
[13:39:48] <halfak>	 o/ Amir1 
[13:39:59] <halfak>	 Probably missed him
[13:49:12] <halfak>	 akosiaris, thoughts on https://github.com/wiki-ai/ores-wikimedia-config/pull/63 ?
[13:49:36] <halfak>	 Re. configs coming from /etc/ores/*.yaml
[13:54:00] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Deploy ores precaching in production - https://phabricator.wikimedia.org/T137370#2367838 (10Halfak) >> How destructive is running two precached daemons? Utterly unacceptable / minor annoyance? >It would hit the API twice which one of scores goes to vain, so basically ha...
[14:09:21] <akosiaris>	 halfak: commented there. LGTM
[14:10:00] <halfak>	 Thanks.  how badly are we blocked on this right now?
[14:10:09] <halfak>	 I'd like to wait until amir1 gets back to comment
[14:10:15] <halfak>	 But not if we're held up
[14:10:42] <akosiaris>	 well, not much. just avoid deploys and we are ok
[14:10:49] <halfak>	 OK will do
[14:14:36] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Create grafana dashboard for ores.wikimedia.org - https://phabricator.wikimedia.org/T137367#2367858 (10Halfak) I got the server CPU and memory stats working.  https://grafana-admin.wikimedia.org/dashboard/db/ores
[14:17:49] <halfak>	 Weird.  I just randomly found ores-compute-01 offline
[15:14:23] <Amir1>	 halfak: I'm back
[15:14:33] <Amir1>	 ores-compute-01 is in shutoff mode
[15:14:39] <Amir1>	 I don't who did that
[15:54:08] <Amir1>	 akosiaris: one important thing: CORS is disabled for ores.wikimedia.org and we need for gadgets that uses ores. 1- Is it okay to enable CORS *. ORES is a read-only service and all data is public 2- If the answer is yes, Is it okay to do it in service::uwsgi or I should go deeper and do it in flask or app? doing it via service::uwsgi would be much easier for
[15:54:08] <Amir1>	 me
[16:03:57] <halfak>	 o/
[16:04:01] <halfak>	 Just got out of meeting
[16:04:02] <Amir1>	 halfak: o/
[16:04:27] <Amir1>	 halfak: 1- as you saw, grafana dashboard is up \o/ thanks for fixing it
[16:04:27] <halfak>	 +1 for CORS in puppet
[16:04:33] <halfak>	 No prob
[16:04:42] <halfak>	 You can see my running performance tests now. 
[16:04:57] <halfak>	 I got a 500 a little bit ago.  Maybe a reboot of the service?
[16:05:35] <schana>	 halfak: are your performance tests set up in a way where they could be run as part of a test suite?
[16:05:39] <Amir1>	 not so sure
[16:05:41] <Amir1>	 let me check
[16:06:07] <Amir1>	 halfak: 2- I enabled using ores in prod for fawiki users of ScoredRevisions (it's a gadget there)
[16:06:22] <Amir1>	 but I got error, so we nee to fix CORS
[16:06:30] <akosiaris>	 Amir1: I think it's fine to enable CORS in ores. I am not sure what you mean by fixing it in puppet
[16:07:20] <halfak>	 in service::uwsgi, we can configure for CORS
[16:07:25] * halfak digs for wmflabs example
[16:08:02] <Amir1>	 halfak: we had uwsgi::app before now we need to use another thing
[16:08:12] <Amir1>	 let me do some searches
[16:08:21] <halfak>	 Oh yeah
[16:08:47] <Amir1>	 akosiaris: I mean fix it by adding a parameter to the service::uwsgi definition in ores::web
[16:08:48] <akosiaris>	 lemme get this straight, you will be relying on uwsgi to set the CORS headers? not ORES itself ?
[16:09:00] <halfak>	 akosiaris, yes. 
[16:09:07] <halfak>	 The uwsgi config for ores-web
[16:09:33] <halfak>	 akosiaris, if you feel strongly, we could have ORES itself do that
[16:09:38] <halfak>	 But that didn't seem right to me. 
[16:09:46] <akosiaris>	 uwsgi::service has a config parameter that accepts a hash as a parameter
[16:09:53] <Amir1>	 we need to add some dependencies 
[16:10:04] <Amir1>	 akosiaris: yeah we did that for logformat
[16:10:07] <akosiaris>	 which is more or less passed to uwsgi::app as is
[16:10:11] <akosiaris>	 will that be sufficient ?
[16:10:15] <Amir1>	 yup
[16:10:26] <akosiaris>	 sounds fine by me then
[16:10:28] <Amir1>	 akosiaris: do you want to make the patch or I can do it and you review
[16:10:35] <akosiaris>	 I 'll review
[16:10:38] <Amir1>	 sure
[16:14:11] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Move CORS to uwsgi - https://phabricator.wikimedia.org/T137433#2368269 (10Ladsgroup)
[16:20:46] <Amir1>	 akosiaris: https://gerrit.wikimedia.org/r/293521
[16:20:50] <Amir1>	 afk for dinner
[16:20:59] <Amir1>	 be back in ten min.
[16:23:07] <akosiaris>	 ts ts, it's dinner, take your time
[16:33:10] <Amir1>	 akosiaris: I don't think uwsgi support that
[16:33:20] <Amir1>	 http://uwsgi-docs.readthedocs.io/en/latest/InternalRouting.html
[16:34:00] <Amir1>	 http://uwsgi-docs.readthedocs.io/en/latest/Options.html
[16:34:02] <akosiaris>	 Amir1: http://uwsgi-docs.readthedocs.io/en/latest/Options.html#add-header ?
[16:34:23] <Amir1>	 okay
[16:34:25] <akosiaris>	 and I 've checked on scb1001 that the option is indeed supported
[16:34:30] <Amir1>	 super strange
[16:34:32] <akosiaris>	 on the version we got
[16:36:01] <Amir1>	 thanks for pointing out, done now, akosiaris :)
[16:37:39] <Amir1>	 halfak: one thing: We need to determine the exact steps before making an announcement and deploying the extension. For example 1-enable cors, 2- setup grafana, etc.
[16:38:44] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: Announce the deployment of ORES to fawiki - https://phabricator.wikimedia.org/T137436#2368346 (10Halfak)
[16:38:44] <halfak>	 https://phabricator.wikimedia.org/T137436
[16:39:19] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Move CORS to uwsgi - https://phabricator.wikimedia.org/T137433#2368360 (10Halfak)
[16:39:21] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: Announce the deployment of ORES to fawiki - https://phabricator.wikimedia.org/T137436#2368359 (10Halfak)
[16:39:48] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: Announce the deployment of ORES to fawiki - https://phabricator.wikimedia.org/T137436#2368346 (10Halfak)
[16:39:50] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Create grafana dashboard for ores.wikimedia.org - https://phabricator.wikimedia.org/T137367#2366245 (10Halfak)
[16:40:05] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Load test ores.wikimedia.org - https://phabricator.wikimedia.org/T137365#2368368 (10Halfak)
[16:40:07] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: Announce the deployment of ORES to fawiki - https://phabricator.wikimedia.org/T137436#2368346 (10Halfak)
[16:40:37] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: Announce the first deployment of the ORES extension (fawiki) - https://phabricator.wikimedia.org/T137436#2368346 (10Halfak)
[16:41:07] <halfak>	 Where should we announce for fawiki?
[16:41:14] <Amir1>	 I do that for fawiki
[16:41:14] <halfak>	 Is there a village pump?
[16:41:22] <Amir1>	 yeah there is
[16:41:26] <Amir1>	 don't worry about it
[16:41:38] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: Announce the first deployment of the ORES extension (fawiki) - https://phabricator.wikimedia.org/T137436#2368371 (10Halfak)
[16:41:39] <Amir1>	 but I think we should announce it in wikitech-l 
[16:42:00] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: Announce the first deployment of the ORES extension (fawiki) - https://phabricator.wikimedia.org/T137436#2368346 (10Halfak)
[16:42:34] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Enable ORES extension as a beta feature in fawiki - https://phabricator.wikimedia.org/T137437#2368373 (10Halfak)
[16:42:43] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Enable ORES extension as a beta feature in fawiki - https://phabricator.wikimedia.org/T137437#2368387 (10Halfak)
[16:42:46] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES, 10ORES: Announce the first deployment of the ORES extension (fawiki) - https://phabricator.wikimedia.org/T137436#2368346 (10Halfak)
[16:42:55] <halfak>	 OK.  I think I've got this task spec'd https://phabricator.wikimedia.org/T137436
[16:43:08] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10Wikidata, 05WMDE-Tech-Communication-Mentoring-And-Events: Wikidata item quality classification - https://phabricator.wikimedia.org/T127470#2368388 (10Sumit) a:03Sumit I've created a very basic page at [[https://meta.wikimedia.org/wiki/Research:Wikidata_Item_Qualit...
[16:43:40] <akosiaris>	 the CORS changed has been merged but is not live yet. Problems with changeprop. I 'll let you know when it will go live
[16:44:44] <halfak>	 thanks akosiaris 
[16:44:48] <Amir1>	 thanks
[16:49:03] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10Wikidata, 05WMDE-Tech-Communication-Mentoring-And-Events: Wikidata item quality classification - https://phabricator.wikimedia.org/T127470#2368434 (10Lydia_Pintscher) Awesome. Let's have a call to figure out the best way to get you input from the editors. http://doo...
[16:51:14] <Amir1>	 halfak: https://grafana.wikimedia.org/dashboard/db/ores
[16:51:24] <Amir1>	 check scores errored :D
[16:51:55] <Amir1>	 were you trying to run a "knockover the service test"?
[16:52:19] <halfak>	 Not knock it over.  More replicate a common load pattern. 
[16:52:28] <halfak>	 I could up the request rate >:) 
[16:52:56] <Amir1>	 :D
[16:52:57] <halfak>	 I wonder what those errors were
[16:54:13] <Amir1>	 it wasn't overloading error
[16:54:21] <Amir1>	 otherwise it would end up in time outs
[16:54:38] <Amir1>	 and I checked grafana admin, we hadn't get a timed out score yet
[17:08:25] <halfak>	 We should ensure that this logging is working right. 
[17:10:23] <halfak>	 We should add a timeout statsd event here: https://github.com/wiki-ai/ores/blob/master/ores/score_processors/score_processor.py#L96
[17:12:28] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Metrics for TimeoutError - https://phabricator.wikimedia.org/T137442#2368536 (10Halfak)
[17:12:32] <halfak>	 https://phabricator.wikimedia.org/T137442
[17:17:38] <Amir1>	 nice
[17:17:49] <Amir1>	 halfak: I need to travel to the airport
[17:18:00] <Amir1>	 be back in a few hours until morning 
[17:18:20] <Amir1>	 if there is anything I should do, I have about ten minutes halfak 
[17:36:08] <halfak>	 Sorry.  Was just finishing up a meeting
[17:36:20] <halfak>	 I hope you have a good trip, Amir1  :) 
[17:36:47] <akosiaris>	 access-control-allow-origin:*
[17:36:51] <akosiaris>	 :-)
[17:36:53] <akosiaris>	 ok done
[17:37:24] <akosiaris>	 CORS is enabled in production
[17:50:04] <Amir1>	 halfak: thanks
[17:50:10] <Amir1>	 akosiaris: thanks yay
[17:50:40] <Amir1>	 https://grafana.wikimedia.org/dashboard/db/ores
[17:51:01] <Amir1>	 since the scoredrevisions in fawiki uses ores.wm.o now
[17:51:15] <Amir1>	 human usage of ores got up!
[17:52:01] <Amir1>	 https://fa.wikipedia.org/wiki/%D9%88%DB%8C%DA%98%D9%87:GadgetUsage
[17:52:19] <Amir1>	 scoredrevisions is being used by 64 users now (34 active users)
[17:53:06] <akosiaris>	 \o/
[19:15:51] <yuvipanda>	 halfak: do we have static dumps of results from the quality thingy?
[19:15:56] <yuvipanda>	 s/thingy/model/
[19:16:04] <halfak>	 Article quality?
[19:16:07] <yuvipanda>	 yes!
[19:16:22] <yuvipanda>	 (am at a conference for distributed web, so wondering)
[19:26:12] <halfak>	 yuvipanda, I have one that I could wrap up quickly from last year
[19:26:25] <halfak>	 I'm planning to make a new one and start generating them regularly next quarter. 
[19:26:42] <halfak>	 Load testing complete!  
[19:26:46] <halfak>	 Now to write a report
[19:26:57] <halfak>	 yuvipanda, ^ re. article quality stuff
[19:50:35] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Load test ores.wikimedia.org - https://phabricator.wikimedia.org/T137365#2369297 (10Halfak) https://commons.wikimedia.org/wiki/File:ORES_performance_in_grafana.png  So, I ran the following in parallel to generate a rate of requests that roughly mimics what we have in la...
[20:04:36] <yuvipanda>	 halfak: ok! +1 for autogenerating them over time etc :D
[20:30:17] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Move labs parts in GUI to its own loader - https://phabricator.wikimedia.org/T137362#2369470 (10Halfak) Looks like we are pulling assets from https://wikimedia-ui.wmflabs.org/MW/mediawiki.min.css and https://munmay.github.io/WikimediaUI/MW/elements.html  Seems like both...
[20:30:53] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ORES: Move labs parts in GUI to its own loader - https://phabricator.wikimedia.org/T137362#2369500 (10Halfak) Forgot to say, but of course we can just download them and make them assets of ORES.  That would be sad, but not that sad.
[23:56:40] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10Wikimania-Hackathon-2016: A training session on supporting ores in more languages in Wikimania Hackathon - https://phabricator.wikimedia.org/T134628#2370287 (10Rfarrand) Go ahead and schedule this is you like @Ladsgroup ! https://wikimania2016.wikimedia.org/wiki/Hacka...