[00:20:57] (03PS1) 10: Use ores.wikimedia.org instead of ores.wmflabs.org [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292508 [00:21:42] (03CR) 10: [C: 04-2] "Not yet. Tomorrow or the day after." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292508 [00:40:27] (03PS1) 10: Remove a done TODO [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292510 [00:40:53] (03CR) 10: [C: 032] "super trivial." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292510 [00:41:58] (03Merged) 10jenkins-bot: Remove a done TODO [extensions/ORES] - 10https://gerrit.wikimedia.org/r/292510 [00:42:43] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Improve mwext-ORES integration with RecentChanges - https://phabricator.wikimedia.org/T120183#2351788 (10Ladsgroup) >This is a follow-on iteration of T112856's Recent Changes hooks, starting with these known bugs: >* The "R" flag should hid... [00:43:09] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Improve mwext-ORES integration with RecentChanges - https://phabricator.wikimedia.org/T120183#2351789 (10Ladsgroup) 05Open>03Resolved [00:44:07] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Add description when a user reviews an edit - https://phabricator.wikimedia.org/T136903#2351790 (10Ladsgroup) [01:50:36] 06Revision-Scoring-As-A-Service, 10rsaas-articlequality : Article quality models for Russian Wikipedia - https://phabricator.wikimedia.org/T131635#2351887 (10Ladsgroup) [06:30:25] 06Revision-Scoring-As-A-Service, 10bwds, 10revscoring: Language assets for Norwegian - https://phabricator.wikimedia.org/T131855#2352120 (10Ladsgroup) https://github.com/wiki-ai/revscoring/pull/278 [06:33:09] wiki-ai/revscoring#735 (nowiki - 4d1666b : Amir Sarabadani): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/134947915 [07:45:48] 06Revision-Scoring-As-A-Service, 03Scap3: Scap3 checks should not be random - https://phabricator.wikimedia.org/T136253#2352251 (10Ladsgroup) Needs review https://phabricator.wikimedia.org/D249 [08:08:17] akosiaris: hey, 1- if you are free please review https://gerrit.wikimedia.org/r/#/c/292517/ and https://gerrit.wikimedia.org/r/#/c/292516/ 2- I talked with releng, they told me that scap3 keeps the last 5 revs. https://github.com/wikimedia/scap/blob/master/scap/context.py#L14 so I'm not sure if we would run into storage issues now. [08:56:51] 06Revision-Scoring-As-A-Service, 06Operations, 06Research-and-Data-Backlog, 10Research-management, and 3 others: [Epic] Deploy Revscoring/ORES service in Prod - https://phabricator.wikimedia.org/T106867#2352395 (10akosiaris) [08:56:53] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Setup LVS for ORES - https://phabricator.wikimedia.org/T124202#2352393 (10akosiaris) 05Open>03Resolved ORES LVS works fine in both DCs, hence resolving [09:01:08] yes, yes, yes [09:09:02] 06Revision-Scoring-As-A-Service, 10ORES: Setup varnish endpoint for ORES - https://phabricator.wikimedia.org/T124203#2352409 (10Ladsgroup) [09:09:50] akosiaris: hey, I didn't assign this to you, since you might not want to do it but I guess that's the last step: https://phabricator.wikimedia.org/T124203 [09:14:22] afk for a while [09:35:24] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 10Research-management, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2352473 (10jcrespo) 05Resolved>03Open p:05High>03Normal a:05Halfak>03jcrespo This has been approved, but not yet executed. Blocked... [10:52:37] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Setup redises on oresdb hosts - https://phabricator.wikimedia.org/T124200#2352640 (10akosiaris) [11:06:43] Amir1: now we are in waiting mode. Let's see what brandon and emanuelle have to say [11:07:05] sure akosiaris [11:07:27] In the mean time I work on the setup in labs to see if everything works as expected [11:17:21] Amir1: btw, are there any plans to add swagger specs ? [11:17:40] akosiaris: we have swagger specs [11:17:46] but very badly implemented [11:18:07] they help a lot with monitoring. We already got clients that use them for thorough monitoring and alerting [11:18:24] ah, under which endpoint are they available ? [11:18:32] https://ores.wmflabs.org/v1/ [11:18:43] https://ores.wmflabs.org/v2/ [11:19:09] ok, lemme see what the service_checker says for those 2 [11:19:26] awesome [13:17:50] 06Revision-Scoring-As-A-Service, 10ORES: [Spike] Implement & test dependent tasks in Celery - https://phabricator.wikimedia.org/T136875#2352897 (10schana) @halfak, how is this going to share/get the extracted features between requests? It seems that this is just another caching area for results. I think I'm al... [13:19:47] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2352900 (10schana) >>! In T134782#2349237, @Halfak wrote: > Sorry, I'm confused. The scenario is that a web node goes down. The reason shouldn't m... [14:03:57] Amir1: /usr/local/lib/nagios/plugins/service_checker -t 5 10.64.0.16 http://10.64.0.16:8081/v2/ [14:03:57] Generic error: No valid spec found [14:04:09] so, it's somehow not conformant... [14:04:16] need now to figure out why and how [16:10:46] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2353365 (10Halfak) > How are internal-to-labs services currently being monitored? @Yuvipanda or @Dzahn might be able to answer this question. [16:16:19] 06Revision-Scoring-As-A-Service, 10ORES: [Spike] Implement & test dependent tasks in Celery - https://phabricator.wikimedia.org/T136875#2353415 (10Halfak) The "supertask" is where the sharing takes place. Note that multiple models are applied in the "score_many_models". > I'm also against the idea of combi... [16:18:20] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2353416 (10yuvipanda) Depending on what you mean by 'internal-to-labs', the answer might be 'lol, we do not' to 'hope' to 'this graphite+shinken bas... [16:18:27] o/ Amir1 [16:20:07] 06Revision-Scoring-As-A-Service, 10ORES: [Spike] Implement & test dependent tasks in Celery - https://phabricator.wikimedia.org/T136875#2353418 (10Halfak) Oh! I just realized that you might be imagining something different from me. I don't want to cache features between requests. Instead, I want to re-use t... [18:57:18] * halfak chews on the single-revision special case problem [19:07:51] So... I want to register that a score is being worked on as soon as soon as the first request arrives. [19:14:59] 06Revision-Scoring-As-A-Service, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2354117 (10Dzahn) We can monitor something from prod Icinga if it has a public IP and we consider it a special "semi-prod" service or so. Ores is b... [20:35:09] I think I found a way to get rid of the special case. [21:14:03] OK! [21:14:08] I think I'm getting pretty close. [21:14:14] I've cut down the code again [21:14:48] I've added signals to the base ScoringSystem class so that we can do some supplemental things in Celery and other distributed processors [21:15:00] This allowed me to drop the special case for single-revision scorings too [21:15:57] One problem I see though is that we're likely to get in a weird state if something goes wrong in the code between when a new task is registered and when it is sent to the celery queue. [21:16:05] I'll have to think about that one some more. [21:16:14] mmmm distributed systems! [21:16:38] The price we pay for efficiency and minimized duplication of work [21:16:45] == lots of sweat and debugging [21:17:06] OK. I'm to pack up and bike home. [21:17:11] offline for ~40 mins. [21:17:18] *I'm --> Time! [21:18:27] halfak: o/ [21:18:43] I was asleep (told you in telegrm :) [21:19:08] I'm still waking up :d [22:16:50] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train/test `reverted` model for nowiki - https://phabricator.wikimedia.org/T131856#2354745 (10Ladsgroup) a:03Ladsgroup [22:23:12] Who are you people [22:23:29] SigmaWP: I'm Amir! [22:24:25] We have halfak who is not here, Yuvi who helps us in operational stuff, and so many other volunteer or staff that work with us [23:47:46] 06Revision-Scoring-As-A-Service, 10ORES: Deprecate flower - https://phabricator.wikimedia.org/T137003#2354889 (10Ladsgroup)