[04:17:47] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: [Spike] Proof of concept damage detection with hash vectors - https://phabricator.wikimedia.org/T132581#2274188 (10Sabya) There were few discussions over email. I'm pasting that here for future reference. It is in reverse chronological order. **On... [04:19:36] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: [Spike] Proof of concept damage detection with hash vectors - https://phabricator.wikimedia.org/T132581#2274192 (10Sabya) a:03Sabya [05:35:15] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 10wikilabels: Enable CORS for error responses from ORES - https://phabricator.wikimedia.org/T119325#2274268 (10Ladsgroup) a:03Ladsgroup [05:35:44] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 10wikilabels: Enable CORS for error responses from ORES - https://phabricator.wikimedia.org/T119325#1823542 (10Ladsgroup) Same happens with #wikilabels. It's pretty easy to fix. [07:28:04] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 10wikilabels, 07Icinga: icinga-wm report on wikimedia-ai too in case of puppet failures for ores and wikilabels nodes - https://phabricator.wikimedia.org/T134726#2274515 (10Ladsgroup) [07:29:47] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 10wikilabels, 07Icinga: icinga-wm report on wikimedia-ai too in case of puppet failures for ores and wikilabels nodes - https://phabricator.wikimedia.org/T134726#2274527 (10yuvipanda) oh, for puppet failure on labs we use shinken. Look at `./modules/beta/man... [07:42:34] 06Revision-Scoring-As-A-Service, 10ORES, 10wikilabels: Enable CORS for error responses from ORES - https://phabricator.wikimedia.org/T119325#2274548 (10Ladsgroup) [07:42:58] 06Revision-Scoring-As-A-Service, 10ORES, 10wikilabels: Enable CORS for error responses from ORES - https://phabricator.wikimedia.org/T119325#1823542 (10Ladsgroup) [08:00:24] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Configure ORES load balancer to rebalance on 500 error - https://phabricator.wikimedia.org/T111806#1616840 (10Ladsgroup) Per what I've learned from nginx [[http://nginx.org/en/docs/http/load_balancing.html|load balancing manual]]. It does have a very basic syste... [08:00:24] 10[1] 04https://meta.wikimedia.org/wiki/http://nginx.org/en/docs/http/load_balancing.html [08:01:30] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Configure ORES load balancer to rebalance on 500 error - https://phabricator.wikimedia.org/T111806#1616840 (10yuvipanda) We do not have any proprietary software, including nginx plus. [08:02:01] 06Revision-Scoring-As-A-Service, 10ORES: Configure ORES load balancer to rebalance on 500 error - https://phabricator.wikimedia.org/T111806#2274577 (10Ladsgroup) [08:06:09] 06Revision-Scoring-As-A-Service, 10ORES: Configure ORES load balancer to rebalance on 500 error - https://phabricator.wikimedia.org/T111806#2274580 (10Ladsgroup) >>! In T111806#2274575, @yuvipanda wrote: > We do not have any proprietary software, including nginx plus. \o/ Like :) [08:49:29] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 10wikilabels, 07Icinga: icinga-wm and shinken report on wikimedia-ai too for ores and wikilabels nodes - https://phabricator.wikimedia.org/T134726#2274697 (10Ladsgroup) [08:49:44] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 10wikilabels, 07Icinga: icinga-wm and shinken report on wikimedia-ai too for ores and wikilabels nodes - https://phabricator.wikimedia.org/T134726#2274515 (10Ladsgroup) I think we need to support both :) [13:48:04] Running a little late today [13:48:25] Will be on for real in 15 [13:53:28] o/ [13:53:33] halfak: o/ [14:01:21] halfak: 1- we enabled CORS for all responses https://phabricator.wikimedia.org/T119325 [14:01:52] 2- https://gerrit.wikimedia.org/r/#/c/287590 [14:02:28] 3- I'm not so sure about this, should we use nginx as lb or another proxy? https://gerrit.wikimedia.org/r/#/c/287570/ [14:02:43] 4- Ran some tests: https://github.com/wiki-ai/ores/pull/142/files [14:03:30] We should ask YuviPanda about why he recommended nginx. [14:03:45] One nice thing for us is that we can re-use config from ORES. [14:04:04] Right now, we use in-memory storage to manage session information for users in wikilabels. [14:04:11] he actually merged my patch in the first thing [14:04:24] We'll need to switch to redis or something like that for session management if we want to have multiple web nodes behind an lb. [14:04:27] :) [14:04:55] Caching is a little funny. How do you know when something needs to be invalidated? [14:05:11] and connecting with postgres db becomes super complicated because of courser issues [14:05:38] it's a trade-off [14:05:53] too much caching == losing functionality [14:06:10] too loose == performance sucks [14:06:14] courser issues? [14:06:22] let me find it [14:07:45] https://phabricator.wikimedia.org/T133557#2242615 [14:08:19] when there are two instances that changes database, courser acts strangely [14:08:36] we had this issue with staging and experiment [14:11:54] I'm confused about what the problem is. [14:12:15] FWIW, from postgres' point of view, it gets commands from many connections simultaneously. [14:12:37] uwsgi starts up may processes that handle requests in parallel. [14:14:03] hmm [14:14:05] yeah [14:14:19] it might worth looking again [14:20:46] Amir1, could you help me understand the expected behavior of this caching set up, I'd appreciate it. [14:20:50] Looking at https://gerrit.wikimedia.org/r/#/c/287570/1/modules/wikilabels/templates/nginx.erb [14:21:08] Let's say that I submit a request for /campaigns/10?campaign=stats [14:21:21] And then 5 minutes later I submit the same request. [14:21:26] Should I get a cached result? [14:21:37] yeah [14:21:41] That's not right [14:21:44] halfak: but we can simply disable it [14:21:44] We shouldn't do that. [14:21:55] or put a very low number [14:21:58] e.g. 1 min [14:22:43] That could work, I guess, but in this case, I don't think we want to return cached responses when the user requests a response whose status changes frequently. [14:23:09] It would probably be OK to have "/campaigns/enwiki/" be cached. [14:23:23] We don't need to see a new campaign right away. [14:23:46] But we'll need users to be able to request a workset and not get a cached response. [14:23:56] Or get the status of a campaign and have it reflect the real current status. [14:23:58] * halfak thinks. [14:24:09] If there were a better way to invalidate, that would be great. [14:24:43] e.g. if we could have cache invalidation happen on an action -- like saving a label or requesting a workset. [14:24:51] That seems tedious and error prone. [14:25:05] But it would get us both caching and the flexibility we want. [14:26:39] I'm not sure that would be possible with nginx [14:27:26] Agreed. [14:27:35] We might be able to switch to redis. [14:27:46] Actually, now that I think about it, we could use redis for everything... [14:27:49] Hmmm.... [14:27:54] That would be BLAZING fast [14:28:03] Our DB is pretty small. [14:28:35] And the good news is that we already have a data model abstraction so the only hard part would be translating SQL to redis commands. [14:29:48] but why not send the workset to the client machine and as the editor finished labeling the client sends the labeled data back to the server, by this method method only two query request is needed for each workset [14:30:42] hmm [14:32:37] GhassanMas, what if the user browses away while the workset is open? [14:32:50] I guess we could implement a "save workset" flow in the UI [14:33:00] And a warning prompt if browsing away from an unsaved workset [14:35:59] could access the cookie [14:36:29] GhassanMas, not a bad idea. Generally worksets won't contain all that much data. [14:36:43] We might have free-form text fields though. And in that case, the cookie could get very large. [14:37:06] Also, you could have multiple worksets open at a time (for different campaigns) [14:37:23] I think I like this idea of a workset as the editable unit. [14:37:36] We'd make fewer requests to wikilabels, but they'd contain more data. [14:38:07] We could send a diff when saving so that we don't have to give wikilabels the whole workset every time. [14:38:50] e.g. workset == task_ids: [1,2,3,4,5,6,7,8,9,10]; save_changes([1,5,8,9]) [14:40:05] yeah halfak, that seems well to me [14:40:46] * halfak will file design task to get us started. [14:40:52] Would be nice to pull in a designer for this. [14:41:12] Amir1, any thoughts re. ^? [14:41:34] yeah [14:42:08] I think we gave some flexibility to worksets by adding abandoning option [14:42:12] but we need more [14:50:33] 10Revision-Scoring-As-A-Service-Backlog, 10wikilabels: [design] Workset as editable unit - https://phabricator.wikimedia.org/T134766#2276302 (10Halfak) [14:50:35] https://phabricator.wikimedia.org/T134766 [14:50:40] Damn you wikibugs. [14:50:44] Beating me to the post [15:00:07] 10Revision-Scoring-As-A-Service-Backlog, 06Research-and-Data, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2276339 (10Halfak) [15:00:32] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2272293 (10Halfak) [15:00:54] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Configure ORES load balancer to rebalance on 500 error - https://phabricator.wikimedia.org/T111806#2276342 (10Halfak) [15:01:46] halfak: I'm actually working on this: https://phabricator.wikimedia.org/T111806#2276342 [15:01:55] I'm waiting for your input [15:02:11] 06Revision-Scoring-As-A-Service, 06Research-and-Data-Backlog, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2276348 (10Halfak) [15:18:50] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: Configure ORES load balancer to rebalance on 500 error - https://phabricator.wikimedia.org/T111806#2276456 (10Halfak) Our scoring timeout is 15 seconds. See https://github.com/wiki-ai/ores-wikimedia-config/blob/master/config/00-main.yaml#L63 This timeout exist... [15:20:13] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: [Spike] Proof of concept damage detection with hash vectors - https://phabricator.wikimedia.org/T132581#2276458 (10Halfak) Awesome! Thanks for the dump. Here's my email response. ---- With our currently deployed model, we get ROC-AUC of 0.914... [15:31:10] halfak: tell me when you have some time [15:35:57] that was fast :D [15:38:29] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Configure ORES load balancer to rebalance on 500 error - https://phabricator.wikimedia.org/T111806#2276584 (10Ladsgroup) [15:40:59] 06Revision-Scoring-As-A-Service, 10ORES, 13Patch-For-Review: Configure ORES load balancer to rebalance on 500 error - https://phabricator.wikimedia.org/T111806#2276596 (10Ladsgroup) a:03Ladsgroup [15:45:32] Hey Amir1 [15:45:33] What's up? [15:48:27] I was afk this time [15:48:30] o/ Sabya [15:48:42] o/ halfak [15:48:44] halfak: first, now we changed timeout [15:49:09] akosiaris merged the patch really fast (\o/) [15:49:22] https://gerrit.wikimedia.org/r/287640 [15:49:59] Great! [15:50:11] So this will rebalance on a 500 too, right? [15:50:45] We should have incinga send us a warning with that happens. I wonder if our health checks can skip the lb. [15:50:57] * halfak gets prepared for sync meeting [15:51:50] halfak: anything you wanted to discuss? [15:51:57] yeah [15:54:42] halfak: can you join now just to check if my microphone is working [15:54:48] kk [15:54:54] I did some stuff it worked on skype but not so sure about hangout [15:55:06] sabya, not really beyond my last message. I'm hoping that Justin will hop onto phab to share some suggestions too. [15:55:25] got it [15:55:53] i'll also work on skipgram [15:56:06] Great. Does it look like my grammer will work for you? [15:56:34] * halfak feels weird about calling it a grammer because people will think he is mispelling grammar. :D [15:56:46] But it is a thing that produces grams :) [15:58:33] :-D will look into that and let you know. didn't get a chance yet [16:30:55] 06Revision-Scoring-As-A-Service, 06Research-and-Data-Backlog, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2276823 (10DarTar) I strongly support this request, @Ladsgroup is a core contributor to ORES and I'd love to see him set up with an NDA to remove these blockers. [16:31:28] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 10Research-management, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2276825 (10DarTar) [16:31:45] 06Revision-Scoring-As-A-Service, 10revscoring: Improve parsed feature performance (mwparserfromhell) - https://phabricator.wikimedia.org/T134780#2276827 (10Halfak) [16:31:58] 06Revision-Scoring-As-A-Service, 10revscoring: Improve parsed feature performance (mwparserfromhell) - https://phabricator.wikimedia.org/T134780#2276841 (10Halfak) a:03Halfak [16:32:23] 06Revision-Scoring-As-A-Service, 10revscoring: Improve parsed feature performance (mwparserfromhell) - https://phabricator.wikimedia.org/T134780#2276827 (10Halfak) https://github.com/wiki-ai/revscoring/pull/271 [16:33:11] 06Revision-Scoring-As-A-Service, 10revscoring: Make cache be preserved (in place) when solving dependencies - https://phabricator.wikimedia.org/T134781#2276861 (10Halfak) https://github.com/wiki-ai/revscoring/pull/272 [16:33:13] 06Revision-Scoring-As-A-Service, 10revscoring: Make cache be preserved (in place) when solving dependencies - https://phabricator.wikimedia.org/T134781#2276847 (10Halfak) [16:33:34] 06Revision-Scoring-As-A-Service, 10revscoring: Make cache be preserved (in place) when solving dependencies - https://phabricator.wikimedia.org/T134781#2276866 (10Halfak) [16:40:03] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: [spike] Find out if we can still get health check warnings after lb rebalance - https://phabricator.wikimedia.org/T134782#2276894 (10Halfak) [16:45:32] 10Revision-Scoring-As-A-Service-Backlog, 10wikilabels: Deploy cron job for expiring wikilabels tasks - https://phabricator.wikimedia.org/T134783#2276913 (10Halfak) [16:46:48] 10Revision-Scoring-As-A-Service-Backlog, 10ORES, 10revscoring: Deploy new ORES with revscoring performance improvements - https://phabricator.wikimedia.org/T134784#2276935 (10Halfak) [16:59:48] 06Revision-Scoring-As-A-Service: Develop training session material for Wikimania Hackathon - https://phabricator.wikimedia.org/T134785#2277016 (10Halfak) [17:01:14] oh god [17:01:19] Hey! Getting food. Will be back soon. [17:01:26] I have a really bad headache [17:01:29] got to go [17:01:34] see you soon [17:05:12] o/ [17:05:14] take care [17:10:10] 06Revision-Scoring-As-A-Service, 10Wikimania-Hackathon-2016-Organization, 10bwds, 06Developer-Relations (Apr-Jun-2016): Generate bad words for all languages more than 100K articles - https://phabricator.wikimedia.org/T134629#2271709 (10QuimGil) Is there a doc explaining to volunteers what they need to cont... [17:18:47] Amir1: halfak didn't fully read backscroll but I responded at https://gerrit.wikimedia.org/r/#/c/287570/ about nginx [17:56:06] YuviPanda: thank you for your note [17:56:13] I'll probably abandon it [17:56:56] the biggest reason for me was enabling CORS (since I want to move to using POST method which not possible right now because we use jsonp instead of json) [17:57:16] I talked with halfak quite a bit about it [17:58:23] * YuviPanda nods [17:58:28] you can add it with uwsgi itself [18:32:16] o/ [18:38:34] I'm here [18:38:46] but about 50%, got a headache [18:39:08] probably I'm in hibernate mode until someone pings me :D [18:39:14] kk. [18:39:29] Take care of yourself first :) [18:39:43] I do :) [19:13:26] 06Revision-Scoring-As-A-Service, 10Wikimania-Hackathon-2016-Organization, 10bwds, 06Developer-Relations (Apr-Jun-2016): Generate bad words for all languages more than 100K articles - https://phabricator.wikimedia.org/T134629#2277587 (10Ladsgroup) Hey @QuimGil, Thank you for showing interest in this project... [19:48:27] halfak: do you have a min to write the weekly update [19:48:28] ? [19:53:16] Sure. Can do. [19:53:45] awesome [19:53:47] thanks [21:37:51] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 10Research-management, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2272293 (10DarTar) p:05Triage>03High [21:38:12] 06Revision-Scoring-As-A-Service, 06Research-and-Data, 10Research-management, 06WMF-NDA-Requests: NDA for Amir Sarabadani - https://phabricator.wikimedia.org/T134651#2272293 (10DarTar) a:03Halfak [21:51:37] hey halfak, do we currently have anyone helping with visual/UX design on the ORES beta feature? [21:52:11] Nope. Used to be that we'd get a bit of help from violetto, but not for a while [21:52:21] got it [21:52:42] I’m attending the DR meeting in a moment and I’ll flag that [21:57:37] kk thanks [22:21:58] halfak: done. Jonathan agreed to help with some heuristic-based evaluation of the design, if we want to, and I’ll be giving designers the heads up tomorrow during the cross-dept UX meeting [22:22:43] if we want to use DR help, we should send J-Mo a link to the most recent version of the design of the beta feature [22:23:05] Amir1 ^