[10:25:09] halfak: You on? [10:29:53] halfak: When you see this, or whoever does, how does one find an ORES score from a diff? [11:23:29] Vermont: https://ores.wikimedia.org/ [11:23:58] the docs should be quite explanatory and there is ofc https://ores.wikimedia.org/ui/ which might be helpful [12:12:31] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Complete edit quality campaign for Arabic Wikipedia - https://phabricator.wikimedia.org/T131669#4046310 (10Ghassanmas) [12:12:43] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Complete edit quality campaign for Arabic Wikipedia - https://phabricator.wikimedia.org/T131669#2174291 (10Ghassanmas) [12:13:22] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Complete edit quality campaign for Arabic Wikipedia - https://phabricator.wikimedia.org/T131669#2174291 (10Ghassanmas) [14:02:27] Thanks akosiaris :) [14:02:34] o/ Vermont [14:02:39] Let me know if you still have questions :) [14:39:28] halfak: I always have questions [14:41:07] o/ [14:41:39] Amir1: I know you're off, but feel free to pitch me any pending CR [14:42:05] awight: sure, where is it? [14:42:28] other way around! I'm available all day if you need to keep me going on stuff. [14:42:50] Maybe tomorrow we can talk about the plans for the ORES-JADE connector. [14:53:09] o/ awight [14:57:42] Yes, sorry for the miscommunication! [14:58:09] I thought I remembered something about you suggesting that I add anything I wanted to the sync agenda since I would be missing it [14:58:28] oops /pm [14:58:31] Oh yeah! We did talk about that :) [14:58:33] lol [14:58:42] No worries, it’s been a looong weekend all around. [14:59:13] Ultimately the calendar == truth [14:59:18] I’m working on a tweet about how people thinking about joining a gym should start with *all* the housework. [14:59:26] +1 I’ll cover that next time. [14:59:31] Cool :) [14:59:34] Already have my Mar 27-29 break marked [14:59:40] Nice. [14:59:42] * awight gets scared and double-checks [14:59:55] {{done}} [14:59:56] How cool, awight! [15:00:00] BTW, you should consider pivoting the JADE position paper into a Wikimania talk [15:00:10] saucy! [15:00:19] Deadline for proposals is Sunday [15:00:29] People just might want to hear about that. [15:00:30] oh [15:00:33] hehe okay then [15:01:03] halfak: Did I also mention, I’m fired up to work on paid editing? [15:01:37] Yes. Biggest hurdle we have to addressing promotional-type language is deploying parsers in production. [15:01:47] There's a task somewhere about dealing with the memory issues involved. [15:02:02] Might be able to reuse some of the insights from codezee's work to solve for that. [15:02:15] Alternatively, we can deploy a parallel service that hosts parsers only. [15:02:34] Preliminary work suggests that we can recognize spammy content. [15:03:25] I’ve been wondering about specialized workers... [15:03:31] It might also make sense for drafttopic [15:04:17] mmm…. ah and the conclusion of my thoughts about that was, first we need to profile memory usage to see if we have leaks that could be corrected by moving initializations pre-fork. [15:05:03] I don't think initializations are an issue. But python behaves strangely with different types of globals -._o_.- [15:05:18] For parsers, do you think the memory issue related to reference data, cache, or actual in-flight data? [15:05:24] *is related [15:06:03] I think that it's a shared memory issue. [15:06:30] For some reason, when we set a big memory hog as a "global" explicitly, it is shared. [15:06:57] But if we just import the module (which *is a global*, but implicitly) it duplicates memory usage for each process. [15:07:11] In python 3.5 anyway [15:07:25] I should make a super simple demo of this behavior. [15:08:29] * halfak checks how difficult that will be. [15:10:03] oh that’s a huge bug [15:10:10] but yeah I’m sure there’s a workaround. [15:10:34] We’re sure that the import happens before fork? [15:10:36] The "global" keyword is working pretty well so far. I wonder how many other cases of this we might identify. [15:10:42] Yes. [15:11:05] Can you show me an example of where the global is declared? [15:11:07] E.g. maybe we can dramatically reduce memory usage by loading all models into a global map in ORES before forking. [15:11:33] https://phabricator.wikimedia.org/T189364 [15:11:37] See sumit's work ^ [15:11:43] ty [15:13:17] I’m not seeing where the “global” was added… [15:13:22] in __init__? [15:13:35] Oh I asked sumit the same thing. He did it inside revscoring. [15:13:40] In the word2vec thingie [15:13:52] Hence why I want a simple demo [15:13:58] rather than relying on revscoring itself. [15:14:54] halfak: side note, I think you’re up: https://github.com/wiki-ai/editquality/pulls [15:15:13] Does our team use the idiom, “licking the cookie”? [15:17:07] * halfak looks [15:17:55] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores is alerting: 5xx rate (Change prop) alert. [15:18:47] Anyway, I still haven’t find where “global” gets added, but I trust that it’s the right solution. I just want to know how to apply myself. [15:18:58] i.e. it can wait. [15:20:21] oh, 5xx’s are coming from prodcution? [15:20:39] Looks like it [15:21:02] Whatever it was, it dropped down again [15:21:20] looks correlated to a small bump in overall scores errored [15:21:45] I wish our “overload” graph could be trusted. We need to add wsgi in there somehow, for when it’s the limiting factor. [15:22:23] timeout errors make it look like the MediaWiki API is the root cause [15:23:31] Hmm, it’s possible that timeout metrics are a blend of different types of timeout. [15:25:44] * awight wanders away from the panic room [15:27:01] About the word2vec global, one coding error I can imagine creeping in here and there is failure to declare a module-global as “global” when assigning to it in a function. That would create weirdly orphaned vars. [15:28:23] ^ wat [15:29:47] 10Scoring-platform-team (Current): Investigate word2vec memory issues with multiprocessing - https://phabricator.wikimedia.org/T189364#4039997 (10awight) See also {T182350}, I think you're onto something important and we can learn from this, and use it to fix other places where we fail to use copy-on-write. @Su... [15:29:47] lol [15:29:55] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores is not alerting. [15:43:53] Hmm... Something just occurred to me. I wonder if our pickling process re-imports the module after forking and that this skips shared memory. [15:44:18] We definitely initialize pre-fork, but then the reference is sitting in the pickled object. [15:45:23] Amir1: Are you presenting at today’s Diversity Alliance meeting? [15:45:33] halfak: holy cow [15:45:51] re-importing isn’t a thing in python, but that’s certainly an interesting lead to folow. [15:46:20] ooh and it would be interesting if the unpickling *is* a way to defeat python and re-import modules [15:46:30] awight: yup [15:46:38] Amir1: \o/ thanks! [15:47:15] awight: I just saw your message, today I have a marathon of meetings :( [15:48:25] no worries—I determined that I’m not blocking you [15:49:12] (03CR) 10Awight: "recheck" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/418877 (https://phabricator.wikimedia.org/T166427) (owner: 10Ladsgroup) [15:49:55] awight: i saw your comment on the memory issue task, do you know what that 310MB corresponds to by celery workers? [15:50:23] 10Scoring-platform-team (Current), 10JADE, 10Security-Reviews: Security review for Extension:JADE - https://phabricator.wikimedia.org/T188308#4046984 (10awight) >>! In T188308#4040037, @Bawolff wrote: > Review passed. > > Some minor comments Thanks, good points! > > * "JudgmentValidator.php" line 172 - s... [15:50:38] codezee: hi! No, I have no idea what’s in there... [15:50:59] It could be perfectly innocent, un-garbage-collected in-flight stuff. [15:52:25] (03CR) 10Ladsgroup: [C: 04-1] "This needs more work" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/418877 (https://phabricator.wikimedia.org/T166427) (owner: 10Ladsgroup) [15:52:28] but it seems interesting if there could be a solution to a wide array of problems just through an experiment :D [15:52:48] codezee: high-five! [15:53:03] (03PS1) 10Awight: Security review followups [extensions/JADE] - 10https://gerrit.wikimedia.org/r/419211 (https://phabricator.wikimedia.org/T188308) [15:53:50] 10Scoring-platform-team (Current): Investigate word2vec memory issues with multiprocessing - https://phabricator.wikimedia.org/T189364#4047000 (10Sumit) >>! In T189364#4046883, @awight wrote: > @Sumit please link to the code changes you're making that seem to improve memory sharing. Refer to the gist in the firs... [15:54:21] awight, I think re-importing is a thing when unpickling. [15:55:13] Wow! Exciting possibility. [15:55:24] IMO we should unpickle in the pre-fork process either way. [15:55:42] Just cos that’s repeated work for no good reason, if we’re unpickling in children [15:59:37] awight, that's impossible [15:59:49] The pickle is *for* sending data to the processes. [16:00:01] oh, gotcha [16:00:19] Oh wait. well. sort of. [16:00:20] Hmm. [16:00:29] I'm not sure that is true in ORES. [16:00:31] halfak: so in earlier experiments word2vec was getting pickled? [16:00:55] codezee, I think word2vec might be getting referenced in a pickled object [16:00:56] Just a quick advertisement for T173244 [16:00:56] T173244: [Investigate] Use PMML for prediction model serialization - https://phabricator.wikimedia.org/T173244 [16:01:16] nvm. that’s just for models [16:01:24] wiki-ai/revscoring#1453 (w2vfix2 - e84950c : Sumit Asthana): The build was broken. https://travis-ci.org/wiki-ai/revscoring/builds/352910060 [16:01:25] awight, is that a serious proposal? 'cause it sounds like a huge about of work for little gain to me [16:02:08] The potential gain is that we can train and test our models using a generic backend, for now. Yeah pretty small gain! [16:02:11] halfak: if its referenced in a pickled object, does that mean it should be replicated? [16:02:49] codezee, the unpickling process involved importing dependencies [16:04:42] * halfak --> meeting [16:05:40] it makes sense now, the word2vec is referenced in the features object which is pickled when we parallelized [16:05:47] *parallelize [16:06:11] good find! [16:06:39] but now since the vectors reside in the same module as the datasource definition and that too as a global, no import is needed [16:06:46] even after de-pickling in children [16:07:58] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES: Migrate ORES extension threshold config from old to new syntax - https://phabricator.wikimedia.org/T181159#4047060 (10awight) [16:08:19] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES: Migrate ORES extension threshold config from old to new syntax - https://phabricator.wikimedia.org/T181159#3781494 (10awight) This isn't working. I'll look for debug info and then revert the current patch. [16:09:25] codezee: That makes sense, and if correct then my CR suggestion to keep a local reference to the global data should cause the memory bloat to regress, I think. [16:12:02] awight: if we assign any *large* thing as a datamember of the object which gets pickled it will bloat [16:12:15] so if we do self.keyed_vectors = keyed_vecs [16:12:18] even then it'll fail [16:12:51] I see. Awesome that this is solved! [16:14:31] \o/ [16:24:32] 10Scoring-platform-team (Current), 10editquality-modeling, 10User-Ladsgroup, 10User-Tgr, 10artificial-intelligence: Train/test damaging and goodfaith model for Hungarian Wikipedia - https://phabricator.wikimedia.org/T185903#4047100 (10Tgr) Thanks! The models are in [[https://github.com/wiki-ai/editqualit... [16:27:08] 10Scoring-platform-team (Current), 10editquality-modeling, 10User-Ladsgroup, 10User-Tgr, 10artificial-intelligence: Train/test damaging and goodfaith model for Hungarian Wikipedia - https://phabricator.wikimedia.org/T185903#4047101 (10Tgr) Next step on the checklist is [[https://www.mediawiki.org/wiki/OR... [16:29:50] 10Scoring-platform-team, 10ORES: Beta cluster ORES is emitting statsd errors - https://phabricator.wikimedia.org/T189605#4047110 (10awight) [16:56:35] Does this look right? [16:56:36] 'likelygood' => [ 'min' => 'maximum filter_rate @ recall >= 0.9', 'max' => 1 ], [16:59:33] ok yeah, “maximum” makes sense there. [17:31:20] halfak: since I've not got edittypes repo yet, you can let me know if parsers code is somewhere around on github which i can have a look at and try to get it working with multiprocesses [17:32:55] codezee, check out wikigrammar [17:33:16] Essentially, the strategy is to load the models in that library into memory and run them on sentences. [17:33:33] Then we'd take aggregate measures of sentences based on that. [17:33:46] wiki-ai/wikigrammar [17:33:50] ok [17:33:58] do you have the drafttopic models in a PR? [17:34:33] halfak: no wait, nice reminder, I'll add a PR for the models and word2vec features [17:34:45] :) [17:36:09] awight: are you aware of/involved in https://phabricator.wikimedia.org/T111416 ? it talks about topic suggestion for dashboard program [17:38:37] codezee: Apparently I was subscribed a few years ago :-) Great to see that it's in play again. [17:39:43] i was looking if it our topic model could fit in somewhere there [17:39:51] *if our... [17:42:21] I bet it would; even better, if WikiEd knows about your model, they'll come up with ways to use it :) [17:48:23] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Patch-For-Review: Migrate ORES extension threshold config from old to new syntax - https://phabricator.wikimedia.org/T181159#4047411 (10awight) Looks good now. [17:50:28] halfak: i realized tuning report is not there, so i'm generating one, till then i've submitted word2vec features file - https://github.com/wiki-ai/drafttopic/pull/18 [17:50:45] i'll submit makefile, tuning-report, model in a single commit [17:53:48] Sounds great [18:00:12] 10Scoring-platform-team (Current): Scoring platform team FY18 Q2 - https://phabricator.wikimedia.org/T176324#4047462 (10awight) [18:00:15] 10Scoring-platform-team (Current), 10JADE, 10Patch-For-Review: Deploy JADE prototype in Beta Cluster - https://phabricator.wikimedia.org/T176333#4047460 (10awight) 05stalled>03Open [18:19:19] Amir1, I'm looking at https://github.com/wiki-ai/editquality/pull/136 and I think I've finally got to the bottom of WTF was in our old makefile and I still want to light my hair on fire. [18:19:21] lol [18:19:41] :)))) [18:19:46] What the hell were we doing? [18:20:05] Somehow, we were training the old model on a random sample of a random sample, but I can't figure out why! [18:20:38] I think making the Makefile templated helps us to review things :D [18:21:35] heh. I think i might take a pass on making this as simple as possible. Is that OK with you? [18:21:51] I'll start with a brand new sample since it seems like we're not using the current one. [18:22:21] halfak: sure, today I have too much (wikidata-sdie) [18:22:23] *side [18:30:07] OK :) [18:31:52] Demo complete! [18:31:59] https://github.com/halfak/demo_shared_memory [18:32:06] I get the exact same output for both strategies! [18:32:07] WTF [18:32:11] awight|lunch, ^ [18:32:23] See the ".dat" files for the memory profile. [18:32:35] The scripts in the base represent the two different entry points. [18:32:41] Maybe I did something wrong. [18:55:30] 10Scoring-platform-team (Current): Investigate word2vec memory issues with multiprocessing - https://phabricator.wikimedia.org/T189364#4047619 (10Halfak) I made a demo of this problem to try to see if I could reproduce it in isolation. See https://github.com/halfak/demo_shared_memory TL;DR: it didn't work. I... [19:03:35] hehe [19:04:33] that's terrible. [19:20:08] yes. [19:20:54] awight, any early bird registration deadlines we should be aware of for WebSci? [19:21:05] I just realized I missed one for CHI :( [19:21:07] $200 mistake [19:21:15] yes... hold on [19:21:28] May 1st [19:21:34] https://websci18.webscience.org/index.php/registration/ [19:21:55] OK cool. You should hear back from the workshop organizers way before then. [19:22:33] +1 they're suggesting that they're be able to respond before March 23rd [19:22:51] weow. that'd be fast [19:22:57] meowy weowy [19:32:58] 10Scoring-platform-team (Current): Investigate word2vec memory issues with multiprocessing - https://phabricator.wikimedia.org/T189364#4047706 (10Sumit) >>! In T189364#4047619, @Halfak wrote: > I made a demo of this problem to try to see if I could reproduce it in isolation. See https://github.com/halfak/demo_s... [19:33:26] halfak: ^ [19:33:46] codezee, I responded to your PR [19:34:07] I don't see why we need to have demo_kv be an argument to the function [19:34:20] Of course that would be a problem because we'd pickle it for every call! [19:34:36] halfak: we don't what i'm trying to convey is that its the same as when we do self.kv=something [19:36:02] halfak: in your demo if you use a class and assign demo_kv to self.kv and make process use that you'll see the difference, just that using a partial function makes it much more easy to see [19:36:47] codezee, hmmm. Let's test that then [19:37:22] Also, if that is the problem we don't need a global to get around it :) [19:40:00] i've realized we don't need the 'global' keyword it works without it, by just having the keyed-vec outside the class definition [19:44:40] halfak: https://github.com/halfak/demo_shared_memory/pull/1 [19:44:44] updated [19:45:39] also proves the point that partial functions and the self.xyz approach is the same [19:45:47] *are the same [19:51:32] the above code is closest to the extractor script i have been using [19:52:13] alright, gotta go, let me know your thoughts on the task or pr [20:08:59] halfak: I haven't been able to find any task about the COI parser and memory usage, let me know if that's easy for you to dig up. [20:09:06] I did find https://phabricator.wikimedia.org/T120170 [20:09:31] Ahh this is about the parsers that sumit was working on. [20:09:49] Right the PCFG for spam articles. [20:10:09] With the idea that POV language is like spam language and can be detected linguistically. [20:10:18] Do you have any other ideas for how we'd gather signal? [20:13:24] Editors were dropping great feature ideas into our task, and into the related one that lzia started. [20:13:45] Oh goodness. We have two tasks :| [20:14:11] I think editor maturity vs account maturity is gonna be an interesting source of signal. [20:14:55] halfak: k, I found the task you mentioned earlier: https://phabricator.wikimedia.org/T157041 [20:15:48] 10Scoring-platform-team, 10ORES, 10artificial-intelligence: Address memory usage issues for deploying PCFG-based features - https://phabricator.wikimedia.org/T157041#4047855 (10awight) [20:15:51] 10Scoring-platform-team (Current), 10draftquality-modeling, 10editquality-modeling, 10revscoring, and 3 others: [Epic] Implement PCFG features for editquality and draftquality - https://phabricator.wikimedia.org/T144636#4047854 (10awight) [20:16:15] 10Scoring-platform-team, 10ORES, 10artificial-intelligence: Address memory usage issues for deploying PCFG-based features - https://phabricator.wikimedia.org/T157041#2993446 (10awight) [20:16:19] 10Scoring-platform-team, 10Research Ideas, 10artificial-intelligence: Paid editing (COI) detection model - https://phabricator.wikimedia.org/T120170#4047859 (10awight) [20:16:56] 10Scoring-platform-team (Current), 10Research Ideas, 10artificial-intelligence: [Epic] Paid editing (COI) detection model - https://phabricator.wikimedia.org/T120170#1847096 (10awight) [20:17:39] This list is human-centric, but we can probably automate a few if we think they'll be useful: https://en.wikipedia.org/wiki/Wikipedia:Identifying_PR [20:33:08] +1 awight [20:33:18] We should add that as a Q4 goal if you want to put some time into it. [20:33:25] That's coming up at the end of the month. [20:33:25] Sure do! [20:33:42] I think we have a dataset for that too. I've been working with doc james. I should get that published :) [20:34:43] I'm looking at a Java PCFG NLP library just for reference. I think it's got the same problem, that the model itself is reasonable (20MB) but the intermediate structures for parsing are huge (500MB for normal sentence length). https://nlp.stanford.edu/software/lex-parser.shtml [20:36:32] A service sounds nice, or it might be lighter-weight to use specialized celery workers, e.g. http://docs.celeryproject.org/en/latest/userguide/routing.html [20:40:07] Just pinged the thread re. the paid editors dataset. I'm just confirming the author list and some details about cleanup I want to do. Then I'll get it up on figshare and we can start experimenting. [20:40:16] The dataset is just a list of know paid editors. [20:40:39] So we'll probably want to model them against the average newcomer assuming that most of them aren't paid COI editors. [20:41:51] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Re. init enwiktionary reverted model & labeling campaign - https://phabricator.wikimedia.org/T188271#4047952 (10Halfak) Ran into {T188564} while working on this. I'm waiting for it to ge... [20:42:41] \o/ [20:48:58] * halfak books a bunch of travel for April :( [20:49:10] I hate traveling but I like the things I travel to [21:05:15] hehe, I think I'm getting jacked by "New user landing page" now. https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Wikipedia:New_user_landing_page&page=Jade%3ADiff%2F376901 [21:25:24] o/ Need to leave a bit early, probably won't be back until tomorrow! [21:25:30] o/ [21:25:35] Have a good one awight [21:25:40] Ext:JADE is... nearly deployed to beta [21:25:42] * halfak prepares for his management training [21:25:48] I'm scoping it out now. [21:25:48] bahahaha [21:25:55] Enjoy [21:26:11] "There is no revision history for this page." [21:26:15] wha? [21:26:20] https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Jade:Diff/376901&action=history [21:26:35] Oh. Must have just been cleared.