[09:54:35] Hi all! quick question RE http://ores.wmflabs.org/scores/wikidatawiki/ . is accuracy a guess of how goo the AI is? 1 = perfect? [13:45:04] addshore, hey! [13:45:58] There's many ways you can measure the fitness of a model. We go by AUC because it's very close to a common use-case (sorting of edit by the probability of damage) and it's hard to game. [13:46:36] Accuracy is the proportion of predictions that are right (in the test set). [13:46:40] It is very easy to game. [13:47:00] We could build a classifier with 99.9% accuracy for WikiData by always predicting False. [13:47:50] xD [13:47:57] See https://en.wikipedia.org/wiki/Receiver_operating_characteristic [13:48:30] FWIW, I *just* trained a model for WikiData that gets 0.90 AUC! [13:48:38] Hopefully, we'll have that released soon. [13:49:07] (For both AUC and Accuracy, 1.0 would be perfect) [13:51:44] ooooh :P [13:52:16] We've been working out a few specific problems that wikidata-editors have pointed out. I think this will be a substantial update :) [13:52:18] * addshore is super excited [13:52:41] so also, what exactly am I being shown when I go to http://ores.wmflabs.org/scores/wikidatawiki/?revids=253391330&models=reverted ? [13:52:57] prediction is false means the ai doesnt think it is a bad edit? [13:53:13] Yeah. That's right. [13:53:34] But I wouldn't put too much weight on the prediction itself. It's not really optimized for recall. [13:54:04] We usually set the threshold for "is worth reviewing" at true = 0.70 [13:54:14] okay [13:54:17] wiki-ai/wb-vandalism#105 (new_balance - 9aa4a57 : halfak): The build passed. https://travis-ci.org/wiki-ai/wb-vandalism/builds/94164088 [13:54:22] And "is likely to be vandalism at true = 0.80-0.85 [13:54:24] true + false wont equal 1 though? or will it? [13:54:27] Shuddup travis [13:54:31] :> [13:55:57] addshore, should equal 1. [14:02:01] okay! [14:02:02] coolio! [18:29:37] can anyone help me here please? [18:29:37] I want to buy an amd gpu and use opencl to run a massively parallel neural network of sorts... [18:29:37] 12http://www.amazon.co.uk/dp/B00ZPOAYG6/ref=wl_it_dp_o_pd_S_ttl?_encoding=UTF8&colid=321QN7BI6ECFH&coliid=I24ARRG4LZGXFE [18:29:38] this thing has 2560 stream processing units [18:29:38] would that actually work? I just basically port a massive neural network into 2560 arrays and execute, and make sure they're written in opencl form so the card can execute the code? [18:30:08] sorry, duplicating this question -- I want to know what's best for parallel processing for a massively parallel neural network kind of structure [18:54:31] jenelizabeth, regretfully I don't have any experience with ANN or GPU-based ML work. I'd like to buy that video card to play fallout 4 :) [18:54:45] amir1 has done some ANN work, but I don't think it was on a GPU. [18:55:01] What type of ANN are you working with and why go with a GPU? [18:55:17] * yuvipanda vaguely waves [18:55:38] o//////// [18:55:44] 7 arms waving!!! [18:55:50] Or 8 [18:56:08] :D [18:56:15] * yuvipanda has 3h of meetings ahead [18:56:32] halfak: ALSO DID I SHOW YOU tools.wmflabs.org/paws/hub/oauth_login [18:57:16] Interesting. "My Server" -- does that mean I have a whole set of python processes to myself? [18:58:18] halfak: yes [18:58:31] halfak: and there's a terminal with pywikibot stuff [18:58:36] halfak: and your $HOME is persisted [18:58:45] Can haz mw_...? [18:58:47] yeah [18:58:49] next step [18:58:56] halfak: but you can pip install stuff already [19:00:03] also need to be able to access db (not yet) [19:00:36] Access to XML dumps? [19:01:05] on the way [19:01:07] today or tomorrow [19:01:46] Cool! [19:02:27] Would like to brainstorm with you about moving some of my XML/db stuff to jupyter. [19:02:33] Do you have a plan for an example bot? [19:02:52] no [19:03:05] I'm basically at 'build it and someone will tell you!' [19:03:08] similar to quarry [19:04:27] halfak: I guess my current aim is at interactive things on terminals [19:04:52] Cool. I have lots of terminal thingies? [19:04:57] halfak: yeah! [19:05:38] halfak: do you have ideas? [19:05:41] :D [19:05:44] halfak: did you check out the terminal [19:05:49] * halfak slips into a meeting :( [19:12:49] halfak, just a very simple neural network... I wouldn't even call it a neural network, it's a personal diy design I've been working on for the past year [19:13:33] the operation applied would be simple yet dynamic, adaptive... an action potential, but here's my problem I suppose... since these are stream processing units and it would be SIMD processing [19:14:41] the input array of nodes could comprise of thousands of nodes, and the operation applied to them all at once would be that action potential... but for each node in the input array, the result would be different [19:15:19] after the operation (action potential, scalable and dynamic) has been applied: I want it to be like addition [19:16:30] ahh yeah, it would work my bad - the differentiations of each node action potential would be of its own floating point value, so... the network as an array when active... would look a bit like a light bulb, with some nodes having less potential, releasing less output etc [19:17:18] I couldn't apply to operations simultaneously could I, for two separate arrays? one for the nodes, and one for the connections determining the positions of each node's I/O [19:17:27] :/ [19:18:27] I was going to diy a cluster of systems, but working out the math formulas for determining the network (external) and processing (internal) timings, where each system has a different architecture... and are not all of the same generation... [19:18:55] plugging them all into a switch, and then an output into a master control system for monitoring what the cluster is doing... [19:19:20] I could write it and design it, but... it sounds like a bit of a headache, since they'd all need benchmarking several times over each [19:19:58] and network benchmarks several times over to determine low and peak limits to figure out how to pseudo multi-thread the cluster [19:20:23] would probably enjoy designing it tbh... for experience... [19:21:03] they're just spare motherboards, cpus and ram laying about, all that's needed is about 8x cases with ~400 or ~500 watt psus [19:21:57] I've got three switches, one in a box never opened.. and not used any of them, they came from a friend from some company that was getting rid of its hardware after going bankrupt a few years ago [19:22:37] I kept them in case of need/use to play with, but never really got round to clustering a few systems and using a switch to play with their networking [19:52:10] halfak: You around? [20:06:14] * halfak is meeting day :\ [20:15:50] k [20:17:15] The reward for great work is more meetings! [20:18:18] I think someone else might be able to answer this: [20:18:28] How are worksets for wikilabels selected? [20:18:45] Randomly from incoming edits? [20:21:38] aetilley, randomly from the last full year (to deal with periodic patterns) [20:21:39] ahh, just checked one of my switches... from 2006... 24 ports [20:21:53] says some crap about needing a grounded power cable [20:22:07] and not to look into the fibre optic ports (there's only two) [20:23:45] It's a Nortel Business Ethernet Switch (100 or 200 series) [20:24:01] manual I've got here is from April 2006 [20:24:09] can't be that bad of a switch, right? [20:24:31] it has two additional ethernet ports, which I assume to be for network input and output streams [20:24:52] and what looks like an inside out vga port (the vga like port with pins instead) [20:25:01] that would be for a screen to connect to it, right? [20:48:59] jenelizabeth, I wonder if AWS would support what you want. [20:49:12] Probably not GPU, but they'd manage a cluster for you so you don't need to. [20:49:15] AWS has GPU [20:49:18] It's a surprisingly large amount of work. [20:49:19] Cool! [20:49:45] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html [20:49:49] halfak: Thanks. We might consider (probably not in the immediate future) this new craze of active learning [20:49:54] http://repository.cmu.edu/cgi/viewcontent.cgi?article=1007&context=machine_learning [20:50:34] We can come back to this though. [20:50:42] aetilley, ooo. That looks interesting. [20:51:12] The algorithm asks for labels? [20:51:59] Seems relevant: http://scikit-learn.org/stable/auto_examples/semi_supervised/plot_label_propagation_digits_active_learning.html#example-semi-supervised-plot-label-propagation-digits-active-learning-py [21:00:08] Well the idea is that unlabeled examples are cheap and labeled examples are expensive, but we can efficiently train a classifier if we are judicious about which incoming unlabelled examples to hand to our experts for labelling. [21:01:02] In particular, workssets would not be random. They would consist of edits that, if labelled, would be very informative to training our classifier. [21:01:24] I'll check out that link. [21:04:32] Im thinking that in a year's time, you could have a framework that does feature extraction on every single incoming edit. If the features are "interesting", the edit is passed to a queue of "to-be-labled" edits (perhaps bumping off another edit if this queue is alread too long. [21:04:53] Then periodically this queue is passed to wikilabels. [21:12:20] Sorry for being in and out. Reading scrollback now. [21:12:58] aetilley, this sounds very interesting. [21:13:06] So, the model would learn "online" [21:13:07] ? [21:27:33] halfak: btw, bmansurov offered to help on the staff list is a mw dev :D [21:28:40] halfak: Kind of a hybrid. The labels would only be processed periodically. [21:29:09] But I think there are several options. There's definitely such things as batch vs selective active learning. [21:34:23] aetilley, gotcha. Would labels be requested online? [21:34:23] Anyway, that would be a long term project. My immediate goal today is to write you some new features. :) [21:34:29] Yay!!! [21:34:32] :) [21:34:51] "requested" = put into the next work-set [21:34:53] yuvipanda, +1 good point. I suppose he hasn't contributed yet though. [21:35:07] aetilley, I suppose that makes sense :) [21:35:12] halfak: he hasn't [21:35:58] * halfak looks around for bmansurov [21:36:09] I should have invited him to this channel. [21:36:22] halfak: you can find him on #wikimedia-mobile [21:45:01] * halfak camps out in #wikimedia-mobile