[00:05:07] hi [01:17:02] Amir1, need new feature list pushed to the repo [08:04:56] halfak: great. I will do it asap [12:51:23] halfak: hey, https://github.com/Ladsgroup/wb-vandalism/blob/master/wb_vandalism/features_list/wikidata.py [12:51:32] Hey Amir1 [12:51:34] saw that [12:51:40] great [12:51:56] Is the model beig trained? [12:51:59] *being [12:52:07] (damn you Konversation) [12:52:33] ha. Still iterating on the input. [12:53:09] Also, working on a ping for you in phab because qgil wants us to discuss hacksessions before Nov 6th [12:53:39] halfak: I found a wonderful alternative to standing up a wikibase for media metadata. [12:53:52] Oh? [12:54:02] Inference via Commons categories on Wikidata. [12:54:46] Inference sounds complicated. [12:54:56] Unless it's one-off. [12:55:20] Well, an image is in the category "white dogs". Category is associated with items for white and dog. Images in that category are white and dog by association. [12:56:30] Also will help when we transition to proper image tagging on Commons since half the work will already be done. It's doable really because Commons is absurdly specific with their categories. [12:57:17] Ladsgroup/wb-vandalism#23 (master - e630809 : amir): The build was broken. https://travis-ci.org/Ladsgroup/wb-vandalism/builds/87878513 [12:57:22] Amir1, it's working! [12:57:33] * halfak waits patiently [12:57:39] Awesome [12:58:40] :) [13:07:16] Damn. looks like one of the features is not pickleable. But it looks like we're getting ~80 AUC, which is probably acceptable. [13:07:21] Amir1, ^ [13:07:46] do you know which feature? [13:07:54] We'll need to see what the scores look like in comparison to the edits that get flagged, but I think we're close to good. [13:08:05] Good enough to deploy anyway. [13:08:13] wb_vandalism.features.diff. [13:08:16] There [13:08:20] 's a lambda in there [13:08:23] \o/ \o/ \o/ [13:08:29] lambdas can't get pickled. [13:08:33] I can take a pass on this. [13:08:36] I've been meaning to. [13:08:58] Okay [13:09:06] I will use a function instead of lambda [13:09:12] halfak: is it okay? [13:09:20] I have a couple hours I could spend on this right now if you don't mind looking at the PR. [13:09:37] Will you be around in a couple hours? [13:09:56] yeah [13:10:05] I will be here until we deploy [13:10:18] Cool. [13:10:26] I can't see any PRs in wb-vandalism [13:10:31] do you mean in ORES? [13:10:51] Sorry. meant the PR I'll be putting together right now. [13:11:03] I've been just self-merging recently :S [13:11:07] :))) [13:11:19] Okay [13:23:29] Amir1, I'm not seeing any tests for the features in wb_vandalism. Am I missing something or is that a todo? [13:25:44] Also comments like "AF/8". What does that mean? [13:26:00] we have tests [13:26:06] (code coverage 95%) [13:26:13] Hmm [13:26:16] they are in tests folder [13:26:17] * halfak looks harder [13:26:23] in root [13:26:33] Oh! [13:26:37] Not using nosetests [13:26:45] AF is abbreviation of Abuse filter [13:26:45] ewww.. unittest [13:26:49] Gotcha! [13:27:18] when you write tests using unittest you can test them with nosetests [13:27:22] Would you mind if I switched from unittest to something like nosetests? It's totally OK if you'd prefer to stick with unittest. [13:27:29] travis does exactly this [13:27:40] Yeah. I just think the way you do it with unittests is gross and harder to test subsets of the code. [13:27:46] BUT I'll side with your judgement [13:28:22] Can I see some examples? [13:29:38] https://github.com/wiki-ai/revscoring/blob/master/revscoring/features/tests/test_diff.py [13:29:41] Amir1, ^ [13:30:16] So, if I go into the revscoring/features/ directory and run "nosetests", it will only run the tests that are in revscoring/features/tests/ [13:30:26] This is nice in revscoring since we have some tests that take a long time to run. [13:30:42] E.g. the model building tests and anything that requires NLTK. [13:31:07] It also places the tests close to the tested code. [13:34:34] I'm reading this to understand how we can write tests with nose [13:34:53] revscoring and ores use nose. [13:35:21] (actually all of the libraries I've built since 2010 use nose) [13:35:39] Since travis and other CI tools run nosetests, it is not hard to move tests to another folders [13:35:47] Indeed :) [13:35:47] I can test and let you know soon [13:36:18] but rewriting whole tests based to use nose [13:36:31] that is inefficient for me [13:36:36] I need to learn nose [13:37:03] once I learned, I start rewriting tests for practice [13:39:51] Amir1, I wouldn't mind taking a first pass. I'm extending the tests right now to catch pickling issues. [13:44:30] Amir1, can you help me find a revision wikidata that adds a language? [13:45:01] I knew one [13:45:05] let me search history [13:45:16] * halfak boosts test coverage slightly [13:48:04] And the pickle error is fixed :) [13:49:07] halfak: https://www.wikidata.org/w/index.php?title=Q2380760&diff=264056515&oldid=197154871 [13:49:11] I just made one [13:49:14] Woot! [13:49:21] Best test cases lead to contributions [13:49:55] it works! [13:50:55] Amir1, can you make me a collaborator for https://github.com/Ladsgroup/wb-vandalism? [13:51:45] halfak: you are already there [13:52:21] Oh! My bad. [13:52:24] PR coming soon. [13:54:02] :) [14:01:15] * halfak tests the tests [14:05:10] Amir1, https://github.com/Ladsgroup/wb-vandalism/pull/1 [14:07:36] halfak: merged [14:07:45] OMG you rewrote all tetst [14:07:50] :D [14:07:50] OMG [14:07:55] * halfak types fast [14:08:18] Also my editor is pretty powerful with regexes ;) [14:11:11] woot merge [14:11:45] Now to build a Makefile and encode my commandline params for building the model. [14:20:40] Amir1, BTW, I think that we ought to use a similar strategy to building a balanced sample for idwiki. [14:20:45] idwiki seems to be a botpedia [14:24:59] * halfak builds the models. :D [15:38:38] Ladsgroup/wb-vandalism#24 (pickleable - ed1149a : halfak): The build has errored. https://travis-ci.org/Ladsgroup/wb-vandalism/builds/87895474 [15:38:38] Woops. [15:38:38] Errors [15:38:39] Looks like we got up to 82 AUC. That's as good as our old enwiki model that Helder buil ScoredRevisions around. [15:38:39] So I think that's *probably* good enough. [15:38:40] LOL, github doesn't believe that my TSV is a tab separated file because it can't find any tabs. IT'S ONLY ONE COLUMN!!!! [15:38:40] Arg! [15:38:42] :) [15:38:42] halfak: technically it's also a CSV then [15:38:42] It is both and also neither [15:38:42] harej, so just pick one and format it! lol [15:38:42] * halfak tries to figure out where to file a bug. [15:38:43] You could just arbitrarily end each line with a tab? [15:38:43] Not sure. But that's at least an anti-pattern. [15:38:43] Should be fixed at the source. [15:38:50] * halfak sees the merges [15:38:50] :) [15:38:50] Woot! Amir1, we need to either incorporate the new wikibase features into revscoring or make wb-vandalism a dependency of ores-wikimedia-config. [15:38:50] I like the latter best. [15:38:50] So, that means we need to get a nice, clean version up on pypi. [15:38:50] Do you want to do that now? [15:38:50] let's have it in ores-wikimedia-config while it's developing [15:38:50] once it's finished [15:38:50] I suggest incrementing the version to 0.1.0 since I changed the module structure for "feature_lists" a bit. [15:38:50] sure :) [15:38:51] let me release the new version [15:38:51] Then we get to troll YuviPanda by telling him that we have a new dependency [15:38:51] Or... I suppose we could have another submodule. [15:38:51] That might be better. [15:38:51] :D [15:38:51] the latter sounds fine to me [15:38:51] no [15:38:51] a dependency is better [15:38:51] OK. Cool with me. [15:38:51] since submodule gets developing release [15:38:51] (we can have a branch dedicated to releases too) [15:38:51] but that would be tricky [15:38:51] Ladsgroup/wb-vandalism#27 (models_models - fdd3a5e : halfak): The build failed. https://travis-ci.org/Ladsgroup/wb-vandalism/builds/87900778 [15:38:53] Okay, it just released [15:38:53] releasing in pypi [15:38:54] https://pypi.python.org/pypi/wb_vandalism [15:38:54] Ladsgroup/wb-vandalism#28 (models_models - 4b6b90b : halfak): The build has errored. https://travis-ci.org/Ladsgroup/wb-vandalism/builds/87900821 [15:38:55] BRB, gotta shower and get ready for work stuff. I'll be back on in about an hour working to get the wikidata model onto staging. [15:38:55] Amir1, ^ [15:38:55] o/ [15:38:55] thanks halfak :) [15:38:55] It's great [15:43:06] Ladsgroup/wb-vandalism#30 (halfak-patch-1 - 7da78f8 : Aaron Halfaker): The build failed. https://travis-ci.org/Ladsgroup/wb-vandalism/builds/87901637 [16:15:43] halfak: remember, new dependencies gotta go through security review [16:15:58] and i really hope this doesn't touch pwb [16:16:04] Nope. [16:16:06] lol [16:16:23] I've got your back there. I insisted. Amir1 came through and split all the things we needed from pwb. [16:16:49] YuviPanda, also, I don't think this needs to go to prod right away. [16:17:04] In the long run, I think we'll have an ores-wikimedia-config and an ores-wmflabs-config. [16:17:28] The wmflabs one will pull in a lot of experimental stuff [16:17:53] * halfak doesn't want to give up rapid experimentation [16:17:57] Science and all that [16:18:43] Amir1, can you merge this and send a new version to pypi? https://github.com/Ladsgroup/wb-vandalism/pull/5 [16:18:57] Regretfully the install via pip doesn't work. See referenced bug. [16:19:23] YuviPanda, what do you think about the submodule vs. installation dependency? [16:19:53] Eventually, I want to manage the models independently via wiki-ai/editquality, wiki-ai/wikiclass and wiki-ai/wb-vandalism [16:20:20] Right now, I'm copy-pasting models. [16:23:22] * halfak watches Amir1 merging [16:23:26] halfak: done [16:23:28] :) [16:23:33] new version in pip too [16:23:33] Thanks [16:23:56] \o/ [16:25:25] Bah! Another problem. [16:25:37] find_packages() doesn't work when there's a missing __init__.py [16:26:34] https://github.com/Ladsgroup/wb-vandalism/pull/7 [16:26:46] Woops. Gotta increment version [16:26:47] One sec. [16:27:15] OK Pushed. [16:27:29] Amir1, ^ [16:28:42] halfak: depends on how tightly tied it is [16:28:56] halfak: if it is super tightly tied, submodule. else, library [16:29:04] done [16:29:10] uploading into pypi [16:29:41] YuviPanda, the repos I am talking about (editquality, wikiclass, wb-vandalism) will include (1) the models and (2) utilities for building and using the models. [16:30:01] e.g. http://pythonhosted.org/wikiclass/ [16:30:05] They are full packages themselves. [16:30:15] Complete with command-line utilities and stuff. [16:30:39] Further, they define "features" that will need to be importable in order to unpickle the model files that they build. [16:31:43] These packages use revscoring as a framework to build scoring utilities. [16:36:00] * halfak needs to build a diagram [16:37:04] halfak: right but I think a more defining line might be [16:37:25] halfak: 'if something in ORES breaks during a deploy, how probably is that the fix might be in this library?' [16:37:36] YuviPanda, quite probably. [16:37:40] for revscoring that's fairly high and we've been bit by it before and hence submodule [16:37:46] while it isn't that high for say sklearn [16:37:50] +1 [16:37:51] That's right [16:38:01] halfak: yeah, so if it's 'quite probably' then submodule [16:38:17] Gotcha. I'll try to get that worked out. [16:38:21] I hope that makes sense as a guideline and is a fairly clear line in the sand [16:38:32] I think it is clear. Thank you :) [16:38:44] yw [16:39:32] Amir1, can we move the wb-vandalism repo to wiki-ai? [16:39:43] of course :) [16:39:44] If you'd like it to stay in Ladsgroup, that's cool too [16:39:56] I just want to not have to update the submodule location later :D [16:40:18] Cool! You can make the move in the settings for the repo [16:40:32] I need to setup gerrit mirrors at some point [16:40:39] I already moved BWDS to wiki-ai [16:41:01] halfak: ok, csteipp says it's mostly done and should be back today or tomorrow [16:43:09] halfak: https://github.com/wiki-ai/wb-vandalism [16:45:43] YuviPanda, great! Thanks :) [16:45:57] * halfak copies new github link [16:46:00] Thanks Amir1 [16:48:42] yw [16:48:46] Thank you :) [16:49:18] Amir1, we've been using DBNames to represent wikis. [16:49:27] But wikidata's dbname is 'wikidatawiki' [16:49:36] shall we make an exception and just call it 'wikidata'? [16:49:48] Same thing with 'commonswiki' [16:49:55] and 'mediawikiwiki' [16:50:15] mediawikiwiki is just priceless [16:50:21] lol [16:50:33] I don't think we need an exception [16:50:40] You want to keep 'wikidatawiki'? [16:50:46] it's unnecessary [16:50:48] yes [16:50:52] OK. [16:50:55] :) [16:51:27] This is good anyway. [16:51:34] Gadgets will be able to use the dbname var. [16:51:44] Maybe we can include some aliases too. [16:51:59] Feature requests for ORES [16:52:02] maybe [16:52:22] :D [16:54:38] The model thinks that this looks like vandalism: https://www.wikidata.org/wiki/?diff=10000 [16:54:43] Not that confident. [16:54:45] 70% [16:55:01] Let me try to get one with a higher percentage. [16:56:06] What about this one? https://www.wikidata.org/wiki/?diff=220328471 [16:56:51] Uh oh. I get an error for that one "TypeError: Failed to process : 'set' object is not subscriptable" [16:56:57] Let's figure that out! :) [16:57:12] * halfak loves the collection of utilities for revscoring. [16:57:20] Makes it easy to replicate this on the commandline [17:02:02] Amir1, looks like the "claims_differ" can return None when 'added()' is called [17:02:05] Here: https://github.com/wiki-ai/wb-vandalism/blob/master/wb_vandalism/datasources/diff.py#L86 [17:03:19] It shouldn't [17:03:30] Yeah... Trying to figure it out myself. [17:03:40] I think I fixed this in my repo [17:03:46] let me try to fin it [17:03:49] *find [17:04:50] yeah [17:04:55] in my repo I fixed it [17:05:01] it should be added_claims += current_item.claims[p_number] [17:05:29] I will fix it and push it ASAP [17:05:41] halfak: should I release a new version? [17:05:57] Add this revid to the test case: 220298625 [17:06:01] Make sure the test passes first [17:06:03] And then yeah. [17:06:15] See anywhere else that we might be making this mistake? [17:06:22] exactly [17:06:32] Also, maybe we need to extract features again? [17:06:39] How did this feature extraction work in the first place? [17:06:46] I apparently have feature for this revision. [17:07:23] Oh wait. no I don't. [17:07:43] Woops. Wrong rev_id [17:07:52] This one: 220328471 [17:09:00] Heh. You've got a print() in the feature_list too. [17:09:07] line 76 [17:09:09] :P [17:12:11] Here are the feature values I get for that revision: https://gist.github.com/halfak/c0f66298dca427bd5c40 [17:12:21] "number_added_claims" == 1 [17:13:21] Ladsgroup/wb-vandalism#36 (manifest - 95ea07a : halfak): The build failed. https://travis-ci.org/Ladsgroup/wb-vandalism/builds/87926297 [17:15:51] >>> list(extractor.extract(220328471, [diff.number_changed_claims])) [17:15:53] [0] [17:15:56] it's okay now [17:15:59] I'm pushing [17:16:15] I fix the features_list [17:18:22] halfak: RTRC got ORES support today apparently [17:18:25] pushed [17:18:26] halfak: I've invited Timo here [17:18:43] halfak: Uploading to pypi [17:19:11] What's the difference between 'Revscoring' and 'Ores' on Phabricator? [17:19:25] hi Krinkle [17:19:38] Hi Krinkle! ORES is one of our systems [17:19:39] I think ORES is the web API and revscoring is the underlying library [17:19:42] So it's a tag. [17:19:53] Revscoring is the project name (and also one of the systems) [17:20:05] In contrast, Wiki labels is another system we maintain. [17:20:15] I believe we have a wikilabels tag. [17:20:32] So what's the tree relation between ores, wiki labels and revscoring (for Phabricator purposes) [17:20:48] siblings, or two childs of revscoring [17:22:32] I've added the above to the descriptions of those three projects. [17:22:36] Two childs of revscoring [17:22:39] Thanks [17:23:31] So in which one do I request a task to work on adding support for this stack to another wiki? [17:23:34] revscoring? [17:23:53] revscoring is great. We'll do labeling. :) [17:26:20] Amir1|afk, it works! [17:26:29] * halfak continues testing other revisions [17:26:34] Hm.. no phab wikibugs here? [17:27:10] phab wikibugs? [17:27:57] New errors with the wikidata features for this revid: 220328478 [17:28:22] halfak: I mean, to have wikibugs (as in #wikimedia-dev and other channels) report relevant task activity here. [17:28:30] halfak: it's a bot that announces changes to phab tickets [17:28:48] Oh yeah. We already have too much spam in this channel [17:28:56] Any time i work on revscoring or ores it's aweful. [17:29:02] I kind of want to turn that off, honestly. [17:29:13] Maybe we can just quiet it down. [17:29:14] it's called /ignore :P [17:29:26] By why should everyone need to ignore it. [17:29:37] wikimedia-dev is a wasteland from all of the bothavior. [17:29:49] I kinda agree with halfak [17:29:54] It's disucssed every other year. For a lot of folks it helps increase visibility, collaboration and productivity [17:30:12] yeah it's a fairly deep split, I guess. [17:30:15] Maybe I just need to dig into the config. [17:30:25] But for others, they may want to ignore it. It depends on your workflow. [17:30:43] I wouldn't mind a message for every merge, bug report, new phab task, resolved phab task. [17:31:15] But a message for "halfak committed something to a random branch" is too much [17:33:41] * halfak runs off for a quick lunch and then I'll be back to hacking on wb-vandalism to get these feature issues cleaned up. [17:35:48] halfak: I filed https://phabricator.wikimedia.org/T116939?workflow=create [17:56:14] wiki-ai/wb-vandalism#39 (manifest - 6aa25e6 : halfak): The build failed. https://travis-ci.org/wiki-ai/wb-vandalism/builds/87928883 [18:00:06] halfak: hey, I'm back. Everything is ready? [18:06:44] wiki-ai/wb-vandalism#41 (manifest - e8df4a6 : halfak): The build failed. https://travis-ci.org/wiki-ai/wb-vandalism/builds/87929184 [18:13:45] Amir1, still working on the last bug. [18:13:50] Check the issues for wb-vandalism [18:14:06] I was mostly out to lunch, so I haven't looked at it much yet. [18:15:43] It looks like the error happens when there is no parent revision (e.g. page creation) [18:15:58] Amir1, what did you pass as parent_revision.text when processing the XML dumps? [18:17:22] halfak: haha great response on the bug :D [18:18:16] lol [18:18:25] I'm going to get a bunch of "Hi" back now [18:18:40] halfak: yeah already got one :P [18:28:25] halfak: text in parent revision [18:28:43] For page creation [18:28:53] When there's no parent [18:29:41] I'm working on a PR that sets the past_item to None when there is no past item. [18:29:49] Then I'm handling the None case in the datasources [18:30:08] I defined a default [18:30:14] What was the default? [18:30:18] let me show it to you [18:30:29] I tried to pass an empty dict to item.get() as content and that failed [18:30:38] But it works if I pass {"hi": 1} [18:30:40] ha [18:31:08] Maybe if you pass nothing? [18:31:09] defualt_item = {"type":"item","labels":{},"descriptions":[],"aliases":[],"claims":[],"sitelinks":{}} [18:31:15] Ahh. [18:31:21] json.dumps(defualt_item) [18:32:02] No badges? [18:32:07] Is that a list by default? [18:35:25] badges are part of sitelinks [18:35:43] each member of sitelink is dictionary [18:35:57] *is a dictionary [18:36:01] Gotcha. [18:38:28] wait to see travis builds are passsing [18:38:39] halfak: have you tested PR 9? [18:39:18] Amir1, yeah. Just added a test case to make sure it is all working. [18:39:34] great [18:39:46] One we resolve https://github.com/wiki-ai/wb-vandalism/issues/2 we can make sure to explicitly test all cases where parent_revision doesn't exist. [18:40:08] We'll probably want to use miniconda to get around our travis CI issues. [18:40:14] Since revscoring is a dependency. [18:40:21] I could take a look at that quick. [18:42:05] Do we use any of the language-specific stuff? [18:43:46] no [18:43:48] AFAIK [18:44:15] Cool. I'll try skipping those in travis. [18:45:58] merged :) [18:46:23] do you want a new release? [18:47:08] Maybe. let me do some more testing of random revids [18:49:36] lol missed one. [18:49:44] Looks like we're erroring later now! [18:52:46] Works now [18:52:47] :) [18:52:53] Let me push [18:54:23] Amir1, https://github.com/wiki-ai/wb-vandalism/pull/11 [18:59:22] Travis is so slow! [18:59:49] https://shogofawafa.files.wordpress.com/2015/08/tumblr_npnpkttr9b1tqtfrjo1_500.gif?w=700&h=438 is my favorite gif now [18:59:54] wiki-ai/wb-vandalism#45 (parent_revision - db943cf : halfak): The build failed. https://travis-ci.org/wiki-ai/wb-vandalism/builds/87957297 [19:00:16] yeah [19:01:11] Let's not wait :D [19:01:37] I've got that other PR that is intended to fix travis and I'm waiting for it to even make an attempt! [19:01:54] c'mon travis, ya lazy bum! [19:02:25] I suppose this is prime development time while the both coasts of the US and Western Europe are awake and working. [19:06:58] OK. This is looking pretty good. [19:07:11] I'm going to try to send it to staging. [19:08:21] Amir1, did you push a new version to pypi yet? If not, could you? [19:08:56] * halfak is having a lot of fun getting WikiData vandalism detection online :) [19:09:58] halfak: sure [19:11:10] I had a presentation a month ago about Wikipedia in Tehran SFD. I talked about you. This is video of that presentation. In the mean time try to find that part :D [19:11:27] * halfak waits for link [19:11:30] halfak: https://www.youtube.com/watch?v=3BNNu_hh74k [19:12:21] Audio is weird. I can't understand what you are saying ;) [19:13:39] Cool that I can mostly make it out by your slides :) [19:14:27] :D [19:18:00] Amir1, I like your presentation style. Few lists, lots of props (slides are photos/screenshots/examples) and (what sounds like) a conversational style. [19:18:58] thanks :) [19:19:27] I particiated at a lot of good presentations and tried to learn specially in Wikimania [19:19:33] *participated [19:19:34] Maybe we should have you give the next revscoring presentation at :) [19:19:59] :D [19:20:40] This is super high quality video/audio too. You think we could get this kind of thing at Wikimania next year? [19:20:43] heh [19:21:09] Tehran SFD was held in best university in Iran, Sharif [19:21:32] in the main auditorium [19:21:57]