[02:00:05] Helder: if you get those, please file bugs in phabricator so someone can look up the offending query [02:00:29] I'm getting lots of them [02:00:57] on the same types of queries? or different? [02:04:59] Apparently, all the same [02:07:45] legoktm, https://phabricator.wikimedia.org/T121333 [02:09:15] Helder: thanks. I added the MW-API project and a traceback so anomie will probably take a look on Monday [02:13:29] thanks! [14:24:54] halfak, any ideas why my trained model could have such a low accuracy? http://dpaste.com/3X6RHFT [14:25:17] (and also a smaller ROC-AUC) [14:26:51] PS: I didn't make any changes to revscoring before training [14:46:09] Hey Helder [14:46:26] So, not sure. There's a little bit of randomness involved. [14:47:08] So it could be that the train/test split was less favorable. You could try tunning against `tune` to get a better send for what the range is. [14:47:52] Still, I'm surprised to see such a wide difference in AUC. [14:48:17] Oh!!! [14:48:42] Wait... no. I thought I had an idea, but that doesn't make sense after all. [14:48:56] * halfak checks out the branch [14:50:32] Ok. I was actually right. It looks like the last commit to master was during our transition re. user-features. [14:50:52] Your model lacks the very useful features "user.age" and "user.is_anon" [14:51:04] But slightly problematic too. [14:51:36] I have a branch where I an creating both models -- models that include these user dimensions and ones that do not. [14:56:35] Helder, I just started up a job on ores-computes.labs to make the right models for comparison. [14:57:53] When I compare to what I have for the same feature set, it looks like you're doing a little better. [14:58:46] For the no-`user.age'-`user.is_anon` damaging model, I get http://dpaste.com/08Z9454 [14:59:05] I'm rebuilding the model that includes user.is_anon right now. [15:14:02] o/ Amir1 [15:14:08] I'm preparing for that deploy. [15:14:11] o/ halfak [15:14:15] :) [15:14:17] Awesome [15:14:29] let's check what's good and what's bad [15:14:34] I still haven't trained the wikidata rf model, but at least I can get the testwiki fake model up. [15:15:02] BTW, I was thinking about this test model. I think that we should reverse the order of the last two digits so that we can get high probabilities earlier. [15:15:11] e.g. 325343457 == 75% probability [15:15:19] or 9 = 90% [15:15:28] 91 = 19% [15:15:31] good point [15:15:33] 19 = 91% [15:15:34] +1 [15:15:36] kk [15:15:38] Will implement [15:15:47] I'm currently trying to do some edits [15:15:57] by bot [15:16:56] Did you get BAG approval ;) [15:17:02] * halfak imagines a testwiki bag [15:18:13] It works! [15:18:21] On to the next thing. [15:19:16] This new extractor needs no internet connection to function. [15:19:28] Not even a connection back to the wiki [15:19:31] :D [15:21:09] Amir1, https://github.com/wiki-ai/revscoring/pull/229 [15:21:38] :D :D [15:22:37] amazing [15:22:39] thanks [15:24:02] Woops. Don't merge yet. [15:24:04] I have issues. [15:24:07] Amir1, ^ [15:24:18] merged already [15:24:20] :D [15:24:25] Oh well. Can fix in a minute [15:27:41] Arg. ORES has it's own revscoring still. [15:28:01] * halfak self-merged https://github.com/wiki-ai/ores/pull/108 [15:30:55] Yup... looks like I do have an issue in the fromconfig of the new extractor. [15:30:56] boo [15:32:13] wiki-ai/revscoring#391 (text_extractor - cb49f7f : halfak): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/96583529 [15:32:31] Shuddup travis [15:33:39] wiki-ai/revscoring#392 (text_extractor - f29894e : halfak): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/96583673 [15:33:47] http://tools.wmflabs.org/ores/mediawiki/index.php?title=Special:RecentChanges [15:33:50] halfak: ^ [15:38:10] Arg. This is annoyingly more complex than expected. [15:51:19] halfak: should I merge #230? [15:51:27] waiting for adam [15:51:34] Yup. It's ready to go and tested :) [15:51:45] Working on reversing the digits now. [15:53:54] awesome [15:54:00] I need to go [15:54:06] be back in five or six hours [15:54:11] (going sleep) [15:54:25] OK. Will be ready for your testing [15:54:26] o/ [16:10:55] * halfak updates wikiclass for the new version of revscoring [16:22:37] HA! http://ores-staging.wmflabs.org/scores/testwiki/reverted/32234234/ [16:22:40] Woooooo [16:22:56] http://ores-staging.wmflabs.org/scores/testwiki/reverted/ [16:41:32] Boo... I cant upload a new version of wb-vandalism to pypi. [16:41:39] Will try to see if I can do this anyway. [16:43:23] * halfak crosses fingers [16:44:06] also, i've just got to reflect quickly... It's pretty awesome to have makefile and unix utilities to work with. It took me no time to build this new model. :) [16:45:59] http://ores-staging.wmflabs.org/scores/wikidatawiki/reverted/?revids=264410816|262421685|261592403|259275765|258928880 [16:46:12] Scores look pretty good for water (Q283) [16:46:27] The only edit with a high score is changing the image. That seems like a good edit to review. [16:47:14] Hey YuviPanda [16:47:19] Around? [16:55:54] wiki-ai/wb-vandalism#118 (wikidata_tuning - 197702d : halfak): The build has errored. https://travis-ci.org/wiki-ai/wb-vandalism/builds/96592197 [18:16:50] o/ ToAruShiroiNeko [18:17:58] * Helder_ needs coffee [18:18:09] Hey Helder! [18:18:22] I'm just waiting for the build before I merge your new PR. [18:18:34] * Helder presses F5 [18:18:45] It'll be nice to re-run the model tuning with the new badwords :) [18:18:53] BTW: is there a way to test the build locally? [18:19:07] I didn't see any improvements :( [18:19:28] As in no improvement to AUC? [18:21:47] Helder, ^ [18:23:04] yep [18:24:18] You probably missed my one-way conversation with myself earlier. [18:24:21] * halfak gets a copy-paste [18:24:53] Helder, http://dpaste.com/3BVC9TQ [18:25:25] So I think you *are* doing better. [18:25:52] But we might get even more with model tuning since we tend to get better performance with GradientBoosting and RandomForest. [18:25:57] So I want to do some tuning. [18:26:31] Woops. Looks like a test failed. [18:26:36] Helder, ^ [18:26:52] Looks like the regex is not catching "boiolão" [18:27:25] Horray for test cases! \o/ [18:27:58] halfak, here is the new info: http://dpaste.com/02K9QR6#line-47 [18:28:27] I went from 0.867 to 0.842 [18:28:36] Oh! Gotcha. [18:28:52] Still, this could be a problem that turn frequency will help us deal with. [18:29:30] It makes it less troublesome to have an inclusive badwords list. [18:29:39] so "my-first-trained-models" doesn't include the new badwords [18:29:40] ? [18:31:35] yep [18:35:14] Helder, yeah. That's interesting. [18:35:23] I'm surprised that it isn't better. [18:35:34] I still think we should merge it. [18:35:44] Once the tests pass :) [18:35:46] yeah [18:35:56] * Helder waits [18:37:44] Helder, tests passing locally? [18:37:54] don t know [18:38:05] You should be able to run them in the branch. [18:38:17] nosetests? [18:39:01] yup [18:39:28] er... nope.... they dont pass [18:39:31] :( [18:39:33] * Helder fixes [18:39:38] :D [18:39:42] Tests are good. [18:39:52] I like seeing tests fail. [18:40:07] I get nervous when I see them all passing unless I've just gone and fixed a bunch of things. [18:44:17] Ok, I think the tests are ok now... the only errors I get are those from myspell [18:44:46] Those are the random "hunspell..." stuff, right? [18:45:04] Seriously annoyed at how that stuff pollutes our stderr. [18:59:12] halfak, things like "ERROR: Failure: ImportError (No enchant-compatible dictionary found for 'vi'. Consider installing 'hunspell-vi'.)" [18:59:52] I don't want to install them because it makes my life worse (a lot of unwanted items in the browser menu for spell checking) [19:00:26] Oh yeah. No worres. [19:00:39] Those *should* work on travis though [19:01:58] "All checks have passed" [19:02:00] :D [19:02:35] Yay! [19:03:05] {{merged}} [19:16:04] halfak, at least it is more confident that this edit is damaging the page [19:16:04] https://pt.wikipedia.org/w/index.php?diff=44194532 [19:16:08] http://dpaste.com/1X3H63G [19:23:57] \o/ [19:24:10] It's worth noting that our metrics aren't everything. [19:24:15] They are just a number. :) [19:25:28] Ha. I should really set up a default user agent for the score utility. [19:28:00] o/ ellery [19:35:45] halfak: sure; I'm wrapping up something, start writing and I'll catch up in a few minutes? [19:37:22] Uh oh. Looks like we're splitting. [19:37:25] * halfak tries to hold the channel together [19:37:37] Sure. Thanks :) [19:37:38] * halfak gets links [19:37:49] This is the best place to start: https://meta.wikimedia.org/w/index.php?title=Research:Revision_scoring_as_a_service/Word_lists/fr&action=edit [19:37:50] We have a list of words that are common to reverted edits that you can start with. [19:38:11] We'd like the list sorted into "informal" words and "badwords". You can see that we have some templates on the page for putting those two lists. [19:38:33] We'll use these lists to write a set of regular expressions that we'll apply to edits in frwiki [19:38:56] But don't do the regexes now. We need the real words for our test case to make sure the regexes work! [19:39:16] You should feel free to supplement our generated list with all of the variants you can think of. [19:40:06] E.g. "hi" is an informal in English. I have test cases for "hi", "hihi", "ha", "haha", "hehe", "ho", "hoho", "hu", "huhu", "muhaha", [19:42:16] Something else came up so I'll look at this in ~10 minutes instead. Will ping you :) [19:49:26] guillom, no worries. I'll be around when you're ready :) [20:01:42] Alright, I'm back. Looking at the page now. [20:02:15] hi halfak [20:02:23] o/ YuviPanda [20:02:30] my building was on fire yesterday so I might not be around for long today [20:02:38] Holy crap! [20:02:40] You OK? [20:02:43] Your stuff OK? [20:02:50] yeah [20:02:59] my appt was only filled with toxic smoke [20:03:03] I got out in time [20:03:11] !!!! [20:03:13] the one below ours is completely trashed [20:03:13] Yikes. Did you wake up to it? [20:03:19] YuviPanda has the craziest stories. [20:03:21] Glad you're ok. [20:04:05] halfak: no, I had woken up 30mins before it but was just sitting in my room writing code [20:04:16] this was at 8:30 pm tho... [20:04:20] "Writing code saved my life." [20:04:37] At least prevented an early onset of lung cancer. [20:04:41] Anyway, I'm glad you are OK. [20:04:50] I wanted to do a deploy today and have you nearby. [20:04:54] Should we wait until Monday? [20:04:58] YuviPanda, ^ [20:05:13] heh [20:05:23] how complex do you expect it to be halfak [20:05:31] I can be around for about 25mins more now [20:05:38] Not at all, but there's a lot of new versions. [20:05:47] I think we can do it in that time. [20:05:51] let's do it then :) [20:06:01] thanks guillom / halfak :) [20:06:07] I had also just broken puppet on two hosts [20:06:10] ori pointed that out [20:06:20] and I had to go 'I think my building is on fire, can you fix that? sorry!' [20:06:34] btw YuviPanda I drafted https://office.wikimedia.org/wiki/User:Guillaume/The_bright_future yesterday, thought you might want to know about it. [20:06:35] talking about your building being on fir on IRC seemed straight outta bash.org [20:06:46] ah, officeiki [20:06:49] *officewiki [20:06:50] "I think my building is on fire" -- Lamest excuse ever. [20:06:51] :P [20:06:54] :D [20:06:57] I need to reset my password on it again... [20:07:09] "My building /is/ on fire" however, is a great excuse :p [20:07:30] YuviPanda: Basically, "let's replace mailman by discourse" [20:07:43] guillom: ah. Did you see me flame out on -staff earlier? :) [20:07:55] I did not. [20:08:14] Our hours don't overlap much :p [20:08:28] like a few days ago. I think I don't have any energy left to push things like this in the face of consistent 'but if I could figure this out when I was 12 clearly other people should!' and 'it is not our problem that people do not use tech X that I clearly find easy' [20:08:35] (He says, days after they saw each other on the street, when one was leaving work and the other was getting there.) [20:08:41] guillom: haha, yes. I remember me walking towards the office as you were walking out [20:08:46] :D [20:09:11] YuviPanda: We're in for the long haul; we'll be victorious! (eventually) [20:09:25] * halfak starts deploy and monitors celery workers [20:09:26] anyway [20:09:49] halfak: So, I can split the list into informal and truly bad words, but I'm not sure what template to use. [20:09:59] guillom: :D yeah, I hope so. I guess I just burnt (HAH!) out on that one for now at least! [20:10:09] guillom, if we get some lists from you, I can work out the details. [20:10:14] ok [20:10:24] The page should already have the template fields on it though. [20:12:36] Ah, I need to add |list-informal= [20:12:39] guillom, I see the problem. If you add a field for "list-informals" and the supplement the list for "list-badwords" [20:12:45] Yeah. [20:12:48] Not plural [20:12:50] You have it right [20:13:07] So... a lot of copy/pasting around. /me gets mouse :) [20:13:19] trackpad isn't ideal for this [20:13:42] Yeah. Sorry about that. I don't know a good strategy for doing this. It would be great if we could provide you an UI with check boxes or something like that. [20:14:19] \o/ our new models are online and everything seems to be OK [20:16:34] It's fine, I just need to adapt because this armchair is very comfy but not idea for mousing :) [20:16:43] Alright, editing now. [20:17:05] I'm probably going to do this in a split text editor. [20:17:12] +1 for that. [20:17:23] Oh! One other thing that would help. If you could provide a rough english translation, that would be very helpful for curating this list later. [20:17:59] ouch. That's a lot of translations. [20:18:05] See this recent pull request for portuguese -- https://github.com/wiki-ai/revscoring/pull/228/files [20:18:12] No sweat! I can source that later. [20:18:29] It's way less important than getting the lists together in the first place. [20:18:35] o/ AsimovBot [20:18:39] [[ORES]] [20:18:56] Daw. It didn't work. [20:19:11] YuviPanda, all confirmed. We're in the clear. [20:19:15] \o/ [20:19:18] Thanks for being available. :) [20:19:32] our deployments are more like trains than rocket launches now [20:19:36] maybe not fully, but getting there :D [20:19:43] Wikidata now has a *super* high fitness model :) [20:19:49] And we have a testwiki model [20:19:57] I'm really proud of the testwiki model. [20:19:58] :) [20:20:20] halfak: so, I have a problem with the very first word... "Anus" is not a "bad" word per se, nor a common word, nor an informal word. [20:20:55] guillom, I agree. If you find something that doesn't belong in the list, ignore it. [20:21:10] The machine will benefit from your judgement. [20:21:20] halfak: ignore it = remove it from any list? [20:21:35] guillom, leave it in the generated list, but remove it from the badwords list. [20:21:39] gotcha [20:21:43] thanks [20:21:52] :) no. thank you! [20:21:53] :D [20:21:53] 04https://www.wikidata.org/wiki/Special:ItemByTitle// [20:22:10] o/ Guest77692 [20:23:42] halfak: Sorry, another question. If a word like "gay" is in the "badwords" (where it doesn't belong) but not in the generated, what do I do? [20:24:23] I'm going to go afk now [20:24:32] guillom: <3 for discourse / mailman alternatives! [20:24:44] guillom, add it to the badwords. [20:24:51] * YuviPanda waves at everyone and goes afk for real [20:25:03] Is "gay" a french word too or is it an english word that people were probably using for vandalism? [20:25:05] halfak: feel free to call me if you need to reach me later today [20:25:15] It's a French word too. [20:25:16] Have a good one YuviPanda [20:25:29] guillom, good to know. Thank you! [20:25:39] halfak: So it shouldn't be a bad word, right? [20:26:15] guillom, this one is a judgement call. With words like this, I think it's more commonly used as curse of some sort. [20:26:35] https://meta.wikimedia.org/wiki/Friends_of_gays_should_not_be_allowed_to_edit_articles [20:26:49] Even the most direct curse words have some appropriate use. [20:27:18] We're developing strategies for counting the frequency of these words that make it less likely we'll predict vandalism if the word is already used on the article. [20:27:27] In this case, I think that "gay" should be a badword. [20:27:47] If the article uses "gay" a lot and an editor adds another instance, that will be downweighted substantially. [20:27:47] Hmmm. I see. [20:29:10] Ok. However, I think the template should say explain that "bad" words aren't necessarily "bad". Because I'm uncomfortable saying that "gay" or "homosexuel" belong in "Bad words are words unwelcome on any page. This would include curse words, spam and other content that would be reverted regardless of where it is inserted." [20:29:32] That's a good point. [20:30:19] :) [20:31:15] It should say, "Bad words are words that would be commonly associated with vandalism. They are generally used to insult or be vulgar. This includes curse words, racial slurs, assertions of- and prejudices against sexual preferences." [20:31:33] halfak: OK, so I think I'm almost good to go. My last question is: when I put a word from "generated" in "informal", do I remove it from "generated" or so I leave it there and /also/ put in in "informal"? [20:31:47] Leave it there. [20:31:54] ok! [20:31:57] :) [20:34:21] "#hamster". lol [20:34:24] Poor hamsters. [20:34:58] Poor Joconde, too. [20:35:17] I keep seeing Deskana appearing in these lists. [20:35:21] Not sure what that is about. [20:37:16] Deskana was one of the targets of a persistent vandal a few years ago IIRC. [20:37:50] It's funny how this shows up in our analyses. [20:38:01] The history of Wikipedia vandals is still trolling us :) [20:50:01] I'm still wondering what the hamsters did :D [20:52:08] Also, it's interesting that "alternative" spellings of bad words are in the "generated" list, but not the "proper" spelling. [20:52:14] Not always* [20:52:58] guillom, that's very good to know. Could you make sure that both the alternative and proper spellings make it into your badwords list? [20:53:21] It would be great if you could put on your vandal hat (everyone has one of those, right?) and imagine other alternatives that you think we should catch. [20:53:35] * halfak wonders what a vandal hat would look like. [20:53:36] halfak: ok; I just saved my edit (sorting); I'll try to add alternative/proper spellings now. [20:53:44] great. Thanks :) [20:58:23] done [20:58:38] * guillom doesn't have a lot of imagination for alternative spellings. [20:58:48] \o/ thanks guillom [20:58:55] This is a huge help for our quality models. :) [20:59:06] sure :) [21:00:22] Just made a card and assigned it to myself to incorporating your work into the code. See https://phabricator.wikimedia.org/T121358 [21:00:35] I think I'll be able to get to that tomorrow or Tuesday. [21:00:46] I'll let you know how it affects the fitness of the model :) [21:01:02] Thanks! [21:01:11] hah, interesting... [21:01:24] I was looking at how you rephrased the "bad word" description [21:01:33] Do you think it's OK? [21:01:33] (thanks, by the way!) [21:01:35] yes [21:01:37] :) [21:01:38] but it got me thinking [21:01:56] the fr list doesn't have anything about racial slurs [21:02:07] That *is* interesting [21:02:16] we have plenty of sex/lgbt related words [21:02:29] but nothing like "jap" or "nigger" like on the en list [21:02:37] Presumably there *are* common french words used as slurs, right? [21:02:41] yes [21:02:44] Just maybe the vandals don't use them [21:03:12] yeah. It was just interesting to see the difference. [21:05:57] Alright, I'm going to go work on something else. I'm around if needed :) [21:06:22] OK. Thanks again for your help. :) [21:06:28] Happy Sunday :) [21:06:35] You too! [21:35:19] ToAruShiroiNeko, did you want to chat about how to track out datasets/models at some point today? [22:36:56] halfak sure [22:40:47] So, every time we do a deploy to prod, we log it with a labs service so that it will be recorded on the wiki. [22:41:14] It seems like we should have a checklist that could include uploading datasets to a directory. [23:01:59] * aetilley is in bed with the flu. Wash your hands. [23:02:45] aetilley, oh no! I hope you feel better soon :/ [23:02:50] Take care of yourself. [23:03:03] thnx [23:03:12] * halfak sends immune system thoughts aetilley's way [23:05:25] lol [23:27:25] [[ORES]] [23:27:25] 10[1] 1010https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service - Redirección desde 10https://meta.wikimedia.org/wiki/ORES?redirect=no [23:27:30] It works again!