[00:01:51] halfak: Around? [00:02:04] first of all, read this: https://www.wikidata.org/wiki/Wikidata:Project_chat/Archive/2015/06#Revert_analysis [00:02:39] Is it okay for an AI tool to be racist? Maybe we should investigate adding some countries [00:03:41] another thing, Do you know when we can deploy damaging model in ORES for fa.wp? people are asking for that [00:17:05] halfak: does the quality labelings correspond pages with wikiproject names? [00:18:23] also, generally, AIs shouldn't be racist. [00:19:59] :D [13:02:30] o/ Amir1 [13:02:32] logging in [13:02:50] to skype [13:03:01] hey [13:03:06] I'mm coming to skype [13:03:30] halfak: Amir1 I'm in the WMDE office! [13:04:21] one sec, I'll be there [13:04:27] YuviPanda: cooooool [13:04:45] i work for them, I'd never been there :D [13:05:32] YuviPanda, cool! [13:05:37] Say Hi to Birgit for me :) [13:09:55] I have an endpoint for WikiBrain! (At long last) [13:09:59] http://shilad.github.io/wikibrain/tutorial/web-api.html [13:10:55] Ran into some memory issues that set me back a day or two, but it's up on my own server. [14:16:22] Amir1, why did you decide not to use any ANN libraries for python? [14:21:42] Imagining the UI... How would you manage the training set for new predictions? [14:21:50] for several reasons, labs is hard on dependencies (installing scientific dependencies is a pain the ass), I wanted to learn how ANN works, [14:22:00] Gotcha. [14:22:05] * halfak shakes fist at labs [14:22:16] But seriously, people should learn virtualenv :) [14:22:28] And virtualenv should be easier to use [14:22:32] and pip! [14:22:34] I use virtualenv [14:22:36] and pip [14:22:51] but e.g. you can't do pip3 in labs [14:23:02] (inside virtualenv) [14:23:13] Yeah. You should create the virtualenv for python3 and then use its pip. [14:25:25] waaah, labs is hard on dependencies what? [14:25:31] you can do virtualenv! [14:25:33] and python3 [14:25:41] virutalenv -p python3 [14:25:42] works fine [14:25:44] A wild yuvipanda appears [14:25:47] :D [14:26:02] Amir1, so what about training new models. [14:26:25] wow [14:26:28] If you provide a UI, I'll need to provide something in order to give it some observations. [14:26:34] YuviPanda: you should things like this somewhere [14:26:42] You definitely should [14:26:44] ;0 [14:26:46] hmm [14:26:51] it's just the standard python3 virtualenv way [14:27:01] Yeah, I'm just saying things. :D [14:27:04] e..g python3 doesn't work directly in jsub, you need to add -l release=trsuty [14:27:07] you told me [14:27:15] ah yes.... [14:27:23] our docs suck [14:27:24] I didn't know this for a very long time [14:27:40] we're very overstretched, though :( It's basically me and Valhalla and Tim now... [14:27:53] halfak: Kian goes and gets the observations [14:27:55] Tim? [14:28:06] halfak: Tim L [14:28:09] scfc [14:28:22] gotcha [14:28:37] Amir1: do edit and add those! A 'python3 for toollabs' faq would be great :) [14:28:53] yeah [14:29:05] no one can spell his family name [14:29:07] like mine [14:29:28] YuviPanda: I added to my todo list [14:29:43] thanks [14:29:56] Support the devs (by SOFIXIT)! [14:31:57] I appreciate your YuviPanda in labs, keep doing the job [14:38:08] halfak: I forgot to say, I made some progress on diff features in wb-vandalism [14:38:35] Get that card going :) [14:38:42] Also, which ones? [14:39:19] I used your diea [14:39:20] *idea [14:39:25] let me show you the commit [14:39:28] :D thanks [14:39:50] https://github.com/Ladsgroup/wb-vandalism/commit/a7349a7e2d4239e5c8c369bf40d658e95b1ea0f4 [14:40:35] * Amir1 is mourning for Flow [14:41:41] Yeah. Too bad for Flow. I think that project barely got a start. [14:42:10] I mean... it got a start, but not a chance on-wiki. [14:42:15] Honestly, I like FLow a lot. [14:43:26] It definitely needs lots of improvements but it could make a better experience in communicating with others [14:43:36] it's LQTv2 now [14:43:51] Yeah. [14:44:17] unfortunately [14:44:33] but apparently it's going to be available as a beta feature [14:44:38] legoktm: EXPLAIN TO US WHAT HAPPENED TO FLOW [14:45:40] I was prepping an RFC to get flow enabled on the Research namespace on Meta [14:46:01] yeah [14:46:03] that can still happen [14:46:41] Yeah. Unmaintained talk system? I'm not sure I want to anymore. [14:46:53] Then again, it's not like wikitext talk pages are "maintained" [14:47:24] I made that RfC for fa.wp today just in case we had any strange RTL bug, we can catch while we have development resource [14:48:05] halfak: does your thing that pulls quality ratings from talk pages also associate articles with WikiProjects? [14:48:51] heh [14:48:55] harej, yes [14:49:18] Is this information available over an API? [14:49:46] harej, negative. But I could send you the dataset [14:51:20] I mean I *also* have that dataset. But presumably you're a real programmer and your approach works better than mine. [14:52:46] (The harej approach is called project_index.py and it sucks) [14:53:13] I'd use categories generally, but in this case, I process the template name to get the project name. [14:53:50] Doesn't always work! Some projects hitch a ride on other projects' templates [14:54:05] They do. And so they get bad data. [14:54:10] :) [14:54:19] Shouldn'ta done that [14:54:49] I don't need the quality rating, just the association of projects and pages. My script generates that list by accounting for as many setups as possible. [14:56:05] Gotcha. I don't care to get the edge cases. [15:04:09] Do you have any news from the storyteller halfak? [15:04:17] I really need those images :P [15:04:27] Amir1, nope. I'm worried I scared them off. [15:04:51] I think they're used to people like us [15:07:07] which storyteller? [15:08:37] Victor and Jonathan [15:09:07] They were going to do a story about revscoring featuring ToAruShiroiNeko only. [15:09:47] I talked to them about how the project had many members who would take issue with not being involved in the story -- if it was in fact about revscoring -- not ToAruShiroiNeko's work generally. [15:09:55] And then they stopped talking to us AFAICT. [15:25:28] halfak: oh I think we don't need to launch a wiki-labels campaign in wikidata, we have enough data from Stefan :D [15:25:51] Amir1, na. I think their data is really bad actually. [15:26:19] halfak: I'm going to get started on a structured database of citations that appear on wikimedia projects [15:26:20] They only consider rollback-based edits vandalism [15:26:23] really, they told us that, it had some kind of human review [15:26:33] And they consider all rollbacks to be rollbacks for vandalism [15:26:46] (how they came up with idea of 86%) [15:26:53] harej, you know about my datasets and code for extracting citations? [15:26:57] yeah [15:27:01] Amir1, yeah. Spot checking. [15:27:02] ok [15:27:29] halfak: I believe I do, but can you link to that code repository? [15:27:32] But they are perfectly aware of a huge amount of vandalism they miss that is reverted in a different way -- which suggests a clear bias in their results. [15:27:45] https://github.com/halfak/mwrefs [15:27:52] https://github.com/halfak/mwcites [15:28:00] Soon to be merged. [15:28:09] But in the meantime, they work pretty good. [15:28:20] They'll process XML dumps to get you history or a snapshot [15:28:27] mwrefs looks for tags [15:28:49] mwcites looks for scholarly ids -- like DOI, ISBN, PubMed, etc. [15:28:54] niiice [15:29:07] man, I really wanna stop doing this infrastructure stuff for a while and just write some code [15:29:08] but noooo [15:29:18] coooool halfak [15:29:20] https://www.wikidata.org/wiki/Wikidata:Project_chat#Request_for_vandalism_fighters_in_Wikidata [15:29:20] Get back to your infra pit! [15:29:25] :P [15:29:25] I just made the notice [15:29:33] * YuviPanda goes back to infra pit :P [15:29:37] Amir1, great! :) [15:29:51] YuviPanda, how deep is this infra pit? [15:30:00] i.e. how long until you are out? [15:30:49] at least a year, I reckon [15:30:53] when I finally kill NFS and GridEngine [15:31:06] unless my body /mind / soul gives out by then, of course :) [15:31:33] Booo [15:32:24] harej, re. reference extraction -- even if you don't want to use the utilities, I think you'll find the library useful. [15:32:48] E.g. my DOI extractor is the fastest and most powerful around because I spent a lot of time testing alternative detection and extraction strategies. [15:33:21] The extractor is also blazing fast and deals with in a sane way. [15:35:23] halfak: oh, for clustering centers [15:35:27] we already have them [15:35:28] https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Clustering_in_reverted_edits#enwiki [15:35:51] 6.02661374e-02 is for first feature in first center [15:36:08] -7.96489059e-03 first feature center in second center [15:36:16] (note that all of them are scaled) [15:36:34] See https://github.com/wiki-ai/ores-wikimedia-config/blob/master/feature_lists/enwiki.py#L6 for feature order [15:36:39] comparing it https://github.com/wiki-ai/ores-wikimedia-config/blob/master/feature_lists/enwiki.py [15:36:45] (I was writing that) [15:36:46] So, added_symbolic_chars_ratio [15:36:52] yeah [15:36:53] ha [15:37:15] we can now easily analyse it [15:37:22] what do you think? [15:37:34] What about the 4th feature value? [15:37:54] That's the first one I think we can clearly associate with damage. [15:38:34] and that means first cluster is about bad faith vandalism [15:38:53] because it has higher value [15:39:31] Likely. What about the 32nd feature? [15:39:48] and 31st [15:40:59] let me check... [15:44:42] first has higher value in 32nd feature [15:44:48] *first cluster [15:45:13] same applies for 31st feature [15:45:23] halfak: ^ [15:46:48] it continues for all of bad-word misspelling [15:50:56] first center is higher for user.is_anon, is lower for is_bot. [15:51:27] definitely first cluster is for bad faith reverts [15:58:12] Amir1, cool. How big is that cluster? [15:59:52] hmm, let me check for a sec [16:05:19] (1303, 179)(ones, zeros), 1303 == good faith, 179 == bad fiath [16:07:57] So the cluster is small? [16:08:11] * halfak is confused. [16:11:53] the reverted edits for english wp is this big [16:12:35] Is the bad-faith cluster small? [16:13:30] yeah [16:13:54] sorry [16:13:57] sorry [16:14:05] I was wrong [16:14:17] it's the way around [16:15:19] * halfak continues his confusion [16:15:39] nope I was wrong again [16:16:05] bad faith is small [16:16:13] :((( [16:16:19] That's OK. [16:16:25] Now for cluster 2. What's that? [16:16:47] good faith probably? [16:16:54] What about cluster 3? [16:16:59] medium faith? [16:17:06] no faith? ;) [16:17:59] we only clusterd them into two [16:18:14] Oh! I thought you did 10 [16:18:36] this is the result for two clustering [16:18:41] kk [16:19:07] I can get you results for any other numbers [16:19:17] if you want [16:20:17] I think that laying out the statistical differences between the 10 will be interesting to see. [16:28:49] sure thing [16:49:39] Amir1, can you review something for me? [16:49:41] https://github.com/wiki-ai/revscoring/pull/165 [16:50:26] sure but in a meeting right now [16:50:38] in ~10 min. [16:50:58] Thanks [16:51:36] yw [17:00:13] halfak: https://github.com/wiki-ai/revscoring/pull/165/files#diff-d982ecab407e9258b2e70ad3bf412c81R46 [17:00:25] in general it's better to use '' instead of "" [17:00:34] specially for short strings [17:00:38] Amir1, why? [17:00:53] some kind of coding convention [17:00:59] http://stackoverflow.com/questions/56011/single-quotes-vs-double-quotes-in-python [17:01:09] I'm not sure if it's part of PEP8 [17:01:24] I follow the same pattern as the top answer there [17:01:32] double quotes for string data [17:01:38] and single quotes for string symbols [17:01:49] An example of a symbol is a key in a dictionary [17:01:57] A value would always be double quotes. [17:02:02] Honestly, I don't mind either way. [17:02:07] flake8 doesn't complain [17:02:11] pep8 doesn't comment [17:02:15] "single quotes for small symbol-like strings" [17:02:34] yeah [17:02:41] It's nothing [17:02:57] Anyway, if you want to drop double quotes, I'm game, but I'd suggest doing it all in one big PR [17:02:59] return mwp.parse(revision_text or "") [17:03:05] no no [17:03:07] oh no [17:03:09] ? [17:03:17] I'm talking about future changes [17:03:25] please don't change this one [17:03:30] Oh. OK [17:06:45] halfak: I made another note about PEP8 [17:06:48] but merged it [17:07:01] kk. Checking note [17:07:22] before any first order function "def" there should be exactly two empty lines [17:07:31] :P [17:07:42] We need to flake8 the whole thing. [17:07:54] I nearly did that but decided to do it in a followup PR [17:08:00] I'll do that next :) [17:08:44] I love to fix PEP8 issues [17:08:50] I know them by heart [17:09:15] (because other devs reaped my ass off in pywikibot) :D [17:12:49] Just doing some cleanup [17:13:03] Amir1, easy one since I've got your attention: https://github.com/wiki-ai/revscoring/pull/173 [17:13:20] Just something to say "Include me in the docs please" [17:14:48] I got to run [17:15:06] it's note safe to be outside at this time [17:15:17] at downtown [17:15:24] Thanks for your help! [17:15:28] See you later [17:15:31] o/ [17:15:46] you are more than welcome, I have to thank you for giving me the chance [17:15:52] o/ [17:18:15] * halfak looks for other reviewers [17:18:32] ToAruShiroiNeko, are you ping-able? [17:19:02] This one should be easy and it will bring revscoring master up-to-date with my work: https://github.com/wiki-ai/revscoring/pull/177 [18:08:13] I heard a smart author talk about her recent book, "Measuring Manhood", on Pacifica this morning. It sounds relevant to our bias work, she's writing about how scientists perpetuate prejudices, and about the lie that pure numbers are objective. [18:09:41] was that about measuring penis sizes? [18:11:21] awight, sounds like critiques raised 50 years ago [18:12:17] https://en.wikipedia.org/wiki/Donna_Haraway [18:12:21] 70 years ago [18:12:34] Well... actually, Haraway raises the counter-critique. [18:12:47] So, I guess it's much older than that. [18:13:49] YuviPanda: hehe u went there [18:14:46] Seems like reigniting the critiques of strawman positivist is back in style. [18:15:13] Not saying I read the book, but what you said has the hallmarks. [18:15:15] halfak: yeah I don't think the critique is original, at least the bit that was shared on the radio, but it sounded like good historical work [18:15:37] maybe. Did she use the word "lie" about the objectiveness of numbers? [18:15:51] Because I'd like to point her towards the notion of multiple objectivites and standpoints. [18:15:51] mmm nope [18:15:55] OK. [18:16:09] Because numbers are objective for some chosen objectivity. [18:16:19] The choice of objectivity is subjective. [18:16:42] Using values to choose an objectivity is a brief way to describe a "standpoint". [18:17:35] I think I'm partial to https://en.wikipedia.org/wiki/Intersubjectivity in the way Habermas wrote about it. [18:17:36] And having multiple standpoints evaluate reality via differing objectivities to do better science is what you get from Haraway and Harding. [18:17:39] https://en.wikipedia.org/wiki/Sandra_Harding [18:17:49] so much philosophy [18:17:51] They call it "Successor Science" [18:18:34] You don't change objectivities to do better science. No one has a *better* point of view. You *expand* objectivities to do better science. [18:19:11] Interesting. Why do people still use the term "objectivity"? [18:19:17] It's silly. [18:19:21] See also https://en.wikipedia.org/wiki/Science_wars [18:19:28] Oh! I think it is very useful. [18:19:37] It's just not a basic thing like was once assumed. [18:19:50] Objectivity is about repeatable measurement. [18:19:53] but, objective as in in thrall to a paradigm? [18:19:59] hrm [18:19:59] But we choose which measurements to take and how to value them. [18:20:10] consistent subjectivity [18:20:33] * awight is glad to press these buttons. Thanks for the pointers! [18:20:47] :D [18:21:05] Fun topics. Next time I'm in town, I'd like to introduce you to THE GEIGER. [18:21:16] He gave me an intro to this stuff a few years ago. [18:21:27] It's the theoretical foundation for revscoring, it turns out :) [18:21:29] I had my head pretty well blown open by Kuhn, not sure I took away the right lessons though. [18:21:40] all science is wrong. But it'll take ten years to find out. [18:22:22] Here, the word "wrong" is even more outdated as objective. [18:22:42] If science can't be Right, it certainly can't be Wrong. [18:22:50] It's a matter of modeling, really. [18:22:54] I think I saw Stuart talk about his vandalism bot research [18:23:08] Knowledge forms a model of the world. [18:23:08] this clearly needs more alcohol, or I'm worriyingly less intelligent than I used to be [18:23:11] All models are wrong. [18:23:14] Good models are useful. [18:23:17] Well, science as increasingly accurate approximations of the Truth is plain Wrong [18:23:29] yah ^ that [18:23:36] useful for whom :p [18:23:39] Yeah. That's not as useful a way of thinking and some other ways. [18:23:45] But it is a very useful way of thinking. [18:24:26] Will we make better predictions through scientific progress? Sure! [18:24:33] That is literally higher accuracy [18:24:53] But does scientific progress necessarily march towards truth. Well... WTF is Truth? [18:25:23] We could be on the verge of observing a side of the world we'd never seen before -- and that might completely change how we think it works. [18:25:38] Economics is a good example of how the better predictions thing is bogus. I really appreciate Galbraith on that. [18:25:40] At that point, we might say that we were not getting closer to the truth before that point -- that we were stuck. [18:25:50] awight, huh? [18:26:04] I like the investigations of science which talk about the motivations of people doing the work [18:26:39] awight, sure. Many people who do that build strawmen though. :/ [18:26:53] JK Galbraith writes this amazing stuff about economics from an insider's perspective, but writes stuff like, economists have put scientific trappings on pure rationalization of the status quo [18:26:53] That's one thing I find very frustrating as a practicing scientist. [18:27:07] get it all out :) [18:27:25] Which economists? [18:27:31] Which economist cultures? [18:27:46] the neoclassical model, including marxists [18:27:53] There's an economist culture in HCI that is torn between the human experience and marketing style behavioral optimizations right now. [18:29:47] for example, updated classical models don't take unemployment into consideration. [18:29:50] bah [18:30:08] Sure. This sounds like a methodological concern though. [18:30:27] I'd like to hear more about why second-guessing researchers is a bad idea [18:30:42] Not second-guessing. [18:30:48] That's the name of the game :) [18:30:56] I agree that the biographical approach leads to pretty bad reading of literature... [18:31:34] Second guessing and critique is great. Setting up a strawman is not going to help us make progress. [18:32:17] Well, when I talk about motivations, and I've been burned by this personally, I mean things as simple as following the money. Is this work among the >50% of American scientists funded in part by the military? [18:32:57] I worked at UC Berkeley's integrative biology lab, while it was basically owned by Novartis. My lab was funded by the office of naval research. [18:33:21] It depends how those things play out in your work. [18:33:47] So, my funding came from the NSF. I knew what kind of things the NSF was excited about recently and I tried to write my funding proposals in that direction. [18:33:55] The PI and post-docs called themselves "rational pacifists" and deeply believed that the robotic spy fly was going to be used to rescue earthquake victims.... [18:34:31] sure. I'm sure that many people who read my work would look at what we are doing with machine learning and call us fascists. [18:34:34] Yah. I think the concepts of Marxist class consciousness and postmodern psychoanalysis are really helpful for understanding how bias is structured [18:34:42] But I don't think they understand why we're doing it. [18:34:44] I won't lie, the machine learning is scary :) [18:35:07] BTW, re. scientific capacity, see this awesome stuff: http://sss.sagepub.com/content/45/2/214.full.pdf+html [18:35:25] "Research infrastructure" -- what questions are you enabled to ask/answer? [18:35:38] My infra is better when I have people with more standpoints involved in discussions about value. [18:35:55] infra? [18:36:01] oic [18:36:09] research infrastructure [18:37:22] Yeah. [18:37:33] * awight learns to read backscroll [18:37:38] :D [18:38:25] Anyway, critique is awesome, but those under critique should be understood if our goal is improvement. If the goal is to sell books (follow the money), then misunderstanding peoples motivations is a great strategy! And quick! [18:38:47] This is why I'm on the fence about Tufecki [18:39:21] Just to finish the thought, the thing that's scary to me about ML, and I assume is the reason you wanted to look at bias, is that we're gearing up to build a really strong feedback system that reinforces the status quo [18:40:22] awight, not really just that. Status quo is one concern. Also, the status quo is already well reinforced :\ [18:40:31] But it's a very good reason to look into bias. [18:40:54] Honestly, I think that ML is secondary to the affordances of the software that uses it. [18:41:14] We could have really shitty, racist ML with really solid UI, and we'd be doing much better. [18:41:16] I think. [18:41:32] But we should have good ML and good UI. [18:41:41] So let's figure out how good our ML is! [18:42:29] o/ ToAruShiroiNeko [18:42:33] I see you editing the wiki [18:42:37] Come review my pull requests :D [18:42:49] Totally. I suspect that one of the main on-wiki problems is that the software makes it easy to be a dick to people. [18:43:14] Yeah. If not even actively encouraging bad social behavior. [18:43:27] I'll troll u more soon, I got to attend to some fundraising things [18:43:53] My goal with revscoring is to bring a lot more standpoints to the design of UIs around quality control. [18:43:58] o/ awight [18:43:59] Speaking of dark money :p... actually, this is why I'm proud of the WMF: https://www.mediawiki.org/wiki/User:Adamw/Contribution_distribution [18:44:00] fun chats :D [18:44:39] We're not funded by G$$gle, yet! [18:44:45] \o/ [18:45:02] it's regular people and their teeny contributions. A lot of them. [18:45:07] Hopefully they aren't sneakily donating $5 a million times [18:45:14] hahaha no we check for that [18:45:18] :D [18:49:21] wikidata was funded by google [18:50:30] I noticed ;) [18:52:09] select pl_from_namespace, page_title from page join pagelinks on pl_from = page_id and pl_from_namespace = page_namespace where pl_namespace = 4 and (pl_title in ("WikiProject_directory", "WikiProject_Directory") or pl_title like "WikiProject_Directory/%"); [18:52:15] this is gonna take a while . . . [18:52:50] Watcha doing, that looks like an interesting project! wikiproject X? [18:53:38] I'm trying to get a list of pages that link to one of my directory pages [18:53:46] To see people using it in conversation, on WikiProjects, etc. [18:54:40] WikiProject Medicine, for example, links to https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Directory/Description/WikiProject_Medicine [18:54:42] And I didn't even tell them to do that! [18:55:52] that "or" is going to kill u [18:55:59] a union might be faster? [18:56:27] donno. I'm a mediocre sqlist [18:56:34] as is the "like" [18:56:43]