[15:00:29] miriam: hi. I meant to share this with you since Friday. ;p I found one of my favorite Wikimedia Commons categories. Category:Icons and Category:Icons_by_theme . There are some really nice icons in there. I figured I should inform you as our visual expert. :D [15:00:52] leila: good morning!! [15:00:56] * leila goes to a meeting. will be back in 1 hour. [15:02:30] leila: looks fantastic :) actually that is good to know, the quality model has some issues with icons and graphics in general, given the low presence of this kind of images in the training data! [17:16:15] 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999) [17:17:54] 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999) [17:18:41] 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999) [17:23:01] miriam: which quality model? the one for predicting images for WD items? [17:23:16] leila; yes [17:23:28] 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999) [17:23:30] sorry, in my mind there is one quality model only :) [17:23:32] :D [17:24:03] haha. :D [17:24:10] miriam: yeah. that makes sense. [17:24:33] miriam: there are some amazingly cute ones in there. let me find a few that are bookmarked [17:26:36] 10Quarry, 10Patch-For-Review: Do the big Quarry migration - https://phabricator.wikimedia.org/T202588 (10zhuyifei1999) [17:29:30] miriam: check https://commons.wikimedia.org/wiki/Category:Creative_Tail_Round_Object_Icons [17:30:03] miriam: or https://commons.wikimedia.org/wiki/Category:Creative_Tail_Round_Animal_Icons . These are really amazing. :) [17:31:00] leila: <3 so cute!! [17:32:17] miriam: on a related note, I tried to reach out to the creator of https://www.stockvault.net/photo/237741/cartoon-astronaut-communicating---with-copyspace to ask them to upload the work on Commons (as the current license is not Commons-compatible as far as I understand and I can't use it on meta). [17:32:53] miriam: stockvault has made it quite impossible to reach the author, I think. I tried to leave a comment for the photo and that didn't go through either. [17:33:31] leila: wow really? [17:33:34] miriam: I WANT that for the taxonomy of WP readers. I love that the figure is reading something (though it's really communicating) and it doesn't have a specific gender as far as I can tell. [17:34:15] We should bring all of these people to Commons to upload their work their. At least they can get proper attribution. [17:35:51] miriam: FYI, I also learned through that process (thanks to effeietsanders) that Commons has an {{Unspalsh}} template that allows you to upload photos from Unsplash, as long as they were uploaded prior to June 2017 to Unsplash. Apparently Unsplash changed their licensing terms around that time and the new form is (in my read) a bit contradictory, and I see how Commons may not like it. [17:36:15] leila: would this platfom make it any easier? https://freerangestock.com/photos/110384/cartoon-astronaut-communicating--with-copyspace.html [17:36:22] leila: you have the same image there [17:36:51] * leila holds her breath and reads the license terms of freerange [17:37:54] miriam: IANAL or an expert in Commons, but I /think/ the second bullet point in https://freerangestock.com/licensing.php is problematic. [17:38:08] miriam: the "redistribute" part specifically. [17:38:16] at the very least [17:38:51] "you cannot sell products which derive their primary value from the image" is problematic, too, I think. [17:39:12] miriam: If you run into the person's contact, let me know. I want to convert them to Commons. :D [17:40:22] leila: sure :) [17:41:19] miriam: how are things on your end? what's keeping you busy? [17:42:31] leila: I just ran a pilot on mturk replicating a part of a wikilabels data collection experiment. First results seem encouraging, we might be able to get a lot of data correctly annotated in crowdsourcing! [17:43:00] miriam: are you collecting multiple labels for each entry? [17:43:19] miriam: I'm asking as we may have to do a similar thing for evaluating section alignments [17:44:59] leila: one label per sentence; 3 independent annotators. [17:44:59] We simplified the task a lot for mturk. The original task was: does this sentence need a citation? Can you tell us why? [17:44:59] The task now is: we know this sentence needs a citation. Could you tell us why? Selecte from this X possible reasons. [17:45:46] miriam: makes sense. [17:46:21] miriam: and how do you evaluate how careful/accurate the mturker is? Do you have some items for which you know the answer? or you get multiple labels per item? [17:46:45] leila: it's very easy to use the native MTurk interface to build the task. Please let me know if you or dsaez need help :) [17:47:18] leila: yes, we have groundtruth from Wikilabels, so the data annotated by experts. [17:47:19] miriam: ok ok. thanks. [17:47:31] miriam: excellent. cool. [17:48:24] leila: we are thinking to make an analysis of the 2 sets of judgements similar to this: https://www-users.cs.umn.edu/~bhecht/publications/goldstandards_CSCW2015.pdf (paper suggested by dr Morgan) [17:49:47] leila: what is keeping me busy is also having fun helping SDC with computer vision <3 [17:50:52] miriam: I'm looking forward to see your choice of title. :D [17:51:50] miriam: what insights you want to gain out of comparing the two in depth? [17:52:44] miriam: we could observe in https://arxiv.org/abs/1804.05995 (where we also had labels from Wikipedia editors and turkers) that Wikipedia editors keep the bar higher than turkers, and that's good to know for future experiments, of course. [17:52:50] miriam: (which was btw, expected) [17:57:02] leila: in practice, we want to see whether such an experiment can be easily ported to MT (accuracy/agreement); see where the limitations are (qualitative inspection of errors) and whether Mturkers and wikipediasn rely on the same textual cues when assigning labels (correlation labels/textual features) [17:57:14] *Wikipedians [17:57:51] miriam: got it. that's helpful as we keep running to labeling issues across many experiments. [17:58:17] leila: anything I can do to help? [18:01:56] miriam: I'm not sure. :D I'm going to talk to Dario today about what's left from Q1 goals and based on that he may come to you all with some requests. otherwise, please enjoy. [18:02:33] leila: sure. Here to help. We can also discuss tomorrow in our 1:1 [18:03:17] ow right. we have one of those. GREAT. :D