[12:01:18] HelloOoOo... ;) [12:01:40] hi DrTrigon [12:01:47] hi jayvdb! [12:03:05] which parts of the MVP background didnt make sense ? [12:03:49] Oh - they might make sense but my english might be to bad to understand... ;)) [12:04:10] First paragraph "Background Information:" [12:06:14] jayvdb: ^^^ [12:09:10] ya, which part? [12:10:28] Explain the 1 part please may be then I do get the other 2 as well... [12:11:26] ok, you were proposing that the MVP might be the uncategorised files categories [12:12:08] that is more complex than newfiles, as those old files are often categorised in meta categories [12:12:39] but do still appear in "uncategorized"? [12:13:12] ya [12:13:16] let me find a better example [12:14:19] https://commons.wikimedia.org/wiki/File:Angeles_de_Plinio.jpg [12:15:33] so why are they marked as uncategorized anyways? is that a misstake? [12:16:00] or is Category:Libro antiguo a bad category that "does not count"? [12:16:15] here is an example of one that shows why using the API categories would be difficult [12:16:17] https://commons.wikimedia.org/wiki/File:10_planta_de_agua_piedras_(12522557055).jpg [12:17:29] so whats your oppinion on this? not doable? do you now haow many % of all uncategorized are like that? [12:17:36] it has flickr status categories. it would be hard to determine which categories are meta categories (and therefore the image still needs categorisation), and which are real categories (and therefore the image does *not* need categorisation) [12:18:26] I would see it like if there is an "uncatgeorized" tag - the bot is allowed or even forced to add what he has to say... then a human can check, right? [12:19:15] not many , it seems. maybe we can ignore that problem. and assume that most of the images in those categories are uncategorised [12:19:38] ya, im ok with that. [12:20:28] as long as we do not *remove* categories - all should be fine? humans might have to look at it, but they just need to decide which category to keep, so it's like going through a list and add some checkmarks [12:21:09] nod. [12:21:18] i cant easily find other examples [12:22:41] cool - now I fully understand your point and furthermore we mention a way of dealing with it. [12:25:05] So are you fine with this MVP? Do you want to add or change something? Have you seen my 2nd generation of notes? (e.g. "percentages I considered technical errors not the ones reported", etc.) [12:25:14] the next paragraph was another example [12:25:32] any problems with the third paragraph? [12:26:57] my concern is that 10% error rate is not a careful bot [12:27:22] which will make obtaining a bot flag difficult [12:27:23] If you consider CV algos it is - people have to some extend to live with that... [12:27:50] it's AT MAX 10% of the categorized ones (not of all files). [12:28:13] so if we categorize 50 faces, 5 might be wrong - that's not bad honestly! [12:29:12] nod. if we're saying 10% error rate only for "hard" CV decisions, I can accept that (but dont like it), but color vs monochrome shouldnt have a 10% error rate [12:30:14] of course - also mime-types need to be way better! I was mentioning an overall worst case error rate - we want to be better [12:30:28] ...but if we are worse we have to seriously revise our code! :) [12:30:40] 3rd paragraph I added some comments [12:30:55] and there is a 100% reliable way to avoid miscategorisation for color vs monochrome -- skip the file if the file is mostly one color with dull pigments in other colors, or has a few spots of color [12:31:45] yes - we can have a "grey-area" of cases were we just say our algo is not precise enough so we skip them for now. good point. [12:32:48] * DrTrigon tried to build a PAWS docker but the network refuses gpg key download at work *grrrr* [12:34:12] AbdealiJK around? [12:35:02] the MVP needs to be clear that the error rate should be very low (2%?) for normal most decisions, but can be 10% only for hard CV problems. im not sure how we describe the difference, except listing the "hard" ones explicitly [12:36:12] the "skip if unsure" option should be used by default, in the automatic mode and when determining our MVP, and then validating the MVP is obtained. [12:36:29] the list won't be that long for now I guess... and I would anyhow like to see such lists ;) [12:36:54] nod [12:37:13] we know that nobody is going to watch the bot process thousands of images. they will let it go in automatic mode, at high volume, and the operator could be nasty when problems are found [12:38:42] on the other hand: people not understanding at least some minimal background on cv algos (can be read up in wiki) should not run the bot, right? as they anyhow do not know what the bot and thus they themselfes are doing... [12:39:08] Hello [12:39:23] the bot will be run by people who have no idea about cv algos. that the blessing and curse of this project [12:39:26] hey AbdealiJK [12:39:45] Started the meeting already o.O ? [12:39:46] that *is the blessing and curse of this project [12:39:57] only chatting about the MVP [12:40:00] wiki you mean? [12:40:01] Aha, ok. [12:40:22] AbdealiJK: just used some free time - we can start the meeting now! [12:40:27] By the way - I found an interesting way where the bot running can be simplified. [12:40:39] ? [12:40:41] well this project makes the bot so simple to use, and the result is some simpletons will use it [12:40:41] DrTrigon, jayvdb: Have you played the WikiData game ? [12:40:48] AbdealiJK, ya [12:41:02] jayvdb: Why don't we make a commons game ? [12:41:18] DrTrigon: https://tools.wmflabs.org/wikidata-game/# [12:41:20] AbdealiJK, wikibase is coming to commons soon ;-) [12:41:45] WikiBase ? Is this the structured data saving thing ? [12:42:03] a game version can be your next project, and you can use the Wikidata Game code with some tweaking ;-) [12:42:09] Exactly :) [12:42:29] ya. Wikibase is the software that runs Wikidata [12:42:32] So, using the game version, we aren't restricted to only Uncategorized files [12:42:55] DrTrigon - Apologies. Are you following ? Are you aware of wikidata-game ? [12:42:58] Whats the difference between a "game" and a a javascript interface? [12:43:14] DrTrigon: I'd suggest you login and play https://tools.wmflabs.org/wikidata-game/# [12:43:40] ya, lots of problems disappear after structured data arrives. and, maybe categorisation doesnt need to exist either [12:44:08] jayvdb: yep. In WikiBase we'd be adding properties like wikidata to commons. i think the game approach would be easily extensible to commons then [12:44:11] at all...?! [12:44:15] structured data + natural language queries == categorisation [12:44:44] the bot can help playing games? [12:44:54] So, DrTrigon [12:45:12] The wikidata-game basically "finds" things you can very very easily do as a human which cannot be automated completely [12:45:53] we would need to rename the wiki writing code pywikibot-classifyfiles ;-) [12:45:57] So actually that game should have a learning algo in background that try to suggest solutions as long as it can do it itself... [12:46:00] For example, it has a wikidata page on a footballer. But the wikidata article has no "Gender". So, it shows the user a summary (from wikipedia) if it exists, and asks the game player whether the person is male or female [12:46:28] Now when the user presses "male", it adds the "male" property to the Item in wikidata. [12:46:40] DrTrigon: Yes. exactly [12:47:07] So, we show the picture and existing categories in the image. And say "Should we add Category:Groups of People ?" [12:47:12] So another GSoC: Add learning algos to wikidata games [12:47:24] DrTrigon: Noo, I was thinking we make a commons-game [12:47:45] nod, another GSoC [12:47:49] hehe [12:47:59] I think I'll probably do it before next GSoC :P [12:48:12] So, to give you an idea as to why I like the game [12:48:12] AbdealiJK: does that add to or replace your former plans of an web interface? [12:48:21] DrTrigon: Adds to it [12:48:34] ;) cool - count me in! [12:48:34] ;-) [12:48:49] Some stats: It took me a week to get 1000 edits in commons. It took me less than half a day to get 500 edits in wikidata [12:49:13] it took me years to get my first 1000 edits in wikipedia ;-) [12:49:29] those were the days [12:49:29] (I still don't have that many - I think... ;)))) [12:49:34] Wikipedia is quite a bit more difficult because so many people :P [12:49:42] English wikipedia * [12:49:49] hehe [12:49:56] So. coming back. Sorry to disrupt your earlier conversation [12:49:57] did you meet Yann ? [12:50:06] jayvdb: He did not come [12:50:09] awww [12:50:10] AbdealiJK: Why that difference? [12:50:28] DrTrigon: Because of wikidata game ! [12:50:47] * DrTrigon hast to think about that a bit [12:51:04] jayvdb : There were absolutely NO commons developers there. I asked around a lot [12:51:49] I met about 5-6 people who are photographers and are in fact lazy to upload pictures to commons from flickr because they don't want to categorize them. [12:51:52] commons admins ? [12:52:04] They were all interested in the script a lot :) [12:52:14] Nope - No commons admins that I saw [12:52:29] hehe. they need two bots - upload then categorise [12:52:41] Upload is flickerbot right ? [12:52:47] ya [12:53:46] and pywikibot has a flicker uploader, which can be used together with other python code (such as your bot) to build a bigger solution [12:54:05] first categorize then upload [12:54:08] flickrripper [12:54:27] Nod, but I believe there was a big in that - will check [12:54:35] bug * [12:55:05] nod - you could even turn the process around -- search on leaf category names in flickr, and then upload anything that matches all the rules [12:55:20] There are categories in flickr ? [12:55:29] ya, it is hard to operate at the moment [12:55:49] I see, hmm. Anyway, Coming back to your discussion [12:56:08] it uses tags, but it also has free text fields which could be searched [12:56:12] yup. [12:56:13] I apologize for disrupting - but I wanted to mention this was a nice way to get Non-Computer Vision people to use the script effectively [12:56:18] meeting coming to order!! [12:56:20] Without installing a thing ! [12:56:50] it is a very good point -- people do avoid uploading because of that [12:57:01] I agree - I planned to work on paws docker - is that useful for you? [12:57:12] And as you mentioned - Integrating with UploadWizard would be cool. [12:57:27] DrTrigon: I still have not tried PAWs. I don't think it would be very useful though [12:57:29] UploadWizard is php? [12:57:34] DrTrigon: Yep [12:58:03] And it would call our code as externals "binary" then? [12:58:13] So, I have an idea on how to integrate it. I can make catfiles run as a webservice, and give the categories in JSON and get that using a JavaScript extension [12:58:23] So, we're essentially hosting a "Service" for it. [12:58:31] * DrTrigon sees... [12:58:59] The reason to keep it as a separate webservice is because mediawiki has a very strict policy for stuff it runs. And that security procedure is painful. [12:59:26] Integrating with UploadWizard would be *very* cool [12:59:41] ...and solve some complex issues. [12:59:51] yup [13:00:14] anyway ... unit tests ... there was some confusion about them in the Z [13:00:22] Where to host the service? Who to contact to make that happen? [13:01:00] unittests: I don't know how clever it would be - but I feel the need for an install-unittest [13:01:38] something like: https://travis-ci.org/drtrigon/install-file-metadata-wikibot [13:01:42] DrTrigon: Hosting service would be in ToolLabs [13:01:44] the best way to do that is to run the installer in lots of different dockers [13:02:27] * AbdealiJK checks the travis build mentioned above [13:02:31] the installer alone is noth enough - we have to test the bot-scripts afterwards [13:03:07] to check proper operation [13:03:46] but may be this is more testing of pywikibot than file-metadata ... I not so sure about that. [13:03:57] DrTrigon: But that's unusual IMO. When I install VLC or Firefox or Gedit, it does not do a host of tests on my system to check if it works [13:04:00] the best way to do that is to run the installer in lots of different dockers, and then run small sanity check in each docker [13:04:21] sanity check? [13:04:33] jayvdb: So you mean basically have a "test grid". And your sanity check would actually be the unittests I guess ? [13:04:41] AbdealiJK: of course not for end-users - just for us to keep track of proper operation after changes [13:05:14] So, who is it intended for ? [13:05:26] I mean if I want to run unittests I'd just do py.test right ? [13:05:38] For us to keep track wheter we have incompatibilities with e.g. pywikibot or else [13:05:43] yup [13:06:00] (mostly pywikibot) [13:06:13] I still do not seem to follow. So, us testing incompatibilities is why unittests+travis exist [13:06:36] I think what you're proposing can be done by having unittests for the scripts too ? [13:06:56] for example, yes! [13:06:59] ya; we just need to add some docker jobs to the travis builds [13:07:12] Nod. I completely agree that unittests would be nice ! [13:07:30] jayvdb: other than https://travis-ci.org/pywikibot-catfiles/docker-file-metadata [13:07:38] I have not added it yet, because the scripts were changing almost daily and writing tests for it seemed double work [13:07:49] jayvdb: Docker jobs for ? Different OSes ? [13:08:16] AbdealiJK: so you kind of have it on your/our TODO? [13:08:22] DrTrigon: Yep [13:08:26] perfect! [13:08:36] I'd rather wait a bit more for the scripts to mature though [13:08:53] I think thats ok for me. [13:09:46] may I mention my first long test-run: https://commons.wikimedia.org/wiki/User:DrTrigon/logs/eth-bib [13:10:21] checking results [13:10:33] I was a bit suprised by the fact that only a small number of distinct categories was added [13:10:46] but that might be due to the fact that as they are very similar [13:10:50] Hmm. The images are very specific. [13:10:54] yeah - very similar [13:11:28] Then I was wondering whether we could alos categorize the "files with exceptions" as such? [13:11:33] Majorly distinct categories was being popoulated by "Created with " and similar. Which is always Hasselblad H3DII-39 and Photoshop here [13:12:20] DrTrigon: Nod. Better handling for those is required. Either need to categorize it appropriately or check if the script has a bug I'm unaware of [13:12:23] As I see the exceptions do not apperar in any bin right now, correct? [13:12:30] They do not [13:13:01] As it was always a small number it's not so bad, but actually the should be at least counted to the 100% reference. [13:13:19] (I know that makes or stats an epsilon worse) [13:14:21] Do we currently have a cut-off threshold for landmarks faces score? I guess not, right? [13:14:26] jayvdb: Still here ? [13:14:35] DrTrigon: We do, Don't remember the number though [13:15:04] jayvdb: ??? [13:15:46] (may be the docker link I sent overwhelmed him...?) [13:16:03] DrTrigon: File:ETH-BIB-Das Serengeti-Camp-Kilimanjaroflug 1929-30-LBS MH02-07-0047.tif is an example of an image with a dlib detection but with no Category => The score is too low [13:16:47] File:ETH-BIB-Gebirgslandschaft-Persienflug 1924-1925-LBS MH02-02-0236-AL-FL.tif [13:16:48] etc. [13:18:23] ...they are very low... I see - I asked because of https://commons.wikimedia.org/wiki/User:DrTrigon/logs/eth-bib#File:ETH-BIB-Bachvegetation_am_Makungu-River-Kilimanjaroflug_1929-30-LBS_MH02-07-0459.tif and 1 down and 4 down [13:18:52] aa, ok 0.037 is not soo low... [13:19:26] yes, ok - to tweak that sucessfully we need proper stats - and that is for later [13:19:36] now I am concerned about jayvdb [13:19:42] :D [13:20:05] sorry, my little man needed to go to the toilet [13:20:23] Alrighty - So, MVP [13:20:26] pinged him on skype - the good thing is wm-bot logs it for us and jayvdb (! ;) [13:20:47] jayvdb: aaa - now THAT is what is important - not we, right! ;))) [13:21:04] XD [13:21:24] yup; ive caught up reading [13:22:22] ok, I am done except for a few strange bugs I want to mention to abdeali ... [13:22:43] other questions? issues? concerns? [13:23:05] so MVP ... before you came AbdealiJK I said I dont mind "10% error rate" if we restrict that to only a set of "hard problems". i.e. color vs monochrome is not a hard problem. detecting faces is. [13:23:05] Well, I am still unsure what the MVP is [13:23:45] Error rate == ? [13:23:50] we need to list which ones are both hard to guess, and hard to avoid bad guesses. [13:23:57] "Possible Proposals for MVP:" in https://etherpad.wikimedia.org/p/gsoc-catimages-MVP [13:24:16] Is error rate: Wrongly categorized images ? [13:24:22] ya [13:24:31] ok [13:24:38] (false positives) [13:24:49] just call it an error :P [13:24:56] Oki [13:25:51] what other algorithms have a high error rate, and it is hard to know when an error has happened [13:26:38] Well, IT8. [13:26:48] Hmm, /me finds a list [13:26:49] what other algorithms have a high error rate, and it is hard to know when the result *may* be erroneous [13:27:56] e.g. with color vs monochrome, there are a few types of images with mostly one color but a few other colors where the algorithm can say "this is too close to decide; I must skip it" [13:28:22] Barcodes (But that's reallyt rare anyway) [13:28:42] but if both face detectors get it wrong, there isnt much that can be done. [13:28:47] Icons and Football stuff [13:28:58] football stuff should be easy ... ? [13:29:16] icons, agree, can be difficult [13:29:20] jayvdb: Well, I'm currently just using the size of the image [13:29:22] jayvdb: circle detection sound but is not soo easy [13:29:31] But yes. I agree that if the error is large I can easily make it better. [13:29:35] circle detection definitely hard [13:29:58] I would say all "new" or "experimental" algorithms are in the class "more errorprone" [13:30:22] I can't think of anything else at the moment [13:30:32] (jayvdb: you meens the icons... sorry!) [13:31:15] ok, so what other decisions need to be made before we lock down the MVP [13:31:38] "Finalizing the GSoC Project:" [13:31:51] are we going to include both newfiles and category;uncategorised_files in the MVP ? [13:32:31] Also: People with glasses. [13:32:31] uncategorized: We cannot include that as long as we don't have stats on that! [13:34:36] ok, maybe we simply say that part of the MVP is to do a bot run on uncategorised, and include stats in the final report [13:35:24] ya, why not... [13:36:07] jayvdb: In what form(at) does this MVP be to be? [13:36:08] we can say that the focus has been on handling a diverse set of files from special:newfiles , in order to stop the uncategorised set from increasing uncontrollably [13:36:31] and working on uncategorised has not been focused on. [13:37:21] ...and we intend to use the bot to slowly clean up history of uncategorized ones too. [13:37:38] The MVP needs to be a written description of what the code will do, in as simple writing as possible, [13:37:47] So: 1. stop uncategorized from increasing further 2. try to reduce historic left-overs over time [13:37:55] It belongs on one of the tasks [13:39:11] jayvdb: As you are more native speaker than I am - can you formulate the draft? Just ping me whenever I can be handy, would that be fine? [13:40:17] then AbdealiJK needs to run the bot to prove it meets that criteria. it doesnt need to write to the wiki, and probably shouldnt write to the wiki, as that would probably violate the bot policy. Then AbdealiJK needs to document the bot run results in a data filled report . charts and stuff [13:40:38] can do [13:40:55] ...similar to https://commons.wikimedia.org/wiki/User:AbdealiJKTravis/logs/newimages [13:41:20] ... with some prose to explain it to someone unfamiliar with the project ;-) [13:41:29] jayvdb: can https://etherpad.wikimedia.org/p/gsoc-catimages-MVP serve as a strating point - I tried to write it like that [13:41:37] DrTrigon, yup [13:41:40] that is fine. [13:42:00] Is it ok if I use 2% error rate for easy algorithms? [13:42:19] AbdealiJK: ^^^ ? [13:42:22] I think that should be Ok. [13:42:25] (and 10% for hard algorithms, we have agreed upon) [13:42:46] Btw - redlink are not error right ? [13:43:01] ... and may be give an example: 5 out of 50 faces means 45 do not have to be processed by humans! still not sooo bad... ;) [13:43:10] So, are redlinks considered for the denominator of error % ? [13:43:40] I would like to see a bot run properly analysed to determine the error rate , so we can talk about those errors and hopefully agree about what is an error and what isnt. [13:43:58] Ok [13:44:02] Will do [13:44:06] AbdealiJK: what do you exacltiy mean by red-link here? [13:44:43] e.g. IMO face coordinates that overlap a face are OK, but face not overlapping the face isnt OK, but I am sure we will see some grey areas when we talk specifics [13:45:38] Argh [13:45:44] ? [13:45:49] jayvdb: Also - the "Taken with" should be 10% error rate [13:46:10] AbdealiJK, arnt you using templates for that now ? [13:46:14] Because all camera models aren't supported yet, and those will keep coming up [13:46:54] I am, but There are some camera models that just have for example "P700" in their exifdata. And that would be considered wrongly categorized IMO [13:47:07] Couldnt we build the smarts into the template (or a chunk of Lua code, instead of making the categorisation static) [13:47:10] ^ Red link + Wrongly categorized [13:47:43] (tks) [13:47:58] jayvdb: Not sure, and I'd rather not commit on it in the MVP with 2 weeks left [13:48:18] nod; ok. [13:49:23] ok, ill have the MVP ready by Friday, and we can fine tune it then [13:49:31] anything else? [13:49:33] :+1: [13:49:50] jayvdb: tell me whenever you want me to comment on anything in the MVP [13:49:57] DrTrigon, will do [13:50:04] I am done for today - tahanks a lot guys! [13:50:51] I have nothing else to add either specifically [13:50:57] Except that you should try the wikidata game :D [13:51:00] DrTrigon: ^ [13:51:01] lol [13:51:16] AbdealiJK: I have it open and 1.5hrs to travel home - so yes.... ;) [13:51:23] Naiice :) [13:51:42] AbdealiJK: btw. we did not ask on how the conpherence actually was, or did we? [13:52:00] It was really nice ! Met really fun people [13:52:13] I met the ED of wikimedia - Catherine. [13:52:26] "ED"? [13:52:39] And also met Asaf - I'm not sure what he does in WMF exactly, but he was a really fun person to interact with [13:52:52] ;)) [13:53:04] ED = Executive Director [13:53:11] Katherine * [13:53:27] (physicist: ED=electrodynamics) [13:53:39] I also met a bunch of admins and logn time contributors in tamil, Malayalam, Odia, etc. wikis [13:54:02] And I also met Tony Thomas - the GSoC admin from wikimedia as well as a few last year GSoC students [13:54:19] oooh, very cool! [13:54:45] PS: I also participated in the Hackathon with Tony. to make a mediawiki extension which enabled Desktop Notifications for the recent changes [13:55:26] nice [13:55:37] That's the summary of it. Lots of fun and neat set of contacts [13:57:05] perfect - that sound like a very familiar and relaxed atmosphere - that is actually where inventions get made! Very nice! I a bit jealous which is a good sign for you... ;))) [13:57:07] cool! [13:57:37] So I guess you will profit from that in future a lot! [13:57:57] jayvdb: Question [13:58:16] Is there a wikimedia project which aims to accumulate knowledge of Softwares ? i.e. crowd sourced documentation ? [13:59:25] And would that fall into the "accumulate and share knowledge of the world" aim of wikimedia ? [14:00:02] I have to appologize since if there is nothing else left I have to leave now. [14:00:10] DrTrigon: Sure - Cya