[00:31:20] Sk!d: thanks, stathat burped, and i didn't catch it [00:31:36] Sk1d: restarted [07:16:55] Hi [07:17:05] Is there a way we can have https://www.wikidata.org/wiki/Category:Wikidata:Guideline only transclude one copy of the page? [07:40:12] [02:39:11 AM] <+RyuBot-wikidata> User [[User:Aude]] Tiny create [[User:Aude/todo]] (+4) URL: http://www.wikidata.org/w/index.php?oldid=27145796&rcid=27160087 "create" [07:40:12] 10[3] 10https://www.wikidata.org/wiki/User:Aude13 => [07:40:14] 10[4] 10https://www.wikidata.org/wiki/User:Aude/todo [07:40:19] aude: :P [08:54:48] legoktm: heh :) [08:55:34] the bots are always watching :P [08:56:02] good [08:58:38] any one knows of qualifiers already used on the page? [08:58:54] Denny_WMDE: live you mean? [08:58:59] hi all [08:59:06] am i supposed to join a hangout? [08:59:13] https://www.wikidata.org/wiki/Q4794331 [08:59:16] can I give the daily here? [08:59:40] * DanielK_WMDE will just try [09:00:17] UI-Issue: clicking on "I accept these terms for my future edits. Do not show this message again." doesnt close the tooltip [09:00:20] DanielK_WMDE: give it here. Abraham is not there to set it up [09:00:41] rschen7754: thanks! [09:00:45] np [09:00:51] Denny_WMDE, aude: "in the fly" deserialization of changes; "late" injection of langlinks on the client; reviews & follow-ups; dispatch-via-job-queue proposal. [09:01:05] err, *on* the fly. [09:01:16] maybe i should take a nap first :P [09:01:19] can we please move the "on the fly deserialization" [09:01:40] seperating langlinks into local and remote would be more important [09:01:58] as it is more urgent [09:02:07] i thought this was the result from yesterdays talk? [09:02:15] there is still no date-type? [09:03:09] DanielK_WMDE: work in the langlinks is good [09:03:20] lbenedix: no there isn't. we deployed quite a number of things yesterday, but dates were not among them [09:03:54] * lbenedix has very unstable connection right now [09:03:58] * lbenedix has very unstable connection right now [09:04:30] [04:03:19 AM] lbenedix: no there isn't. we deployed quite a number of things yesterday, but dates were not among them [09:04:42] since i think you missed that. [09:05:04] yes [09:05:13] !logs [09:05:14] http://bots.wmflabs.org/~wm-bot/logs/ [09:05:15] you have a lot of examples with date-types [09:05:50] Denny_WMDE: i wrote the "on the fly deserialization" in 30 minutes on the train yesterday [09:05:53] just need to submit it [09:06:03] i'd postpone compression [09:07:21] Denny_WMDE: i don't remember a clear priorization of the changes stuff, but the lazy deserialization stuff was quick and easy, so it was a good fit for the train ride :) [09:07:52] lbenedix: dates are quite complex, it's not easy to get them right [09:08:11] there are not only a lot of different notations, there are also different calendars... [09:08:26] * lbenedix thougt the last change in dates was done about 500 years ago ;) [09:09:06] whatever. I hope it won't take much time, and the link thing is up next anyway :) [09:09:37] lbenedix: you are soooo western centric [09:09:58] true [09:10:12] but are the different system completely uncompatible? [09:10:32] no, they just need to be supported [09:10:51] it's pretty far anyway, it will be there this season [09:12:26] it's the next feature on the UI actually :) [09:20:26] hello Denny_WMDE [09:20:51] hi [09:22:28] Denny_WMDE: is there any plan of implementing language fallback in wikibase? [09:26:43] plans yes, but we are lacking a concrete plan of how to do it [09:26:50] we want to do it, it's on the roadmap for the year [09:26:55] but we don't know yet exactly how [09:27:07] it would be great if someone would work that out for us, at least a first draft [09:28:02] Denny_WMDE: I'm considering doing it as a gsoc project, but not sure about its workload [09:29:06] hi [09:30:15] I have a list of thousands of items to delete. What should I do? [09:31:11] New review: Aude; "(2 comments)" [mediawiki/extensions/Wikibase] (master) C: -1; - https://gerrit.wikimedia.org/r/59030 [09:31:57] liangent: i'm not exactly sure all the steps involved, but if they can be split into steps, that would help to get a better sense [09:32:07] the draft on how it might be done is the first step [09:33:54] aude: I don't really have much knowledge about wikibase internals [09:34:08] liangent: hmmm, ok [09:34:46] New patchset: Daniel Kinzler; "Unserialize on the fly to Save memory in Change." [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/59992 [09:35:10] aude: and wikibase keeps changing in my POV ... [09:35:26] when it comes to languages, i don't think we're doing that much different than core [09:35:30] so it may be not so easy to keep up to date [09:35:36] except the $wgLang and content language are the same [09:36:35] New patchset: Daniel Kinzler; "Expand legend on Special:DispatchStats" [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/59030 [09:36:46] liangent: i think the fundamental pieces fo wikibase, like the data model are fairly stable [09:36:54] of [09:37:17] aude: what about PHP interfaces? [09:37:44] the api is due for some rework, hopefully or the goal is to make it easier to understand and work with [09:39:33] liangent: we have interfaces in many parts of the code [09:40:05] on the front end side, not sure as much how the languages work [09:40:15] uses the same i18n system though [09:40:32] the jquery msg stuff so shouldn't be too much different and shoudl be stable [09:40:46] and they get stuff from the api in whatever specified lang [09:42:37] btw. as we talked about mocking recently, this could be interesting for you people http://qafoo.com/blog/037_phake.html [09:43:00] hi tobyS [09:43:02] Kizar: hi [09:43:07] can you post that list for me somewhere? [09:43:14] hi :) [09:43:17] whats the reason for deletion? [09:44:28] the list has this format [09:44:30] Q5655592 | Q9567887 [09:44:32] Q5655920 | Q9574947 [09:44:33] Q5655964 | Q3944674 [09:44:35] Q5655975 | Q10398602 [09:44:36] Q5656596 | Q10399319 [09:44:38] Q5390659 | Q10399320 [09:44:39] Q5656644 | Q9581019 [09:44:48] original item, merged item [09:44:51] sure [09:44:56] can you pastebin it for me? [09:45:08] sure [09:45:11] I guess fallback/langconv stuff shouldn't be done at frontend, (and even conv can't be done in this way) [09:45:14] but isn't complete [09:45:32] well i can delete what you've done so far [09:45:37] I will post in your talk page when done [09:45:40] the api should be changed so when frontend is requesting data for sr-ec, it returns data from sr and saying "this data is from sr" [09:45:54] that works too [09:46:09] the link to pastebin i mean :) [09:46:15] liangent: makes sense [09:46:16] ah ok :P [09:46:17] when frontend is saving data, api should just store it to the specified language [09:46:23] i already have a script that does this so its pretty easy [09:46:46] frontend should ask your what they want to do I think [09:46:53] liangent: sure [09:47:23] but ... this may cause data in variants out of sync [09:48:14] for example, an item has data[en]='color' [09:48:39] when a en-gb user is reading, they would *add* data[en-gb]='colour' [09:48:46] anyone feels like adding a news item about yesterday's deployment to the main page? :) [09:49:22] however some time later, someone writes data[en]='cat' [09:49:25] liangent: think that's the right approach ,though not sure of the details [09:49:45] and they forget to clear data[en-gb] [09:50:31] oh nvm - it is there already actually [09:50:32] Lydia_WMDE: i can [09:50:34] just missed it [09:50:37] legoktm: :) [09:50:50] but in return, can you add what you know to https://www.wikidata.org/wiki/Wikidata:Social_media :) [09:50:55] liangent: so we'd need to know which piece of data they edited [09:50:57] ah ok [09:51:08] and apply just that edit and not the entire item [09:51:11] legoktm: i'll have a look [09:51:15] thanks [09:51:45] legoktm: ok those are all handled by me plus a few others - i will update it but i'll not get to it today - will put it on todo [09:52:04] no rush :) [09:52:04] (plus a few others = a few other accounts) [09:52:09] I don't really understand your last setence [09:55:46] New review: Aude; "(1 comment)" [mediawiki/extensions/Wikibase] (master) C: -1; - https://gerrit.wikimedia.org/r/58464 [09:56:24] liangent: Tobi_WMDE would know more details but i think some parts of the api take the entire json, the code creates a diff on the json, and applies the difference [09:56:48] in some places, it's more atomic, so e.g. just the label part of the item is passed to the api and diff/apply for just that pwrt [09:56:51] part [09:57:01] it's somewhat inconsistent but we are working to improve that [09:57:41] i think the json has lang -> value type structure everywhere, like for labels [09:58:08] so it might have en-gb -> colour mixed with en -> something else [09:58:21] legoktm: any final thoughts on http://www.wikidata.org/wiki/Wikidata:List_of_bots before I templatafy it? [09:58:40] i don't think it would be a problem but we'd need to preserve the language code for each part when passing back and forth [09:58:42] Is "admin" for bot flag? [09:59:05] yes [09:59:10] just incase it ever happens ;p [09:59:52] aude: I don't think it's a well-designed api.. [09:59:58] liangent: agree [10:00:13] it's high priority to fix the issues [10:00:14] addshore: I'd really like to see 1,2,3,4 links rather than just one [10:00:19] mhhhm [10:00:21] ill see [10:00:22] and think how best to do things [10:00:34] just have a bot fill it out [10:00:35] not that hard [10:00:38] liangent: API will undergo heavy refactoring soon [10:00:39] if it's stored in json internally, the api module should expand it on server [10:00:59] legoktm: how about if it is 1, 2, 3, 4, 5, 6, 7, 8 ? [10:01:00] so, don't expect it to stay the same [10:01:01] Change merged: Aude; [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/59030 [10:01:14] or does it start getting too long? :P [10:01:30] works too [10:01:54] taking language fallbacks into account when restructuring the api would be important [10:01:54] Tobi_WMDE: any scheduled date? [10:02:36] liangent: it will be more an ongoing change, not a huge change at once [10:02:50] Tobi_WMDE: if it isnt going to be backwards compatible, please make sure bot writers/framework authors have enough time to migrate [10:03:02] * aude thinks sooner than later [10:03:30] legoktm: from bot writer's view, a huge change at once may be better [10:03:35] legoktm: I'm rather talking about the code design [10:03:41] legoktm: and we'll try our best to keep it backwards compatible, but if any reason to break that, then yes there will be plenty of advanced notice [10:03:47] so they don't need to follow your changes to keep their tool working [10:04:50] i think most, if not all of the changes will be transparent to the bots [10:06:01] * legoktm 's ethernet cord randomly fell out [10:06:05] aude: so new apis returning data parsed from json will be working together with old apis returning json? [10:06:12] aude: thanks :) [10:06:53] liangent: not exactly sure the details but think so [10:07:50] btw how'll phase3 implemented if stuff are stored as json blob? [10:09:00] liangent: http://meta.wikimedia.org/wiki/Wikidata/Development/Queries [10:09:22] that's a general but up-to-date outline of how it will work [10:10:22] at least in terms of searching, having an extra backend data storage service (e.g. solr) is on the roadmap [10:10:54] not sure yet if that might be useful for queries..... [10:16:22] I don't know about solr but will it render current json structure redundant ? [10:16:24] afk [10:16:52] a quick search says it's a search tool hmm [10:18:00] liangent: no [10:18:43] anyway, probably questions for denny :) [10:18:54] or jeroen who's working on queries [10:20:08] right now the geodata extension is using solr, so you can retrieve coordinates nearby, etc. [10:20:22] it can support stuff like that that would be not so efficient or easy with mysql [10:20:42] and stuff gets indexed, etc. in more optimized way [10:21:00] but doesn't replace mysql and the main storage [10:36:13] legoktm: I have another task for u [10:36:22] sure, whats up? [10:37:53] I can give you a list of the remaining items to import from eswiki [10:38:19] Some have problems but many missing [10:39:03] hmm [10:39:10] you should work with addshore on that [10:39:16] his bot is more robust at the moment [10:39:51] Not removing [10:40:03] just create the new items in Wikidata [10:40:37] oh [10:40:38] sure [10:40:45] if you give me a list :) [10:40:57] for example [10:41:01] http://es.wikipedia.org/wiki/A-319 [10:41:22] This article exists in 2 languages but has not Wikidata item [10:42:25] You have exceeded the maximum file size of 500 kilobytes per paste. PRO users don't have this limit! [10:42:27] humm [10:42:28] wait [10:45:26] try dpaste.de? [10:51:06] sorry for the wait [10:51:08] https://dl.dropboxusercontent.com/u/57331437/wikidata_eswiki.txt [10:51:45] got it [10:52:27] perfect :) [10:52:56] When are you going to start? [10:56:40] in a few minutes, was waiting on my IDE to start up [11:03:37] ok [11:07:28] should be running now [11:08:31] i see it [11:09:23] humm [11:09:36] isn't working well [11:09:44] http://www.wikidata.org/wiki/Q11480343 [11:09:52] http://es.wikipedia.org/wiki/ARCO [11:09:57] there are 2 iw links [11:10:17] erwut [11:10:23] hmm [11:13:26] ohhhhh [11:13:29] stupidstupidme [11:13:50] don't worry [11:14:35] my bot can fix that [11:15:11] editions already made [11:17:35] well i just mass deleted them all >.< [11:18:40] great, fixed the bug [11:19:35] the but was i was parsing your list and expecting the line to be the exact pagename, except i forgot that the way i was reading the file would tack on a \r\n [11:19:56] the bug* [11:21:39] Denny_WMDE: picked up bug 45535 as discussed. [11:22:21] now it's fine [11:31:42] aude: go back to the topic of gsoc. is there something suitable can be done here in wikidata related to fallback or conversion? [11:32:25] Kizar: actually im gonna go to sleep now, so i killed the script since i dont trust it 100% right now (there are still some weird issues), so ill restart it once i wake up [13:13:59] DanielK_WMDE: https://bugzilla.wikimedia.org/44844 requires poking ariel to remind him [13:14:34] * aude badly wants these dumps :) [13:15:12] liangent: i think there language fallback and/or conversion is something of reasonable size and scope for gsoc [13:15:28] 1) fallback required first 2) if time allows, conversion would be cool [13:16:02] something that we all want very much, but have no particular plan yet and can use help with it [13:19:09] +1 [13:22:06] aude: one drawback of this idea I can think of is that the dependency (wikibase core/api) is not stable enough (so what I'm doing may have to keep changing), but it can be one advantage too, because I can influent newly designed ones so they can better adapt to what I need [13:49:43] liangent: i think your experience and input would be very valuable [13:50:27] the timeline for us to work on the api is soonish and i hope doesn't take that long (but never know) [13:51:55] ps - denny is a bit distracted today with a visitor here, but normally here more to answer questions [13:58:25] aude: any documentation about api - or do I have to extract it from source myself? [14:00:18] liangent: i was working on some documentation [14:00:24] we might have oldish docs [14:01:02] https://www.mediawiki.org/wiki/Extension:Wikibase/API (outdated) :( [14:08:41] hmm [14:10:00] the bot authors have asked about the docs,and something we need to do asap anyway [14:10:26] * aude experimenting with writing bots, in my spare time :) [14:20:19] liangent: probably what's helpful is the data model documentation [14:20:21] http://meta.wikimedia.org/wiki/Wikidata/Data_model [14:20:32] http://meta.wikimedia.org/wiki/Wikidata/Notes/Data_model_primer [14:20:58] the api (and the code generally) follows that closely [14:24:16] legoktm: around or asleep? [14:28:23] * addshore is going to change his bot to also remove interwikis to sections from his db (i think..) [14:38:40] aude: I saw that before but didn't read it thoroughly [14:39:47] liangent: ok [14:40:14] it helps for understanding how/why things are the way they are, though reading the code works too [14:40:40] and i'll see about updating api docs soon [14:41:22] New review: Aude; "(1 comment)" [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/59992 [15:14:11] Danwe_WMDE: there? [15:15:57] hoo: yeah [15:17:06] about the wikibase.Store refactor... would you rather only have the abstracted functions within wb.AbstractedRepoApi and the other ones still use wb.RepoApi or wb.Abstracted... hold the full functionality [15:17:56] hoo: why? [15:18:10] huh? We need to decide on one :P [15:18:20] oh sry [15:19:23] so you're asking me wether we need the non-abstracted functions in wb.AbstractedRepoApi? [15:19:43] Exactly [15:19:56] I guess we don't and it might even be cleaner to not have them there [15:20:04] That's a design decision to make [15:20:28] I think so as well... tough that'll require a lot of changes outside of the module itself [15:20:50] That will mix both in the code, but certainly it will be more clear that they do [15:21:22] liangent: sorry, been afk. still there? [15:21:51] yes. Will it be that many changes though? E.g. "removeClaim" should perhaps be in the a "abstracted" one anyhow. Perhaps the abstracted one is just something we need until we have a proper store implementation [15:22:15] It could be extended later, I presume [15:22:44] $ grep -r '\.RepoApi' | wc -l [15:22:44] 59 [15:23:19] Should be doable... given that I already did that with wb.Api -> wb.RepoApi earlier this year [15:24:18] yeah, but here you have more changes since only some of those will require replacement while others should still use the RepoApi and not the AbstractedRepoApi [15:24:42] Denny_WMDE: I'm here [15:24:58] Sure... that will need a lot of testing and can easily run into merge conflicts... I hope to get code review for that on monday if possible?! [15:25:57] sorry, i got disconnected for a moment, don't know why, in case you typed something [15:26:47] hoo: sure, perhaps even Sunday [15:27:09] hoo: you could use a reges like /RepoApi\.\w*(?:claim|snak|statement|...)\w*/ for searching the ones requiring replacement [15:27:51] I'll do something like that... [15:28:02] New review: Daniel Kinzler; "(1 comment)" [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/59992 [15:29:25] Denny_WMDE1: nothing more than a "I'm here" [15:33:27] ah cool :) [15:34:34] for anyone with 2 spare mins Q226622 and Q321381 need sorting :/ [15:34:39] so here are a few ideas re language fallback: [15:35:39] when a label (and/or description) is missing we should fallback to some other language [15:36:25] but make it visible in the ui that it is another language [15:36:32] and ideally make it locally editable [15:36:41] so that the user language can be set there [15:37:03] New review: Aude; "(1 comment)" [mediawiki/extensions/Wikibase] (master) - https://gerrit.wikimedia.org/r/59992 [15:37:24] the hard questions are: should there be global language fallback chains? [15:37:32] should there be user-defined ones for each user? [15:37:53] and should we use the automatic script conversion available for some scripts? [15:38:00] step 1 would be to write this all down [15:38:13] the api should be already sufficient. as far as i can tell all of this should be front end work [15:38:14] Denny_WMDE1: probably both, but the question is how to mix them. also, per-user stuff makes caching harder (but atm, we don't cache anyway) [15:38:38] could be. but I'd like to see it written down before implementation [15:38:53] especially for any global language fallback chain. [15:39:00] Denny_WMDE1: we'd also want to fallback to apply when rendering item labels as data values on the client. but only global fallbacks can apply there [15:39:18] do we? [15:39:26] sure. [15:39:43] but we don't have on site editing [15:39:48] yet [15:39:49] otherwise small languages will see a lot of empty values for properties that actually do have a value [15:39:58] it has nothing to do with on site editing [15:40:23] but it should say be somehow displayed differently [15:40:29] to make it obvious it is a different language [15:40:42] maybe. the distinction should be available in lua [15:40:47] right [15:40:50] don't think it fits into the parser function [15:40:51] that would probably be fine [15:40:52] Denny_WMDE1: isn't it acceptable to choose to serve logged-in users with better fallback? this avoids caching issues [15:40:59] or do we have more caching layers? [15:41:18] Denny_WMDE1: if we'd not have that, {{#property:father}} on [[Bach]] would be blank in many many small languages, I suppose. [15:41:18] 04[1] 04https://www.wikidata.org/wiki/Bach [15:41:38] "and should we use the automatic script conversion available for some scripts?" this is the root cause when I want to do this. I'm from zhwiki [15:42:15] liangent: a user might say "i speak spanish", and it that case it might make sense to fallback to spanish, even if their language is hindi [15:42:29] liangent: so we should use the script conversion :) good [15:42:45] DanielK_WMDE: as said, for Lua I agree [15:43:10] i wonder if we should not kill {{#property:}} and use only lua... [15:43:13] addshore: done ;) [15:44:47] Denny_WMDE1: so don't say "their language is hindi" in this case [15:45:05] let users be able to have multiple "acceptable" languages defined [15:45:21] but they would have one major language [15:45:23] just like now [15:45:32] there can be some ordering of course [15:45:34] i use english, but i also say that i speak croatian [15:45:38] Denny_WMDE1: Someone already complained about some things only working w/ Lua... there are many bound to the old, stupid wikitext, I fear [15:45:59] hoo: you can wrap Lua in old stupid wikitext [15:46:12] That's what I told him... [15:46:23] it would look like {{property|xxx}} instead of {{#property:xxx}} [15:46:23] 04[2] 04https://www.wikidata.org/wiki/Template:property [15:46:28] not that huge a difference [15:46:58] Maybe we should provide such modules in case there ever is a central repository later... [15:47:35] yeah, that would be nice [15:49:08] Denny_WMDE1: I guess we may want to change fallback chain to let every level a "quality degrading score" [15:49:38] for example, zh-hk falls back to zh-hant, then zh, finally en (ignore conversion here) [15:50:01] the first two levels are almost the same [15:50:04] that sounds good. but how do you mix the global chain with a local one? [15:52:19] i.e. global: zh-hk, zh-hant, zh, en — local: main language zh but also fr, es, ro [15:52:34] that kind of thing I'd like to see written down and agreed on first [15:52:35] for example, I'm defining my Accept-Language: zh-hk;q=1,es;q=0.8,en;q=0.5 and the zh-hk fallback chain has zh-hk;q=1,zh-hant;q=0.9,zh;q=0.7,en;q=0.1 [15:52:52] so if you serve me zh, it has 0.7*1=0.7 [15:52:55] hmm, can we go around using scores like that? [15:53:03] or if you serve me es, it's 0.8 [15:53:06] so give me es :) [15:53:08] where do we get the scores for the languages from? [15:53:52] I guess we don't have many (non-en) fallback yet, so defining them one by one can be acceptable [15:54:06] though the exact values need some discussion [15:54:18] Denny_WMDE1: the scores express the user's preference. the essentially just impose an order of preference, they don't have any absolute meaning [15:54:22] it's in the HTTP spec [15:54:37] no, i know http [15:54:47] i meant, where do we get them for wikidata [15:54:57] we are not going to take the browsers http header, are we? [15:55:07] that would add another layer to global chain and userdefined [15:55:18] and would kill caching even more thoroughly [15:55:31] (just in case it ever thought about resurrection) [15:55:50] Denny_WMDE1: sorry I'm just borrowing the concept from http [15:56:08] I was thinking about storing zh-hk;q=1,es;q=0.8,en;q=0.5 as users' preferences [15:56:18] Denny_WMDE1: we could get them from the browser once when the user creates an account. [15:56:18] we have something similar [15:56:34] for reference: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.4 [15:56:36] anyway... [15:56:38] http://www.wikidata.org/wiki/User:Denny [15:56:47] gives an ordered list [15:56:57] about cache: how many caching layers do we have? [15:57:17] more than you think :) [15:57:21] liangent: web cache for logged out visitors. (currently) no parser cache. [15:57:35] (for data items on the repo) [15:57:56] various things are cached in the object cache, but I think that's it for actual page content. [15:58:33] we'd like to get the parser cache enabled again. which would mean no per user fallbacks (or fallbacks get applied via JS). [15:58:48] so I was asking the question above: is it acceptable to decide to serve logged out users worse than logged in users? [15:59:16] liangent: we already do. anons can't set the interface language at all. [15:59:25] they get english content, period. [15:59:30] DanielK_WMDE: not true [15:59:35] Denny_WMDE1: no? [15:59:37] they have a javascript script from commons [15:59:41] they can set it as they like [15:59:45] works well [15:59:58] Denny_WMDE1: beware then that that bypasses the web cache [16:00:09] if it becomes popular, we are going to have load issues [16:00:49] as far as I can so, *no* caching is done for page content in such a case. [16:00:55] *see [16:00:55] let's make sure we don't get too popular then… :P [16:02:33] is http://www.wikidata.org/wiki/Q1?uselang=de being cached? [16:02:50] or is it always created fresh? [16:03:05] Denny_WMDE1: No, it's Squid cached for anons [16:03:18] well, then there's no issue [16:04:48] hoo: then how do you make sure they get purged when someone edits the item? [16:05:55] hoo, Denny_WMDE1: it's *not* cached, any query parameter disables the cache. [16:06:14] DanielK_WMDE: Not *any* AFAIR [16:06:19] liangent: that's one of the really nasty issues with per-language urls: we have to purge 300x [16:06:49] hoo: i'm pretty sure this one bypasses it. precisely because of the issue mentioned by liangent [16:06:56] i discussed this at length with rob & co at wikimania last year [16:07:05] mhm, that would suck as we have a lot of such links towards commons [16:07:05] maybe something has changed since then, but i highly doubt it [16:07:14] it does suck :) [16:07:25] I think we had whitelisted some [16:07:46] hoo: sure, if they are *ignored* in the cache key. [16:08:06] but cachign that means to fracture the cache by value of uselang [16:08:13] and would make efficient purging impossible [16:08:27] the only way to get around this is some pretty big hackery in the squids... [16:08:39] maybe that has been done, but i clearly recall being told that this does not work. [16:09:26] anyway [16:09:35] Varnish would solve that [16:09:46] we'd need mark or soem other squid viz for a definitive answer [16:10:02] hoo: not easily, last time i looked... but would sure make it easier [16:10:46] It's more dynamic than squid [16:11:03] but it also van't vary on individual cookies [16:11:06] which i find very silly [16:11:12] anyway, off to fixing dinner [16:11:58] Guten Appetit ;) [16:12:18] anyway, liangent, I think there's plenty of fallback work that can be done without upsetting caches [16:12:34] like conversion [16:13:30] yep and I've already hacked around the conversion stuff before [16:13:37] sweet [16:13:45] and a draft on how local + global can be combined [16:13:54] there are not that many possibilities anyway [16:14:03] and something simple, not too smart [16:14:33] Denny_WMDE1: about caching, are there any documentation available, for our current architecture? [16:14:47] sorry, i don't know [16:14:56] some wikidata-specific one is enough [16:16:30] i don't think there's anything wikidata-specific [16:16:37] written up [16:24:36] by the way, to all the bot runners: awesome job [16:24:55] it is amazing to see how you push the edit rate up but keep the dispatch lag down [16:28:45] lbenedix: you there? [16:45:07] off see you [17:14:11] New patchset: Daniel Werner; "Reorganization of DataValues extension's JS resources" [mediawiki/extensions/DataValues] (master) - https://gerrit.wikimedia.org/r/60025 [17:42:06] hey wikidata people, are you happy with the redirect now [17:42:15] Tim merged patch set 12:) [17:45:03] mutante: it's good :) [17:45:06] yay:) [17:45:23] we still need to think about redirecting the "language" subdomains though [17:45:38] maybe figure that out together in amsterdam? [17:46:12] mutante: you'll be there right? [17:46:16] sorry it took so long on our side, but as you see it had quite a discussion and i really preferred others to confirm the technical part after 12 patch sets.. since i already broke it once [17:46:49] agree [17:46:50] yes,i'll be there. let's get Roan or someone from platform as well [17:47:07] took time to understand all the parts but think i get it now [17:47:15] * aude don't want to break wikidata :o [17:47:25] the subdomains are special [17:47:30] heh, yea, me neither, and last time it was when we tried getting rid of the www [17:47:37] not good [17:47:38] and that meant it wasnt hitting the loadbalancers at all [17:47:44] yep [17:48:06] it's like you have to use a CNAME to load balance, no way to use the "naked" domain [17:48:15] oh well, glad it's in pretty good shape now [17:48:19] http://meta.wikimedia.org/wiki/Wikidata/Notes/URI_scheme is what we have in mind though think we are open to suggestions, as needed [17:48:25] ok [17:48:48] we might also want https://gerrit.wikimedia.org/r/#/c/56153/ [17:48:58] no need to have cors for wikidata.org (w/o the subdomain) [17:49:22] you are the only project that is both.. a regular project .. (main.conf) and additionally stuff in redirects.conf afaik [17:49:34] not sure about the *.wikidata.org [17:49:43] yeah, that's unique [17:49:53] aude: www.wikidata.org should be enough in case the config. is correct [17:50:00] hoo|away: sure [17:50:02] I hope it si [17:50:03] no * [17:50:11] but the * does not hurt [17:50:23] we may want that later [17:50:32] removing you from CommonSettings.php .. suggesting Reedy as reviewer [17:50:36] lemme add him [17:50:45] ok [17:50:46] Unless people start to do requests to de.wikidata.org which we probably don't want them to do?! [17:51:01] hoo|away: agree [17:51:25] i heard from Roan that the plan is to use DNS to set the language, like en.wikidata.org would redirect to www.wikidata.org/../bla?lang=en or so [17:51:32] ? [17:51:37] Going to finish the store refactor tomorrow... of for BBQ now ;) [17:52:02] mutante: not quite sure [17:52:30] Lydia_WMDE: heya, do you have a good one url for the phase 2 stuff that is happening next week? [17:52:41] like a blog post or some other announcement that you've already done? [17:52:55] we have this: http://meta.wikimedia.org/wiki/Wikidata/Technical_proposal#Phase_2:_Infoboxes already [17:53:00] aude: ok, let's talk more later and/or in AMS [17:53:24] we could have en.wikidata.org/wiki/New_York_City redirect to www.wikidata.org/wiki/Q60?uselang=en (or setlang=en --- sticky) [17:53:27] so yes [17:54:09] we have a special page that can do some of the redirecting and tim thinks we could have a special entry point like that to handle it [17:54:22] not really sure, but can chat later [17:54:38] shall need to experiment on my test setup [17:55:13] greg-g: we had something for the first 11 wikipedias [17:57:34] https://blog.wikimedia.de/2013/03/27/you-can-have-all-the-data/ (not too informative) [17:57:37] http://meta.wikimedia.org/wiki/Wikidata/Deployment_Questions [17:57:59] Lydia_WMDE: might know more [17:58:35] aude: i'm glad to hear apparently Tim already has some thoughts on that part [17:58:43] yeah [17:59:01] it's a bit unique what we're trying to do [17:59:11] so needs some thought [17:59:29] nods [18:06:28] aude: ooo, nice one, the Deployment Questions one [18:06:31] aude: thank you [18:07:34] greg-g: yw [18:09:01] Lydia_WMDE: I remember, that you said, that the information in wikidata will be maintained by the community and not the bots... is there any article where you describe how and why this goal should be accomplished? [18:11:06] * Moe_Epsilon waves and disappears [18:18:30] Lydia_WMDE: Do we already have a idiots guide to qualifiers? Could use one myself :P [18:18:37] (to see if https://www.wikidata.org/wiki/Wikidata:Property_proposal/Generic#Image makes sense) [18:19:56] multichill: http://meta.wikimedia.org/wiki/Wikidata/Notes/Data_model_primer#Qualifiers might be best we have [18:20:09] (btw, need to add you to maps project and make the instance) :) [18:21:48] Thanks aude, I remember reading that. That makes data modelling a lot easier [18:22:25] multichill: added you the project, you can make the instance if you prefer [18:23:02] No, I don't prefer that. I didn't have a lot of sleep in London and I went into standby right after it [18:23:05] for qualifiers, i'm not quite sure which properties to use or make (as editor) [18:23:16] multichill: heh [18:23:26] malfunctions have kept me up for the last three nights so I'm completely exhausted :P [18:23:27] k [18:24:47] legoktm: New fun stuff to implement: http://meta.wikimedia.org/wiki/Wikidata/Notes/Data_model_primer#Qualifiers [18:25:07] Aude, any idea when the missing datatypes like time will be added? [18:25:15] multichill: no [18:25:20] don't want to overpromise [18:25:29] Time is very very complicated [18:25:33] it is [18:27:33] Would be nice to have it somehow work together with things like https://www.wikidata.org/wiki/Q1998 [18:28:08] ah, yes [18:28:30] * aude like to qualify properties like president with date ranges [18:28:40] Yes, exactly [18:28:55] That would be great for the infoboxes for previous and next [18:29:00] yep [18:29:33] And you could make such an awesome search engine with all this extra structured data [18:30:28] hope so [18:30:39] Q: Give me a list of all heads of state on 5 feb 1917, something like that would be possible [18:30:53] * aude nods [18:43:14] greg-g: yeah the two links that aude posted (sorry was away) [18:43:45] lbenedix: i don't think i said it this way ;-) [18:43:51] multichill: i fear not, sorry [18:44:12] Lydia_WMDE: no problem, you were popular while you were away, too! Thanks! [18:44:18] greg-g: hah indeed [19:37:08] please add to http://www.wikidata.org/wiki/Wikidata:Status_updates/Next really soon if there is anything you want in the weekly summary [19:37:39] greg-g: btw - not sure anyone tells you this but the deployment highlight emails are really useful [19:40:08] Lydia_WMDE: thank you for saying that! [19:40:24] this was probably the best one yet, much more time was put into it [20:14:08] Lydia_WMDE: done with weekly summary [20:14:20] sorry forgot earlier :( [20:14:36] aude: np - have not started preparing it yet so it's fine :) [20:14:45] k [20:55:55] I have simple questions regarding cross referencing from say an en article to articles on other language version of wikipedia [21:00:22] Rjlabs: sounds like first you want to check is if Wikidata has already been enabled on the languages you are interested in [21:00:32] because it might do the interwiki links [21:00:48] !ask [21:01:24] is there a central "term index" that is controlled and language independent? [21:01:56] with spokes out to each foreign language article on that centeral "term" [21:02:29] yea, that term index is wikidata [21:02:48] How do I learn how its actually constructed? [21:03:06] http://www.wikidata.org/wiki/Wikidata:Introduction [21:03:45] is it "literal" or fuzzy, that is are the cross language links very strict or are they more "fuzzy"? [21:06:10] Im seeing som cool stuff on that intro site! OWL? [21:09:53] oops was cut off [21:11:23] how is the central term index that is "language independent" constructed? [21:16:41] Rjlabs_: these are the pages on wikidata, they are all called "Q" and a number, example the term that is called "universe" in English is Q1 http://www.wikidata.org/wiki/Q1 [21:17:16] but i'm sure others who actually code on it can give you way more details [21:18:20] I keep loosing connections [21:20:09] http://www.wikidata.org/wiki/Q1 this is fantastic! wow!!! is it croud sourced and do those terms get specific / esoteric [21:21:37] and any attempt ,at more "fuzzy" cross referencing? See also, used for, broader term, narrower term... [21:22:07] it's like any other wiki, anyone can edit pages [21:22:59] Rjlabs: http://meta.wikimedia.org/wiki/Wikidata/Notes/Data_model_primer [21:23:19] http://meta.wikimedia.org/wiki/Wikidata/Notes/Future [21:26:39] lots of drops [21:28:22] http://meta.wikimedia.org/wiki/Wikidata/Notes/Data_model_primer and http://www.wikidata.org/wiki/Q173800 are WAY COOL!!! [21:29:27] how are Q's assigned/administered? only one Q per knowledge topic, correct? [21:29:46] Q itself is language independent [21:30:04] only that which is attached is language specific [21:31:10] it just started with Q1 and then counting up [21:31:19] whenever people add a new page [21:32:22] write a new article in any language and a Q is created by "real people" [21:32:22] ? [21:33:39] "Also known as" must be to another Q? [21:34:26] yes, Q's are created by real people [21:34:36] you could add one [21:34:41] Can one Q point to 2+ articles in the same language that are similar? [21:35:00] yet have radicaly different titles [21:35:48] i'm not sure, i'm not an expert in the data model, i do think though there have been ideas about using more "fuzzy" links like "see also" etc [21:36:12] way cool [21:37:07] what is hard is going from esoteric terms (eg. busines terms or psych terms) in one language to all the possible, "fuzzy" related terms in other languages [21:37:17] particularly crossing west and east [21:38:18] I'm envisioning a term search in one language, cull to all relevant articles in that language ,then start looking those up in the Q system? However would that give you good "expansion" on the other side.... [21:38:47] That is how do you get the nearby subjects? [21:38:56] in the foreign language? [21:41:00] we need access to the wikilinks inside the article and the see also (presumably related) to some how pump into a Q like system and expand othe ther side (the foreign language), so you ge the nearby articles on that end [21:41:34] based on the "locals" who put in the wikilinks and see also's in their nativie language [21:41:45] sort of a many to many relationship [21:42:39] addshore: Hi [21:42:43] Don't do section links [21:42:51] [01:24:47 PM] legoktm: New fun stuff to implement: http://meta.wikimedia.org/wiki/Wikidata/Notes/Data_model_primer#Qualifiers <-- will be implemented tonight! [21:42:54] Do you follow? is there any slick way of doing that now? or just have to start with several pages on one language, hit the q system, get the pages in the other language, then decend into those pages, get the wikilinks and the see also's and go from there [21:47:00] Why do I feel like I'm back in my Prolog days with predicate logic as I look at http://meta.wikimedia.org/wiki/Wikidata/Notes/Data_model_primer#Qualifiers? [21:54:08] legoktm: i had been keeping them in my db until now [21:54:12] gonna get rid of them al [21:54:26] Well you shouldn't remove section links from page text [22:27:36] legoktm: I dont :P but i feel they may be the bulk of my db now ;p [22:27:45] :D [23:13:29] heh my bots report page has broken xD [23:16:26] xD