[01:05:08] question: does SPARQL have an equivalent to the MySQL "in ('a', 'b', 'c')" ? [01:10:26] ...I guess it doesn't matter, if I can just get all of the values for a given property anyway. [01:10:43] That might be more performant. [01:35:14] harej: https://www.w3.org/TR/sparql11-query/#objLists ? [09:31:14] I added the VIAF identifier to Wikidata and as a consequence en.wp provides the Worldcat information ... here https://www.worldcat.org/identities/containsVIAFID/79065511 [09:31:39] This provides other identifiers.. do we have a tool that gets such information ? [09:32:01] and adds it to wikidata? [09:48:36] I'm looking for a copyleft triplestore [09:48:50] any one of these you recommend https://en.wikipedia.org/wiki/List_of_subject-predicate-object_databases ? [09:49:52] could be a full graph database too [09:50:43] graph database and triplestore are internally different but graphs can be constructed from triplets and graphs can be expressed in triplets [14:29:07] Can someone invite me into to admin channel? addshore? [14:41:27] Amir1: you are on the access list, you should be able to invite yourself [14:41:44] Vogone: I didn't know that [14:41:45] thanks [14:42:57] "#wikidata-admin: You're not on that channel" [14:43:03] /cs invite #wikidata-admin [14:43:05] ;) [17:10:55] Thiemo_WMDE: https://phabricator.wikimedia.org/T160436 [17:11:02] Hallo. [17:11:39] Maybe it was announced somewhere and I missed: What's the status of Wikibase Quality External Validation? Is it used? Is it developed? Is it deployed anywhere? [17:12:50] aharoni: stalled [17:13:21] it was developed as part of a bachelor thesis. the thesis is done. the extension is not. [17:15:02] DanielK_WMDE: No problem, just wondering. I have a list of projects in translatewiki that I take a look at periodically. The highest priority ones are those that are deployed on the live Wikimedia sites and I translate them every day. Those that may be deployed some day are lower priority and I only translate them when there's nothing to translate in higher-priority. [17:18:56] aharoni: i see. you may be interested to know then that the constraints check extension is deployed, and under development again. [17:20:24] DanielK_WMDE: in the list already, as part of "Extensions used by Wikimedia" :) [18:57:18] <|Jeroen|> hellow! [18:58:40] DanielK_WMDE: around? [18:58:56] DanielK_WMDE: wanted to talk a bit about the search things [19:00:20] <|Jeroen|> is there a way to get the page content using SPARQL ? [19:00:35] <|Jeroen|> or at least the first part that describes the item [19:02:01] |Jeroen|: you can get a description, with schema:description [19:02:13] SMalyshev: i'm around, and Lydia is sitting besides me. we were just saying that we need to finally reply to you :) [19:02:18] beyond that, not much, the sparql db does not have artile tests [19:02:29] SMalyshev: so yeah we can talk now [19:02:47] DanielK_WMDE: heh :) so do you have minute to discuss https://www.wikidata.org/wiki/User:Smalyshev_(WMF)/Wikidata_search? [19:02:58] |Jeroen|: what would "page content" be? HTML? There is an API for getting ther JSON representation of entities [19:03:35] <|Jeroen|> DanielK_WMDE, yeah page html would be ok also [19:03:47] SMalyshev: i edited it, did you see? [19:03:56] DanielK_WMDE: yes I did [19:03:58] SMalyshev: and I'll edit again right now, to add an idea i had last night [19:04:06] ok go ahead [19:05:10] <|Jeroen|> what about getting the real wikipedia url via sparql? [19:06:54] |Jeroen|: http://tinyurl.com/zgn8u5b [19:08:26] <|Jeroen|> thx WikidataFacts [19:08:31] np [19:08:48] <|Jeroen|> quite weird this qparqsl :p [19:08:52] <|Jeroen|> i prefer regualt sql :p [19:08:59] * Lydia_WMDE watches DanielK_WMDE type and type and type :P [19:09:34] |Jeroen|: heresy! :O [19:09:40] <|Jeroen|> lol [19:09:40] Lydia_WMDE, SMalyshev: https://www.wikidata.org/w/index.php?title=User%3ASmalyshev_%28WMF%29%2FWikidata_search&type=revision&diff=466322124&oldid=466319080 [19:09:49] <|Jeroen|> wel i probaly just don't get it [19:10:14] |Jeroen|: what'S a "real" wikipedia URL? [19:10:38] <|Jeroen|> wel an url to the page [19:10:49] |Jeroen|: well, there are no tables in RDF, it's more graph-like, so you need a language specialized for this. [19:11:05] <|Jeroen|> what i actualy want is a list of all beers with there picture and url and even body if possible [19:11:13] |Jeroen|: the wikidata page? or the corresponding page in a specific wikipedia? [19:11:29] what would "body" contain? [19:11:40] <|Jeroen|> the page you can c on wikipedia [19:11:41] the html from wikipedia? [19:11:46] <|Jeroen|> or a description of the beer [19:11:53] <|Jeroen|> yeah somthing like that [19:12:09] DanielK_WMDE: I am not a big fan of "hook returning opaque HTML" thing. It'd be trouble once we want to change something and have no control over what extensions return [19:12:17] |Jeroen|: there are neearly 300 wikipedias, one per language. you'll have to know which one. BUt if you do, you can get the link from sparql, yes [19:12:37] <|Jeroen|> wel the english one would do [19:12:44] SMalyshev: i'm fine with returning a data structure. [19:13:25] DanielK_WMDE: otherwise, your idea seems to be pretty much what I meant by "visually separated" results... GUI code calls two API functions and displays the results separately [19:13:40] |Jeroen|: https://www.wikidata.org/wiki/Special:EntityData/Q1.ttl is the raw RDF for the item about "universe". You should be able to see what kind of properties and relations exist there, and then query them using sparql [19:13:44] DanielK_WMDE: it's not related to Cirrus, Cirrus part won't even know it's happening [19:15:02] <|Jeroen|> yeah i am trying that, but i can't seem to grasp the logic of this sql [19:15:04] <|Jeroen|> there is no from! [19:15:07] SMalyshev: yes, the point is: 2 API calls. I'd like to have a generic extrension point for that. [19:15:37] |Jeroen|: you are looking for triples of the form schema:about wd:Q12345. [19:15:39] <|Jeroen|> i would like SELECT * from wikipedia where type ='beer' :p [19:15:39] DanielK_WMDE: ok, I don't object to N API calls instead of 2, driven by extension points [19:16:07] |Jeroen|: sadly, our data model is a bit more compelx than that? [19:16:21] <|Jeroen|> yeah i undrstand [19:17:05] this query should work, but for some reason I get “unable to display result”… I’ve never seen that in the table view before o_O http://tinyurl.com/grde53y [19:17:21] (seems to be between results 300 and 350, according to some quick LIMIT bisecting) [19:17:41] SMalyshev: it seems more robust and more flexible than trying to combine title and lable search in cirrus somewhow [19:18:12] <|Jeroen|> well i have : SELECT ?item ?itemLabel ?pic ?brewery WHERE { [19:18:13] <|Jeroen|> ?item wdt:P31 wd:Q44. [19:18:13] <|Jeroen|> OPTIONAL { ?item wdt:P18 ?pic. } [19:18:13] <|Jeroen|> OPTIONAL { ?item wdt:P1071 ?brewery. } [19:18:13] <|Jeroen|> SERVICE wikibase:label { bd:serviceParam wikibase:language "en". } [19:18:13] <|Jeroen|> } [19:18:15] <|Jeroen|> LIMIT 200 [19:18:24] <|Jeroen|> but i would like the url with it for example [19:18:34] DanielK_WMDE: the question here would be - is one of the searches wbsearchentities here? Does this mean we don't need to support mixed search for entity/article namespaces? [19:19:25] |Jeroen|: OPTIONAL { ?url scham:about ?item } [19:19:42] (i think - i'm not a sparql qizard) [19:19:46] or wizard, even [19:20:20] SMalyshev: yes, it would use wbsearchentities. my thinking is that we do not need to support it for quick serach, at least. [19:20:30] I'm not sure how this should work on Special:Search, though [19:20:46] <|Jeroen|> Query is malformed: QName 'scham:about' uses an undefined prefix [19:21:41] schema:, not scham :) [19:22:10] DanielK_WMDE: yeah mixed search is a problem I keep encountering in different scenarios. I'd say maybe for now we just not allow it and produce some warning and just choose one option [19:22:44] DanielK_WMDE: right now the API does not really allow to return warnings, we probably need to refactor wbsearchentities helper API to return Status [19:23:04] so that we could attach warnings to the results [19:24:03] <|Jeroen|> lol, i was whondering what sham was :p [19:24:34] sounds like a good prefix for testing purposes ;) [19:25:01] DanielK_WMDE: so, do you want to go over TBDs there one by one and discuss them? [19:25:49] DanielK_WMDE: Hey, do you know how to get the list of valid language codes for wikidata.org? [19:25:55] For https://www.wikidata.org/wiki/Wikidata:Project_chat#Languages_not_localized_in_English [19:25:59] SMalyshev: talking to lydia how to combined the results of two api calls. lydia favors interleaving. but that decision can be left for later [19:26:28] multichill: IIRC those come from some Language class [19:26:55] \MediaWiki\Languages\Data\Names::$names [19:27:17] Jonas_WMDE: Lydia_WMDE: See latest comment at https://phabricator.wikimedia.org/T158941 [19:28:34] SMalyshev: yea, let's do that [19:29:16] Nevermind, I think I found it at https://www.wikidata.org/w/api.php?action=query&meta=siteinfo&siprop=languages [19:29:26] multichill: we have some extra codes configured somewhere. also, languages supported for labels are not 100% the same as languages supported for monolingaul text. but i don't remember the details [19:29:34] Krinkle: thx [19:30:08] Krinkle till when should this be done? [19:30:41] SMalyshev: so this is about how to merge results from entity namespaces and other namespaces? [19:30:54] DanielK_WMDE: ok, 1: mixed namespace search for APIs. The thing here currently we try to guess the namespace from query text, so query startiing with Category: implies category namespace [19:30:54] DanielK_WMDE: Must be some api function to check if a language code is valid? [19:30:59] time to eat [19:31:06] then i think getting X result from entity namespace and then X from the other namespaces seems like a workable solution [19:31:17] Jonas_WMDE: The sooner the better, it's currently blocking discovery of further tech debt, which in turn blocks the upgrade itself. [19:31:22] multichill: try to use it, it will fail if it is invalid :) [19:31:25] I was hoping this quarter, but will probably have to be next. [19:31:35] Lydia_WMDE: well, before "how to" there's "whether to". In general, the question is - do we do mixed NS search, and if we do, then how [19:31:38] SMalyshev: i'm not sure we need an API that does mixed search [19:31:44] jQuery 3.0 brings a lot of performance improvements [19:31:52] Lydia_WMDE: and whether something like Category:Fo would do mixed search [19:31:55] SMalyshev: people are asking to get results from other namespaces, yeah [19:32:08] but maybe doesn't need its own api yeah [19:32:08] Lydia_WMDE: what exactly are they asking? [19:32:18] many of which will benefit Wikidata too, considering it uses jQuery animate() and other expensive methods quite a lot. [19:32:22] mostly getting project namespace results in the top search box [19:32:37] when looking for wiki projects for example or help pages [19:32:38] Lydia_WMDE: without prefixing it with proper namespace? [19:32:49] i think they'd use the namespace [19:33:25] Lydia_WMDE: i.e. do they want Wikidata:Proj to find https://www.wikidata.org/wiki/Wikidata:Project_chat or just "Proj" to find it? [19:33:36] Lydia_WMDE: if they want Wikidata:Proj that's not a problem - it's how it works now [19:33:53] from what i know the first is what is being asked for - the second would be nice to have [19:34:04] SMalyshev: on some 3rd party wikis, there would be text in the main namespace, and items would live somewhere else. on such a wiki, searching without the namepsace prefix should also find titles. [19:34:07] but that doesn't give you suggestions [19:34:17] and completion [19:34:22] DanielK_WMDE: titles of what? [19:34:25] you have to type in the full page title [19:34:35] SMalyshev: pages in the main namespace [19:35:08] DanielK_WMDE: ah, sure, if you don't prefix you get pages in main NS. In fact, it probably should be configurable parameter which namespaces are searched by default [19:35:21] yes, probably. [19:35:38] and other namespaces are search (in addition to entities) if a namespace prefix (or alias!) is given [19:36:27] i guess the UI doesn't know about this. it just makes two API queries and (somehow) combines the results [19:37:22] SMalyshev: btw, searching for Wikidata:Proj in the quick box currently does not work on wikidata.org [19:37:25] right, that'd be the simplest [19:37:30] ...because that box is an entity selector [19:37:43] DanielK_WMDE: that's because wikidata's quick search is not the standard one [19:37:55] but we'll change that back to standard one [19:38:27] yes, that'S the idea [19:38:44] the standard one, with a callback that calls wbsearchentities [19:39:40] SMalyshev: ok. so: no API for combined search. Quick search can be extended using callbacks. UI takes care of combining results. [19:39:54] wbsearchentities gets an implementation based on Cirrus/Elastic [19:39:57] DanielK_WMDE: so, what we say is that API:opensearch would support searching in item namespaces only when given item namespaces or item namespace prefix in the query [19:40:49] SMalyshev: hm, oh, opensearch. that *should* return combined results, no?... [19:41:07] DanielK_WMDE: well, that's exactly the question :) [19:41:25] DanielK_WMDE: I tend to say no, you have to search either article or item namespaces [19:42:08] DanielK_WMDE: for simplicity, I'd say just if you mention any non-item namespaces (either in search string or in namespaces param) then you get article search [19:42:13] SMalyshev: i'm a bit confused about opensearch vs prefixsearch vs completionsearch. [19:42:30] with exception being default NS (0), which does not really set a specific namespace [19:42:35] opensearch is for 3rd parties, prefixsearch for the quick search box, but could be replaced by compeltionsearch soon? is that it? [19:42:39] but uses defaults configured [19:42:53] DanielK_WMDE: no, quick search box is using opensearch [19:42:59] *sigh* [19:42:59] at least on enwiki it does [19:43:13] will it stay that way? should it? [19:43:56] well I have no idea by why not? completion is just a mode of the search, internal detail of which algorithm is used (i.e. you can turn it on and off without changing the gui) [19:44:04] IIRC at least :) [19:44:15] SMalyshev: if opensearch needs to be useful for 3rd parties, it needs to do a combiend search. for the quick search box, i don't want it to do a combined search [19:44:24] not sure what API:Prefixsearch is doing now even... [19:44:38] also, can opensearch search multiple namepsaces? how are namespaces specified? [19:44:45] are we conforming to this? http://www.opensearch.org/Specifications/OpenSearch/1.1 [19:44:47] DanielK_WMDE: how would you do it combined? [19:45:13] DanielK_WMDE: hmm good question not sure about multiple NS... let me check [19:45:39] i don't know how, tbh. [19:46:35] we can trigger the search profile on namespace, if a single namespace is given explicitly. but only then [19:46:41] we have the same problem on special:Search [19:47:17] ah nice, opensearch lets you define a search language. [19:48:17] SMalyshev: Lydia suggested that just interleaving results from different searches may be ok for prefix matching. The results shouldn't be as bad as they would be for a general search using that approach. [19:50:48] DanielK_WMDE: well I guess it's possible... may look a bit weird. the question is do we want to do it? I.e. I'm not even sure how well current opensearch api supports multiple namespaces [19:51:07] completion suggester doesn't support non-article ones for example [19:51:21] SMalyshev: we should distinguish between the opensearch API module, and the public OpenSearch interface we expose to 3rd parties. While they are physically the same, conceptually, they are not. The public OopenSearch interface does not support namespaces, afais [19:52:18] DanielK_WMDE: hmmm true that. Which is easier for us, since that means we can have only one non-default NS - one detected from search string [19:52:27] SMalyshev: for quick search wikidata.org, we need *especially* non-article namespaces. btu we may requrie the namepsace to be typed in [19:52:57] yes. [19:53:03] DanielK_WMDE: if we say "we support only in-string namespaces and ignore namespace param on opensearch" that may work [19:53:04] btu the api mudule still supports multiple namespaces i think [19:53:08] let me check the code quickly [19:53:25] it supports it, but how is a tricky question [19:53:29] https://www.wikidata.org/w/api.php?action=help&modules=opensearch [19:53:42] well, it recognizes the param, but what happens next... [19:55:12] DanielK_WMDE: check out normalizeNamespaces() in SearchEngine.php. That's where the magic happens [19:55:36] $ns = [ $title->getNamespace() ]; [19:55:55] basically if the search string contains non-default namespace, namespaces parameter is completely ignored [19:56:05] that makes sense [19:56:41] also there's PrefixSearchExtractNamespace which allows exts to inject namespaces at will [19:57:59] also, completion search does not work if namespace is not main [19:58:09] it falls back to old DB prefix search then [19:58:42] so branching on namespace is not unheard of [19:59:04] no, but combining strange namespaces is tricky. [19:59:19] like, say, you want to search properties, and the help namespace. [19:59:30] makes little sense anyway... but what to do? [19:59:42] DanielK_WMDE: the problem with wikidata is that in regular wiki, DB prefix search does something useful. In wikidata, it doesn't for items [20:00:14] DanielK_WMDE: I'd say we don't support searching both help and properties via API. Via GUI maybe, through multiple API calls [20:01:25] opensearch API would return then item search if search string has no namespace prefix, and the proper NS search if it does [20:01:37] which means Category:Fo will find only actual category [20:01:38] the multi-namespace-issue mostly arises for Special:Search [20:02:05] "don't supprot" means what? fail? use only the first namespace? fall back to something dumb? [20:02:09] DanielK_WMDE: not only. The question was - if you search for Category:Foo - do you get only the category or also wikidata item with such label? [20:02:38] SMalyshev: atm i think, from the api, you get only the title. in quick search, you get both, because of two api calls. [20:02:50] DanielK_WMDE: probably will use only one of the search modes. And issue warning [20:03:08] SMalyshev: well, I am talking API right now [20:03:25] we'll get to the GUI later :) [20:03:51] DanielK_WMDE: so, if you pass 'Category:Foo' to opensearch, you get a category (article title) search [20:04:35] same for prefix search which is now hardwired to do DB search only anyway [20:04:50] so I'm not even sure we can do anything different for prefix search API [20:05:08] I guess we could but should we? [20:05:29] I'm nto sure what the intended semantics of prefixsearch is [20:05:35] is it like SPecial:PRefixIndex? [20:05:44] yes basically [20:05:50] in that case, no. that is defined to search for titles, and it's nto supposed to be smart. [20:06:21] it's most basic DB title search. It has a hook though so it can be overridden [20:09:09] DanielK_WMDE: actually, I am wrong, I was confused between class and API... [20:09:26] the prefixsearch *API* is basically the same as opensearch but with different result format [20:09:42] and i'm confused about using opensearch for prefixes in quick search. to me, opensearch should be full text... no? [20:09:46] with some options, it would use PrefixSearch *class*, which is the DB one [20:09:54] Ah, I see [20:09:55] but with other options, it would use completion search [20:10:14] what would use completion search? [20:10:15] nah, opensearch is title search afail [20:10:33] DanielK_WMDE: search in main namespace, if enabled by configs [20:10:50] via what api? [20:11:09] both opensearch and query/prefix [20:11:13] i think we really need to figure out the intended use case of, and the relationships between, the different search apis [20:11:23] without that, i can't really say which should do what. [20:11:39] anyway, i'll run away real soon. need dinner. [20:11:56] DanielK_WMDE: ok, want to set up another time to continue this? [20:13:33] SMalyshev: hm, not tomorrow. thursday, but earlier than today. afternoon my time, of that's ok for you [20:13:52] SMalyshev: i think that it would be best to specify expected behavior from the user perspective first. [20:14:05] Lydia_WMDE should have some ideas for that [20:14:31] SMalyshev: let's talk that through in the call tomorrow? [20:14:37] DanielK_WMDE: we need to remember there are several users here - API, quickearch, suggestion box, Special:Search... [20:14:44] Lydia_WMDE: sounds good! [20:14:50] cool [20:15:25] I'll do some edits to the page in the meantime in light of what we talked about today. [20:15:35] sounds good [20:17:46] SMalyshev: i wouldn't call teh API a user. but there are 3rd parties using opensearch, and there's the entity selector, too [20:18:15] right [20:18:32] entity selector would probably use wbsearchentities [20:18:49] yes [20:18:49] but quick box uses opensearch API... [20:18:56] i just mentioned it for completeness [20:19:00] sure [20:19:07] wgesearchentities will also be using the elastic index [20:19:11] so should be considered [20:19:25] right