[09:35:37] WikidataFacts: don't suppose you have any ideas for how to make https://www.wikidata.org/wiki/Wikidata:Request_a_query#en.wiki_humans_with_no_gender work? [09:41:20] I tried to count how many humans without p21 there are but that times out which is a bit annoying [09:48:16] nikki: no success [09:48:27] aww [09:48:32] we tried at least :) [09:49:10] I guess you could get a list of all humans with LDF and then filter out those with P21 with API calls [09:49:15] but that would take pretty long too [09:49:23] it would just shift the load from WDQS to you :P [12:12:34] Is it possible to get the full data for multiple wikidata entities in a single api request? [12:17:10] johtso: I think so. Is this what you are searching for? https://www.wikidata.org/w/api.php?action=wbgetentities&format=json&ids=Q1%7CQ2 [12:17:21] johtso, https://www.wikidata.org/w/api.php?action=help&modules=wbgetentities [12:17:35] gets it for Q1 and Q2 [12:19:09] Ainali: oh right, wbgetentities does return the actual data! [12:19:57] I just need to know all the property ids I'm interested in.. [12:21:27] https://www.youtube.com/watch?v=ELD2AwFN9Nc [12:37:49] User:QuickStatementsBot doesn't have a botflag? My watchlist is dead :( [12:39:41] It doesn't, ffs [12:39:49] https://www.wikidata.org/wiki/Wikidata:Requests_for_permissions/Bot/QuickStatementsBot [12:41:08] Oh dear. https://www.wikidata.org/wiki/Wikidata:Database_reports/Unauthorized_bots [12:52:49] that seems pretty unacceptable [12:55:05] I can understand not being too strict about policies, but several orders of magnitude more edits than actually allowed is far beyond reasonable [12:58:09] and it seems like it's also making edits from multiple people/quick statements batches at the same time, so special:contributions for the account is pretty useless if you want to check someone's edits [13:34:16] Is it possible to include labels when making a wbgetentities request? [13:34:36] labels for entities referenced in claims for example [13:53:11] or will I have to fetch all the referenced entities to get their labels? [13:56:29] johtso, You might want to do something like this: https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q42&props=labels [14:00:14] amisha: right, I can limit the data to just labels, but I'd still need to request all the entities I want the labels for right? [14:07:56] johtso, I think it just fetches the labels, when you specify &props=labels [14:08:29] I verified it in network tab in developer tool. [14:09:02] Try this : https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q42&props=labels and https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q42 [14:09:18] And see network tab in developer options for both of them [14:10:07] One with just labels is getting only ~65kb data whereas the other request gets 383kb data [14:10:24] johtso, check if that helps. [14:10:56] amisha: sure, that's useful, was just wondering if I could avoid making any extra requests [14:12:02] Yes that will request only whatever property you want . : ) [15:14:26] halfak: I wanted to talk more about use cases for statement tracking. Also have a draft of the Lua function to do so and was wondering about doing a pull request in Github [15:15:27] hoo is probably the best person to look at the lua code. [15:15:49] What use-cases were you thinking about? Is this related to the discussion over what DB we put the statement tracking table in? [15:16:12] hall1467, ^ [15:17:27] Yes, it is related to that discussion [15:18:34] hey [15:18:52] You had mentioned that there were use cases from hoo and Lydia [15:18:55] Hey [15:20:38] Oh yeah. I noted that in the phab task. [15:20:40] * halfak digs it up [15:21:50] hoo, I was thinking that the main MW integration of statement tracking would be watchlists. jynus has some notes in phab about where he recommends storing the data. [15:22:07] See https://phabricator.wikimedia.org/T151717#2966462 for my post and jynus' followups are below [15:22:10] I saw that in my mail, but didn't yet read it [15:22:12] will do in a sec [15:22:18] (if you have time now :D) [15:22:24] great. Thanks :D [15:23:32] hoo: Could I have you look at a draft of code that I put together to do the tracking? I forked from the GitHub repo. Should I do a pull request? [15:23:52] If you have time, as well :) [15:28:50] Ok, read it… hm [15:30:09] What we need to support is (basically) UsageLookup in Wikibase client [15:30:29] for the RC integration I think we're using very messy JOINs right now [15:31:31] ah, wait, no [15:32:00] we basically insert the WD changes into the client's RC table via asynchronous jobs [15:32:15] so supporting UsageLookups accessors would work out for now [15:32:24] halfak: hall1467 ^ [15:33:23] https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/d91d79725bf9b45545214be12de6bff4b04e0b0a/client/includes/Usage/UsageLookup.php [15:33:35] that's UsageLookup… just so that's we're on the same page [15:35:10] All read and write queries related to usage are currently only touching the wbc_entity_usage table (AFAICT), so putting this into a separate shard sounds fine to me [15:42:29] Interesting. The separate shard could make this a lot easier to justify. [15:42:51] I wonder if Lydia_WMDE has any concerns about integration and recording this data in a separate shard? [15:47:21] this mostly depends on the use cases we need to support, so we definitely need to consider her [15:47:41] We also need to think about what to do with the current tracking table… also move it? [16:07:47] hoo: Aside from the current conversation, I was wondering if I could run my draft of the Lua code for tracking by you? Should I do a pull request? Thanks! [16:08:25] * andre__ would expect Wikimedia Gerrit instead of GitHub? [16:08:50] hall1467: What draft exactly? [16:08:56] What did you change [16:09:09] oh and yes, our code is on gerrit, I just linked to github here for convenience [16:11:40] -> https://www.mediawiki.org/wiki/Gerrit/Tutorial [16:14:55] I added in a new "logTableAcccess" function. The idea is to track access to the entity table. [16:14:59] https://github.com/hall1467/mediawiki-extensions-Wikibase/blob/master/client/includes/DataAccess/Scribunto/mw.wikibase.entity.lua [16:15:53] I could make the changes in Gerrit instead? Perhaps I'm not going about the implementation in the right way, but I wanted to give it a try [16:42:51] hoo: Should I make the changes I mentioned above in Gerrit instead? I could go ahead and do that if you like [16:48:07] hall1467: AFAICT your change will break iterating over the claims, wont it? [17:00:06] hoo: I'm not sure if it will. The intent was to provide a wrapper around the entity table to do the tracking, but not affect anything using the entity. See the following link for tracking table access: http://www.lua.org/pil/13.4.4.html [17:00:42] (Notice that, unfortunately, this scheme does not allow us to traverse tables. The pairs function will operate on the proxy, not on the original table.) [17:00:53] That was what I expected :S [17:01:47] Ah, I see that. Do you have any idea of a workaround? [17:02:50] ah, that's actually already worked around in Scribunto [17:04:02] https://github.com/wikimedia/mediawiki-extensions-Scribunto/blob/master/engines/LuaCommon/lualib/mwInit.lua#L8 [17:04:21] So you would also need to define __pairs and __ipairs in your proxy [17:09:24] Okay. I'm not entirely sure what those definitions would look like. Its a bit confusing to me [17:18:32] hall1467: I guess you could just call the respecting methods on the proxied table [17:20:32] hall1467: halfak: Do you plan to gather this data before we have the actual storage in place for measuring purposes? If so, where? [17:27:03] PROBLEM - Blazegraph Port on wdqs2003 is CRITICAL: connect to address 127.0.0.1 and port 9999: Connection refused [17:27:23] PROBLEM - Blazegraph process on wdqs2003 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 997 (blazegraph), regex args ^java .* blazegraph-service-.*-dist.war [17:27:34] PROBLEM - Check systemd state on wdqs2003 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [17:31:10] hoo: Okay, I'm not sure if I see the solution for pairs and ipairs you had in mind right now but I can think about it. Sounds like __pairs and __ipairs will need to be defined [17:32:27] yeah, just like __index [17:33:03] hoo: Okay, I'll look into it. Thanks [17:38:17] hoo: halfak: Would writing to a log file make sense in the mean time perhaps before the actual storage exists? [17:38:40] That's what I was thinking about [17:43:47] Okay. I'm not sure where this would be written to. halfak probably has a better sense of how logging might be done [17:44:43] hoo, hall1467: sorry. In meeting. It seems like jynus thought we could write to a different server without much concern. [17:44:51] I was under the impression that this already existed. [17:45:13] something like "the extension database" or something like that. [17:50:24] If we want to actually track this continuously, this is going to become more cumbersome [17:50:32] but I guess we need to do that at some point anyway [19:47:19] nikki, I have a look to your gender-query [19:47:36] maybe I can find all of them that have a country, by country, without the time-out [19:47:59] it's not my query, as such, I was just trying to help someone else :) [19:50:54] never mind... there is at least one country that still breaks the query [19:51:24] then I would have to loop by country by profession ... [20:06:17] and even then a time out [20:06:30] we need more memory on the sparql-server I guess [20:37:06] RECOVERY - Check systemd state on wdqs2003 is OK: OK - running: The system is fully operational [20:37:16] RECOVERY - Blazegraph Port on wdqs2003 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 [20:37:27] RECOVERY - Blazegraph process on wdqs2003 is OK: PROCS OK: 1 process with UID = 997 (blazegraph), regex args ^java .* blazegraph-service-.*-dist.war [21:44:25] hey! I can't find it or maybe it's not possible. So... Is it possible to get taxonomy of specimen in one request? [21:45:27] I see only parent taxon and I need to get whole chain [21:45:59] but then it would make too many request [21:46:46] I mean sometimes it would be 1000 per day, more often 0... I think I could get banned. Currently I've two requests per one animal/plant. [21:47:26] heh, I wonder how much space would take whole tree. Probably it would be pointless to get it all and have it locally cached. [21:47:45] if somebody would know the answer, even if it would be negative, please ping me [21:47:52] and I'm going to search for it [21:49:17] krzysiu: well, you can get the tree with a SPARQL query: http://tinyurl.com/z972abu [21:50:39] WikidataFacts, thanks! :) [21:54:46] krzysiu: you can also see the entire graph at https://angryloki.github.io/wikidata-graph-builder/?property=P171&item=Q146 (note that it’s not just a straight line) [21:55:02] and the graph builder can show you the query that it uses to construct the graph if you select the “WDQS” mode [22:22:42] WikidataFacts, that's horrible in UX perspective! I'm pulling elements instead working on it ;) Seriously - nice visualisation. I thought that when you go up the tree, you'll get flat array :( [22:24:33] but just a photographer. I wanted to automatically generate description - English and Latin names, aliases (if available) and taxonomy. That taxonomy looks like totally different task than the rest of things (I've already programmed eng+lat+aliases) [22:24:48] ^ I'm not a biologist, but ... [23:09:42] hm https://grafana.wikimedia.org/dashboard/db/wikidata-datamodel-terms stopped at Dec. 25 [23:11:00] addshore: ^ [23:17:52] hoo: https://phabricator.wikimedia.org/T154601 [23:18:20] needs to be rewritten to use the dump / have the query rewritten to be better [23:18:26] night! :D [23:20:43] I see :/ [23:20:44] Good night [23:42:30] is there a reason why Wikibase does not register its content handlers in $wgContentHandlers? [23:42:52] I assume no [23:43:26] for some reason it uses ContentHandlerForModelID hook instead of the config. Not sure why [23:43:26] for some reason it uses ContentHandlerForModelID hook instead of the config. Not sure why [23:44:04] Maybe conditional registration [23:44:26] (in the past) [23:44:46] hmm maybe... causes problems for me now since I can't enumerate available content models [23:45:03] or, rather, I can enumerate them, but wikibase ones are not among them anymore [23:46:31] it's handled in onContentHandlerForModelID instead... maybe config is too early to create it? I don't know... [23:50:04] maybe it's because it is configured in special config file... but then I still need some means to find out which handlers are there...