[06:39:21] Hi, I am trying to upload a batch of language translations using a script which uses pywikibot. But for few of records, it gives Exception API error, when running it again, the exceptions lower down [06:39:33] Is it something related to throttling ? In my user-config file, put_throttle is set to 5 [06:39:40] Any help? [06:45:17] amisha: what does the error message say? [06:54:36] legoktm, logged just exception name. which says API Error [06:54:49] can you pastebin it? [06:58:30] legoktm, here is the script: https://pastebin.mozilla.org/8950200 [06:59:40] amisha: sorry, can you pastebin the output of the script? [07:09:09] legoktm, Here it is: https://textb.org/t/jjx8u6n4n6/ [07:10:04] in your code you have [07:10:05] except Exception as e: [07:10:05] excepName = type(e).__name__ [07:10:05] l['logs'] = "Exception" + excepName [07:10:15] I think you can use e.code or something to get the API error code [07:10:20] that will help debug it [07:11:02] * amisha is trying [07:16:09] legoktm, failed-save: As an anti-abuse measure, you are limited from performing this action too many times in a short space of time, and you have exceeded this limit. [07:16:09] Please try again in a few minutes :( [07:16:21] yeah, there you go :) [07:16:31] how to handle that? [07:18:09] legoktm, right now put_throttle is 5 [07:18:17] increase it to 10? [07:18:19] Will increasing it can help? [07:19:34] okay [07:21:59] it will slow your bot down so it's less likely to be hit by rate limits [07:24:24] legoktm, Yeah now trying with throt [07:24:30] throttle as 10. [07:24:40] Thanks legoktm : ) [07:30:01] legoktm, When are 'upper bound exceptions'? [07:30:09] *What [09:05:51] LeaAuregann_WMDE: Can you mark https://www.mediawiki.org/wiki/Extension:Wikibase_Repository for translation? [10:01:34] When I am trying to read wikidata object, item.get() gives KeyError , Any help? [10:22:10] addshore: hey! are you plannign to re-run the quantity stats script? should be create a ticket for this task? [10:31:08] amisha: that looks like a python issue [10:31:22] amisha: .get will raise a key error if whatever you are trying to 'get' does not exist [10:31:41] so maybe the label / property you were trying to get isn't set? [10:37:20] YuviPanda, Actually it's quite strange. So the situation is if I don't use item.get(), it gives me exception, But when I try doing something like https://textb.org/r/83bww7lilx/, it works well [10:38:24] Seems it whatever values from item the further code requires is fetched by the time it receives exception [10:38:47] amisha: hmm, so that code looks like it's just bypassing item.get() completely, and yeah that's pretty confusing :| [10:40:39] Is there a way to fetch only those values from item, which I require instead of everything? YuviPanda [10:40:43] amisha: am going to look at https://github.com/wikimedia/pywikibot-core to see what's going on [10:41:01] amisha: I don't think so [10:41:05] amisha: What version are you running? [10:42:09] multichill: also providing the stacktrace when you're getting KeyError might be hlepful [10:42:42] YuviPanda: I bet the l['id'] is causing that [10:43:01] bah, I meant amisha ^ for my prev. message not multichill [10:43:22] multichill: yeah, could be, depending on what's the 'except' for that try [10:44:20] YuviPanda: I found https://pastebin.mozilla.org/8950200 in the log [10:46:23] amisha: So basically you're trying to add missing labels and aliases, right? [10:47:16] Output at https://textb.org/t/jjx8u6n4n6/ suggests you run into APIErrors, but you're not including the full output. [10:53:20] multichill, YuviPanda Here are the stack traces https://pastebin.mozilla.org/8950238 [10:53:49] multichill, , yeah right adding missing labels and aliases [10:54:16] amisha: is that a KeyError? I can't tell from the traceback [10:54:54] amisha: in general, instead of doing a 'try: except' and then logging the error manually, it's better to just not have that and let the program crash. python's default traceback format provides nicer info [10:55:46] amisha: but I'll admit I'm stumped too, not sure what's going on there [10:58:10] How about this YuviPanda https://pastebin.mozilla.org/8950239 [10:58:27] amisha: much better :D [10:58:34] : ) [11:01:35] amisha: hmm, what version of pywikibot are you using? [11:02:31] amisha: or rather, how did you install it? [11:02:44] amisha: it might be an older version, since I don't see any uses of eval at all in that file [11:03:35] perhaps the pywikibot version is missing https://github.com/wikimedia/pywikibot-core/commit/ba027bc7880f568bc4676b8c6708904a88207c83 ? I think I had a similar problem a few days ago [11:04:08] YuviPanda, its 2.0 [11:04:42] YuviPanda, Followed these instructions https://www.mediawiki.org/wiki/Manual:Pywikibot/Installation [11:05:50] WikidataFacts: that sounds super relevant [11:06:08] amisha: ah interesting. what operating system are you on? [11:06:30] OSX [11:07:01] Y [11:07:03] amisha: ok, can you try installing from git? pip install --upgrade git+https://github.com/wikimedia/pywikibot-core.git [11:07:23] Yeah [11:08:08] amisha: ok! [11:09:01] been trying to debug this keyerror mystery amisha YuviPanda . can say one thing, it happens only on certain qids [11:09:10] YuviPanda, Do I need to change mu user-config files as well? [11:09:50] amisha: shouldn't be needed [11:09:56] Yes planemad , because in others, item.get() doesn't fail as it has all the keys present in it [11:10:04] okay YuviPanda [11:10:43] amisha: planemad yeah, it looks like it's failing on things that have a Quantity with upper bound missing [11:12:43] what is meant by upper bound YuviPanda ? [11:13:56] amisha: some properties are of type 'quantity' (https://www.wikidata.org/wiki/Help:Data_type#Quantity) [11:14:19] like, 'volume of lake' or something - they have a unit, and an optional upper and lower bound [11:14:34] so you can't have a lake with 500 billion trillion m3 of water for example (bad example) - so that'll be an upperBound [11:14:52] so if you look at the stacktrace and try to go find the code that is throwing the exception [11:14:57] you'll see it's trying to read upperBound and failing [11:14:59] YuviPanda: isn’t the upperBound part of the uncertainty? see https://www.wikidata.org/wiki/Wikidata:Project_chat/Archive/2016/11#Changes_to_quantity_precision_handling [11:15:02] because there's no upper bound [11:17:04] I see YuviPanda , I upgraded the pywikibot-core. That helped. Thanks a lot from pulling me out of this trap :) [11:17:21] amisha: does it work fine now? [11:17:41] WikidataFacts: wow, you're right and my understanding is completely bogus [11:17:44] hey [11:17:50] amisha: don't listen to me about what upperbound means, listen to WikidataFacts [11:19:26] does anyone know how can i stop Validator from registering one of its parser function hooks? [11:19:58] its basically conflicting with GeoData, as they have different syntax for {{#coordinates:...}} function [11:20:47] and since it's Validator that catches the function call, coordinates never read GeoData, and don't get indexed [11:21:18] never reach* [11:21:41] YuviPanda: I had the same problem on PAWS, and solved it by pulling /srv/paws/pwb and then “updating” /srv/paws/lib/python3.4/site-packages with `cp` [11:22:09] do you know what version of pywikibot is installed there? Is it some static version for everyone, or did I just get the problem because my server was created a while ago, before that commit happened? [11:22:11] WikidataFacts: ah, nice. I need to rebuild PAWS containers [11:22:15] Haha YuviPanda . Yep. Checking the link [11:22:23] WikidataFacts: Did you actually try sort by random? ;-) Random is just one float, not a function like in mysql [11:22:26] WikidataFacts: yeah, could be either. [11:22:56] WikidataFacts: if you started the server a while ago it won't update even if I build a new container [11:23:46] multichill: oh, are you mdammers? :) I did try it, and the results were different than without the ORDER BY RAND(), so I thought it was having some effect at least [11:23:58] YuviPanda: ok, thanks [11:24:58] multichill: SPARQL spec 17.4.4.5 says “Different numbers can be produced every time this function is invoked.” so that sounds like a bug in BlazeGraph if it’s really the case [11:25:10] It ended up just always being sorted by Qid (or same as without ORDER BY) [11:26:11] okay, apparently different invocations do get different randomness, but the same invocation gets the same result across all results. weird http://tinyurl.com/zw4vtqr [11:28:14] WikidataFacts: You do get some sort of different results, but it's not random at all. Right now I get almost all paintings in the :Q23xxx range [11:29:04] * YuviPanda needs to integrate wdqs into PAWS better [11:29:37] It's two types of random. Just RAND() will get n results that are next to each other in the resultset, the other one n results evenly spread over the full set [11:42:37] AmishaSingla_: What account are you using? Make sure to read https://www.wikidata.org/wiki/Wikidata:Bots [11:52:57] multichill, its an upload of a limited set of translations by individual reviewers. Not sure if it falls under the bot category. Background https://www.wikidata.org/wiki/Wikidata:Project_chat/Archive/2016/11#Data_donation_of_translated_place_names [11:55:10] YuviPanda, WikidataFacts _/\_ for the pywikibot help [12:00:47] planemad: np [12:03:39] YuviPanda: what would be the best way to upgrade pwb on PAWS? This feels wrong: https://paws-public.wmflabs.org/paws-public/46618563/Latest%20Population.ipynb#Preface:-pywikibot-update [12:03:57] WikidataFacts: going to /srv/paws/pwb and doing a git pull [12:04:08] that alone didn’t seem to have any effect [12:04:15] WikidataFacts: you don't need to do the 'mv' [12:04:19] I think? [12:04:26] if you do, I can fix that [12:04:44] I did the git pull first and still got the same error afterwards in a new notebook (i. e. new python process, I think) [12:05:51] WikidataFacts: right. I am making a change that should fix that now [12:05:56] cool, thanks [13:51:25] WikidataFacts: https://paws-public.wmflabs.org/paws-public/46618563/Latest%20Population.ipynb#Preface:-pywikibot-update is awesome btw! \o/ [13:55:46] YuviPanda: no, PAWS is awesome, so thank you for making it! <3 [13:56:53] WikidataFacts: :D [13:57:13] WikidataFacts: hopefully in a week or two I'll have a much improved terminal there (new release coming up) [13:58:28] WikidataFacts: I'm pushing new container now :) [13:58:42] should have newer version of pywikibot + able to pull and not need to do any other hijinks [13:58:56] nice! [13:59:09] can I update my existing container somehow? [14:06:06] WikidataFacts: you'll have to restart it unfortunately. but not done pushing yet, gonna be another 15min [14:06:11] these containers are huge now :D [14:06:43] oh, but restart will do the trick? nice [14:07:03] so between restarts, only the notebooks are persisted, the rest is fresh from the image each time? [14:08:29] ah, okay, /home/paws is a separate mountpoint [14:10:12] whoa, there are wiki dumps available in /public/dumps/public? That’s awesome! [15:00:55] WikidataFacts: you can restart now :D [15:00:59] WikidataFacts: yes :D [15:01:12] WikidataFacts: /home/paws is a separate mount yeah [15:01:26] wow, that really was a long time :D thanks for the heads-up! [15:01:49] WikidataFacts: yeah! it has a ton of stuff pre-installed, including multiple languages (R and Python) [15:01:54] and needs to be deployed to all of tools' k8s machines [15:34:57] YuviPanda: I just restarted my server and got this when trying to `import pywikibot` in a new notebook: “ImportError: No module named 'pywikibot'” [15:36:03] WikidataFacts: waah [15:36:04] let me look [15:36:58] let’s hope it’s an error on my end and not in the container :/ [15:37:07] WikidataFacts: do a 'pip install -e /srv/paws/pwb'? [15:37:47] YuviPanda: yup, that fixes it [15:37:57] (I restarted the notebook kernel too, not sure if that was necessary) [15:38:03] WikidataFacts: ok, let me fix that [15:53:13] WikidataFacts: found the issue and a fix is on the way [16:14:35] * DanielK_WMDE waves to YuviPanda [16:15:13] hey DanielK_WMDE [16:23:16] WikidataFacts: if you restart now it should be fine [17:45:47] WikidataFacts: anyway, going to go afk soon, but - if you git pull, it should be instantly visible now :) [17:46:18] YuviPanda: great, thanks! [17:46:34] WikidataFacts: np. you need to restart container first tho :D [17:46:48] I'm going to hopefully spend more time on it soon [18:45:21] YuviPanda: here’s a crazy idea that I had: https://paws-public.wmflabs.org/paws-public/46618563/Embed%20WDQS.ipynb [18:49:53] WikidataFacts: nice! I was trying to do something like that with http://github.com/yuvipanda/python-wdqs [18:51:05] WikidataFacts: I'm not sure why your code doesn't work [18:51:07] it should [18:51:15] it does for me, in the interactive notebook [18:51:24] I see the embed [18:51:28] but I can’t show it to anyone [19:16:56] WikidataFacts: interesting. [19:17:03] WikidataFacts: can yo file a bug in phab? :D [19:17:14] I'll check tomorrow (it's past midnight for me here) [19:18:04] okay, will do [19:18:45] YuviPanda: is it a bug that the widget doesn’t show up, or should it be a feature request ticket? [19:19:20] WikidataFacts: just tag it as 'PAWS', bugs are just feature requests in disguise ;) [19:19:54] haha, okay :D [19:38:56] YuviPanda: https://phabricator.wikimedia.org/T152623, apparently the ipywidgets/nbconvert people are working on this *right now*, with some comments in the related issues/PRs less than a day old :D [19:39:19] WikidataFacts: oh, where? [19:39:29] I added some links to the ticket [19:39:38] I think the tracking issue is https://github.com/ipython/ipywidgets/issues/16 [19:39:58] and this PR seems to be part of it, though the description doesn’t make it very clear, so I’m not sure: https://github.com/jupyter/nbconvert/pull/482 [19:40:14] WikidataFacts: ah, nice :D [19:40:20] ( I know a good chunk of those people :D) [19:40:25] great :D [19:41:14] WikidataFacts: instead of ipywidgets.HTML() can you try using IPython.display.HTML? [19:41:18] and see if that works? [19:41:39] sure [19:41:44] WikidataFacts: there's actually IPython.display.IFrame [19:42:36] oh that sounds like an obvious candidate [19:44:25] YuviPanda: it works!! [19:44:47] WikidataFacts: save it? I'll test paws-public :D [19:44:55] https://paws-public.wmflabs.org/paws-public/46618563/Untitled.ipynb [19:45:18] (width isn’t perfect) [19:46:36] :D [19:46:37] nice [19:47:43] I guess I can move that back to Embed%20WDQS.ipynb and close the bug [19:47:50] that was very fast, thanks :D [19:48:26] WikidataFacts: :D yw! [21:51:31] "[13:16:57] Just can't get his face. I'm with faces than we names (and a complete disaster with nicknames)" > he didn't like to be taken in photos, so on the few we have of him, we generally see only his back... [21:53:49] multichill: he is here https://commons.wikimedia.org/wiki/File:WikiGrenier_20150227_-_01.jpg and here https://commons.wikimedia.org/wiki/File:WikiGrenier_20150227_-_10.jpg with Harmonia_Amanda [21:54:09] ? [21:54:14] ah yes, that me [21:54:20] and Coyau [21:54:21] Great face pics. :P [21:54:24] scanning a bool [21:54:26] book [21:55:38] we'll never scan together the rest of the Almanachs :'( [21:58:09] Ash_Crow: Jean-Fred shared one in private [21:58:37] ok [21:58:56] It's sad [22:00:07] yes, he has been a friend since... I don't know, 2008 or 2009... :s