[00:07:23] RECOVERY - Check if the Hadoop HDFS Fuse mountpoint is readable on notebook1004 is OK: OK [02:30:03] PROBLEM - Check if the Hadoop HDFS Fuse mountpoint is readable on stat1005 is CRITICAL: CRITICAL [07:01:23] RECOVERY - Check if the Hadoop HDFS Fuse mountpoint is readable on stat1005 is OK: OK [07:13:05] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: RFC: Modern Event Platform: Schema Registry - https://phabricator.wikimedia.org/T201643 (10kchapman) TechCom is placing this on Last Call ending Wednesday December 5th 10pm PST(November 22nd 06:00 UTC, 07:00 CET) [07:13:50] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: RFC: Modern Event Platform: Stream Intake Service - https://phabricator.wikimedia.org/T201963 (10kchapman) TechCom is placing this on Last Call ending Wednesday December 5th 10pm PST(December 6th 06:00 UTC, 07:00 CET) [08:04:53] addshore: something else to know about the query you were using yesterday - You'd better use wmf.mediawiki_user_history instenad wmf_raw.mediawiki_user [08:05:25] I thought that might be the case! [08:15:08] let me try and rewrite it :D [08:15:13] also addshore, I encourage you to use spark instead of hive whenever you can :) [08:15:19] addshore: I have a version of it now [08:15:37] i had another look at super set last night too [08:15:43] some people have made some very cool things there [08:18:31] indeed addshore - I'm eager to have better sql-on-hadoop engine so that we can connect them (ping elukey :-P) [08:19:12] addshore: https://gist.github.com/jobar/efa203f03208764b021d04b376b8f7c7 [08:19:24] addshore: When used with spark, the querying takes a few seconds [08:20:09] addshore: The reuslting dataframe has 43369 rows, meaning it's small enough to be collected onto the spark driver and written as a single file [08:24:55] addshore: I updated the gist to show how to write the file [08:24:56] https://gist.github.com/jobar/efa203f03208764b021d04b376b8f7c7 [08:24:59] addshore: --^ [08:30:56] hmm, running spark2-shell seem to freeze up [08:31:02] on stat1007 for me [08:31:12] Mwat? [08:31:15] hm [08:32:25] the last thing it said is "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel)." but it wont let me input anything yet [08:32:44] addshore: you have 2 yarn jobs running currently as spark-shell, so I'm assuming stat1007 has an issue indeed [08:32:52] addshore: Do you mind tring from stat1004? [08:33:36] addshore: Killing your existing yarn spark apps [08:33:55] it gets further on stat1004 [08:34:01] \o/ [08:34:03] and yes, its loaded up :) [08:34:21] stat1007 has something wrong - I'll ask elukey when he comesin [08:34:43] --verbose joal :) [08:34:50] :D [08:34:51] Morning elukey [08:34:55] morning elukey [08:34:57] o/ [08:35:29] elukey: "spark-shell2 --master yarn" blocks when launched from stat1007, while it succeeds from stat1004 [08:35:41] lovely [08:35:54] elukey: Could it be related to upgrade not fully finished on stat1007? [08:36:29] not really, it is working like stat1005 [08:36:54] so on stat1005 spark2-shell --master yarn works [08:37:38] ah indeed it blocks on stat1007 [08:38:13] joal: i guess within that dataframe it is then pretty easy for me to essentially do a group by and count number of users in each group? [08:39:00] groupby([by, axis, level, as_index, sort, …]) :D [08:39:18] addshore: the group by and sum is already done, no? [08:39:48] so this is giving me individual users, but I want to break it downa tiny but furthers [08:40:03] addshore: Ah - Are you after how many users have done how many creations? [08:40:19] yes, or at least for the bottom half of the users [08:40:34] addshore: Updating the query [08:40:46] probably everyone over 100,000 can remain as individuals [08:41:19] but below that it might be nice to have some groups of say..... 0-1, 2-10, 11-100, 101-1000 etc [08:41:34] I was going to do that with post processing, but also not sure if there is an easier way [08:43:37] addshore: very doable in postprocessing, also very doable in SQL - Currently doing in sql [08:43:53] your sql magic can do far more fancy thiings than mine ;) [08:44:17] addshore: When keeping individuals, I have 1214 lines [08:44:32] individuals over 100,000? [08:44:38] addshore: meaning, when I group by number of creations and count number of users [08:44:54] I have a distinct number of created-items of 1214 [08:45:07] I will provide better group [08:45:09] well, thats 40 times fewer results now :D [08:50:16] addshore: https://gist.github.com/jobar/40f7f62920692477376817de0c406e25 [08:50:22] addshore: I'm assuming this should do :) [08:51:06] aaah okay, not too much magic, just boring old sql :D [08:51:17] addshore: And I'm not sure about your perception, but now that I've used spark, I don't want to use hive anymore :) [08:51:31] addshore: no magic in SQL unfortunately, just a lot of copy pasting [08:52:08] that makes me feel better about some of the other queries i have written in the past ;) [08:53:07] addshore: And we said, it depends how you feel about doing postprocessing - I find it easier for these use cases to do it in SQL - One might prefer post-processing [08:55:08] 10Analytics, 10Product-Analytics: Event counts from Mysql and Hive don't match. Hive is persisting data from crawlers. - https://phabricator.wikimedia.org/T210006 (10JAllemandou) @Nuria : Do we know why those events don't get inserted into MySQL? [08:56:08] joal: and, in theroy with this daat set in the history table, it would actually be possible to get the data for these gorups over time right? :P [08:56:17] (not that im asking for that now too) just checking [08:57:13] addshore: I think we could relatively easily get how many users have made how many item-creations per time-periods (for instance month), yes :) [08:57:44] addshore: Reminder - The current method strips away anonymous users (counting all as 1) [08:57:52] joal: yup [08:58:13] in theory could this also be done in super set to spit out some sort of chart straight away? [08:59:04] (really weird, can't find the blocking issue on stat1007) [08:59:16] addshore: in theory I assume yes, but we don't have the sql-engine as of now, and I also don't know enough of supersetting-sql-queries (I have used it with Druid almost only) to be sure of how [09:02:59] addshore: https://gist.github.com/jobar/40f7f62920692477376817de0c406e25 edited to show you numbers per month [09:04:58] joal: so when I create the spark2-shell, it gets accepted by yarn but it doesn't get to RUNNING state [09:05:01] for some reason [09:05:36] elukey: I have seen that yes - This is wird :S [09:08:23] ah I was wondering why spark-shell was complaining about address in use [09:08:31] does it work for you? [09:08:45] elukey: nope :( [09:09:12] that's good, it means that it could be firewall/network related [09:09:59] * elukey blames addshore [09:10:01] :P [09:10:09] elukey: I think it could well be - I can't even get logs from yarn on the application, meaning it has not even started [09:10:10] :D [09:10:21] yeah it is the firewall indeed [09:10:27] silly firewall [09:11:07] i think that stat1007 uses base firewall [09:11:19] (that collects basic firewall rules etc..) [09:11:39] but we had to turn it off for stat1005 since spark uses random ports [09:11:45] IIRC at least [09:12:01] but in theory stat1005 and stat1007 have the same puppet config [09:13:49] mmm maybe it is a leftover from role::spare::system [09:15:22] addshore: batcave for a minute:? [09:18:08] what's this batcave you all always talk about [09:18:19] ema: https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave [09:18:28] come say hello ;) [09:18:45] ha! [09:18:53] ema: analytics secret meeting place ! [09:27:13] spark now works :) [09:27:52] \o/ elukey :) [09:28:25] addshore: ping cave or later? [09:28:51] joal: sorry, didnt see that earlier ping! [09:28:55] np addshore :) [09:29:00] yes, coming! [09:29:08] Want to share some more tricks ;) [09:29:31] wait, where is the link to the batcave ? :P [09:29:37] i swear it used to be in the subject? [09:29:43] / channel topic [09:29:49] oh wait [09:29:51] you linked it [09:29:51] bah [09:33:32] addshore: ssh -N notebook1003.eqiad.wmnet -L 8000:127.0.0.1:8000 [09:43:37] addshore: cp /home/joal/investigations/wikidata-item-creators.ipynb . [09:49:19] addshore: https://yarn.wikimedia.org/cluster/scheduler [10:09:12] addshore: playing with wikidata - https://gist.github.com/jobar/aa3f45b78a154ae15bdf2a32ad70c412 [10:45:49] joal: for that wikidata example what size yarn kernal do i need? [10:50:21] also [10:50:23] bubble chart :D [10:50:26] https://usercontent.irccloud-cdn.com/file/Q0eX2jPO/image.png [10:57:29] hey teaaam [10:59:11] elukey, what do you think of: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/475231/ [10:59:54] nuria suggested to deploy this, and see the alarms trigger, before we deploy the fix to refinery-source (which is already merged) [11:00:08] and then we'll see the alarms cease [11:00:28] we might get some volume of emails though, thoughts? [11:01:24] mforns: could be yes, we might just do it before the next deployment [11:01:42] elukey, would you be OK if I deploy today? [11:02:20] I don't have problems, it is just team europe so let's possibly not deploy huge things [11:02:25] joal: what do you think? --^ [11:02:44] let me see what's in the changelog [11:04:33] on the refinery-source side there's 2 small changes, not critical [11:04:52] and on the refinery side there's nothing [11:05:19] I think we can deploy, this way I can unbreak EL sanitization [11:08:49] +1 [11:15:30] addshore: bubble chart is great :) [11:16:23] no prob with deploying a-team [11:16:34] k! [11:16:35] mforns: I think that in refinery, there is fdans path [11:17:06] joal, I don't see it merged: https://github.com/wikimedia/analytics-refinery/commits/master/artifacts [11:17:49] mforns: not in artifacts [11:17:53] https://github.com/wikimedia/analytics-refinery/commit/fd44524cd5fd1fa75ea025349db13b3dbb341546 [11:17:55] oh sorry [11:19:05] ok, so this is more critical [11:20:43] mforns: no problem, it can be deployed without being restarted, and it actually needs to be deployed :) [11:20:51] K [11:22:07] cool, so elukey, can you do me the honor? :P https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/475231/1 [11:22:52] 10Analytics, 10Beta-Cluster-Infrastructure, 10User-Elukey: TCP connections between analytics and deployment-prep - https://phabricator.wikimedia.org/T208870 (10elukey) Thanks a lot for the link Tyler, I battled a bit with some settings (and updated the wiki as well) but eventually I was able to deploy on my... [11:23:44] mforns: merging [11:23:55] :D [11:23:58] thanks [11:29:13] done! [11:29:26] so in theory it should spam very soon right? [11:31:35] elukey, it should email every hour until last 24 hours are sanitized [11:33:23] looking forward for it :D [11:33:35] so we should wait until at least one error is emitted before deploying right? [11:59:38] joal: the wikidata example you left me with failed to run, do i need a bigger kernel for it? [12:03:09] * elukey passes https://hops.readthedocs.io/en/latest/overview/introduction/hopsFS.html to joal [12:21:29] no alarms so far... :((((( [12:26:07] oh! refineMonitor only runs once per day at 4am... [12:26:08] ok [12:27:08] mforns: shall we run it manually? [12:28:41] elukey, hmmm, I prefer it failing by itself no? [12:29:02] I think I will start backfilling of event_sanitized, but leave the last couple days empty [12:29:10] to ensure that tomorrow the alarm fires [12:29:21] we can deploy refinery-source on monday [12:29:32] or do you prefer tomorrow? [12:31:40] 10Analytics, 10Analytics-EventLogging: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10phuedx) It looks like [[ https://github.com/wikimedia/puppet/blob/8b4f503c5186f2d4b26538e237f828d2e1b11a13/manifests/site.pp#L1334-L1338 | `logstash1007.eqiad.wmnet` is... [12:34:13] mforns: refinery monitor just checks right? if so we can execute it now (same thing as waiting for 4am via cron) and then if it alarms we can eploy [12:34:16] *depoy [12:34:18] *deploy [12:34:24] no? [12:34:34] I was under the impression that you needed this soon [12:34:57] elukey, yes theoretically, but this way we do not know 100% for sure whether the production refineMonitor will alarm, no? [12:35:09] mforns: why? [12:35:16] I fear us executing it in a way different from prod [12:35:27] nah it doesn't change anything [12:35:31] because prod uses spark_job.pp [12:35:40] an we would be using raw command line [12:35:42] ok [12:36:01] I trust you then! [12:36:09] will execute it! [12:36:25] if we want to be super sure [12:36:27] we do [12:36:32] 1) sudo -u hdfs /bin/bash [12:36:52] 2) crontab -l [12:36:53] maybe puppet has deployed a command line script to run it no? looking at the crontab [12:36:58] 3) copy the command and execute [12:36:59] yea [12:37:08] it didn't run it [12:37:16] it only deployed it in a crontab [12:37:22] yea [12:37:53] Wow elukey - I didn't know about HopFs ! [12:38:09] me too! [12:38:25] addshore: Have you managed to get the wikidata example running? [12:38:46] nope, just about to head to lunch too [12:38:54] addshore: please do (min [12:39:03] again - I just did mine ) [12:40:14] elukey... I can not find the cron job in hdfs's crontab in an-coord1001... WTH? [12:40:18] 10Analytics, 10Analytics-EventLogging: Resurrect eventlogging_EventError logging to in logstash - https://phabricator.wikimedia.org/T205437 (10phuedx) Mibad. We actually remove the `eventlogging_EventError` tag before logging the log line: https://github.com/wikimedia/puppet/blob/134aff8e082c26e1c28ff2f0f76759... [12:41:05] so it is called monitor_sanitize_eventlogging_analytics afaics [12:41:19] elukey@an-coord1001:~$ sudo -u hdfs crontab -l | grep monitor_sanitize_eventlogging_analytics [12:41:22] # Puppet Name: monitor_sanitize_eventlogging_analytics [12:41:24] mforns: --^ [12:41:26] 15 4 * * * /usr/local/bin/monitor_sanitize_eventlogging_analytics >> /var/log/refinery/monitor_sanitize_eventlogging_analytics.log 2>&1 [12:42:13] ah, cool, I was greping for -i refinemonitor [12:42:14] wow elukey - How have you managed to insert that manual line in the middle of pasted stuff ... [12:42:18] makes sense [12:42:42] joal: manual line? [12:42:53] the one that say --^ [12:43:11] ahhh you received it in the middle of the paste? [12:43:14] ahahhaha [12:43:16] yes [12:43:17] :D [12:43:39] k executing [12:43:39] There are IRC skills I still need to work ;) [12:44:11] ahhaah [12:44:14] going to lunch! [12:44:21] Laster elukey :) [12:44:51] byeeee [12:45:17] joal: yes i got this [12:45:18] Message: Job aborted due to stage failure: Task 0 in stage 5.0 failed 4 times, most recent failure: Lost task 0.3 in stage 5.0 (TID 14, analytics1040.eqiad.wmnet, executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Container marked as failed: container_e96_1542030691525_34930_01_000015 on host: analytics1040.eqiad.wmnet. Exit status: 143. Diagnostics: Container killed on request. Exit code is [12:45:18] 143 [12:46:16] addshore: I'm assuming it could come from the line "spark.table("wd1").cache()" -- The data we're trying to cache is too big for the "small" kernel [12:46:37] addshore: I managed to get an answer by commenting that line [12:46:40] so i should try with a bigger kernel? [12:46:44] aaah okay [12:47:11] addshore: 2 solutions, removing the caching line and get slower results (good when you are not in a rush), or use the large kernel :0 [12:47:14] :) [12:47:22] amazing, will try after food! [12:47:26] * addshore is fooding [12:47:36] addshore: currently trying with the large kernel, to be sure it works [13:00:54] back, and it ran without the cache :) [13:01:02] great :) [13:01:30] It runs with the cache on a "large" kernel, but the improvement is not worthwile [13:02:36] so, the references question I was asking / thinking about earlier, i really want to be able to look at that over all time [13:03:05] which i guess means it would be better coming from historical text rather than from just current versions [13:03:13] addshore: This means using text I think, yes [13:03:35] I however have never parsed wikidata content from text, so I can't really help [13:03:46] Or at least not on that aspect of things :) [13:04:06] okay :) [13:04:40] so, one example of a number that we track, is the number of terms by language [13:05:00] this used to run on the sql tables using the wb_terms table until the table became to big and slow [13:05:09] I imagine that would be a very easy question to answer with the wikidata parquet [13:05:24] addshore: a term being either a prop or item, and a language being its label? [13:05:37] term being either label description or alias [13:05:42] or to be precise, label-avialability? [13:06:06] right, any of those one having a value makes availability for that given language [13:06:19] yes [13:07:01] this used to be https://grafana.wikimedia.org/dashboard/db/wikidata-datamodel-terms?refresh=30m&orgId=1&from=now-2y&to=now but as you can see it hasn;t been running for nearly 2 years now [13:07:22] addshore: YOU SHALL HAZ DATAZ ! [13:07:32] https://grafana.wikimedia.org/dashboard/db/wikidata-datamodel-terms?orgId=1&from=1446419521727&to=1486738270153 is the only big of data we had [13:08:02] joal: should i make a wikipage of things that it might make sense to replace? :P [13:08:07] or etherpad or something? [13:08:39] addshore: It might be a little trickier if we want historical, because it involves parsing history, but except from that, I think a monthly value with new dumps shouldn't be too complicated :) [13:08:50] addshore: A wikipage and a ticket :) [13:09:06] yeh, this is only working as a monthly thing currently, the hitorical data is only there from the old runs monthly [13:09:16] this ticket will however be dependant on the one about making the release of json-dumps to HDFS productionized [13:09:18] well, it actually ran daily I think, but whatever [13:09:47] I'm assuming a reliable monthly is better than nothing addshore :) [13:09:53] joal: yup [13:10:00] I want to go and explore this mediawiki text stuff ;) [13:11:05] please do addshore: wmf.mediawiki_wikitext_history [13:11:12] addshore: available in spark and hive [13:11:42] addshore: partitioned by snapshot (usual 2018-10), and wiki_db [13:12:03] addshore: So if you use snapshot = "2018-10" and wiki_db = "wikidatawiki", you'll only read wikidata stuff [13:12:42] for moritzm and elukey: https://image-store.slidesharecdn.com/26f8ac9f-59bf-4dc2-ba57-2033f17f3549-original.jpeg [13:15:14] joal: I find myself wanting to do things like revision_id = MAX(revision_id) [13:15:30] :S [13:15:36] addshore: I understand that [13:15:54] context: i want to find the current revisions, but i guess to do that I want a point to the page table? [13:16:06] addshore: It should be fast enough though, stuff is stored in parquet [13:16:20] line 8:18 Not yet supported place for UDAF 'MAX' [13:16:22] joal: :-) [13:16:54] addshore: can you paste the query? [13:17:28] * mforns lunch [13:17:45] joal: well It wasn't done yet ;) but [13:17:48] https://www.irccloud.com/pastebin/jYFzqWGn/ [13:18:18] also addshore: The caching actually helps when using a huge amount of RAM - Not worth for examples and all :0 [13:18:57] My test currently uses 2Tb of RAM ... Somehow too much IMO [13:19:52] addshore: I'm assuming you'd want max(revision_id) per page [13:19:59] joal: ack [13:21:37] hallo [13:22:08] yesterday I asked whether I can create tables at research_prod [13:22:21] I finally tried it now, and it looks like I cannot [13:22:30] ERROR 1142 (42000): CREATE command denied to user 'research_prod'@'10.64.16.77' for table 'aharoni_test_001' [13:22:49] I ran: [13:22:51] research_prod@db1108.eqiad.wmnet(log)> CREATE TABLE IF NOT EXISTS aharoni_test_001 [13:22:52] etc. [13:23:30] I think it was elukey who said that it's supposed to work :) [13:24:14] a-team [13:25:50] Hi aharoni - I'm no good at mysql and mforns/elukey are gone for lunch - Let's wait and see what they think :) [13:26:13] OK [13:31:20] aharoni: hi! Did you 'use staging' before trying to create ? [13:31:28] (trying to understand how to repro) [13:31:48] elukey: no. I tried it now and: [13:31:53] research_prod@db1108.eqiad.wmnet(log)> use staging; [13:31:58] ERROR 1044 (42000): Access denied for user 'research_prod'@'%' to database 'staging' [13:32:42] If I run `show databases;`, I see information_schema, log, and test. [13:33:02] ah test! lemme check, maybe in there it is called differently [13:33:10] I also noticed only now that you use research_prod [13:33:16] that I've never heard of [13:33:23] (only 'research') [13:33:48] all right so staging is there [13:34:17] and research_prod cannot do what research does [13:34:39] mmm not really [13:34:43] they seeems doing the same [13:34:51] maybe we forgot to add perms for staging [13:35:11] I've been using research_prod for years... that's what I've been told to use in 2014 or so when I started analyzing events. [13:35:42] I don't really care where to run it. It doesn't have to be in research_prod. [13:36:26] I need to test it wherever I should create the table discussed at https://phabricator.wikimedia.org/T189475#4706502 [13:36:52] sure sure [13:37:00] addshore: https://gist.github.com/jobar/9adda130ae39bc6d378ec46d2979e017 <- This one gave me results (pasted as well) and should be reusable on a broader way [13:37:16] aharoni: where do you usually get the research_prod credentials? [13:37:29] *however*, I will need to do two things: To create a table, and to run a query that reads from EventLogging and inserts to this table. [13:38:11] I run: `mysql -u research_prod -h db1108.eqiad.wmnet` [13:38:20] (and I also supply the password using -p) [13:38:32] from mwmaint1002 [13:38:52] sure sure but where do you get the pass? Is it stored in your home, etc.. ? [13:39:09] because on stat1* boxes we have a special file to use for the research user [13:39:34] so people do not have to use copy credentials here and there [13:40:10] heh. I copy :) [13:40:26] (how awful is it?) [13:42:04] a lot of people do it, no intentions to blame, just to figure out where the research_prod user comes from :) [13:42:38] !log allow the research user to create/alter/etc.. tables on staging@db1108 [13:42:40] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:44:03] aharoni: would it be a problem for you to start using the 'research' user on stat1007 or 1006 ? [13:44:33] elukey: I can use whatever you tell me to use, but the really important consideration is something else: [13:44:35] credentials are under /etc/mysql/conf.d/analytics-research-client.cnf [13:45:03] What's the right thing to do for this Phab task I keep mentioning: https://phabricator.wikimedia.org/T189475 [13:45:20] I run hive on stat1007 usually. Can it also run mysql? [13:45:31] If I'm not mistaken, joal and mforns recommend mysql for this task. [13:45:34] aharoni: I was about to say that the research user can now create tables on db1108 [13:45:44] so if you migrate to it then everything will work fine [13:46:00] aharoni: yep you can access mysql as well from there [13:50:42] elukey: hmmm.... how exactly? [13:50:53] I suspect it's not quite the same as on mwmaint1002 [13:57:09] aharoni: so you can read the /etc/mysql/conf.d/analytics-research-client.cnf file [13:57:26] on the stat boxes (since you are part of one of the unix groups allowed, just checked) [13:57:38] basically you do the same thing as you do on mwmaint1002 [13:57:42] yes, I can cat it [13:57:44] Oh [13:58:04] I don't remember exactly how but there is a way to tell the mysql client to use the file directly [13:58:11] so you don't even need to use -u -p etc.. [13:58:43] then if you try to create the table in staging with 'research' it should work fine [13:58:43] joal: mforns milimetric - do you remember? I'm really not a mysql power user :) [13:59:46] or you can simply do as you did up to now [14:01:17] aharoni: --^ [14:06:30] elukey: is it still `-h db1108.eqiad.wmnet` [14:06:31] ? [14:07:05] yep exactly [14:09:49] I'm running `mysql -pblablabliblablabla -u research -h db1108.eqiad.wmnet` and I get [14:10:11] ERROR 1045 (28000): Access denied for user 'research'@'10.64.21.118' (using password: YES) [14:11:38] elukey: Oh, I managed to make it work [14:13:39] aharoni: managed to run create table :) thanks! [14:15:34] goood :) [14:32:53] hello joal, my understanding is that the cassandra bundle can't be restarted until the end of the month, so I should focus on now backfilling per project family uniques right? [14:33:23] fdans: we can restart single jobs, but the bundle will need to wait [14:34:09] fdans: 2 approaches - We stop the uniques-job in the existing bundle and restart a manual one, and we setup an alarm to restart the whole bundle at beginning of December [14:34:24] Or we just setup the alarm and we'll need to backfill until 1st of december [14:34:33] I have no preference [14:34:57] hmmmm I think I like the first option slightly more [14:36:05] joal: because if we restart jobs now, we can expose numbers immediately in aqs right? [14:55:24] joal: any idea about the event.mediawiki_job_* tables? O_o are they all jobs? [15:01:58] hi analytics [15:02:20] any expert on sparql - wikidata here? I have a question :) [15:02:52] dsaez: whats your question? :D but your best bet might be #wikidata ! [15:02:59] oh [15:03:02] ok. [15:03:37] I want to retreive all items with a propierty (P625), but I get a timeout, probably because the response is too big [15:03:51] I'm wondering other ways to do this. [15:03:56] dsaez: whats the exact query your running? [15:04:55] simplest version is: SELECT ?item ?itemLabel ?coord ?coordLabel WHERE { [15:04:55] ?item wdt:P625 ?coord . [15:04:55] SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". } [15:04:55] } [15:05:19] if I add a limit at the bottom it works [15:05:34] https://query.wikidata.org/sparql?query=%23%20Sample%20to%20query%20all%20values%20of%20a%20property%0A%23%20Property%20talk%20pages%20on%20Wikidata%20include%20basic%20queries%20adapted%20to%20each%20property%0A%23%20valueLabel%20is%20only%20useful%20for%20properties%20with%20item-datatype%0A%23%20remove%20or%20change%20limit%20for%20more%20results%0ASELECT%20%3Fitem%20%3FitemLabel%20%3Fcoord%20%3FcoordLabel%20WHERE%20%7B%0A%20%20%3Fitem% [15:05:35] 20wdt%3AP625%20%3Fcoord%20.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%22.%20%7D%0A%7D%0ALIMIT%2010 [15:05:45] getting rid of the label service might make it run [15:07:17] oh! let me try! [15:07:47] let me know :) [15:10:39] dsaez: aaaah, all entities with coordinate location? :) [15:10:50] addshore, yep [15:11:26] too much? [15:11:30] dsaez: take a look at https://tools.wmflabs.org/wikidata-analysis/20181112/ [15:11:43] thats extracted from the dumps and used ot generate the wikidata map that you might have seen [15:13:17] dsaez: you can use wdlabel file if you just want to know how many entities have coordinate locations [15:13:21] see what I do at the bottom of https://addshore.com/2018/10/wikidata-map-october-2018/#more-2077 [15:14:12] if you just want a count :) [15:14:21] but that file should also have all of the entity ids too [15:14:21] amazing! [15:14:24] and labels ;) [15:14:31] how do you generate that? [15:14:34] and it should TM be generated weekly [15:15:10] dsaez: https://github.com/wikimedia/analytics-wmde-toolkit-analyzer/blob/master/analyzer/src/main/java/org/wikidata/analyzer/Processor/MapProcessor.java [15:15:12] I would like to add some categories: for example 'is instance of' [15:15:25] so, the other file actually has more data [15:15:44] and links relations of the entities with coordinates [15:15:47] https://www.irccloud.com/pastebin/KrG5bYEi/ [15:16:05] this used to live in an interactive JS map, but it died as the data got too big [15:16:45] joal, aharoni, I'm back (although I see issues are solved) [15:16:51]