[05:33:26] gooooood morning a-team! [06:48:15] o/ [08:29:04] 10Analytics: Some fields in Pivot should be numbers - https://phabricator.wikimedia.org/T167494#3343721 (10Gilles) For the HTTP response code, both would work, because the set of possible values is finite and relatively small. I think we should pick what makes queries faster (i.e. is it faster to run a regexp li... [08:46:04] * elukey brb! coffee [09:25:22] a-team: I am about to merge the big zookeeper refactoring (https://gerrit.wikimedia.org/r/#/c/354449/) [09:25:25] please be aware :) [09:59:21] joal: you there? [09:59:45] (nothing urgent, just need to double check with you a thing for druid and zk) [10:10:16] !log merged big zookeeper refactoring https://gerrit.wikimedia.org/r/#/c/354449 - Druid's Hadoop client config now correctly points to conf1* and not drud1* [10:10:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:10:24] joal: --^ this is the TLDR [10:10:34] let me know when you'll be online if everything looks good [10:10:56] I rolled out the change everywhere (druid, analytics, kafka, conf, etc..) [10:41:16] I am also fixing deployment-prep, removing the old trusty zk instance [10:57:39] * elukey lunch! [11:18:03] Hi a-team, late start today as planned - Everything seems fine with the druid change (so far) [11:18:26] fdans: What happened to you this morning? 7:30 is way to early :) [11:19:43] joal: I'm an early morning insomniac! sometimes it gets to 5am and I'm unable to sleep anymore ;) [11:27:22] Wow [11:27:39] fdans: When it happens to me it;s the other way around: I start to sleep at 5 am :) [11:29:06] 10Analytics: Some fields in Pivot should be numbers - https://phabricator.wikimedia.org/T167494#3344311 (10JAllemandou) There are differences in performance: [[ http://druid.io/docs/0.10.0/ingestion/schema-design.html | http://druid.io/docs/0.10.0/ingestion/schema-design.html ]] , Numeric dimensions section. I h... [11:42:51] 10Analytics-Kanban: Some fields in Pivot should be numbers - https://phabricator.wikimedia.org/T167494#3344325 (10JAllemandou) a:03JAllemandou [11:43:15] (03PS4) 10Joal: Add sampled-webrequest druid loading oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 (https://phabricator.wikimedia.org/T166967) [11:45:38] (03PS5) 10Joal: Add sampled-webrequest druid loading oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 (https://phabricator.wikimedia.org/T166967) [11:52:25] (03PS6) 10Joal: Add sampled-webrequest druid loading oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 (https://phabricator.wikimedia.org/T166967) [12:03:40] "I start to sleep at 5 am" LOL [12:04:12] * elukey picture Joseph working at this desk and realizing, all of a sudden, that it is 5am [12:04:27] :) [12:05:59] I am impressed that nothing exploded for the zookeeper change [12:06:18] but we need a lot more refactorings [12:06:22] * elukey opens a phab task [12:17:49] 10Analytics: Refactor puppet code for the Hadoop Analytics cluster to roles/profiles - https://phabricator.wikimedia.org/T167790#3344412 (10elukey) [12:18:45] 10Analytics: Refactor puppet code for the Hadoop Analytics cluster to roles/profiles - https://phabricator.wikimedia.org/T167790#3344425 (10elukey) [12:30:19] (03PS7) 10Joal: Add sampled-webrequest druid loading oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 (https://phabricator.wikimedia.org/T166967) [12:34:34] (03PS8) 10Joal: Add sampled-webrequest druid loading oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 (https://phabricator.wikimedia.org/T166967) [12:37:05] (03PS9) 10Joal: Add sampled-webrequest druid loading oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 (https://phabricator.wikimedia.org/T166967) [12:40:19] 10Analytics-Kanban: Some fields in Pivot should be numbers - https://phabricator.wikimedia.org/T167494#3344473 (10JAllemandou) Actually, since we receive http_status codes as string in the original json, let's keep it as string, preventing potential future cast errors. [12:45:53] 10Analytics, 10Analytics-Cluster, 10User-Elukey: Monitor HDFS blocks problems - https://phabricator.wikimedia.org/T163908#3344505 (10elukey) Added Missing and Corrupt blocks graphs to https://grafana.wikimedia.org/dashboard/db/analytics-hadoop [13:00:40] milimetric: wanna define dimensionaldata specs in the batcavoix? [13:01:01] (imma go get a bottle of wataaaaah) [13:09:13] fdans: in a half hour or so [13:09:30] cool! [13:15:05] joal: question about the IP field in webrequest Pivot - is the dropdown sorted by number of requests? [13:15:28] elukey: I assume so but I'm not sure [13:15:30] this is a question from traffic, I don't think there is such ordering but I wanted to check with you [13:15:33] ah! [13:23:25] elukey: which version of druid do we use now? [13:23:50] Have we bumped to 0.10 yet? [13:24:30] joal: I am almost sure 0.9 since 0.10 requires java8 [13:24:35] andrew rolled it back [13:24:41] elukey: Ahhh, yes I recall now [13:24:44] thanks :) [13:25:01] elukey@druid1003:~$ dpkg --list | grep druid [13:25:01] ii druid-broker 0.9.0-2~jessie1 all Druid Broker Node [13:25:04] ii druid-common 0.9.0-2~jessie1 all Druid Common [13:25:07] ii druid-coordinator 0.9.0-2~jessie1 all Druid Coordinator Node [13:25:10] ii druid-historical 0.9.0-2~jessie1 all Druid Historical Node [13:25:13] ii druid-middlemanager 0.9.0-2~jessie1 all Druid MiddleManager Node [13:25:16] ii druid-overlord 0.9.0-2~jessie1 all Druid Overlord Node [13:25:19] there we go :) [13:25:45] 10Analytics-Kanban: Some fields in Pivot should be numbers - https://phabricator.wikimedia.org/T167494#3334762 (10JAllemandou) Moving this task to pause, since it requires druid-0.10 and we have 0,9 due to java major version incompatibilities. [13:26:12] let's upgrade java now on the cluster :P [13:29:20] (03PS10) 10Joal: Add sampled-webrequest druid loading oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 (https://phabricator.wikimedia.org/T166967) [13:30:30] elukey: looking at druid, it seems the filters are shown by descending selected measure [13:30:43] 10Analytics, 10Operations, 10ops-eqiad: analytics1067: Broken BBU - https://phabricator.wikimedia.org/T167797#3344626 (10Marostegui) [13:30:52] it seems so, but the traffic team needed the "split" functionality [13:30:55] super happy people npw [13:30:57] now :D [13:31:00] 10Analytics, 10Operations, 10ops-eqiad: analytics1067: Broken BBU - https://phabricator.wikimedia.org/T167797#3344640 (10Marostegui) p:05Triage>03Normal [13:31:08] split? [13:31:30] yes the second row [13:31:31] Ah, moving a dimension in the split line, right :) [13:31:36] yes sorry :) [13:31:39] np [13:31:46] you know me I am a mechanic [13:32:18] not turk though [13:32:18] not really a fine data cruncher [13:32:32] * elukey tells excuses for his ignorance [13:32:53] anyhow, on the mechanic side.. BBU is broken for an1067! (see above task) [13:33:13] joal: i'm looking into the remove xff stuff [13:33:24] do we use the legacy pageview def stuff [13:33:51] hm, not sure what it is ottomata [13:34:13] 10Analytics, 10Operations, 10ops-eqiad: analytics1067: Broken BBU - https://phabricator.wikimedia.org/T167797#3344646 (10elukey) @Cmjohnson this host is one of the last batch (so under warranty for sure), can you order a new BBU whenever you have time? [13:34:28] ok [13:34:50] * elukey coffee! [13:35:36] 10Analytics, 10Operations, 10ops-eqiad: analytics1067: Broken BBU - https://phabricator.wikimedia.org/T167797#3344650 (10Marostegui) I have not forced the RAID to go to WB, I would leave that to #analytics. If needed, this should be it: ``` megacli -LDSetProp -ForcedWB -Immediate -Lall -aAll ``` And to re... [13:35:54] 10Analytics-Dashiki, 10Analytics-Kanban, 10MW-1.30-release-notes (WMF-deploy-2017-06-13_(1.30.0-wmf.5)), 10Patch-For-Review, 10Wikimedia-log-errors: Warning: JsonConfig: Invalid $wgJsonConfigModels['JsonConfig.Dashiki'] array value, 'class' not found - https://phabricator.wikimedia.org/T166335#3344655 (10... [13:36:18] ok fdans [13:36:19] omw [13:42:18] joal: as far as I can tell, the only thing xff is used for in the code is IsLegacyPageviewUDF [13:42:23] which we don't use anymore at all [13:47:43] helloooo [13:47:46] elukey, yt? [13:49:20] ottomata: I don't mind doing some cleaning and removing that code [13:50:14] mforns: I am! [13:50:33] hello luca :] I forgot to tell you sth about the EL script that I thought the other day [13:51:04] joal: am on it :) [13:51:08] Thanks ottomata :) [13:51:12] checking out hive right now [13:51:13] mforns: something bad? :D [13:51:23] 10Analytics: Refactor puppet code for the Hadoop Analytics cluster to roles/profiles - https://phabricator.wikimedia.org/T167790#3344412 (10akosiaris) > 2. move to profiles/roles, and possibly strive for having a single role in site.pp. I assume you mean a single role per host. That's the end goal after the ado... [13:51:25] to be sure if we don't have the field in json data, that evertyhing will work [13:51:26] elukey, the thing is... remember I convinced you to check if all fields in the whitelist belong to the table? [13:51:47] hallo [13:51:48] so, there is the possibility that the field is not in the table, and it's not wrong [13:52:11] really? [13:52:43] elukey, imagine there's a whitelist for a given table, and then someone alters the schema to add a new field (which creates a new revision and the corresponding table) [13:52:55] and then they update the whitelist with the new field [13:53:12] the older tables for the same schema prefix will not have that field! [13:53:34] but this is OK, the problem is with the check, it's too strict [13:53:59] so, my bad [13:54:20] mmm is it a real use case? [13:54:23] yes [13:54:48] okok we can easily remove the check [13:54:55] there is another issue though [13:54:56] adding a field to a schema is a common use case [13:55:06] if that field is whitelisted, there we have the problem [13:55:24] I had a chat with the dbas and they are concerned about the performance hit of limit offset [13:55:31] mmmmmm [13:55:43] because it might be heavy as offset increase [13:55:50] I see [13:56:01] they are going to review it but one thing that we could do is to provide numbers [13:56:17] so we'll know more or less what kind of queries the script will make [13:57:21] elukey, maybe we can calculate roughly how many events per hour does a table have [13:57:33] and then slice accordingly using timestamps [13:57:47] orrrrrrrr [13:57:52] it could be an option but I'd like to have numbers for the current use case [13:58:02] because it would be great not to change our logic anymore [13:58:38] we could do sth like: we start updating a small slice with timestamp range, like 5 minutes, and then when we get the results, we know how many events there were [13:58:46] and the next slice we adjust accordingly [13:59:11] elukey, totally, it would be awesome to leave it like that ;] [14:01:45] elukey, another option is to get the timestamp of the latest updated event, and pass it to the next query [14:02:43] elukey, maybe it takes less time to change that, than get satisfying numbers and convince dbas? [14:02:50] dunno [14:03:05] a-team: q: should we remove IsLegacyPageview code from refinery-source. [14:03:06] ? [14:03:10] it isn't used anywhere [14:03:22] but, maybe it is nice to have around, in case we ever need to check if something is considered a legacy pageview [14:03:30] you're da boss, elukey, let me know if I can help you with pairing on numbers or anything [14:03:46] hahaah [14:03:47] ottomata: I'd leave it and deprecate it [14:08:16] (03PS1) 10Ottomata: Remove deprecated ClientIpUDF and deprecate Legacy Pageview code. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/358603 (https://phabricator.wikimedia.org/T118557) [14:10:22] 10Analytics, 10Operations, 10ops-eqiad: analytics1067: Broken BBU - https://phabricator.wikimedia.org/T167797#3344826 (10Marostegui) 05Open>03Resolved Looks like it recovered itself: ``` ˜/icinga-wm 16:06> RECOVERY - MegaRAID on analytics1067 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy... [14:14:34] 10Analytics-Kanban, 10Operations, 10Traffic, 10Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#3344847 (10Ottomata) Ooook, I just checked some things. - x_forwarded_for was only being used by legacy pageview code in refinery, which itself... [14:14:35] k fdans, test up for merge and total [14:15:25] milimetric: commit on from now on, we're going to squash them all any way [14:15:34] fear not the commit for it is thy friend [14:15:43] oh yeah, rebase -i is my friend [14:16:12] it used to be all "rooaoaaaar" and I took it behind an alley and broke it. Now it's all "how may I help you sir" [14:17:06] https://media.tenor.com/images/018732737fd4d78812000f1014c48ed7/tenor.gif [14:17:27] nice test, I like this way of doing this, gonna slay this now [14:17:31] milimetric: ^ [14:18:09] HIMYM is only good 'cause of him [14:18:18] how dare you [14:19:01] fdans: I wasn't sure if I wanted the API to return a new DimensionalData or just alter the existing one, you tell me which is better based on the underlying impl, I don't care [14:19:13] like instead of let merged = we can just do dim.merge [14:19:52] definitely change the state of the current one, given the way crossfilter works [14:26:30] milimetric: pong! [14:27:20] fdans: ping [14:31:30] fdans: I think that test is flawed actually, I have to tell it what to roll up on [14:33:11] ok fdans, pushed a better version of that [14:40:15] joal: https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?panelId=40&fullscreen&orgId=1&from=now-3h&to=now [14:40:23] https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?panelId=39&fullscreen&orgId=1&from=now-3h&to=now [14:40:44] nice elukey :) [14:41:03] great elukey! [14:41:17] looking back in time, there were moments with corrupted blocks :) [14:41:31] I am thinking to add an alarm for an1001, like X hours > 0 [14:41:44] would be a good idea! [14:45:35] Is this maven-release-user something from you folks? https://gerrit.wikimedia.org/r/#/c/290630/ [14:49:21] 10Analytics, 10Analytics-Cluster, 10User-Elukey: Monitor that no worker nodes are in the default rack in net topology - https://phabricator.wikimedia.org/T163909#3344922 (10elukey) `sudo -u hdfs hdfs dfsadmin -printTopology` executed on any Hadoop node must not return any mention of `default rack`. This migh... [14:52:12] milimetric: sorry parents called [14:52:27] np, can I push another one fdans ? [14:52:37] oh yeah go ahead [14:52:43] k, ping again then [14:53:01] and that does it, I think! The interface at least is quite a bit easier than imagined [14:53:26] entirely possible my sleepy brain is not thinking of some edge cases [14:57:06] milimetric: so we're generating a new instance with merge? [14:57:15] i guess we can pass over the crossfilter object [14:57:37] it's an open question, I don't know what's best for the underlying implementation [14:57:53] jump in the batcave, let's chat a bit before standup [14:57:53] hmm, i'd prefer to modify the dd than generating new instances [14:57:59] omw [14:58:23] (03Abandoned) 10Nuria: Add refinery-source jars for v0.0.26 to artifacts [analytics/refinery] (jenkins-test) - 10https://gerrit.wikimedia.org/r/290630 (owner: 10Maven-release-user) [14:59:34] milimetric: hi, a very quick question that your sleepy brain should be able to process easily: can you please remind me of an example of scheduling a script with ReportUpdater? [15:00:09] aharoni: scheduling an arbitrary script as opposed to a sql query, you mean? [15:01:17] I am joining :) [15:01:22] aharoni: https://github.com/wikimedia/analytics-reportupdater-queries/tree/master/browser [15:02:07] milimetric: I'd like to examine both. And for now I need more beeline than mysql. Or is it the same? [15:02:29] milimetric: which file in that directory? [15:03:09] aharoni: as with all reportupdater, config.yaml is the config, the keys in that file are the names of the scripts in that directory, and there is a helper python file there as well [15:03:23] OK, that's a start. Thanks! [15:03:27] np [15:05:27] milimetric: If I write something, do I have to do something special to deploy it? Or just merge a patch in analytics/limn-language-data ? [15:06:17] aharoni: merging will deploy it automatically, but add me as a reviewer before you merge [15:06:38] milimetric: OK, I think that's it for now. Thanks! [15:06:46] np [15:07:19] 10Analytics-Kanban, 10Analytics-Wikistats: Data Model using Crossfilter, called DimensionalData - https://phabricator.wikimedia.org/T167678#3344989 (10Milimetric) a:03fdans [15:10:32] 10Analytics-Kanban, 10Patch-For-Review: Make non-nullable columns in EL database nullable - https://phabricator.wikimedia.org/T167162#3344994 (10mforns) a:05mforns>03elukey [15:10:56] 10Analytics-Kanban, 10Patch-For-Review: Make non-nullable columns in EL database nullable - https://phabricator.wikimedia.org/T167162#3344995 (10Nuria) [15:11:21] 10Analytics-Kanban, 10Patch-For-Review: Make non-nullable columns in EL database nullable - https://phabricator.wikimedia.org/T167162#3344997 (10mforns) Sorry, the gerrit patch belongs to another task. This task is about the alter tables to modify non-nullable columns in EL database. [15:11:39] 10Analytics-Kanban: Modify EventLogging so that all table fields are nullable - https://phabricator.wikimedia.org/T167161#3344999 (10mforns) a:03mforns [15:12:18] 10Analytics-Kanban: Modify EventLogging so that all table fields are nullable - https://phabricator.wikimedia.org/T167161#3319587 (10mforns) This is the changeset that belongs to this task. It had been linked with another task by mistake: https://gerrit.wikimedia.org/r/357665 [15:15:06] 10Analytics-Kanban, 10Patch-For-Review: Make non-nullable columns in EL database nullable - https://phabricator.wikimedia.org/T167162#3345007 (10Nuria) [15:16:32] mforns: thanks for review, just responded and submitted new patch [15:16:43] ottomata, k, will look! [15:17:37] 10Analytics-Kanban, 10Operations, 10Traffic, 10Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#3345019 (10Nuria) a:03Ottomata [15:26:35] 10Analytics-Kanban, 10Operations, 10Traffic, 10Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#3345058 (10Ottomata) @bblack, just one last double check: are you sure XFF is not useful for ops purposes? We can easily exclude this data fro... [15:27:57] (03CR) 10BBlack: [C: 031] Remove deprecated ClientIpUDF and deprecate Legacy Pageview code. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/358603 (https://phabricator.wikimedia.org/T118557) (owner: 10Ottomata) [15:31:42] 10Analytics-Kanban, 10Operations, 10Traffic, 10Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#3345069 (10BBlack) No I don't think we need it for non-immediate analysis like this. We still `zero`, `zeronet` and `proxy` in the X-Analytics... [15:31:54] 10Analytics-Kanban, 10Page-Previews, 10Reading-Web-Backlog: Update purging settings for Schema:Popups - https://phabricator.wikimedia.org/T167449#3345070 (10mforns) In the following update to the white-list https://gerrit.wikimedia.org/r/#/c/298721/9..10/files/mariadb/eventlogging_purging_whitelist.tsv I app... [15:32:12] 10Analytics-Kanban, 10Operations, 10Traffic, 10Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#3345072 (10Ottomata) Ok! Will merge this today then, thanks. [15:40:09] 10Analytics, 10Operations: New analytic hosts with BBU learning cycle enabled - https://phabricator.wikimedia.org/T167809#3345083 (10Marostegui) [15:40:49] 10Analytics, 10Operations, 10ops-eqiad: analytics1067: Broken BBU - https://phabricator.wikimedia.org/T167797#3344626 (10Marostegui) Looks like when the server is recharging it might not shown the correct status of the BBU, looks like this wasn't broking, just started an Auto-Learn cycle: T167809 [15:41:10] 10Analytics, 10Operations: New analytic hosts with BBU learning cycle enabled - https://phabricator.wikimedia.org/T167809#3345083 (10Marostegui) p:05Triage>03Normal [15:41:42] 10Analytics, 10Operations: New analytic hosts with BBU learning cycle enabled - https://phabricator.wikimedia.org/T167809#3345083 (10Marostegui) [15:42:57] 10Analytics, 10Operations, 10User-Elukey: New analytic hosts with BBU learning cycle enabled - https://phabricator.wikimedia.org/T167809#3345126 (10elukey) [15:44:13] 10Analytics-Kanban, 10Operations, 10ops-eqiad: analytics1030 stuck in console while booting - https://phabricator.wikimedia.org/T162046#3345131 (10Cmjohnson) 05Resolved>03Open The bbu has bene replaced and the system board that was replaced needs to be swapped again. The service tag that was burned in w... [16:04:26] ping mforns [16:17:36] 10Analytics-Kanban: Use native timestamp types in Data Lake edit data - https://phabricator.wikimedia.org/T161150#3345223 (10JAllemandou) a:03JAllemandou [16:31:05] milimetric: you want to change the tests or should I change them myself? I don't mind if you're busy [16:57:39] joal, ottomata, milimetric: forgot to say THANK YOU for the mediawiki_page_history table. It's awesome and useful and I love you all. [16:58:31] fdans: aaah, sorry, I pushed the change a while back, forgot to say [16:58:39] wikimedia/mediawiki-extensions-EventLogging#656 (wmf/1.30.0-wmf.5 - b3ac3c0 : Amir Sarabadani): The build has errored. [16:58:39] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/compare/wmf/1.30.0-wmf.5 [16:58:39] Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/242504039 [16:58:42] halfak: sweet [16:59:08] :) [16:59:17] milimetric: cooooool beanz [16:59:37] Thank you halfak :) [18:02:53] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Data request for logs from SparQL interface at query.wikidata.org - https://phabricator.wikimedia.org/T143819#3345580 (10Nuria) If @Smalyshev thinks this would be a good idea and can develop the instrumentation for the metrics and own the met... [18:11:54] a-team: going afk now, talk to you next week! [18:12:01] bye elukey ! [18:15:20] ebernhardson: add me to your CR if you ever want to do changes to ext.wikimediaEvents.searchSatisfaction.js to remove the non becaon code, i think we should also change https://github.com/wikimedia/mediawiki-extensions-WikimediaEvents/blob/master/modules/ext.wikimediaEvents.searchSatisfaction.js#L65 [18:16:01] ebernhardson: to not add the timestamp, the token is unique in a 500 million set so no more uniqueness is needed. [18:18:14] laters! [18:20:54] nuria_: have a minute? [18:21:05] joal: yessir [18:21:37] nuria_: I think we're ready to anounce uniques name-change, and also we can tell Tilman to review project-wide [18:21:52] nuria_: excuse me back in a minute [18:22:04] joal: nice, are new jobs running? [18:22:14] joal: did you let data bake a bit? [18:26:06] nuria_: jobs are since yesterday, so not along time, but everything seems fine [18:26:36] nuria_: I have updated the docs as planned, and got an interesting result with the patch on per-domain for offset [18:27:40] joal: let's let them bake a bit before announcing them just in case there is a deployment mistake, let's check tomorrow and thursday to make sure per-domain uniques are populating and that jobs are not erroing due to any typos [18:27:53] joal: aham (re: patch for offset) [18:28:42] nuria_: aham? [18:30:38] nuria_: sounds good about waiting a few days [18:30:51] nuria_: quick batcave about offset diff finding? [18:32:01] joal: sure [18:43:35] also nuria_, following on the thread about sparQL analysis, this could take advantage of tagging+splitting [19:06:44] joal: ya, once they have instrumentation [19:06:49] yup [19:06:54] joal: which i do not think is te case yet [19:06:57] *the [19:07:22] joal: at least from stash reply that is what i got [19:08:55] nuria_: I double checked webrequest, sparql query seems encoded in query [19:12:06] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Data request for logs from SparQL interface at query.wikidata.org - https://phabricator.wikimedia.org/T143819#3345746 (10Nuria) As far as I understand you need to publish not only queries to service but also query results (is this correct @Sma... [19:12:20] joal: but to get the counts they want they need quary results, not incoming queries [19:12:29] joal: as tehy are looking for "usage" of entities [19:12:32] *they [19:12:36] joal: makes sense? [19:12:50] nuria_: I don't know if usage means: is in result, or : is in query [19:13:13] joal: stas says "Hmm not sure how to implement this yet, as we do not track which items were in query results" [19:13:37] right - Then no good with what we have [19:16:07] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Data request for logs from SparQL interface at query.wikidata.org - https://phabricator.wikimedia.org/T143819#3345760 (10Smalyshev) It may be hard to capture query results, given that we don't have any mechanism of tracking them now. We do hav... [19:22:07] gone for tonight a-team - see you tomorrow [19:22:18] bye joal ! [19:25:52] byyye [20:36:24] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Genericize ca-manager script - https://phabricator.wikimedia.org/T166167#3346157 (10Ottomata) Draft WIP work happening here: http://github.com/ottomata/certgen [21:14:04] 10Analytics-Kanban: Implement purging settings for Schema:ReadingDepth - https://phabricator.wikimedia.org/T167439#3346256 (10mforns) Hi @Tbayer :] I modified the white-list to include these changes, see: https://gerrit.wikimedia.org/r/#/c/298721/10..11/files/mariadb/eventlogging_purging_whitelist.tsv Please re... [22:46:59] (03PS14) 10Nuria: UDF to tag requests [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) [23:37:09] (03PS15) 10Nuria: UDF to tag requests [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) [23:39:45] (03CR) 10Nuria: "I tried to address outstanding comments, there is one outstanding question of UDFs extending each other." (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria) [23:41:18] (03CR) 10Nuria: UDF to tag requests (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria)