[03:23:10] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Add pageviews total counts to WDQS - https://phabricator.wikimedia.org/T174981#3578733 (10Yurik) [03:23:25] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Add pageviews total counts to WDQS - https://phabricator.wikimedia.org/T174981#3578746 (10Yurik) [03:35:21] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Add pageviews total counts to WDQS - https://phabricator.wikimedia.org/T174981#3578763 (10Yurik) [05:48:16] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Add pageviews total counts to WDQS - https://phabricator.wikimedia.org/T174981#3578956 (10Yurik) [06:38:42] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Improve purging for analytics-slave data on Eventlogging - https://phabricator.wikimedia.org/T156933#3579027 (10elukey) The new script is running on db1047, restarting from 2009 and ending up to half of 2016 data. Everything seems to work fine, and afai... [06:51:39] hello team, I'd need to do some errands so I'll be afk for a bit, but completely reachable via phone/hangouts/pidgeons/radio-waves/etc.. if you'll need me :) [07:21:44] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3579066 (10JAllemandou) @GoranSMilovanovic : Thanks for investigating and keeping me posted :) Good to know it was not a bug ! I'm also interested... [07:36:39] 10Analytics-Kanban, 10Research: productionize ClickStream dataset - https://phabricator.wikimedia.org/T158972#3579080 (10JAllemandou) Last update: Data is vetted and exactly the same as Ellery's using my last patch for enwiki on month 2017-08 :) @Shilad: My idea was that reviewing this could provide you with... [07:37:15] 10Analytics-Kanban, 10Research: productionize ClickStream dataset - https://phabricator.wikimedia.org/T158972#3579081 (10JAllemandou) [08:16:58] 10Analytics, 10Analytics-EventLogging: Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3579120 (10Tbayer) Now at 10am on August 31: ```lang=sql,lines=10 SELECT DATE(timestamp) AS date, HOUR(timestamp) AS h... [08:39:03] back! [08:41:48] the eventlogging_cleaner script seems to have finally a good shape [08:42:03] ~15s to update 100k rows [08:45:19] before, on dbstore1002 that is faster, it used to take ~380s \o/ [08:57:53] Yay elukey ! [09:11:14] started the eventlogging_cleaner script also on dbstore1002 [09:13:47] joal: in the eventlogging_sync.err log I can see the following [09:13:48] mysqldump: Couldn't execute 'SELECT /*!40001 SQL_NO_CACHE */ * FROM `NavigationTiming_13317958` WHERE id >= 20170420192612 ORDER BY id LIMIT 1000': Unknown column 'id' in 'where clause' (1054) [09:14:10] I am not sure when it got logged the last time, but it recurred a lot [09:14:20] I am now tailing the logs to see if it will re-appear [09:14:38] but it might be that this error stops the replication? [09:14:46] elukey: possible !! [09:14:53] yeah it is recent, just re-occurred [09:15:18] elukey: I have no clue about the replication process, except that it is non-standard and not fun [09:15:42] elukey: Hopefully ottomata will be back later today and will explain :S [09:16:25] joal: the gist of it is that on every slave eventlogging_sync creates a mysqldump from the master (only with the last data that it doesn't have for all its table) and then it executes it in localhost [09:16:47] ok [09:16:47] now NavigationTiming_13317958 doesn't have any id [09:17:50] but maybe it is only a warning [09:17:57] hm, I thought id was created by EKL itself [09:19:39] EKL? [09:19:59] EL sorry [09:20:01] ahh okok [09:20:14] some tables do not have the id for some reason [09:20:42] I think that Marcel explained that to me but it went into swap and I can't recover it now :D [09:22:12] I can't really help :S [09:22:24] elukey: Do we wait for ottomata or do we look further? [09:24:48] joal: I was just checking for curiosity, we can wait for andrew [09:24:54] ok :) [09:25:26] the other idea that I have is the batch size [09:26:04] we execute /usr/local/bin/eventlogging_sync.sh -D 90 -b 1000 -d log m4-master.eqiad.wmnet localhost [09:26:09] so 1000 lines at the time [09:26:47] elukey@dbstore1002:/var/log$ tail -n 10000 eventlogging_sync.log | grep Popups_16364296 [09:26:51] [..] [09:27:03] 2017-09-05T09:24:05 localhost log Popups_16364296_15423246 (no new data on master in last 90 days, skipping) [09:27:06] 2017-09-05T09:24:32 localhost log Popups_16364296 (rows!) ok [09:27:08] 2017-09-05T09:24:48 localhost log Popups_16364296_15423246 (no new data on master in last 90 days, skipping) [09:27:12] 2017-09-05T09:25:31 localhost log Popups_16364296 (rows!) ok [09:27:14] 2017-09-05T09:25:42 localhost log Popups_16364296_15423246 (no new data on master in last 90 days, skipping) [09:27:17] 2017-09-05T09:26:03 localhost log Popups_16364296 (rows!) ok [09:27:20] 2017-09-05T09:26:22 localhost log Popups_16364296_15423246 (no new data on master in last 90 days, skipping) [09:29:11] How bizarre :( [09:35:07] 10Analytics, 10Analytics-EventLogging: Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3579247 (10elukey) The strange thing is that the following happens over and over: ``` elukey@dbstore1002:/var/log$ tai... [09:47:55] 10Analytics, 10Analytics-EventLogging, 10Readers-Web-Backlog: Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3579280 (10ovasileva) [09:48:32] 10Analytics, 10Analytics-EventLogging, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3573902 (10ovasileva) [09:48:46] 10Analytics, 10Analytics-EventLogging, 10Readers-Web-Backlog: Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3573902 (10ovasileva) [09:49:18] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3579287 (10ovasileva) [09:49:33] joal: this is probably one bug https://gerrit.wikimedia.org/r/#/c/375977 [09:49:56] but I am pretty sure something else is going on [09:50:31] yeah becase it does for table in etc.. [09:51:24] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Add pageviews total counts to WDQS - https://phabricator.wikimedia.org/T174981#3578733 (10Lydia_Pintscher) I would prefer if we concentrate on T143424 instead and have that include page views as one indicator. [09:52:19] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3579296 (10elukey) The above patch is not the root cause of t... [09:52:29] elukey: Well spotted !P [09:53:14] elukey: This bug correction will probably prevent the NavigationTiming error I guess [09:54:21] joal: yep.. I think I can merge now, it seems straighforward [09:54:26] what do you think? [09:54:45] Sounds good to me, but I have no idea if it could break anything else :) [09:55:55] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Add pageviews total counts to WDQS - https://phabricator.wikimedia.org/T174981#3578733 (10Esc3300) The simplicity of this approach seems convincing. It could complement number of sitelinks and statements. [10:11:00] 10Analytics, 10Analytics-EventLogging, 10Patch-For-Review, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3579352 (10elukey) >>! In T174815#3579247, @elukey wrote: > T... [10:11:08] joal: --^ [10:11:10] I am stupid [10:11:52] eventlogging_sync works fine, it is just lagging behind due to the small batch size [10:13:03] elukey: My guess is that this small batch size is to mitigate insertion rate for small and big datasets [10:13:24] 10Analytics-Kanban, 10RESTBase-API, 10Services (later), 10User-mobrovac: Expose pageview data in each project's REST API - https://phabricator.wikimedia.org/T119094#3579360 (10mobrovac) 05declined>03Open p:05Normal>03Low @Nuria there seems to some confusion in this conversation. I am not proposing... [10:13:32] If we set the nubmer to small, insertion for small datasets is more regular, if we set the number to big, big datasets get inserted without lag ... [10:13:37] yeah it makes sense, because it doesn't hammer the database [10:13:50] so the Popups table is only too big to replicate, that's it [10:14:05] I mean, to big to be replicated in almost real time [10:14:06] elukey: I guess so?? [10:14:38] yes yes I am pretty sure this is the issue [10:14:50] going to fix the other $column bug on db1047 [10:15:22] Cc: HaeB [10:15:23] awesome - thanks for the investigation elukey [10:16:51] \o/ [10:18:38] joal: NavigationTiming now replicates! [10:18:46] elukey: You rock :) [10:19:00] elukey for the WIN ! [10:19:18] takes a bit though, I should wait a second before celebrating :P [10:21:10] ok it worked, phewww [10:22:03] I'll restart the dbstore1002's daemon to update the script after lunch [10:24:28] we can discuss with ottomata if the 1000 value might be tweaked a bit to allow more throughput [10:24:39] let's see also what HaeB thinks about it [10:25:10] 10Analytics-EventLogging, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3579367 (10elukey) [10:25:27] 10Analytics-EventLogging, 10Analytics-Kanban, 10Readers-Web-Backlog (Tracking): Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3573902 (10elukey) a:03elukey [10:52:08] * elukey lunch! [10:53:17] 10Analytics-Tech-community-metrics, 10Developer-Relations: Understand why merged Gerrit changesets does not equal Git commits - https://phabricator.wikimedia.org/T175003#3579448 (10Aklapper) [10:53:31] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jul-Sep 2017): Understand why merged Gerrit changesets does not equal Git commits - https://phabricator.wikimedia.org/T175003#3579448 (10Aklapper) [11:09:57] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3579494 (10Addshore) I decided to look at the NULL banner stuff described above. So, I dived into webrequest to get some sample data using the fo... [11:17:05] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jul-Sep 2017): Understand why merged Gerrit changesets does not equal Git commits - https://phabricator.wikimedia.org/T175003#3579518 (10Aklapper) 05Open>03Resolved On https://wikimedia.biterg.io/goto/61f06dd49a90fdf7d5e45cf1fde5fcc9 `L10n-bot` has... [11:17:14] 10Analytics, 10Analytics-Wikistats: Provide yearly update of stats for audit report - https://phabricator.wikimedia.org/T174950#3579522 (10Aklapper) > Yet the section on volunteers is not in Wikistats, @Aklapper would you be to able to answer that part? (I should have asked your earlier, my apologies) Note: Ne... [11:18:50] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3579524 (10Addshore) Now for **the differences between the data collected in regards to banner impressions from druid** The 2 values in druid sho... [11:19:28] 10Analytics, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3579525 (10Addshore) [11:20:40] ahhhhhh, it feels good adding all that info to that ticket :) [11:28:54] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Add pageviews total counts to WDQS - https://phabricator.wikimedia.org/T174981#3579558 (10Yurik) @Lydia_Pintscher, having a built in ranking system is awesome, but that's a problem of search optimization - just like the other ticket suggests,... [11:32:42] taking a break a-team [12:06:44] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jul-Sep 2017): Check detached accounts in DB with same username for "mediawiki" and "phab" sources but different uuid's (and merge if connected) - https://phabricator.wikimedia.org/T170091#3579666 (10Aklapper) ...and using the usernames in the output o... [13:23:42] 10Analytics-Kanban, 10Operations, 10User-Elukey: Tune Kafka logs to register clients connected - https://phabricator.wikimedia.org/T173493#3579853 (10elukey) [13:40:16] * elukey brb [13:53:53] 10Analytics-Kanban, 10Operations, 10User-Elukey: Tune Kafka logs to register clients connected - https://phabricator.wikimedia.org/T173493#3580049 (10elukey) Tuning the kafka-authorizer appender is definitely important for us since it contains interesting info like: ``` [2017-09-05 13:39:32,147] DEBUG Princ... [14:46:02] (03CR) 10Ottomata: [C: 031] Correct oozie jobs loading pageviews in druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/375766 (https://phabricator.wikimedia.org/T161824) (owner: 10Joal) [15:01:32] a-team: standuuuup [15:01:41] ottomata,nuria_ ping [15:01:45] trying... [15:30:33] elukey: [15:30:52] is it possible that your fix to the el_sync id/timestamp bug is causing the lag in Popups? [15:30:58] that is a big bug! my fault! [15:31:11] but, it means that it'll find TONS of data to replicate for active tables without id columns [15:31:16] since they wouldn't have replicated in a LONG time [15:31:34] i guess not, since it only does 1000 at a time.. [15:31:34] hmm [15:39:38] ottomata: I fixed the bug 3 hours ago :D [15:40:00] the lag in popups started last week [15:40:33] oook [15:40:33] ya [15:40:35] nm :) [15:41:03] ottomata: I don't understand your question in gerrit about pivot explict measures [15:42:17] joal: i didn't see a dataset named there [15:42:27] just a field name to type mapping, right? [15:42:35] does this mean that another dataset with the same name will have to be that type? [15:42:45] does it also mean that any dimension in any dataset will have to be added here? [15:42:57] ottomata: Actually the patch is about removing introspection, and adding explicit fields [15:42:58] (or did I miss the dataset name...) [15:43:04] right, but for which dataset? [15:43:07] just banner streaM? [15:43:19] yessir [15:43:21] OH [15:43:23] i missed that [15:43:24] somehow [15:43:27] ok then [15:43:27] :) [15:43:35] It's already merged I think ;) [15:43:39] brb, running home for future meetings [15:43:39] ya [15:49:30] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3580442 (10Nuria) You can create dashboards in labs too, they do not need to be on the prod domain, see for example: https://edit... [15:50:57] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3580451 (10Nuria) @kaldari I recommed you use labs so you can iterate faster , see: https://wikitech.wikimedia.org/wiki/Analytics/... [15:51:49] ottomata: whenever you have time today, https://gerrit.wikimedia.org/r/#/c/374332/, I'll merge it tomorrow morning :) [15:53:40] elukey: ya gotta send email real quick [15:53:45] but, i was mistaken, in my last email [15:53:46] i said [15:53:53] "I’ll be sure to run a final home directory rsync from stat1003 -> stat1006 before we do." [15:54:01] so, i'm looking at what would be rsynced to see if i should... [15:54:43] ahhh okok [16:00:01] 10Analytics, 10Beta-Cluster-Infrastructure: deployment-kafka01 - disk is full - https://phabricator.wikimedia.org/T174742#3580490 (10elukey) I tried to add api_version to the kafka Hiera config in https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep and restart eventlogging but it didn't count much, still... [16:00:14] ok, elukey there are newer files, i'm going to run the following [16:00:32] rsync -av --update stat1003.eqiad.wmnet::home/ /home/ [16:00:48] with -n, i compared that output with --ignore-existing, and looked at a few files [16:00:55] there are files that exist in both places by are newer on stat1003 [16:01:04] :( [16:01:04] so, with --update, it should only bring over newer ones [16:01:13] yep yep [16:01:14] i'll keep a list of what copied [16:01:29] and put it in a gist and link to it in the phab task [16:14:29] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Replacement of stat1002 and stat1003 - https://phabricator.wikimedia.org/T152712#3580547 (10Ottomata) Today I rsynced /home from stat1003 -> stat1006 with `rsync -av --update stat1003.eqiad.wmnet::home/ /home/`. Only files that either did not exis... [16:15:27] alright, elukey! proceed at will! thank you [16:17:31] ack! [16:18:23] 10Analytics, 10Beta-Cluster-Infrastructure: deployment-kafka01 - disk is full - https://phabricator.wikimedia.org/T174742#3580565 (10Ottomata) Hm, I think we should just make this go away quickly for now, and then re-do deployment-prep kafka instances as part of the Kafka 0.11 upgrade process. We'll do this b... [16:49:59] 10Analytics, 10DBA, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3580724 (10Halfak) [16:50:03] 10Analytics, 10Project-Admins, 10Research, 10cloud-services-team: Create a phabricator project called "wikireplica-datasets" - https://phabricator.wikimedia.org/T173512#3580722 (10Halfak) 05Open>03declined @bd808, I think that makese sense. [16:52:17] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3580730 (10Nettrom) @Nuria : I'm working on this now, got the metrics added to [[m:Dashiki:CategorizedMetrics]] without breaking a... [16:52:54] 10Analytics, 10Project-Admins, 10Research, 10cloud-services-team: Create a phabricator project called "wikireplica-datasets" - https://phabricator.wikimedia.org/T173512#3580731 (10bd808) There is now a `Datasets` column on the #data-services workboard that we can use to at least group these tickets. [16:55:25] 10Analytics, 10Data-Services, 10Research, 10cloud-services-team: Create a database on the wikireplica servers called "datasets_p" - https://phabricator.wikimedia.org/T173513#3580758 (10bd808) [16:55:56] 10Analytics, 10Data-Services, 10Research, 10cloud-services-team: Document the process for importing a new "datasets_p" table - https://phabricator.wikimedia.org/T173514#3580762 (10bd808) [16:56:59] 10Analytics, 10DBA, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3530850 (10bd808) [16:57:01] 10Analytics, 10Data-Services, 10Research, 10cloud-services-team: Create a database on the wikireplica servers called "datasets_p" - https://phabricator.wikimedia.org/T173513#3530910 (10bd808) 05Open>03stalled p:05Triage>03Normal See {T173511} for higher level discussion. [16:57:37] 10Analytics, 10DBA, 10Data-Services, 10Research, 10cloud-services-team (Kanban): Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3530850 (10bd808) [16:57:40] 10Analytics, 10Data-Services, 10Research, 10cloud-services-team: Document the process for importing a new "datasets_p" table - https://phabricator.wikimedia.org/T173514#3530940 (10bd808) 05Open>03stalled p:05Triage>03Normal See {T173511} for higher level discussion. [17:05:56] a-team: stat1003 is going away from puppet [17:08:19] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Replacement of stat1002 and stat1003 - https://phabricator.wikimedia.org/T152712#3580864 (10elukey) stat1003 is official not a analytics host anymore and ssh keys have been removed accordingly, everything (including your home dirs) should already b... [17:21:48] * elukey off! [17:22:42] 10Quarry, 10DBA, 10Data-Services: CHAR_LENGTH does not return the character count - https://phabricator.wikimedia.org/T174543#3580920 (10jcrespo) 05Open>03Resolved a:03jcrespo See: **https://quarry.wmflabs.org/query/21367 which counts by characters, not by bytes.** Mediawiki on WMF-hosted wikis uses B... [17:36:32] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Patch-For-Review: Make webperf eventlogging consumers use eventlogging on Kafka - https://phabricator.wikimedia.org/T110903#3581033 (10Ottomata) Deployed. I think it looks good! Please double check. [17:56:09] 10Quarry, 10DBA, 10Data-Services: CHAR_LENGTH does not return the character count - https://phabricator.wikimedia.org/T174543#3565363 (10Base) @jcrespo , is this why I also fail to get normal results while attempting to match title against a regex? https://quarry.wmflabs.org/query/21026 [18:06:34] hmm, anybody around for a brain bounce? [18:12:57] 10Quarry, 10DBA, 10Data-Services: CHAR_LENGTH does not return the character count - https://phabricator.wikimedia.org/T174543#3581128 (10jcrespo) > is this why I also fail to get normal results while attempting to match title against a regex? I cannot say, I would tell you to try if it helps :-) Some of the... [18:13:33] Hi ottomata [18:13:55] ottomata: May I help [18:13:57] ? [18:15:24] joal: hmm, i think so! might have to explain a bunch. hmmm ya marcel and luca have context hmm [18:15:27] they are out though [18:15:36] but sure let's try [18:15:37] bc? [18:16:05] sure, OMW [18:16:11] 10Analytics, 10cloud-services-team: Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation - https://phabricator.wikimedia.org/T166712#3581160 (10Krenair) Krenair: you should ask some of the cloud team people about it :) [18:18:19] ottomata: What's the best technology to use to count high-frequency server-side MediaWiki events? We want to count how many non-autoconfirmed users are automatically redirected to the Article Wizard per day during ACTRIAL. We don't need any specific info, just a number per day. It will likely be in the thousands. Is that something we would use statsd for? EventLogging? something else? [18:18:49] It will also only be a temporary need [18:18:56] during ACTRIAL [18:24:39] 10Analytics, 10cloud-services-team (Kanban): Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation - https://phabricator.wikimedia.org/T166712#3581208 (10bd808) This schema was for finding out about how people used our `jsub` and `webservice` commands. We can shut this do... [18:51:52] kaldari: ya statsd sounds fine i think [18:52:05] thousands of events per day? [18:52:07] is pretty low [18:52:11] you could use eventlogging if you lie [18:52:39] not sure if MW uses statsd for stuff already, i would think it would [18:52:46] you could probably just deploy some hook that emits to statsd [18:52:49] that woudl def be simplest [18:53:06] Thanks [18:53:07] check with godog (filippo) to be sure, but i think +1000/day to statsd/graphite would be fine [19:27:39] 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: kafka-jumbo.cfg partman recipe creation/troubleshooting - https://phabricator.wikimedia.org/T174457#3581577 (10Cmjohnson) [19:28:45] fwiw sending lots of events for the same counter to statsd is generally fine, we usually only get into trouble sending lots of unique events that take up a ton of graphite disk space (disk space is roughly constant per-metric) [19:30:09] 10Analytics, 10Operations, 10ops-eqiad: Remove stat1002 - https://phabricator.wikimedia.org/T173094#3581589 (10Cmjohnson) p:05Triage>03Lowest stat1002 is still off-site [19:35:50] hey, any examples of how to get provenance into some externally visible thing? [19:42:23] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Add pageviews total counts to WDQS - https://phabricator.wikimedia.org/T174981#3581660 (10Smalyshev) Personally I am not convinced this is a good match for a graph database. This looks like something that is better as a generic API/database. B... [20:13:22] 10Analytics, 10Data-Services, 10Research: Document the process for importing a new "datasets_p" table - https://phabricator.wikimedia.org/T173514#3581850 (10bd808) [20:13:49] 10Analytics, 10Data-Services, 10Research: Create a database on the wikireplica servers called "datasets_p" - https://phabricator.wikimedia.org/T173513#3581855 (10bd808) [20:22:08] 10Analytics, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3581895 (10GoranSMilovanovic) @Addshore Thank you for this thorough inquiry. Yes, the normalized vs. raw request count differen... [20:22:25] 10Analytics, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3581896 (10GoranSMilovanovic) 05Open>03Resolved [21:48:32] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3582304 (10kaldari) I'm an admin on Meta, but it seems I don't have permission to create Config pages either :P [22:08:44] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Make webperf eventlogging consumers use eventlogging on Kafka - https://phabricator.wikimedia.org/T110903#3582384 (10Krinkle) [22:10:55] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Make webperf eventlogging consumers use eventlogging on Kafka - https://phabricator.wikimedia.org/T110903#3582394 (10Krinkle) [22:13:58] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Make webperf eventlogging consumers use eventlogging on Kafka - https://phabricator.wikimedia.org/T110903#3582398 (10Krinkle) Last one remaining is `coal`. That one currently runs on a different host (graphite1001) and writes directly to Graphite di... [23:01:59] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Make webperf eventlogging consumers use eventlogging on Kafka - https://phabricator.wikimedia.org/T110903#3582603 (10Krinkle) [23:02:12] 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Make webperf eventlogging consumers use eventlogging on Kafka - https://phabricator.wikimedia.org/T110903#1589603 (10Krinkle) 05Open>03stalled