[00:55:28] 10Analytics: Bug: can't make a YoY time series chart in Superset - https://phabricator.wikimedia.org/T210687 (10mpopov) [00:56:03] 10Analytics, 10Product-Analytics: Bug: can't make a YoY time series chart in Superset - https://phabricator.wikimedia.org/T210687 (10mpopov) [00:59:28] 10Analytics, 10Product-Analytics: Bug: can't make a YoY time series chart in Superset - https://phabricator.wikimedia.org/T210687 (10Tbayer) For illustration: It might look like [[https://commons.wikimedia.org/wiki/File:Wikimedia_pageviews_year-over-year_comparison_(since_May_2013).png#/media/File:Wikimedia_pa... [01:18:44] 10Analytics, 10Operations, 10Security-Team, 10WMF-Legal, 10Software-Licensing: Can exfat be used in WMF production? - https://phabricator.wikimedia.org/T210667 (10chasemp) Small bit of background from my perspective, I had discussed this on hangout with a few folks who I will let acknowledge their own le... [03:54:50] 10Analytics, 10Analytics-Kanban, 10DBA, 10Data-Services, 10Core Platform Team Backlog (Watching / External): Create materialized views for performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) p:05Triage>03High [03:55:27] 10Analytics, 10Analytics-Kanban, 10DBA, 10Data-Services, 10Core Platform Team Backlog (Watching / External): Create materialized views for performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) cc @Nuria so she's in the loop that we're following up here about the materialized views work. [04:43:23] 10Analytics, 10Analytics-Cluster, 10Continuous-Integration-Infrastructure (Slipway): Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10greg) p:05Triage>03Normal [07:05:51] 10Analytics, 10Operations, 10Security-Team, 10WMF-Legal, 10Software-Licensing: Can exfat be used in WMF production? - https://phabricator.wikimedia.org/T210667 (10Joe) >>! In T210667#4783582, @Legoktm wrote: >>>! In T210667#4783289, @MoritzMuehlenhoff wrote: >> exfat-fuse itself is free software (GPL) an... [08:20:27] This - https://www.youtube.com/watch?v=WpIAc9by5iU - because that - https://ich.unesco.org/en/RL/reggae-music-of-jamaica-01398 [08:37:45] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Services (watching): Modern Event Platform: Stream Intake Service: Implementation - https://phabricator.wikimedia.org/T206785 (10mobrovac) >>! In T206785#4783544, @Nuria wrote: > "EventGate" as in : "the place where events enter our galaxy"... Event[Bla... [08:45:11] 10Analytics, 10Analytics-Kanban, 10DBA: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10elukey) [08:49:13] 10Analytics, 10SDC General, 10Wikimedia-Stream: Verify that EventStreams work with WikiBase MediaInfo - https://phabricator.wikimedia.org/T210702 (10Abbe98) [08:57:50] 10Analytics, 10User-Elukey: Move turnilo to nodejs 10 - https://phabricator.wikimedia.org/T210705 (10elukey) p:05Triage>03Normal [08:59:00] 10Analytics: Move AQS to nodejs 10 - https://phabricator.wikimedia.org/T210706 (10elukey) p:05Triage>03Normal [08:59:10] there you go --^ :) [09:03:11] * elukey still running errands, should be working properly in ~30 mins [09:19:36] (03PS4) 10Michael Große: Update metric's items and properties automatically [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/475807 (https://phabricator.wikimedia.org/T209399) [09:35:05] (03CR) 10Michael Große: Update metric's items and properties automatically (032 comments) [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/475807 (https://phabricator.wikimedia.org/T209399) (owner: 10Michael Große) [09:39:00] I am fully working now :) [11:10:36] hellooo [11:10:43] joal hiiiii [11:24:15] Hi mforns :) [11:24:19] heloooo [11:24:30] mforns: ready to deploy? [11:24:46] yep, can we merge this patch first? https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/476290/ [11:25:44] joal, and if you're not against it, I will self-merge this one, because I have 2 +1's: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/471279/ [11:26:04] please mforns :) [11:26:07] ok [11:26:32] (03CR) 10Mforns: [V: 032 C: 032] "Got 2 +1's, merging!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/471279 (https://phabricator.wikimedia.org/T199836) (owner: 10Mforns) [11:26:50] mforns: only concern: all crons and timers for deletion need to be patched, right? [11:27:07] joal, there's some refinery changes that have a +1, do we want to look and merge them if OK? [11:27:32] mforns: I deploy what you wish :) [11:27:45] joal, there's no crons or timers for this deletion script yet, we can write one when that is deployed [11:28:04] Ah right, it's as new script, not a replacement- my bad :) [11:28:16] joal, they are not mine, just asking if we should consider including them [11:28:17] :] [11:28:54] 2 of them are yours: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/472506/ [11:28:59] https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/471722/ [11:29:39] the 471722 is not to be merged as of now, still work to be done [11:29:39] should I merge them? I already gave them a +1 [11:30:00] actually, you got already a +1 from Nuria as well [11:30:15] oh, snap rebase error [11:30:17] And the 472506 should be abandonned [11:30:29] So nothing to do for those 2 mforns :) [11:30:43] ok ok [11:31:18] (03Abandoned) 10Joal: Prevent comment loss on sqooped logging table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/472506 (owner: 10Joal) [11:31:44] ok then, the only thing that I'd like to include is: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/476290/ [11:31:51] can you give please a quick look? [11:31:55] it's quite short [11:32:06] (03CR) 10Joal: [C: 032] "LGTM - Merging!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476290 (https://phabricator.wikimedia.org/T202429) (owner: 10Mforns) [11:32:16] thx! [11:32:25] great, I think we can start [11:32:47] Let's wait for jenkins to merge first [11:32:50] ok [11:32:59] I also have another patch to merge in refinery [11:33:59] oh [11:34:49] this one mforns: https://gerrit.wikimedia.org/r/c/analytics/refinery/+/473052 [11:35:47] (03CR) 10Mforns: [V: 032 C: 032] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/473052 (https://phabricator.wikimedia.org/T209179) (owner: 10Joal) [11:37:13] (03Merged) 10jenkins-bot: Upgrade profig lib to 2.3.3 after bug fix upstream [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476290 (https://phabricator.wikimedia.org/T202429) (owner: 10Mforns) [11:37:21] cool [11:42:38] ok, starting the dpeloy with a jenkins job [11:42:50] joal, I'm editing the changelog [11:42:53] is that ok [11:42:54] ? [11:42:56] oh yes !! [11:43:08] I would have gone for that while reading the docs :) [11:43:18] thanks for doing it mforns :) [11:43:32] the new version is 81 right? [11:43:41] the changelog already has bullets for that version [11:47:29] mforns: Andrew usually adds stuff as he merges [11:47:41] aha, yes but uses the snapshot name [11:47:52] the new snapshot indeed [11:48:33] (03PS1) 10Mforns: Update changelog.md for v0.0.81 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476481 [11:48:55] (03CR) 10Mforns: [V: 032 C: 032] "Deploying" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476481 (owner: 10Mforns) [11:49:08] ok, changelog updated [11:51:51] Hallo. [11:52:12] joal, one question, when deploying I always run https://integration.wikimedia.org/ci/job/analytics-refinery-release/build before running https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/ to update the default values in the form. Is that necessary, or can I just retype the correct values in the second form, and thus save me the time of the first build? [11:52:17] hey aharoni :] [11:53:22] mforns: Thanks for the explanation about RU :) You had basically said the same thing the other day, but only now I actually understood it. Now the really fun stuff begins: Superset board. [11:54:06] aharoni, yep, do you have access to Superset? [11:54:13] I think so, yeah, let me see. [11:55:06] Yes, I can open superset.wikimedia.org [11:56:20] ok, I'm trying to add a new database source 'analytics-slave' [11:57:00] mforns: I do the same as you do - I find it better to let the system automatically generate its values [11:57:09] mforns: but theoretically you could type them in [11:57:28] joal, oh, and how do you let the system do it by itself? [11:57:53] mforns: same as you do - run a jenkins-build before the jenkins-release [11:58:10] joal, oh! I was confused, thought you were speaking about superset [11:58:11] ok ok [11:58:21] sorry mforns :) [11:58:36] mforns: Thanks for deploying :) [11:59:30] joal, np running build [12:00:16] joal, do you know how to create a database source in Superset with credentials? [12:06:31] aharoni, do you want to meet in Hangouts to build the dashboard? I managed to query your table from Superset [12:06:51] Yes! [12:06:58] Right now is good [12:08:43] mforns: looks like you've figured it out? [12:08:52] mforns: or the database was already there? [12:10:07] joal, mforns - how does somebody on staff get access to Superset? I'm not sure that other members of my team have it. [12:10:16] Is there a page that explains it somewhere? [12:10:19] joal, the db was already there, it's called EventLogging database, but it's analytics-slave, so we can access staging [12:10:33] cool mforns : [12:11:26] aharoni: I think it's explained somewhere, but I can't recall where - What you need to have for access is: an LDAP account and username, and a ping in here for us to create the user in superset (it's not automatic yet) [12:12:14] aharoni, I'm in https://meet.google.com/mdj-iscd-dhe [12:12:24] coming [12:38:53] PROBLEM - https://grafana.wikimedia.org/dashboard/db/eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/eventbus is alerting: EventBus POST Response Status alert. [12:42:25] RECOVERY - https://grafana.wikimedia.org/dashboard/db/eventbus grafana alert on icinga1001 is OK: OK: https://grafana.wikimedia.org/dashboard/db/eventbus is not alerting. [12:42:58] kinda weird this one, IIRC andrew added it yesterday [12:44:13] (03Abandoned) 10MarcoAurelio: Bump dependencies in requirements.txt to their latest versions [analytics/wikimetrics] - 10https://gerrit.wikimedia.org/r/475142 (https://phabricator.wikimedia.org/T209945) (owner: 10MarcoAurelio) [12:44:52] * elukey lunch! [12:47:09] PROBLEM - https://grafana.wikimedia.org/dashboard/db/eventbus grafana alert on icinga1001 is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/eventbus is alerting: EventBus POST Response Status alert. [12:48:19] RECOVERY - https://grafana.wikimedia.org/dashboard/db/eventbus grafana alert on icinga1001 is OK: OK: https://grafana.wikimedia.org/dashboard/db/eventbus is not alerting. [12:55:06] mforns: no 0.0.81 version in archiva - is that expected? [12:55:47] (03PS1) 10Mforns: Correct date type in cx query [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/476493 (https://phabricator.wikimedia.org/T189475) [12:57:11] joal, the jenkins job failed! :[ [12:57:13] looking [12:57:19] Mwarf :( [12:58:28] joal, Failed to deploy artifacts: Could not transfer artifact org.wikimedia.analytics.refinery:refinery:pom:0.0.81 from/to archiva.releases (https://archiva.wikimedia.org/repository/releases/): Failed to transfer file: https://archiva.wikimedia.org/repository/releases/org/wikimedia/analytics/refinery/refinery/0.0.81/refinery-0.0.81.pom. Return code is: 401, ReasonPhrase:Unauthorized. -> [Help 1] [12:58:49] (03CR) 10Amire80: [C: 032] Correct date type in cx query [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/476493 (https://phabricator.wikimedia.org/T189475) (owner: 10Mforns) [12:58:55] (03Merged) 10jenkins-bot: Correct date type in cx query [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/476493 (https://phabricator.wikimedia.org/T189475) (owner: 10Mforns) [12:59:06] * joal cries in a round and waits for elukey [13:31:00] ah snap sorry didn't see it before [13:31:44] np elukey [13:32:01] elukey: I think this is something we have already seen - is that true? [13:32:17] personally I haven't, kinda strange [13:32:51] hm I don't know why but it rings a bell - Maybe it was Andrew resolving it last time [13:33:24] are there more logs to check? [13:35:10] the only thing that changed recently is the TLS certificate, but nothing more [13:37:30] elukey: I think it might be related to a change in jenkins-infra [13:37:38] mforns: do you have more logs about the failure? Where did you get it? [13:38:16] elukey, https://integration.wikimedia.org/ci/job/analytics-refinery-release/156/console [13:39:17] ah yes thanks I was reading it as well :) [13:47:23] so I do se [13:47:29] 127.0.0.1 - - [29/Nov/2018:12:16:47 +0000] "PUT /repository/releases/org/wikimedia/analytics/refinery/refinery/0.0.81/refinery-0.0.81.pom HTTP/1.0" 401 0 "-" "Apache-Maven/3.0.4 (Java 1.8.0_181; Linux 4.9.0-0.bpo.8-amd64)" [13:48:53] mforns,joal - is it possible to retry and see if it is transient? [13:49:02] just to rule out this possibility while investigating [13:49:22] elukey: it surely is possible :) [13:49:30] mforns: you do, I do ? [13:50:51] joal, elukey, I can do [13:50:59] super [13:52:20] running [13:55:28] mforns: interestingly, us not having deployed yet will help on something I have found I think [14:00:51] * elukey in meetings but will read [14:01:11] job failed again, but because of tags [14:01:26] we should delete the v0.0.81 tag in repo [14:02:30] oh... [14:03:16] doing [14:05:55] joal, done [14:06:03] will try again [14:06:09] Thanks mforns [14:06:30] running [14:07:11] mmmmm, uou [14:07:14] failed [14:07:32] mforns: probably because of commits already existing I assume [14:08:54] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:2.5.1:prepare (default-cli) on project refinery: You don't have a SNAPSHOT project in the reactor projects list. -> [Help 1] [14:09:02] right [14:09:06] pushin a patch mforns [14:09:08] Wo, I don't know how to proceed now... [14:09:10] ok [14:16:09] mforns: fighting will revert - I'll get there [14:24:37] (03PS1) 10Joal: Revert "[maven-release-plugin] prepare release v0.0.81" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476513 [14:24:41] ok mforns - I think I have this right now [14:25:01] k [14:25:06] should I rerun? [14:25:13] with same parameters joal? [14:25:18] mforns: can you wait for my next patch please ? [14:25:25] should I remove the tag? [14:25:26] oh ok [14:26:26] (03PS1) 10Joal: Bump spark-avro dependency to 4.0.0 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476515 [14:26:31] mforns: removing the tag should be done yes [14:26:51] mforns: Do you mind +2 the 2 above? I can do it if you prefer [14:27:45] joal, it says remote tag v0.0.81 does not exist [14:28:06] mforns: It probably didn't even try to create it when failing because of pom format [14:28:17] mforns: Let's merge and retry [14:28:24] (03CR) 10Mforns: [V: 032 C: 032] "LGTM" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476515 (owner: 10Joal) [14:28:52] ok [14:29:35] mforns: you're missing one the 2 patches I think :0 [14:29:43] (03CR) 10Mforns: [V: 032 C: 032] "lgtm" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476513 (owner: 10Joal) [14:29:47] Ah ! [14:29:50] here it comes :) [14:29:59] hehe [14:30:21] ok, we should be good to try again the jenkins [14:30:28] ok, so now it's just running the build right_ [14:30:33] ? [14:30:42] yup [14:31:28] ok runnin [14:33:00] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Contributors-Analysis, 10Product-Analytics: Hive join fails when using a HiveServer2 client - https://phabricator.wikimedia.org/T206279 (10JAllemandou) This should be solved by https://gerrit.wikimedia.org/r/c/operations/puppet/cdh/+/474907 [14:33:26] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Contributors-Analysis, 10Product-Analytics: Hive join fails when using a HiveServer2 client - https://phabricator.wikimedia.org/T206279 (10JAllemandou) a:03elukey [14:37:06] joal, elukey, same error as originally [14:40:04] joal / ottomata: I vaguely remember you two looked at Kafka Streams, I've been reading a lot about it since yesterday, I'm curious to chat about it [14:41:42] ya we played with it some [14:41:50] milimetric: it can be used from the jar on our kafka clusters now if you want to play [14:41:55] 10Analytics, 10Analytics-Kanban, 10DBA, 10Data-Services, 10Core Platform Team Backlog (Watching / External): Create materialized views for performance - https://phabricator.wikimedia.org/T210693 (10Banyek) @bstorm I can assist if you need it, just talk to me [14:42:00] it makes a lot of extra topics, but they don't hurt and we can clean them up later [14:43:09] ottomata: ok, good to know. I'm mostly curious to see how the local storage of KTables works. It says it uses RocksDB by default, so if I started up a Java app on stat1007 wouldn't I need RocksDB installed? [14:43:20] i think its internal [14:43:34] ok, I'll do it and see what happens :) [14:43:35] it comes with the jar, and rocksdb uses memory/local file, kinda like sqllite [14:43:42] yeah [14:43:47] that can't possibly scale :) [14:43:49] milimetric: if you are into this stuff, check out more flink stuff too [14:43:58] i'm more impressed by it all the time [14:44:09] it has similar concepts [14:44:16] well, but flink requires all separate configuration and infrastructure and all that [14:44:26] meh not really, not for local mode stuff [14:44:29] if this scales to what we'd need, it would be amazing, no? [14:44:32] or even for running in yarn [14:45:04] kafka streams is cool, but it is a just a library, so it requires that you figure out how to deploy and scale it yourself [14:45:12] you have to write an application [14:45:29] for KSQL, you have to deploy the KSQL service just like you would flink or spark stuff [14:45:35] mmm, not quite, it does most of the hard part of scaling for you [14:45:52] milimetric: kafka streams won't launch new processes [14:45:54] 10Analytics, 10Analytics-Wikistats: Two bugs to WiViVi visualization tool - https://phabricator.wikimedia.org/T210731 (10ezachte) [14:45:54] you define application ids, and it uses consumer groups to scale across clusters of apps with the same application id [14:46:00] right [14:46:03] ottomata: heya - we have an issue with jenkins and archiva :( [14:46:06] oh ya? [14:46:13] 10Analytics, 10Analytics-Wikistats: Two bugs to WiViVi visualization tool - https://phabricator.wikimedia.org/T210731 (10ezachte) 05Open>03Resolved Both bugs fixed. [14:46:16] ottomata: IIRC you've already seen that [14:46:18] milimetric: ya let's play more, you know i love this stuff [14:46:22] for sure [14:46:31] joal: maybe? sounds familiar remind meeeee [14:47:33] ottomata: jenkins doesn't manage tu push jars to archiva, forbidden [14:48:00] also ottomata / milimetric - You might be interested to look into Beam [14:48:15] It seems to get more and more traction [14:48:15] I did look at Beam when you sent that article way back [14:48:24] jenkins is forbidden? [14:48:29] 10Analytics, 10Analytics-Kanban, 10DBA, 10Data-Services, 10Core Platform Team Backlog (Watching / External): Create materialized views for performance - https://phabricator.wikimedia.org/T210693 (10jcrespo) So you don't have to reuse this- but I am pasting this for reference: https://gerrit.wikimedia.or... [14:48:31] ottomata: seems so [14:48:32] beam + flink oh my :) [14:48:44] joal hm [14:48:46] joal: I realized something about the join -> compat job I'm building [14:49:17] Indeed ottomata - Actually Beam + Flink for streaming, Beam + Spark for batch, with almost no change in code if precessing is to be the same :D [14:49:23] yes milimetric ? [14:49:27] joal: we could skip it completely and just do the join when we select data for page/user/denormalize [14:49:33] I actually have question about the sqoop patch milimetric [14:49:38] joal: bc? [14:49:41] sure [14:50:14] joal: why not beam + flink for batch :p :) [14:51:09] joal so this does sound familiar [14:51:14] last time we released we did it via CLI? [14:51:16] is that correct? [14:52:39] ottomata: I don't think so [14:53:05] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Services (watching): Modern Event Platform: Stream Intake Service: Implementation - https://phabricator.wikimedia.org/T206785 (10Ottomata) EventGate is pretty good. It's fairly descriptive: some events are let in, others aren't. HMMM [14:53:42] mforns: do you remember? [14:54:04] joal: how can I look to see what's happening? do you have link to jenkins job that failed? [14:54:31] joal, ottomata, in Jenkins you can see that prior to run #153 there was a barrage of errored runs as well... [14:54:36] I don't remember no... [14:54:54] ottomata, https://integration.wikimedia.org/ci/job/analytics-refinery-release/ [14:56:28] #155 was done today using https://integration.wikimedia.org/ci/job/analytics-refinery-release/build, this one worked (just build) [14:56:49] #156 failed [14:57:17] the subsequent errors were us trying stuff [14:57:49] #160 is the same error as original #156 [14:58:31] (back) [14:58:57] 10Analytics, 10Analytics-Kanban, 10DBA: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Banyek) a:03Banyek [14:59:30] 10Analytics, 10Analytics-Kanban, 10DBA, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Banyek) [14:59:43] previous errors mention problems with the "ForkedBooter" though, which seems different from what is happening now. [15:02:01] 10Analytics, 10Analytics-Wikistats, 10Internet-Archive: Feedback on Wikistats 2 new edits pages - https://phabricator.wikimedia.org/T210306 (10ezachte) group-bots vs name-bots, Ah I see, makes sense to makes this distinction, at least internally. Not sure if it's relevant to the end-user. AFAIK using the str... [15:03:27] mforns: how do you see that? [15:03:40] i ust see the 401 [15:06:52] sigh i dunno dooooods kinda want to work on other stuff. can we try CLI deploy and make a phab bug for this? [15:09:18] trying to figure out where the jenkins job config stuff is, don't know this stuff too wellllll :D [15:10:01] ottomata, I see the ForkedBooter errors in old runs of https://integration.wikimedia.org/ci/job/analytics-refinery-release/ [15:10:14] But that's the old (already solved) problem [15:10:19] oh [15:10:25] the new problem is the 401 permission denied [15:10:37] which I see in runs #156 and #160 [15:11:17] but yes, I can totally try to deploy via command line [15:12:58] yeah [15:13:12] i can't examine the jenkins stuff either...seems i don't have credentials?? [15:14:10] i just pinged in releng [15:14:14] i think i can't help much atm :/ [15:14:28] mforns: if you like i can try to do CLI release [15:14:50] https://github.com/wikimedia/analytics-refinery-source#releases [15:15:03] no prob, there's documentation, if I have trouble, I'll ping you! [15:27:29] (03PS1) 10Fdans: Refactor test runner and fix tests stalling [analytics/aqs] - 10https://gerrit.wikimedia.org/r/476528 (https://phabricator.wikimedia.org/T209711) [15:29:33] (03PS2) 10Fdans: Refactor test runner and fix tests stalling [analytics/aqs] - 10https://gerrit.wikimedia.org/r/476528 (https://phabricator.wikimedia.org/T209711) [15:38:25] 10Analytics, 10Services, 10Wikimedia-Stream: EventStreams process occasionally OOMs - https://phabricator.wikimedia.org/T210741 (10Ottomata) [15:40:17] joal, luca found the problem with jenkins upload [15:40:34] the password for archiva-ci had expired, and we need a new one [15:40:40] we're fixing it right now [15:40:51] (by we I mean Luca :D) [15:41:47] 10Analytics, 10Analytics-Kanban, 10DBA, 10Data-Services, 10Core Platform Team Backlog (Watching / External): Create materialized views for performance - https://phabricator.wikimedia.org/T210693 (10Bstorm) [15:46:50] great! [15:48:43] but I don't have perm to update the password in jenkins :D [15:48:52] so until somebody helps us, we are stuck [15:59:36] 10Analytics, 10Services, 10Wikimedia-Stream: EventStreams process occasionally OOMs - https://phabricator.wikimedia.org/T210741 (10Ottomata) So currently, KafkaSSE doing several copies of data for each message: - Buffer from node-rdkafka - String from Buffer .toString - Object from JSON.parse String - Stri... [16:10:52] joal: I made a mistake in the sqoop change, didn't realize something subtle and useful about the labs views: they already join with the temp tables and create revision the way it will be at some point [16:11:08] so I'll update the sqoop select, but didn't want you to depend on that change just yet [16:11:26] this means we don't need the temp tables at all [16:39:56] 10Analytics, 10Cloud-VPS, 10DBA: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Nuria) [16:43:22] (03CR) 10Nuria: [C: 031] "I have no objection of this being merged as is. Now, what is our plan for backfilling this data? It seems that the way the jobs are setup " [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476220 (https://phabricator.wikimedia.org/T167539) (owner: 10Fdans) [16:49:25] (03PS3) 10Nuria: Refactor test runner and fix tests stalling [analytics/aqs] - 10https://gerrit.wikimedia.org/r/476528 (https://phabricator.wikimedia.org/T209711) (owner: 10Fdans) [16:51:55] (03CR) 10Nuria: "Thanks for doing this. I corrected what I think is a typo that was preventing test from running and they run clean now on my machine." [analytics/aqs] - 10https://gerrit.wikimedia.org/r/476528 (https://phabricator.wikimedia.org/T209711) (owner: 10Fdans) [16:52:05] (03CR) 10Fdans: Refactor test runner and fix tests stalling (031 comment) [analytics/aqs] - 10https://gerrit.wikimedia.org/r/476528 (https://phabricator.wikimedia.org/T209711) (owner: 10Fdans) [16:53:39] mforns,joal the new archiva-ci pass has been configured by releng [16:53:46] \o/ [16:53:48] jenkins should now work [16:53:59] ok, I'm reverting pom changes [16:55:26] (03PS4) 10Fdans: Refactor test runner and fix tests stalling [analytics/aqs] - 10https://gerrit.wikimedia.org/r/476528 (https://phabricator.wikimedia.org/T209711) [16:55:50] (03PS1) 10Mforns: Revert pom changes done by analytics-refinery-release [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476550 [16:56:42] (03CR) 10Mforns: [V: 032 C: 032] "Self merging to unbreak deployment" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476550 (owner: 10Mforns) [16:57:00] (03CR) 10Fdans: "@Nuria yeah the jobs are tiny so my thought was there would be no problem with re-backfilling" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476220 (https://phabricator.wikimedia.org/T167539) (owner: 10Fdans) [16:57:18] 10Analytics, 10Cloud-VPS, 10DBA: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Banyek) as far as I know we don't have any spare hardware, but I am curious too [16:57:34] 10Analytics, 10Analytics-Kanban: Update refinery-source jobs to join labsdb with actor and comment - https://phabricator.wikimedia.org/T210543 (10Milimetric) [16:59:08] (03PS1) 10Milimetric: Join to new actor and comment tables [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476553 (https://phabricator.wikimedia.org/T210543) [17:00:22] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Set up CI system on AQS - https://phabricator.wikimedia.org/T209711 (10fdans) Helloo @hashar do you think you can hook aqs to Jenkins for CI? We just need to run `npm test`. [17:00:24] (03PS2) 10Milimetric: Update sqoop selects for new mediawiki schema [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476100 (https://phabricator.wikimedia.org/T210541) [17:01:58] milimetric: let's discuss sqoop after standup, I think I don't get the _temp join [17:02:15] 10Analytics, 10Analytics-Kanban, 10DBA, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Banyek) I'll start tomorrow checking / documenting information about the dbstore1002 - I am curious if we make those similar to labsdb1009-1... [17:04:04] (03CR) 10jerkins-bot: [V: 04-1] Join to new actor and comment tables [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476553 (https://phabricator.wikimedia.org/T210543) (owner: 10Milimetric) [17:04:10] 10Analytics, 10Cloud-VPS, 10DBA: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Nuria) We have some hardware orders in progress for db-like hosts, we might be able use those if nothing else is available. [17:07:55] 10Analytics, 10Analytics-Kanban, 10DBA, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10elukey) As far as I know we have to go multi-instance, but I don't have a lot of context if multi-source can or is needed anymore (I guess n... [17:14:48] 10Analytics, 10Cloud-VPS, 10DBA: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Marostegui) We don't have any spare host similar to the labsdb ones. Those are very specific hardware, as they need to contain all the wikis on the same host, they have lots of... [17:15:18] (03PS2) 10Milimetric: Join to new actor and comment tables [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/476553 (https://phabricator.wikimedia.org/T210543) [17:37:08] !log Deployed refinery-source using jenkins [17:37:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:41:56] !log Starting refinery deployment using scap and refinery-deploy-to-hdfs [17:41:59] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:49:22] 10Analytics: Merge metadata from filtered-tables.txt and maintain-views.yaml - https://phabricator.wikimedia.org/T210462 (10Milimetric) p:05Triage>03High [17:49:28] 10Analytics: Merge metadata from filtered-tables.txt and maintain-views.yaml - https://phabricator.wikimedia.org/T210462 (10Milimetric) p:05High>03Normal [17:50:34] 10Analytics, 10EventBus, 10Services (watching): Explore adding validation alarms to eventbus using logstash - https://phabricator.wikimedia.org/T210459 (10Milimetric) p:05Triage>03High [17:53:15] 10Analytics, 10Analytics-Kanban: Link to User Contribution page in wikistats UI rather than user page - https://phabricator.wikimedia.org/T210422 (10Milimetric) p:05Triage>03High a:03Milimetric [17:55:03] 10Analytics, 10Tool-Pageviews: Statistics for views of individual Wikimedia Commons images - https://phabricator.wikimedia.org/T210313 (10Milimetric) Will merge others into this, but keep in mind this nice analysis about the storage in Cassandra implications: T88775#4751882 [17:55:14] 10Analytics, 10Multimedia: Add mediacounts to pageview API - https://phabricator.wikimedia.org/T88775 (10Milimetric) [17:55:17] 10Analytics, 10Tool-Pageviews: Statistics for views of individual Wikimedia Commons images - https://phabricator.wikimedia.org/T210313 (10Milimetric) [17:55:37] 10Analytics, 10Tool-Pageviews: Statistics for views of individual Wikimedia Commons images - https://phabricator.wikimedia.org/T210313 (10Milimetric) p:05Triage>03High [17:56:38] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: [EventLogging Sanitization] Fix passing of input_path_regex params to Refine - https://phabricator.wikimedia.org/T210110 (10Milimetric) p:05Triage>03High [18:04:16] 10Analytics, 10Tool-Pageviews: Statistics for views of individual Wikimedia Commons images - https://phabricator.wikimedia.org/T210313 (10jmatazzoni) Thanks for moving this to High priority @Milimetric. I see the title refers to "images" and "Commons." I'd like to ask: # I assume that instead of images what... [18:05:14] 10Analytics: Refine Monitor should be a systemd timer such if process cannot start we get notified - https://phabricator.wikimedia.org/T210759 (10Nuria) [18:07:27] 10Analytics, 10Analytics-Kanban: druid ingestion should calculate 1/sample rate to be able to normalize event counts - https://phabricator.wikimedia.org/T210099 (10Milimetric) p:05Triage>03High [18:08:12] 10Analytics, 10Product-Analytics: Event counts from Mysql and Hive don't match. Hive is persisting data from crawlers. - https://phabricator.wikimedia.org/T210006 (10Milimetric) p:05Triage>03High [18:08:25] 10Analytics, 10Product-Analytics: Event counts from Mysql and Hive don't match. Hive is persisting data from crawlers. - https://phabricator.wikimedia.org/T210006 (10Milimetric) p:05High>03Normal [18:09:05] 10Analytics, 10Product-Analytics: Event counts from Mysql and Hive don't match. Refine is persisting data from crawlers. - https://phabricator.wikimedia.org/T210006 (10Milimetric) [18:09:52] 10Analytics, 10Cloud-VPS, 10DBA: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10elukey) >>! In T210749#4785874, @Marostegui wrote: > We don't have any spare host similar to the labsdb ones. > Those are very specific hardware, as they need to contain all the... [18:12:48] 10Analytics, 10Cloud-VPS, 10DBA, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10elukey) [18:12:51] 10Analytics: Alert and halt mediawiki processing on schema changes - https://phabricator.wikimedia.org/T209888 (10Milimetric) p:05Triage>03Normal [18:14:00] 10Analytics, 10Analytics-Kanban: Add new wikis to analytics - https://phabricator.wikimedia.org/T209822 (10Milimetric) p:05Triage>03High [18:14:16] 10Analytics, 10Cloud-VPS, 10DBA, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Marostegui) That is up to you guys to decide :-) For context: 1 single host can have all the wikis - we have 3 in labs for redundancy and because we separate the... [18:20:05] 10Analytics, 10Cloud-VPS, 10DBA, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Nuria) [18:24:45] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10Dzahn) [18:24:49] 10Analytics, 10Cloud-VPS, 10DBA, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Nuria) @Marostegui Let's have in mind these boxes are going to be used to scoop data once a month so they likely not need to be as beefy as boxes used every day... [18:29:23] 10Analytics, 10Cloud-VPS, 10DBA, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Marostegui) >>! In T210749#4786204, @Nuria wrote: > @Marostegui Let's have in mind these boxes are going to be used to scoop data once a month so they likely not... [18:29:47] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10Dzahn) what it would mean: all members of "researchers": ` members: [a... [18:30:27] 10Analytics, 10Services, 10Wikimedia-Stream: EventStreams process occasionally OOMs - https://phabricator.wikimedia.org/T210741 (10Milimetric) p:05Triage>03High a:03Ottomata [18:30:50] 10Analytics: Move AQS to nodejs 10 - https://phabricator.wikimedia.org/T210706 (10Milimetric) p:05Normal>03High [18:30:52] 10Analytics, 10User-Elukey: Move turnilo to nodejs 10 - https://phabricator.wikimedia.org/T210705 (10Milimetric) p:05Normal>03High [18:32:22] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10Dzahn) The alternative is creating an entirely new group with a better name... [18:33:27] mforns: Thanks a lot for the deploy :) [18:33:32] np! [18:33:37] oh, BTW [18:33:46] !log Finished refinery deployment using scap and refinery-deploy-to-hdfs [18:33:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:33:51] fdans: how dare you? :D (just read the chat) [18:33:52] mforns: I'll care restarting/updating etc the job that need it [18:34:03] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10Dzahn) [18:34:03] ok, thank you! :] [18:34:27] * elukey off! [18:50:07] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10Nuria) What are recommendation api dumps? If they are destined for productio... [19:03:46] (03CR) 10Nuria: [C: 04-1] Refactor test runner and fix tests stalling (031 comment) [analytics/aqs] - 10https://gerrit.wikimedia.org/r/476528 (https://phabricator.wikimedia.org/T209711) (owner: 10Fdans) [19:04:00] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10bmansurov) @Nuria we are using Spark, Wikidata dumps in Hadoop, and some Hiv... [19:05:01] nuria: sorryyyy yes i undid your change [19:05:33] * fdans that wouldn’t happen in a github PR though 😇 [19:15:41] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Services (watching): Modern Event Platform: Stream Intake Service: Implementation - https://phabricator.wikimedia.org/T206785 (10Ottomata) I'm pretty seriously considering EventGate. (A downside is the connotation that words suffixed with -gate are scan... [19:16:14] ok cool, so deploy has happened mforns? going ot make a patch to make refine use new jar... [19:16:18] first will test one more time [19:16:47] joal: the banner activity dataset in turnilo does not have the 1/sample rate calculation right? [19:38:10] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Services (watching): Modern Event Platform: Stream Intake Service: Implementation - https://phabricator.wikimedia.org/T206785 (10CCicalese_WMF) I would argue that we want **all** of our code to be easily findable and contributable by others. I believe th... [19:50:04] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Services (watching): Modern Event Platform: Stream Intake Service: Implementation - https://phabricator.wikimedia.org/T206785 (10Ottomata) Yeah, I in general agree with that, especially when code is WMF specific. However, I'm hoping that this service/li... [19:53:42] (03PS1) 10Ottomata: Add bin/yarn-logs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476608 [19:55:17] (03CR) 10Ottomata: [V: 032 C: 032] Add bin/yarn-logs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476608 (owner: 10Ottomata) [19:55:37] a-team check it out: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/476608/1/bin/yarn-logs [19:55:48] i got tired of so much verbose output and always typing -applicationId [19:55:51] now you can [19:55:59] yarn-logs [19:56:15] it will give you the yarn app logs, as well as by default filter a bunch of noise [19:56:31] the noise filter I configured was just for spark jobs, plz add more for hive, oozie, etc. as you encounter :) [19:56:33] ooh, nice! [19:56:56] (03PS1) 10Ottomata: Remove bad char in bin/yarn-logs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476609 [19:57:14] (03CR) 10Ottomata: [V: 032 C: 032] Remove bad char in bin/yarn-logs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476609 (owner: 10Ottomata) [19:58:38] Nice ottomata :) I have somthing similar, less evolved, as a bash alias :) [20:00:48] :) yea i've had a few lying around too, but i was going to make a good one just for me...but then decided to share :) [20:02:06] Thanks for that ottomata :) [20:12:33] cool! [20:51:59] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Allow hadoop prod jobs to preempt resource over default queue - https://phabricator.wikimedia.org/T208208 (10Ottomata) [20:54:05] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install cloudvirtan100[1-5].eqiad.wmnet - https://phabricator.wikimedia.org/T207194 (10Ottomata) @Cmjohnson I don't know what the proper disk layout for these are, since they will be Cloud Virt nodes. I doubt RAID 0 is... [20:57:24] 10Analytics, 10Analytics-Kanban, 10DBA, 10Data-Services, 10Core Platform Team Backlog (Watching / External): Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10bd808) [21:00:16] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Core Platform Team Backlog (Later), 10Services (later): Make schemas use required $schema property with absolute path (not absolute URL) to the schema - https://phabricator.wikimedia.org/T208361 (10Ottomata) > One thing that I don't understand i... [21:00:38] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Backlog (Later), 10Services (later): Make schemas use required $schema property with absolute path (not absolute URL) to the schema - https://phabricator.wikimedia.org/T208361 (10Ottomata) [21:00:56] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: is-yarn-app-running script should output the running application id - https://phabricator.wikimedia.org/T206555 (10Ottomata) [21:01:51] 10Analytics, 10Operations, 10Security-Team, 10WMF-Legal, 10Software-Licensing: Can exfat be used in WMF production? - https://phabricator.wikimedia.org/T210667 (10Jrogers-WMF) Hi all, commenting on this from WMF Legal. As I understand the question and context, the issue is using a proprietary format fo... [21:06:50] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install cloudvirtan100[1-5].eqiad.wmnet - https://phabricator.wikimedia.org/T207194 (10Andrew) >>! In T207194#4787008, @Ottomata wrote: > @Cmjohnson I don't know what the proper disk layout for these are, since they wil... [21:08:44] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install cloudvirtan100[1-5].eqiad.wmnet - https://phabricator.wikimedia.org/T207194 (10Ottomata) Hmm, ok, then I think in this case RAID 0 is fine. Since these will have Hadoop, data will be replicated across nodes 3x... [21:11:46] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install cloudvirtan100[1-5].eqiad.wmnet - https://phabricator.wikimedia.org/T207194 (10Andrew) As I understand it, with raid 0 if a single drive dies the whole system (and containing VM) will have to be rebuilt. Also I... [21:17:57] wow this is cool! [21:17:57] https://meta.wikimedia.org/wiki/Wikimedia_Commons_Data_Roundtripping [21:18:03] they are gonna try to use eventstreamas for this [21:22:34] 10Analytics, 10SDC General, 10Wikimedia-Stream: Verify that EventStreams work with WikiBase MediaInfo - https://phabricator.wikimedia.org/T210702 (10Ottomata) Cool! EventStreams is deployed in beta, however, real production data does not go there. If your code that generates events is deployed to beta, you... [22:00:04] that commons roundtripping project looks great, but seems like an opportunity for much cooler solutions that would make this available to anyone hosting content on the wikis [22:26:10] 10Analytics, 10Tool-Pageviews: Statistics for views of individual Wikimedia Commons images - https://phabricator.wikimedia.org/T210313 (10Milimetric) > # I assume that instead of images what we really mean here is "files." E.g., presumably this will also give us a count of pageviews (not plays) to video or a... [23:06:24] (03PS3) 10Milimetric: Update sqoop selects for new mediawiki schema [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476100 (https://phabricator.wikimedia.org/T210541) [23:18:31] (03PS4) 10Milimetric: Update sqoop selects for new mediawiki schema [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476100 (https://phabricator.wikimedia.org/T210541) [23:21:29] (03CR) 10Nuria: [C: 031] Add bin/yarn-logs (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476608 (owner: 10Ottomata) [23:31:50] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10Nuria) @bmansurov We do not recommend to generate these in stats boxes, stat... [23:35:40] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10Nuria) @Dzahn let's please hold on on any changes, stats boxes are mean for... [23:39:11] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10Dzahn) @Nuria understood! thank you for your prompt comments and don't worry... [23:39:25] (03PS1) 10Ottomata: Remove other strange char from bin/yarn-logs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476776 [23:39:44] (03CR) 10Ottomata: [V: 032 C: 032] Remove other strange char from bin/yarn-logs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/476776 (owner: 10Ottomata) [23:39:53] thanks nuria :) [23:40:01] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10Dzahn) 05Open>03stalled [23:44:01] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Services (watching): Modern Event Platform: Stream Intake Service: Implementation - https://phabricator.wikimedia.org/T206785 (10Nuria) I tend to agree with Andrew here, this is very generic piece of code and that github issue sure makes a compeling case... [23:44:38] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10bmansurov) @Nuria OK, that makes sense. I'll work with #analytics on this. @... [23:45:54] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10Dzahn) @Nuria I also have this pending gerrit change that i will put on hold... [23:48:32] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10Nuria) @Dzahn I see, Let's abandon that change. Stats machines are used by... [23:51:06] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10Dzahn) 05stalled>03Invalid Ok Nuria! makes sense. I abandoned the change... [23:58:13] 10Analytics, 10Analytics-Cluster, 10Operations, 10SRE-Access-Requests: Add existing group "researchers" to "hosts that can produce recommendation API dumps" (role statistics::private) - https://phabricator.wikimedia.org/T210757 (10Nuria) Many thanks to everyone for the prompt responses.