[02:48:37] 10Analytics, 10ChangeProp, 10EventBus, 10Services (designing): Requests for new JobQueue monitoring capabilities - https://phabricator.wikimedia.org/T175780#3603041 (10Pchelolo) [03:35:32] 10Analytics, 10WMDE-Analytics-Engineering: dbstore1002 (analytics store) enwiki lag due to blocking query - https://phabricator.wikimedia.org/T175790#3603220 (10jcrespo) [04:04:04] (03CR) 10Shilad Sen: Add Clickstream builder spark job (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/361459 (https://phabricator.wikimedia.org/T158972) (owner: 10Joal) [04:19:19] 10Analytics, 10Research, 10WMDE-Analytics-Engineering: dbstore1002 (analytics store) enwiki lag due to blocking query - https://phabricator.wikimedia.org/T175790#3603278 (10jcrespo) Actually, I am not sure if it is that script, there is other one happening at 3am, too, that seems to block the queries: ``` 4... [04:48:20] (03PS1) 10Shilad Sen: Spark job to create session event log appears to be working. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/377706 (https://phabricator.wikimedia.org/T174796) [04:50:43] 10Analytics, 10Patch-For-Review: Productionize navigation vectors - https://phabricator.wikimedia.org/T174796#3603301 (10Shilad) [04:52:00] 10Analytics, 10Patch-For-Review: Productionize navigation vectors - https://phabricator.wikimedia.org/T174796#3573235 (10Shilad) [04:52:02] 10Analytics-Kanban, 10Research: productionize ClickStream dataset - https://phabricator.wikimedia.org/T158972#3603302 (10Shilad) [06:56:05] 10Analytics, 10ChangeProp, 10EventBus, 10Services (designing): Requests for new JobQueue monitoring capabilities - https://phabricator.wikimedia.org/T175780#3603338 (10Joe) This is very promising, I was in the process of writing down my own requirements and it seems most things are already covered, althoug... [07:22:36] 10Analytics, 10Research, 10WMDE-Analytics-Engineering: dbstore1002 (analytics store) enwiki lag due to blocking query - https://phabricator.wikimedia.org/T175790#3603390 (10Addshore) > as reads will get writes (from replication) I don't quite follow this, what exactly does this mean? > Actually, I am not... [07:22:43] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore: dbstore1002 (analytics store) enwiki lag due to blocking query - https://phabricator.wikimedia.org/T175790#3603391 (10Addshore) [07:39:21] Hi elukey - are you ok for me deploying refinery this morning? [07:40:45] sure! [07:40:48] morning :) [07:42:46] Good morning to you elukey :) [07:42:56] elukey: everything good on your side? [07:43:50] (03PS8) 10Joal: Add Clickstream builder spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/361459 (https://phabricator.wikimedia.org/T158972) [07:44:20] (03CR) 10Joal: "Thanks @Shilad :)" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/361459 (https://phabricator.wikimedia.org/T158972) (owner: 10Joal) [07:45:06] joal: yep! sunny day in bologna [07:45:18] awesome elukey :) [07:45:27] elukey: Will send my message to ops team this morning [07:46:05] elukey: nuria_ suggested to remove the paragraph on why using Druid for pageviews had not been done, I'll do [07:46:08] ok for you elukey ? [07:50:51] sure [08:00:03] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3603435 (10elukey) [08:02:30] elukey: While reviewing stuff to deploy, i noticed https://gerrit.wikimedia.org/r/#/c/361459/ [08:02:56] It's been +1 by ottomata, and data has been vetted (strict equality with the job it replicates) [08:03:05] elukey: I ask permission to merge :) [08:08:29] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3603443 (10elukey) >>! In T170878#3601557, @Nuria wrote: > @elukey: Let me know if you think this is something that will be completed... [08:29:52] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, 10User-mobrovac: Allow easy tuning of the jobqueue concurrency. - https://phabricator.wikimedia.org/T175800#3603486 (10Joe) [08:30:18] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 2 others: Allow easy tuning of the jobqueue concurrency. - https://phabricator.wikimedia.org/T175800#3603500 (10Joe) [09:03:25] elukey: ping --^ ? [09:04:08] wtf I didn't get the pings [09:04:10] sorry reading [09:04:17] no prob :) [09:05:07] I have nothing to object, mostly because I have no idea about what it does :D [09:05:16] elukey: :D [09:05:34] elukey: it creates a clickstream dataset (originally created by ellery) [09:05:52] sure sure :) [09:06:27] in the future (when the day will be extended to 48hrs) I'd really love to get my hands on these things [09:06:48] I feel that I don't know nothing about how apps are executed in hadoop [09:06:57] I only know how to kick it hard enough to make it work :D [09:09:12] huhuhu :) [09:09:24] elukey: I can give some basics on how some things work [09:10:31] I'd love that but it will require a ton of time that we both don't have :( [09:10:46] elukey: I really have no idea why you say that :D [09:11:08] ahahah [09:32:19] (03CR) 10Joal: [C: 032] Add Clickstream builder spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/361459 (https://phabricator.wikimedia.org/T158972) (owner: 10Joal) [09:33:01] (03CR) 10Joal: [V: 032 C: 032] "Self-merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/376235 (https://phabricator.wikimedia.org/T174915) (owner: 10Joal) [09:34:23] (03CR) 10Joal: [V: 032 C: 032] "Self merging for deploy." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/374987 (https://phabricator.wikimedia.org/T174484) (owner: 10Joal) [09:36:00] 10Analytics-Kanban: Add mediawiki-history metrics to AQS - https://phabricator.wikimedia.org/T175805#3603620 (10JAllemandou) [09:36:19] (03Merged) 10jenkins-bot: Add Clickstream builder spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/361459 (https://phabricator.wikimedia.org/T158972) (owner: 10Joal) [09:36:24] 10Analytics-Kanban: Add mediawiki-history metrics to AQS - https://phabricator.wikimedia.org/T175805#3603634 (10JAllemandou) [09:36:33] 10Analytics-Kanban: Add mediawiki-history metrics to AQS - https://phabricator.wikimedia.org/T175805#3603620 (10JAllemandou) a:03JAllemandou [09:43:18] (03PS1) 10Joal: Add mediawiki-history new-articles metric endpoint [analytics/aqs] - 10https://gerrit.wikimedia.org/r/377726 (https://phabricator.wikimedia.org/T175805) [09:46:09] (03PS1) 10Joal: Update changelog.md for v0.0.52 deployment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/377727 [09:46:22] elukey: --^ if you have a minute [09:47:03] (03CR) 10Elukey: [C: 032] Update changelog.md for v0.0.52 deployment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/377727 (owner: 10Joal) [09:47:06] (03CR) 10Elukey: [V: 032 C: 032] Update changelog.md for v0.0.52 deployment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/377727 (owner: 10Joal) [09:47:19] ready to submit whenever you want [09:47:37] elukey: going for it now, deploying just after [09:48:16] Actually looks like zuul will merge it - waiting for a minute [09:50:56] (03Merged) 10jenkins-bot: Update changelog.md for v0.0.52 deployment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/377727 (owner: 10Joal) [09:59:01] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore: dbstore1002 (analytics store) enwiki lag due to blocking query - https://phabricator.wikimedia.org/T175790#3603733 (10jcrespo) Missing word: "//reads will get [blocked] by writes (from replication) [on non transactional engines].//" [09:59:31] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore: dbstore1002 (analytics store) enwiki lag due to blocking query - https://phabricator.wikimedia.org/T175790#3603734 (10jcrespo) > Is it possible to see what query / script is running in the query mentioned in your comment? Not until to... [10:05:06] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore: dbstore1002 (analytics store) enwiki lag due to blocking query - https://phabricator.wikimedia.org/T175790#3603738 (10Addshore) So, looking at the script the same query gets run for 9 different beta features on a cron at roughly 3am, s... [10:08:04] !log Deploying refinery-source using Jenkins [10:08:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:09:03] elukey: I'm sorry I need help :( [10:09:52] elukey: jenkins tells me: joal is missing the Job/Release permission [10:10:03] elukey: this was not happening before :( [10:10:40] weeeeird [10:11:05] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore: dbstore1002 (analytics store) enwiki lag due to blocking query - https://phabricator.wikimedia.org/T175790#3603754 (10jcrespo) Actually, I converted the tables to innodb already- so nothing is to be done unless there is still some non-... [10:11:57] joal: can I see the full error log? [10:12:08] I am checking https://integration.wikimedia.org/ci/job/analytics-refinery-release/lastBuild/ but can't find it [10:12:18] elukey: Acutally nothing more than that - with header: Access Denied [10:12:41] ah there https://integration.wikimedia.org/ci/job/analytics-refinery-release/lastBuild/console [10:13:12] nope elukey - this was my build to refresh versions [10:13:29] The release doesn't even launch anything [10:13:54] so can you give me the link of the failed one? [10:14:30] https://integration.wikimedia.org/ci/job/analytics-refinery-release/m2release/submit [10:14:33] elukey: --^ [10:14:46] Nothing gets schedule nore done [10:15:01] ahhh you mean that Jenkins does not allow you to log in! [10:15:14] I am logged in, but can't release [10:15:26] I am getting [10:15:27] Access Denied [10:15:27] Elukey is missing the Job/Release permission [10:15:31] same for me [10:18:36] lemme ask to releng [10:23:08] joal: https://phabricator.wikimedia.org/T169557 [10:23:28] elukey: can't read - restricted [10:23:52] elukey: I feel like systems don't want to let me work today ;) [10:25:18] ah sorry! So TL;DR there was a refactoring of the Jenkins perms on July and we got pwned [10:25:27] but our use case wasn't listed [10:26:19] hm elukey -- Our last deploy happened in Aug-23 -- Weird for a july refactor [10:28:18] joal: sorry, misread the task (just done now): it was done on the 25th [10:28:35] elukey: Arf :) [10:28:46] elukey: What solution do have for now? [10:29:53] I am asking to the Authority (Moritz) :D [10:29:59] :D [10:30:13] there is a new group that can release but not sure what is the policy for that one [10:38:00] joal: gimme 10 mins and I should sort it out [10:38:14] elukey: You're the man :) [10:41:13] joal: you should be able to launch the job now [10:41:16] can you try? [10:41:26] I can try :) [10:44:35] Works ! You rok elukey, as usual :) [10:47:26] * elukey lunch! [10:51:54] (03PS1) 10Joal: Correct mediawiki-history-reduced bug [analytics/refinery] - 10https://gerrit.wikimedia.org/r/377734 [10:52:15] (03CR) 10Joal: [V: 032 C: 032] "Self merging before deploy." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/377734 (owner: 10Joal) [10:57:05] !log Deploy refinery from scap [10:57:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:59:49] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore: dbstore1002 (analytics store) enwiki lag due to blocking query - https://phabricator.wikimedia.org/T175790#3603912 (10Addshore) Awesome! thanks for the heads up! [11:03:01] !log Deploy refinery onto HDFS [11:03:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:11:30] !log Kill-Restart oozie pageview druid loading jobs (hourly, daily, monthly) [11:11:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:15:45] !log Kill-Restart mediawiki-history-denormalize-coord and launch new coords mediawiki-history-load and mediawiki-history-reduced [11:15:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:19:25] (03PS1) 10Addshore: Remove 'facebook' example in config (unused) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/377737 [11:19:36] (03PS1) 10Addshore: Remove 'facebook' example in config (unused) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/377738 [11:19:39] (03CR) 10Addshore: [C: 032] Remove 'facebook' example in config (unused) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/377738 (owner: 10Addshore) [11:19:44] (03CR) 10Addshore: [C: 032] Remove 'facebook' example in config (unused) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/377737 (owner: 10Addshore) [11:19:47] (03Merged) 10jenkins-bot: Remove 'facebook' example in config (unused) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/377738 (owner: 10Addshore) [11:19:52] (03Merged) 10jenkins-bot: Remove 'facebook' example in config (unused) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/377737 (owner: 10Addshore) [11:21:58] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore: dbstore1002 (analytics store) enwiki lag due to blocking query - https://phabricator.wikimedia.org/T175790#3603981 (10jcrespo) I belive this, or something similar in spirit could be happening now for s5. I need to look more into it to... [11:24:14] elukey: you can see the place that the "secrets" are stored right? [11:30:10] addshore: do you mean the private puppet repo? [11:30:15] I think so :) [11:30:42] do you need to store a passzorz and use it in puppet? [11:30:54] There is a secret called wmde_secrets and I was just wondering if it has a key called "facebook [11:32:33] if there is, it can be removed! :) [11:32:40] it is no longer / hasn't been used in a long time [11:39:34] ah! [11:40:11] yes there is [11:41:48] removed :) [11:42:28] addshore: --^ [11:43:22] elukey: thanks! [12:47:05] elukey: Could you please restart pivot for me to check a schema update? [12:48:24] joal: done! [12:48:28] Thanks elukey [13:03:21] (03PS2) 10Joal: Add mediawiki-history new-articles metric endpoint [analytics/aqs] - 10https://gerrit.wikimedia.org/r/377726 (https://phabricator.wikimedia.org/T175805) [13:05:21] (03PS1) 10Joal: Add mediawiki-history edited-articles endpoint [analytics/aqs] - 10https://gerrit.wikimedia.org/r/377749 (https://phabricator.wikimedia.org/T175805) [13:05:29] taking a break a-team [13:58:00] elukey! 0.11.0.,1 is released! [13:58:08] if you don't mind, i'm going to update apt and install it on jumbos [13:59:30] sure [13:59:52] ottomata: I have a draft of the jmx_exporter stuff https://gerrit.wikimedia.org/r/#/c/377753 [14:04:23] still now able to show a pcc since I need to update the puppet compiler [14:12:12] whoa elukey it runs inside of the JVM?! [14:12:15] interesting! [14:12:25] or [14:12:26] no, does it? [14:12:55] oh [14:12:59] no, there is an instance.... [14:13:00] weird [14:14:58] yep yep [14:15:09] TIL that the javaagent is a pluging that can run a jar [14:16:12] kinda cool [14:16:17] no separarate process [14:16:51] if you like the idea I can keep working on it [14:19:53] 10Analytics, 10Performance-Team: Explore NavigationTiming by faceted properties - EventLogging refine - https://phabricator.wikimedia.org/T166414#3604482 (10Gilles) Awesome, thank you! [14:20:20] elukey: i like it [14:20:28] let's make it happen [14:20:46] lemme know if i can help, maybe we translating jmxtrans queries -> prometheus or something [14:26:42] ottomata: yes that part is something we'd need to discuss, but overall it seems achievable in a couple of days no? [14:27:07] the groundwork si already done [14:28:32] i think so [14:28:35] i think you are right [14:28:36] we should do it [14:32:52] ottomata: coming in a sec to the cave [14:41:52] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3604545 (10Nuria) > The main important point, in my opinion, is to ensure that non WMF staff accounts have an expiration date and someb... [14:44:17] holaaa [14:44:45] joal: hola! one question, does navigation vectors depend on click stream dataset being productionized? [14:44:55] joal; seems like it wouldn't [14:46:25] (03CR) 10Mforns: "thanks for the comments!" (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355601 (https://phabricator.wikimedia.org/T162034) (owner: 10Mforns) [14:46:46] (03PS8) 10Mforns: Add script to purge old mediawiki data snapshots [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355601 (https://phabricator.wikimedia.org/T162034) [14:46:48] 10Analytics-Kanban, 10Research: productionize ClickStream dataset - https://phabricator.wikimedia.org/T158972#3604550 (10Nuria) @shilad: i do not think navigation vectors depends on click tream dataset being completed, does it? i will remove it as a subtask. [14:47:00] 10Analytics, 10Patch-For-Review: Productionize navigation vectors - https://phabricator.wikimedia.org/T174796#3604552 (10Nuria) [14:47:02] 10Analytics-Kanban, 10Research: productionize ClickStream dataset - https://phabricator.wikimedia.org/T158972#3604551 (10Nuria) [14:47:40] (03PS9) 10Mforns: Add script to purge old mediawiki data snapshots [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355601 (https://phabricator.wikimedia.org/T162034) [14:53:51] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Select candidate jobs for transferring to the new infrastucture - https://phabricator.wikimedia.org/T175210#3604587 (10mobrovac) 05Open>03Resolved The job is being double-produced now, so resolving. [15:00:16] a-team: standdupp [15:05:39] (03PS2) 10Mforns: Add script to delete banner activity _SUCCESS files [analytics/refinery] - 10https://gerrit.wikimedia.org/r/353309 (https://phabricator.wikimedia.org/T164497) [15:13:40] 10Analytics, 10Performance-Team: Explore NavigationTiming by faceted properties - EventLogging refine - https://phabricator.wikimedia.org/T166414#3604674 (10Krinkle) [15:16:11] 10Analytics-Kanban, 10Analytics-Wikistats: Handle long project names in Wikiselector - https://phabricator.wikimedia.org/T173373#3604679 (10Nuria) a:03mforns [15:21:52] 10Analytics-Kanban, 10Patch-For-Review: Add zero carrier to pageview_hourly data on druid - https://phabricator.wikimedia.org/T161824#3604734 (10Nuria) Confirming that zero carrier is visble, pinged zero folks about it [15:22:40] 10Analytics-Kanban, 10User-Elukey: Archive PageContentSaveComplete in hdfs while we continue collecting data - https://phabricator.wikimedia.org/T170720#3604736 (10Nuria) 05Open>03Resolved [15:23:01] 10Analytics-Kanban, 10Patch-For-Review: Productionize mediawiki-history-reduced druid ingestion - https://phabricator.wikimedia.org/T174915#3604737 (10Nuria) 05Open>03Resolved [15:23:13] 10Analytics-Kanban, 10Patch-For-Review: Move GraphiteClient from refinery-core to refinery-job module - https://phabricator.wikimedia.org/T175163#3604738 (10Nuria) 05Open>03Resolved [15:24:14] 10Analytics-Kanban, 10Research: productionize ClickStream dataset - https://phabricator.wikimedia.org/T158972#3604739 (10Nuria) @JAllemandou Let's document dataset on https://meta.wikimedia.org/wiki/Research:Wikipedia_clickstream (cadence, availability) announce it to analytics@ and reserach list before we c... [15:24:51] 10Analytics, 10Phabricator: Create phabricator space for tickets with legal restrictions - https://phabricator.wikimedia.org/T174675#3604743 (10Aklapper) So if I get it right: * Create some `#acl*` project for Analytics folks (only used for access control and not for any tasks) * Add one or two Analytics folks... [15:31:59] 10Analytics-Kanban, 10Analytics-Wikistats: Add piwik to wikistats 2.0 site - https://phabricator.wikimedia.org/T171642#3471959 (10Milimetric) verified working and php beacon is gone [15:39:54] joal: I am going to test the spark job on a larger dataset. Do you have an example of spark-submit params that work well for your ClickStreamBuilder job? I thought I could start there. [15:44:29] Shilad: in a meeting, will answer when finished [15:48:46] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (2/4) - Wiki selector - https://phabricator.wikimedia.org/T170936#3448065 (10Milimetric) The first part of the task, dealing with "Something - All Languages" no longer applies. We disabled this until we have endpoints for it in AQS. The cursor bug is... [15:50:10] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 2 others: Allow easy tuning of the jobqueue concurrency. - https://phabricator.wikimedia.org/T175800#3604811 (10Pchelolo) Currently #changeprop indeed only supports concurrency per rule (per job type) and it's hard coded in the config, so alt... [15:50:16] 10Analytics-Kanban, 10Analytics-Wikistats: Fix wikistats 2.0 footer links - https://phabricator.wikimedia.org/T173043#3604812 (10Milimetric) 05Open>03Resolved [15:52:57] 10Analytics-Kanban, 10Analytics-Wikistats: Addition of Unique Devices metric - https://phabricator.wikimedia.org/T170461#3604819 (10Milimetric) [15:52:59] 10Analytics-Kanban, 10Analytics-Wikistats: Productionise line graph - https://phabricator.wikimedia.org/T171766#3604817 (10Milimetric) 05Open>03Resolved Line graph works on the Detail page [15:53:35] 10Analytics-Kanban, 10Analytics-Wikistats: Addition of Unique Devices metric - https://phabricator.wikimedia.org/T170461#3432090 (10Milimetric) The table view doesn't work for this metric on the Detail page. [15:55:20] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (2/4) - Wiki selector - https://phabricator.wikimedia.org/T170936#3604822 (10fdans) @Milimetric that last one has been retasked - T173373 [15:55:23] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (3/4) - Data issues - https://phabricator.wikimedia.org/T170937#3448129 (10Milimetric) The widget now seems to compute the average at all times, but the pageviews metric is additive, so it could show the yearly total instead. [15:55:53] joal: Thanks, no rush! I may go offline but will check chat logs. [15:58:09] team, leaving for meetup, see ya tomorrow! [15:59:23] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0 UI second deployment/iteration - https://phabricator.wikimedia.org/T170460#3604833 (10Milimetric) [15:59:25] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (1/4) - Dashboard and general UI - https://phabricator.wikimedia.org/T170933#3604831 (10Milimetric) 05Open>03Resolved All of these have been addressed, good. [16:01:58] 10Analytics-Kanban, 10Analytics-Wikistats: Use daily granularity for 1-month time ranges - https://phabricator.wikimedia.org/T173372#3525876 (10Milimetric) This works ok from a data point of view, but there are two rendering problems: 1. The X axis shrinks in 1-month and 3-month mode (compared to the 1-year m... [16:08:12] !log restarting druid-brokers with increase in query cache size [16:08:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:11:53] Shilad: My last test with Clistream was using the month of August for webrequest, and snapshot '2017-08' for tables. I took some hours, but finished successfully [16:13:45] joal: Thanks! What were the spark settings on that? # executors, cores, etc.? [16:15:10] I used: 4g for driver, 16g and 2 cores per executor (taking advantage of caching space, maybe more cores can be availbable for you depending on how much caching you need) - and max-dynamic-executors to 32 [16:15:15] Shilad: --^ [16:16:09] joal: Thanks! [16:16:17] No prob Shilad :) [16:55:18] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (4/4) - Detail page - https://phabricator.wikimedia.org/T170940#3448220 (10Milimetric) Still a few problems: * The date range selector does not show selected option when not focused. (try clicking 1 year and then clicking on the graph selector dropdo... [16:56:26] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (2/4) - Wiki selector - https://phabricator.wikimedia.org/T170936#3604984 (10Milimetric) gotcha, editing comment [16:57:22] 10Analytics-Kanban, 10Analytics-Wikistats: Addition of Unique Devices metric - https://phabricator.wikimedia.org/T170461#3605000 (10Milimetric) [16:57:24] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0 UI second deployment/iteration - https://phabricator.wikimedia.org/T170460#3605001 (10Milimetric) [16:57:26] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (2/4) - Wiki selector - https://phabricator.wikimedia.org/T170936#3448065 (10Milimetric) 05Open>03Resolved p:05Triage>03Normal [17:02:26] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3605013 (10MoritzMuehlenhoff) >>! In T170878#3604545, @Nuria wrote: >> The main important point, in my opinion, is to ensure that non W... [17:11:10] 10Analytics, 10EventBus, 10Wikidata, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3605058 (10GWicke) p:05Normal>03High [17:17:51] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Audit users and account expiry dates for stat boxes - https://phabricator.wikimedia.org/T170878#3605089 (10DarTar) >>! In T170878#3605013, @MoritzMuehlenhoff wrote: > All the users with shell access which are not WMF staff have a a... [17:22:59] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (4/4) - Detail page - https://phabricator.wikimedia.org/T170940#3605104 (10fdans) //If we select 1 year in the time-range selector, the graph shows 13 data points. Is that expected? (now it's showing 11, I would think 13 is good, so you can get YoY com... [17:27:34] 10Analytics-Kanban: Provide oozie job running ClickStream spark job regularly - https://phabricator.wikimedia.org/T175844#3605123 (10JAllemandou) [17:27:51] 10Analytics-Kanban, 10Research: productionize ClickStream dataset - https://phabricator.wikimedia.org/T158972#3605136 (10JAllemandou) Actually @Nuria this task is only the spark, not the oozie that will make the saprk job run regularly. I'll modify docs once we have the other one (T175844) done. [17:30:22] 10Analytics, 10EventBus, 10Wikidata, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3605142 (10GWicke) Raised priority, as this is a) blocking the migration to the Kafka job queue backend (T157088), and b) is likely already causing performance and pos... [17:46:40] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 2 others: Allow easy tuning of the jobqueue concurrency. - https://phabricator.wikimedia.org/T175800#3605203 (10GWicke) p:05Normal>03Low We briefly discussed this during today's sync meeting. While there are ways to set up targeted proces... [17:54:41] 10Analytics, 10EventBus, 10Wikidata, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3605227 (10Ladsgroup) Can I examine the job logs in more depth? the pages params can't have more than 100 (old settings) which we changed it to 50 and now to 20. [17:57:54] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Select candidate jobs for transferring to the new infrastucture - https://phabricator.wikimedia.org/T175210#3605236 (10GWicke) Given the useful information we have in this task, I am proposing to widen the scope beyond the first job... [18:00:12] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (4/4) - Detail page - https://phabricator.wikimedia.org/T170940#3605242 (10fdans) [18:01:02] 10Analytics, 10EventBus, 10Wikidata, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3605243 (10daniel) InjectRCRecords batches inserts when running the job, but doesn't chop the batch up before scheduling the job. I can easily fix that. The patch shou... [18:02:33] 10Analytics, 10EventBus, 10Wikidata, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3605249 (10daniel) Note that {T174422} is related, but would not change the fact that the entire set of titles would be put into a single InjectRC job, at the moment.... [18:06:36] 10Analytics, 10EventBus, 10Wikidata, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3605258 (10Pchelolo) Here's an example of a very large event: https://people.wikimedia.org/~ppchelko/large_event It's not an event itself, it's a log message from #ev... [18:16:18] 10Analytics-Kanban, 10Research: Spark job to produce clickstream dataset - https://phabricator.wikimedia.org/T158972#3605295 (10Nuria) [18:17:07] 10Analytics-Kanban: Provide oozie job running ClickStream spark job regularly - https://phabricator.wikimedia.org/T175844#3605123 (10Nuria) Let's make sure we update documentations and anounce release of new data once this task is done. Probably blogpost worthy [18:17:18] 10Analytics-Kanban, 10Research: Spark job to produce clickstream dataset - https://phabricator.wikimedia.org/T158972#3053334 (10Nuria) Ahh, my mistake! [18:17:25] 10Analytics-Kanban, 10Research: Spark job to produce clickstream dataset - https://phabricator.wikimedia.org/T158972#3605303 (10Nuria) 05Open>03Resolved [18:22:09] 10Analytics, 10EventBus, 10Wikidata, 10Patch-For-Review, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3605329 (10daniel) Fix above, backport applies cleanely. Note that the fix must be merged into the wikidata build, which then has to be re-deploy... [18:22:57] Hey milimetric, are you back from duty? [18:23:06] I'm around joal [18:23:13] baby seems relatively calm now [18:23:23] milimetric: would you mind me showing some weird stuff I get from Druid? [18:23:33] let's go! [18:37:33] Can't hear you [18:37:37] milimetric: --^ [19:07:15] milimetric: I have launched a reindex with hyperUnique metrics - I'll test tomorrow morning [19:07:41] milimetric: if it doesn't fit, we're going to need to pivot to hive-precomputed-metrics fast [19:08:20] joal: yea, because you think the nested groupBys are too slow right? [19:08:42] not only too slow - for bigger queries they fail (too many rows for inner-query) [19:09:02] oh yeah, I saw there was some setting for that, but yea [19:11:22] milimetric: It's possible to test to make it work - but I think it put us at risk even more than we already are (timewsise) [19:12:41] yeah, let's pre-compute if the hyper unique metrics don't work [19:14:10] k [19:16:33] leaving for tonight team, see you tomorrow [19:30:00] ottomata: question , were you implying with your comments that we should remove mediawiki code that sends these events before blacklisting? https://gerrit.wikimedia.org/r/#/c/377667/ (rather than other way arround) [19:30:05] *around [19:32:34] HaeB: question, when are we turning the popups experiment off? It started 9/1 so if i remember it right it should run for two weeks meaning that we stop it this friday? [20:04:01] nuria_: yes, we said sep 16 (it needs two full weeks of data), but i just got reminded by sam that this needs a swat deploy ... so it will need to be sep 18 (monday) [21:02:00] HaeB: ok [21:17:41] ottomata: are you ok merging this one: https://gerrit.wikimedia.org/r/#/c/377667/ [21:22:51] sure nuria_ let's do it tomorrow thouguh, s'ok? [21:23:37] ottomata: of course [21:27:11] a-team: I am about to run performance benchmarks for two different implementations of a job. Each will likely tie up ~10% of the cluster for a few hours. Ping me if this is problematic! I'll be checking this channel on and off throughout the evening. [21:28:55] should be fine Shilad, thanks for checking :) [21:29:28] ottomata: Awesome. Thanks! [21:52:16] 10Analytics: Correct pageview_hourly and derived data for T141506 - https://phabricator.wikimedia.org/T175870#3606092 (10Tbayer) [21:53:50] 10Analytics, 10Pageviews-API, 10Reading-analysis: Suddenly outrageous higher pageviews for main pages - https://phabricator.wikimedia.org/T141506#2502585 (10Tbayer) This task is still open after more than a year, and continues to affect pageview data analysis. I have filed T175870 to remedy that. [22:13:32] 10Analytics: Correct pageview_hourly and derived data for T141506 - https://phabricator.wikimedia.org/T175870#3606162 (10Nuria) We already discussed this issue on this ticket: https://phabricator.wikimedia.org/T141506#2575088 and I second @BBlack 's opinion. In a gist: i do not think this traffic should be rem... [22:23:57] 10Analytics: Correct pageview_hourly and derived data for T141506 - https://phabricator.wikimedia.org/T175870#3606252 (10Tbayer)