[03:13:28] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors metrics definition - https://phabricator.wikimedia.org/T247419 (10jwang) Tech team want to define and measure the retention rate of independent developers on Gerrit. **Metric definition ( proposed)** 1. A% of quarterly active subm... [04:07:30] 10Analytics, 10Product-Analytics (Kanban): SQL definition for wikidata metrics for tunning session - https://phabricator.wikimedia.org/T247099 (10Nuria) >I'll continue to think about whether there are simple ways to refine this query (or provide more context for what these 79M Wikidata entities are) but the cu... [06:49:07] good morning! [06:49:39] after checking what Isaac wrote on Presto I was curious to check metrics, and I saw a big "hole" yesterday evening UTC.. [06:49:42] Mar 24 22:06:00 an-coord1001 presto-server[224064]: Terminating due to java.lang.OutOfMemoryError: Java heap space [06:49:45] sigh [06:50:02] https://grafana.wikimedia.org/d/pMd25ruZz/presto?orgId=1&from=now-2d&to=now&fullscreen&panelId=13 [06:50:18] ok https://grafana.wikimedia.org/d/pMd25ruZz/presto?orgId=1&from=now-7d&to=now&fullscreen&panelId=13 is very weird [06:52:22] (for 1 week old metrics we have some duplication, my bad when I set up prometheus) [06:53:00] so the heap has grown up to ~4G without any gc run [06:54:49] no ok there were, didn't see it in the graph's scale [07:01:20] (I am adding some graphs) [07:31:02] I am downloading the hprof to my laptop, maybe I'll be able to see something with visual vm [07:38:34] (4.4G of hprof :P_ [07:54:08] (03PS2) 10Lex Nasser: Configure Oozie job for loading geoeditors data into Cassandra [analytics/refinery] - 10https://gerrit.wikimedia.org/r/582638 (https://phabricator.wikimedia.org/T248289) [07:58:07] found also https://eng.uber.com/jvm-tuning-garbage-collection/ that seems interesting [08:00:58] interesting, they say that CMS works better than G1 for them (but I imagine how big namenodes heaps are at uber :D) [08:14:01] !log restart presto-server on an-coord1001 to remove jmx catalog config [08:14:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:14:23] ok so I used the Eclipse memory analyzer, it was clear that the issue was the jmx catalog [08:14:42] 10Analytics, 10Growth-Team, 10Product-Analytics, 10Security-Team: Hash edit session ID in EditAttemptStep and VisualEditorFeatureUse whitelisting - https://phabricator.wikimedia.org/T244931 (10mforns) Hi @nettrom_WMF, @nshahquinn-wmf and @MMiller_WMF I reviewed the data sets more deeply and understood a co... [08:14:55] is used it to find meaningful metrics and values, but it seems that without proper tuning it leads to memory usage [08:15:14] basically 97% of the heap was jmx historical data [08:15:37] there is some tuning to drop old jmx data if somebody wants to keep the jmx catalog, but I'd say no for the moment [08:18:18] mistery solved :) [08:26:09] brb [08:50:43] created https://wikitech.wikimedia.org/wiki/Analytics/Systems/Presto/Administration [08:50:58] and now I am adding a simple nagios process check for every presto node [08:51:05] so we can get notifications if this happens again [09:30:02] 10Analytics, 10Analytics-Wikistats: [Wikistats2] Broken down data with different time ranges for each line (or column set) breaks the chart - https://phabricator.wikimedia.org/T198630 (10Aklapper) @sahil505, @mforns: Hi, the patch in Gerrit has been merged. Can this task be resolved (via {nav name=Add Action..... [12:06:22] going out to get groceries + lunch :) [13:01:11] o/ [13:01:12] https://wikitech.wikimedia.org/wiki/Analytics/Data_access [13:01:37] IF some wmde people want access to whereever the wikidata json bit is in hadoop and also the mediawiki_history stuff, which group will they need? [13:01:43] Hi itamarWMDE ! :D [13:01:48] and tarrow :D [13:01:59] Hi @addshore! [13:02:53] Getting added to analytics-wmde-users also wouldnt hurt, that will allow you to administer some wmde related scripts that run in a cron on the cluster too [13:03:26] itamarWMDE: if you want to read the docs for that bit they are at https://wikitech.wikimedia.org/wiki/WMDE/Analytics [13:04:11] 100%, thanks addshore [13:18:37] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Inuka-Team (Kanban), 10Patch-For-Review: Set up pageview counting for KaiOS app - https://phabricator.wikimedia.org/T244547 (10nshahquinn-wmf) 05Open→03Resolved @Nuria, thanks for the details! Do you see any problem with us always sending http... [13:18:46] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Inuka-Team (Kanban), 10Patch-For-Review: Set up pageview counting for KaiOS app - https://phabricator.wikimedia.org/T244547 (10nshahquinn-wmf) 05Resolved→03Open [13:18:52] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10Event-Platform, and 4 others: eventgate-wikimedia should support using remote stream configuration - https://phabricator.wikimedia.org/T238657 (10Ottomata) 05Open→03Resolved [13:18:56] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 9 others: Modern Event Platform: Stream Configuration: Implementation - https://phabricator.wikimedia.org/T233634 (10Ottomata) [14:19:02] joal: :] [14:19:16] Hi here hashar [14:19:55] a-team - hashar is testing a new version of refinery-source release using docker, so don't worry if you receive emails about that [14:22:28] so hmm giving it a try [14:22:50] that failed fast :] https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/1/console [14:23:49] hashar: classical I guess :) [14:28:50] * hashar fixes some yaml [14:32:09] ha I have made good finding [14:32:25] there is no ssh client in the container [14:48:51] 10Analytics, 10Analytics-Wikistats: [Wikistats2] Broken down data with different time ranges for each line (or column set) breaks the chart - https://phabricator.wikimedia.org/T198630 (10mforns) 05Open→03Resolved @Aklapper yes, this task should be resolved. Thanks! [14:51:22] meh hashar [14:53:24] joal: I am adding it, building the containers etc [14:53:30] ack [14:53:44] and that would also bump java8 from 8u212-b03 to 8u242-b08 [14:54:11] hashar: ahhh? :) [14:56:27] rebuilding, that will take a while [14:59:45] 10Analytics, 10Product-Analytics, 10Readers-Web-Backlog (Needs Product Owner Decisions), 10covid-19: Weekly updates on editors & readers - https://phabricator.wikimedia.org/T247873 (10Nuria) @MMiller_WMF We know, really. It is a valuable use case, supporting a denormalized dataset that updates weekly or d... [15:17:45] joal: so hmm that will take an extra days. I am hitting a few more issues in two unrelated components when building the container ;D [15:59:09] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors metrics definition - https://phabricator.wikimedia.org/T247419 (10Nuria) Let's also compute changesets per quarter, it is probably helpful to quantify drop/increases on number of contributors that are due to activity being lower in... [15:59:35] no problem hashar - let me know if I acn help [15:59:47] joal: solved it :) [15:59:50] \o/ [15:59:50] I moved to the next one [16:00:03] ssh gerrit requires the ssh fingerprint ;D [16:00:04] * joal bows to hashar - master of CI [16:01:45] a-team standup [16:04:09] itamarWMDE: once you get access to the hadoop stuff then joal is the one that knows all of the magic about querying the wikidata json that is there! [16:04:20] 10Analytics, 10Analytics-Kanban, 10EventStreams, 10Operations, and 2 others: EventStreams drops the connection after 15 minutes, which makes it unreliable - https://phabricator.wikimedia.org/T242767 (10ema) >>! In T242767#5998492, @gerritbot wrote: > Change 583295 **merged** by Ema: > [operations/puppet@pr... [16:04:24] o/ addshore and itamarWMDE :) [16:04:27] In standup now [16:19:12] nuria: who in your team we should add to the doc for the covid-19 data review? (you mentioned yesterday anyone in your team including yourself can help.) We need someone who can get back to us in a little bit more than 24 hours from now. [16:33:28] 10Analytics, 10Analytics-Kanban, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Infrastructure (phase-out-jessie), and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10hashar) I have tried the job, and exposing the SSH agent socket... [16:37:47] leila: Please add mforns and joal , now please have in mind that I do not think at this time we can promise a 24 hour workaround [17:01:39] nuria: ok, I'll do. If you can't do it, that's fine. We will go to the fallback option we discussed yesterday in the meeting. [17:03:58] ping nuria ? [17:06:08] 10Analytics, 10Better Use Of Data, 10Desktop Improvements, 10Product-Infrastructure-Team-Backlog, and 7 others: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 (10Ottomata) @Krinkle can you give me some guidance about https://phabricator.wikimedia.org/T226986#5992337 ? [17:06:42] 10Analytics, 10Operations, 10ops-eqiad: analytics1044 hardware failure - https://phabricator.wikimedia.org/T248413 (10Volans) 05Open→03Resolved a:03Volans So far no more errors in racadm. Let's keep an eye on it, but I'm resolving it for now. Feel free to re-open on re-occurrence. [17:49:03] 10Analytics: clear bot spam-scraping [[en:United States Senate]] not being detected as a bot - https://phabricator.wikimedia.org/T247085 (10Nuria) We can use this one as a validator of our latest bot discussion @JAllemandou [17:50:08] 10Analytics, 10Analytics-Kanban, 10Release Pipeline, 10Patch-For-Review, and 2 others: Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10Ottomata) Ah we need to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/583073 first, before the others, yes? [18:12:51] ok the test cluster is still broken, will try tomorrow :( [18:12:52] * elukey off [18:43:40] joal: +1 on EventSchemaLoader change? [19:01:13] 10Analytics, 10Event-Platform, 10serviceops, 10Patch-For-Review, 10Wikimedia-production-error: Lots of "EventBus: Unable to deliver all events" - https://phabricator.wikimedia.org/T247484 (10Ottomata) Checking in, how goes? [19:01:23] 10Analytics: Refine + EventLoggingSchemaLoader should use api.svc instead of meta.wikimedia.org directly. - https://phabricator.wikimedia.org/T247510 (10Ottomata) I think it would be better for us to hit the internal LVS / discovery endpoints, rather than route out via the webproxy. Either way, if the webproxy... [19:01:41] 10Analytics: Refine + EventLoggingSchemaLoader should use api.svc instead of meta.wikimedia.org directly. - https://phabricator.wikimedia.org/T247510 (10Ottomata) Although, oof, I just looked at my code that does this, and passing this info down the call stack to the point where the HTTP request is made is not s... [19:01:47] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: eventgate-wikimedia should fill in defaults for some important fields - https://phabricator.wikimedia.org/T240477 (10Ottomata) 05Open→03Resolved [19:01:52] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Better Use Of Data, and 9 others: Modern Event Platform: Stream Configuration: Implementation - https://phabricator.wikimedia.org/T233634 (10Ottomata) [19:14:20] 10Analytics, 10Analytics-Kanban, 10Event-Platform: Update MW Vagrant to work with EventLogging and EventGate changes - https://phabricator.wikimedia.org/T240355 (10Ottomata) Hm, @mforns I think we might need to allow events through if event if no stream config has been declared in dev mode. We could force f... [19:23:35] 10Analytics, 10Analytics-Kanban, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Infrastructure (phase-out-jessie), and 2 others: Migrate analytics/refinery/source release jobs to Docker - https://phabricator.wikimedia.org/T210271 (10hashar) So I have played a bit more with the job. Sharing the s... [20:21:40] ottomata: argh, i still need to look at that one [20:25:39] nuria: its ok as long as one of you reviews its fine [20:36:22] ottomata: yeah good for me sorry [20:57:51] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Newpyter - SWAP Juypter Rewrite - https://phabricator.wikimedia.org/T224658 (10Ottomata) a:03Ottomata [22:48:46] 10Analytics, 10Performance-Team, 10Readers-Web-Backlog: Review referer configuration of origin/origin-when-crossorigin/origin-when-cross-origin - https://phabricator.wikimedia.org/T248526 (10Krinkle) [22:50:30] 10Analytics, 10Performance-Team, 10Readers-Web-Backlog: Review referer configuration of origin/origin-when-crossorigin/origin-when-cross-origin - https://phabricator.wikimedia.org/T248526 (10Krinkle)