[02:36:13] 10Analytics, 10Analytics-Wikistats, 10Inuka-Team, 10Language-strategy, and 2 others: Add more popular articles per country data to AQS - https://phabricator.wikimedia.org/T263697 (10Meghajain171192) Thanks a ton for your reply @Nuria :) Will look into "Wikistats Bug - easy to understand language for pagevi... [05:36:23] PROBLEM - Check the last execution of archive-maxmind-geoip-database on stat1007 is CRITICAL: CRITICAL: Status of the systemd unit archive-maxmind-geoip-database https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [08:58:05] 10Analytics, 10Operations, 10Traffic: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10ema) [09:32:56] (03CR) 10Elukey: [C: 03+1] Fix banner_activity_daily job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/630682 (owner: 10Joal) [09:54:43] 10Analytics, 10Operations, 10Traffic: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10elukey) I also checked with `top` on cp5011 to better visualize the graph, and the usage is really too much from what I used to see. We are in the process of evaluating atskafka but it w... [10:03:45] I'm going to take it very slow today because of the potential CTS. [10:08:27] ack :) [10:31:32] 10Analytics: Increase in usage of /var/lib/mysql on an-coord1001 after Sept 21st - https://phabricator.wikimedia.org/T264081 (10elukey) p:05Triage→03High [10:31:42] 10Analytics: Increase in usage of /var/lib/mysql on an-coord1001 after Sept 21st - https://phabricator.wikimedia.org/T264081 (10elukey) The first suspicion is Hue Next, that is what we added recently. I asked to Jaime if it was possible to get a snapshot of the dbs of analytics-meta today and a week ago, and som... [10:32:37] 10Analytics: Increase in usage of /var/lib/mysql on an-coord1001 after Sept 21st - https://phabricator.wikimedia.org/T264081 (10elukey) [10:43:35] 10Analytics: Increase in usage of /var/lib/mysql on an-coord1001 after Sept 21st - https://phabricator.wikimedia.org/T264081 (10elukey) Ok I think I have a lead - I see a ton of entries for the recent days about actions like `loop_mark_hour_done_21` and they seem related to the bulk load of the pagecount-ez (fro... [11:48:35] 10Analytics: Increase in usage of /var/lib/mysql on an-coord1001 after Sept 21st - https://phabricator.wikimedia.org/T264081 (10JAllemandou) My 2 cents on that one: Oozie has a setting about how long it keeps historical information for workflows/coords/bundles. I imagine we can manually tweak it to drop recent i... [12:11:27] 10Analytics: Increase in usage of /var/lib/mysql on an-coord1001 after Sept 21st - https://phabricator.wikimedia.org/T264081 (10elukey) >>! In T264081#6501661, @JAllemandou wrote: > My 2 cents on that one: Oozie has a setting about how long it keeps historical information for workflows/coords/bundles. I imagine... [12:42:27] Huawei Consumer Cloud’s Cassandra deployments have grown to 30,000+ nodes, supporting more than 10 million operations per second with average latency of 4ms, and the maximum number of table records reaches 300 billion. [13:00:07] * elukey afk! [14:13:33] 10Analytics, 10puppet-compiler, 10Patch-For-Review: Puppet catalog compiler fails to diff when change produces non-ascii accented character - https://phabricator.wikimedia.org/T263876 (10jbond) 05Open→03Resolved a:03jbond @razzi Thanks for the report, i have applied a fix to this in [[ https://gerrit.w... [14:20:09] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Put 6 GPU-based Hadoop worker in service - https://phabricator.wikimedia.org/T255138 (10elukey) [14:26:39] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Put 6 GPU-based Hadoop worker in service - https://phabricator.wikimedia.org/T255138 (10elukey) There are 6 workers with GPUs: an-worker1096->1101 Currently we have only an-worker1096 running in the Hadoop cluster, but without any GPU configured.... [14:38:10] 10Analytics-Radar, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker11[02-17] - https://phabricator.wikimedia.org/T259071 (10elukey) @RobH I used the related spicerack cookbook to init an-worker1102 (install all partitions with proper labels etc..) and as far as I can see now... [14:46:49] elukey: are you coming to our apachecon slack channel? cc razzi, mforns, anyone else [14:48:02] I’m leaning towards Camel for the first talk and Flink/Ignite second talk and wanted people to banter with. I’m pretending I’m in a big hotel looking for yall [14:48:22] 10Analytics, 10Operations, 10Traffic: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10ema) >>! In T264074#6501340, @elukey wrote: > I think it is better to know if the increase is brought by the new VUT/VSL api or if it is something else. Other units such as `varnishmta... [14:58:19] milimetric: do we have one? if so I didn't get it [15:01:21] elukey, it’s #wikimedia [15:03:23] 10Analytics, 10Operations, 10Traffic: varnishkafka 1.1.0 CPU usage increase - https://phabricator.wikimedia.org/T264074 (10elukey) https://github.com/wikimedia/varnishkafka/commit/b0675e80c2a059ba3a508d8ebfc16a79bee3e154 shows a big change in usage of VUT/VSL, that afaics should be easier (more responsibilit... [15:32:28] https://sdap.incubator.apache.org/ is very interesting [15:32:38] it is being presented at apachecon [15:48:36] 10Analytics-Radar, 10Technical-blog-posts: Story idea for Blog: The Best Dataset on Wikimedia Content and Contributors - https://phabricator.wikimedia.org/T259559 (10srodlund) @Milimetric Just an update. I had a number of posts this week, and plan on posting this one on Thursday, 1 Oct. Will let you know when... [15:57:04] 10Analytics-Radar, 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Search (Current work): PoC on anomaly detection with Flink - https://phabricator.wikimedia.org/T262942 (10dcausse) Looking at existing solutions based on flink in this area I don't think this is a good fit for the table API and/or SQL unl... [16:01:43] elukey: how does SDAP relate to BigTop? [16:02:23] klausman: no relationship, SDAP is a new project in the incubator that is developed by the JPL [16:02:40] just heard about it, not sure if it is a specialized framework or not [16:02:58] but I think it should sit on top of things like hadoop spark etc.. [16:03:07] (speculation from a minute of reading :D) [16:04:10] Roger [16:06:54] 10Analytics-Radar, 10Operations, 10ops-eqiad: an-presto1004 down - https://phabricator.wikimedia.org/T253438 (10RobH) Self dispatch SR1038108849 entered with Chris as the contact. They should call him to schedule the on-site work. Since this has 'undefined' broken parts, it is easier overall to schedule th... [16:07:18] klausman: https://hadoop.apache.org/ozone/ is very interesting [16:12:51] So an alternative to HDFS (depending on what API you want)? [16:19:02] yep, object-store like [17:06:09] 10Analytics, 10Event-Platform, 10Privacy Engineering, 10Product-Analytics, and 3 others: Remove http.client_ip from EventGate default schema (again) - https://phabricator.wikimedia.org/T262626 (10mpopov) [17:15:14] 10Analytics, 10Analytics-EventLogging, 10Product-Analytics, 10Documentation: Document how ad blockers / tracking blockers interact with EventLogging - https://phabricator.wikimedia.org/T263503 (10kzimmerman) Moving to Tracking on Product Analytics workboard. @jlinehan @sdkim wanted to give you a heads up... [18:44:33] 10Analytics, 10Product-Analytics, 10Structured Data Engineering, 10Patch-For-Review, and 2 others: [L] Instrument MediaSearch results page - https://phabricator.wikimedia.org/T258183 (10egardner) I have an in-progress patch that adds some basic analytics to MediaSearch, using the draft schema from the link... [18:53:46] * elukey afk! [19:19:55] (03PS1) 10Razzi: Release 2.8.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/630928 [19:21:27] (03CR) 10Fdans: [V: 03+2 C: 03+2] Release 2.8.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/630928 (owner: 10Razzi) [20:14:11] fdans, razzi : fi-na-lly ! [20:14:36] nuria: it's live! will send an email tomorrow [20:14:44] thank you for the deploy razzi [20:14:59] fdans: to wikitech-l@ and analytics@ right? [20:15:13] sure [20:15:54] 10Analytics, 10Product-Analytics, 10Structured Data Engineering, 10Patch-For-Review, and 2 others: Develop a new schema for MediaSearch analytics or adapt an existing one - https://phabricator.wikimedia.org/T263875 (10egardner) One thing currently missing from the draft schema is a notion of "language" – I... [20:17:56] * joal sees filters on wikistats !!! [20:18:10] * joal sends wikistatslove to fdans, milimetric and razzi :) [20:18:28] Thank you for your support, fdans :) [20:19:43] I rejoyce [20:26:34] Gone for tonight team - see you tomorrow [20:34:52] milimetric: we all do [20:34:56] REALLY [20:35:00] ~~~~=> tears [20:35:05] of joy [20:43:28] 10Analytics, 10Product-Analytics, 10Structured Data Engineering, 10Patch-For-Review, and 2 others: Develop a new schema for MediaSearch analytics or adapt an existing one - https://phabricator.wikimedia.org/T263875 (10EBernhardson) > If we want to track the interface language that visitors to Special:Media... [20:50:01] 10Analytics, 10Product-Analytics, 10Structured Data Engineering, 10Patch-For-Review, and 2 others: Develop a new schema for MediaSearch analytics or adapt an existing one - https://phabricator.wikimedia.org/T263875 (10egardner) > Two thoughts: > > * I think the interface language for all logged out users... [21:19:27] 10Analytics, 10Operations, 10Traffic, 10User-Elukey: Sort out analytics service dependency issues for cp* cache hosts - https://phabricator.wikimedia.org/T128374 (10BBlack) 05Open→03Declined This is too-stale now and a lot of these bits have been replaced over time and are known to have their deps corr...