[00:54:01] 10Analytics: Check home/HDFS leftovers of rodolfovalentim - https://phabricator.wikimedia.org/T266467 (10Dzahn) @Nuria Thanks! He has been removed by someone else meanwhile. Done. [07:00:55] 10Analytics: Hive log4j logging is misconfigured - https://phabricator.wikimedia.org/T216294 (10elukey) Hello Neil, sorry for this lag in following up. I have tested two things: Stat host (Buster but with hive 1.x deps) ` log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender... [07:06:51] 10Analytics, 10Operations, 10Patch-For-Review, 10User-Elukey: Archival of home directories on servers with very large homes - https://phabricator.wikimedia.org/T215171 (10elukey) 05Open→03Declined Declining this since we have been following another path over the past year and it worked well, will re-op... [07:07:24] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Move https termination from nginx to envoy (if possible) - https://phabricator.wikimedia.org/T240439 (10elukey) [07:08:55] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Remove postal code and longitude / latitude from geocoded data object on webrequest data - https://phabricator.wikimedia.org/T236740 (10elukey) Moving this to done since everything seems already deployed. [08:39:01] 10Analytics-Clusters, 10Operations: Switch Zookeeper to profile::java - https://phabricator.wikimedia.org/T264176 (10elukey) a:05razzi→03None [08:40:24] 10Analytics-Clusters, 10Operations: Switch Zookeeper to profile::java - https://phabricator.wikimedia.org/T264176 (10elukey) a:03elukey Going to takeover the ownership of the task since I need to do some refactoring of some code that I have written :) [08:40:37] 10Analytics-Clusters, 10Analytics-Kanban, 10Operations: Switch Zookeeper to profile::java - https://phabricator.wikimedia.org/T264176 (10elukey) [08:49:56] 10Analytics-Clusters, 10Patch-For-Review: Create a temporary hadoop backup cluster - https://phabricator.wikimedia.org/T260411 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` ['analytics1042.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-r... [09:02:24] 10Analytics: Analytics Presto improvements - https://phabricator.wikimedia.org/T266639 (10elukey) p:05Triage→03Medium [09:02:34] 10Analytics: Analytics Presto improvements - https://phabricator.wikimedia.org/T266639 (10elukey) [09:06:23] 10Analytics: Decide to move or not to PrestoSQL - https://phabricator.wikimedia.org/T266640 (10elukey) [09:10:11] 10Analytics: Test Alluxio as cache layer for Presto - https://phabricator.wikimedia.org/T266641 (10elukey) [09:10:27] 10Analytics-Clusters: Co-locate Presto with Hadoop worker nodes - https://phabricator.wikimedia.org/T256108 (10elukey) 05Open→03Stalled Pending T266641 [09:10:49] 10Analytics: Analytics Presto improvements - https://phabricator.wikimedia.org/T266639 (10elukey) [09:10:51] 10Analytics-Clusters: Co-locate Presto with Hadoop worker nodes - https://phabricator.wikimedia.org/T256108 (10elukey) [09:11:24] 10Analytics, 10Analytics-Kanban, 10serviceops, 10Patch-For-Review, 10User-jijiki: Mechanism to flag webrequests as "debug" - https://phabricator.wikimedia.org/T263683 (10jijiki) @Millimetric, after discussing with @ema, traffic feels that those requests should be visible in turnilo (eg webrequests_sample... [09:14:07] 10Analytics-Clusters, 10Analytics-Kanban, 10Operations: Switch Zookeeper to profile::java - https://phabricator.wikimedia.org/T264176 (10MoritzMuehlenhoff) One gotcha: conf1* is still on jessie (and consequently Java 7), and I don't think anything accounts for Java 7 yet [09:15:14] 10Analytics: Add editors per country data for Wiktionary projects - https://phabricator.wikimedia.org/T266643 (10Pamputt) [09:16:26] 10Analytics, 10Analytics-Kanban, 10serviceops, 10Patch-For-Review, 10User-jijiki: Mechanism to flag webrequests as "debug" - https://phabricator.wikimedia.org/T263683 (10ema) >>! In T263683#6584033, @jijiki wrote: > @Millimetric, after discussing with @ema, traffic feels that those requests should be vis... [09:21:14] 10Analytics, 10Wiktionary: Add editors per country data for Wiktionary projects - https://phabricator.wikimedia.org/T266643 (10Pamputt) [09:24:14] 10Analytics-Clusters, 10Analytics-Kanban, 10Operations: Switch Zookeeper to profile::java - https://phabricator.wikimedia.org/T264176 (10elukey) >>! In T264176#6584036, @MoritzMuehlenhoff wrote: > One gotcha: conf1* is still on jessie (and consequently Java 7), and I don't think anything accounts for Java 7... [09:35:49] 10Analytics-Clusters, 10Patch-For-Review: Create a temporary hadoop backup cluster - https://phabricator.wikimedia.org/T260411 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['analytics1042.eqiad.wmnet'] ` and were **ALL** successful. [09:37:30] 10Analytics-Clusters, 10Patch-For-Review: Create a temporary hadoop backup cluster - https://phabricator.wikimedia.org/T260411 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` ['analytics1043.eqiad.wmnet', 'analytics1044.eqiad.wmnet', 'analytics1045.eq... [10:06:17] 10Analytics-Clusters, 10Patch-For-Review: Create a temporary hadoop backup cluster - https://phabricator.wikimedia.org/T260411 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['analytics1045.eqiad.wmnet', 'analytics1043.eqiad.wmnet'] ` Of which those **FAILED**: ` ['analytics1044.eqiad.wmnet'] ` [10:14:29] 10Analytics-Clusters, 10Operations, 10vm-requests: Create a ganeti VM in eqiad: an-test-ui1001.eqiad.wmnet - https://phabricator.wikimedia.org/T266648 (10elukey) p:05Triage→03Medium [10:19:22] 10Analytics-Clusters, 10Operations, 10vm-requests: Create a ganeti VM in eqiad: an-test-ui1001.eqiad.wmnet - https://phabricator.wikimedia.org/T266648 (10elukey) ` elukey@ganeti1011:~$ sudo gnt-group list Group Nodes Instances AllocPolicy NDParams row_A 4 36 preferred ovs=False, ssh_port=22, o... [10:52:34] 10Analytics-Clusters, 10Patch-For-Review: Create a temporary hadoop backup cluster - https://phabricator.wikimedia.org/T260411 (10elukey) analytics1044 seems to keep PXE booting, so it installs endlessly the OS. I checked the system's setup (reboot + f2) but the hard disk step is configured before the NIC (as... [11:17:39] elukey: so regarding kernel 5.x. I think we can't really test on a Hadoop worker, since they're all on Stretch, and 5.x is already a backport for Buster. I don't think 5.x was backported to Stretch. Which leaves us with 1005 and 1008 for testing [11:18:27] correct yes [11:24:23] On one hand, using 1005 again disrupts the same people, on the other, 1008 seems more popular these days. wdyt? [11:27:49] I think either one is fine, we'll end up impacting people.. what I'd do is to ask Moritz if we have the green light, then announce it to the mailing list [11:27:53] but nothing more [11:30:12] Alrighty [11:39:20] * elukey lunch! [11:43:52] elukey: had a quick chat with Moritz, he's given the green light (mentioning that us testing 5.9 with our stuff is also likely to be useful for Upstream, since they will have to decide on the next LTS kernel (likely 5.10)). [11:44:24] Will send a mail to the usual lists about maintenance on Friday (for stat1005) [11:46:07] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade AMD ROCm drivers/tools to latest upstream - https://phabricator.wikimedia.org/T264408 (10klausman) I had a quick chat with Moritz about the kernel version/rocm siutuation, and we agreed that we'd test 5.8.0 (a backport to Buster) on stat10... [12:09:16] 10Analytics: Check home/HDFS leftovers of rodolfovalentim - https://phabricator.wikimedia.org/T266467 (10diego) @Rvvalentim , please can you double check if you need any of those files? [12:14:08] 10Analytics: Check home/HDFS leftovers of rodolfovalentim - https://phabricator.wikimedia.org/T266467 (10Rvvalentim) Hi, I don't need any of those files. From my end, you can drop it all. [13:31:04] helloooo elukey you wanna look at the thorium files? [13:33:41] hellooo I just got back, ok in 10 mins fdans ? [13:33:49] elukey: sure! [13:35:58] <3 [13:41:33] milimetric: got a couple mins to brain bounce some nodejs/npm optional dependency stuff with me? [13:42:05] https://docs.npmjs.com/cli/v6/configuring-npm/package-json#optionaldependencies [13:42:16] https://github.com/wikimedia/eventgate/pull/11/files [13:42:43] ottomata: yes, one second [13:43:46] ok, omw cave [13:43:57] fdans: all right I am ready, tardis? [13:44:26] elukey: what's the url? [13:45:01] https://meet.google.com/kti-iybt-ekv [14:02:47] 10Analytics-Radar, 10Operations, 10SRE-Access-Requests: Nuria's volunteer account - https://phabricator.wikimedia.org/T266086 (10gsingers) @MoritzMuehlenhoff Thanks! Approved. [14:39:29] ottomata: hi! which is the CR that you were asking yesterday? [14:41:56] 10Analytics-Radar, 10Operations, 10SRE-Access-Requests: Nuria's volunteer account - https://phabricator.wikimedia.org/T266086 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff Excellent, thanks! Closing this task since everything is completed now. I'll merge https://gerrit.wikimedia.org/r/c... [14:42:50] 10Analytics, 10Wiktionary: Add editors per country data for Wiktionary projects - https://phabricator.wikimedia.org/T266643 (10Pamputt) [14:47:04] 10Analytics: Hive log4j logging is misconfigured - https://phabricator.wikimedia.org/T216294 (10nshahquinn-wmf) >>! In T216294#6583801, @elukey wrote: > Do you think that this can wait our upgrade to Bigtop? (that includes hive 2.2.3) ? Yes, of course. I know you have many bigger things to worry about, so I'm h... [14:47:27] 10Analytics-Radar, 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Search (Current work): PoC on anomaly detection with Flink - https://phabricator.wikimedia.org/T262942 (10dcausse) Yes it definitely can support such queries e.g (extract all api requests from mediawiki.apiaction grouped by their action p... [14:48:09] mforns: its ok Pchelolo reviewed it! [14:48:17] oh, sorry... [14:48:48] no prob! [14:48:51] he got to it before you did :) [14:51:40] ottomata: o/ - since netbox now handles IPAM, creating VMs now can be done directly from the cookbook without DNS allocation :O [14:51:48] (I am doing one now) [14:52:03] it handles both ipv4 and ipv6 [14:52:11] NIIIIIICE! [14:52:13] so cool' [14:53:42] an-test-ui1001.eqiad.wmnet [14:53:47] for yarn and hue [14:58:39] a-team, someone java to save me from misery? [14:58:54] :) [14:58:56] sure mforns [14:58:58] what's up [14:59:04] hey milimetric :] [14:59:31] milimetric: I think bc is better, you ok? [14:59:34] omw [14:59:36] ok [15:05:57] * elukey coffee [15:24:39] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Privacy Engineering, and 4 others: Remove http.client_ip from EventGate default schema (again) - https://phabricator.wikimedia.org/T262626 (10CDanis) >>! In T262626#6572889, @Krinkle wrote: > I suppose that could indeed be addressed in various ways, eith... [15:26:47] 10Analytics: Check home/HDFS leftovers of rodolfovalentim - https://phabricator.wikimedia.org/T266467 (10elukey) @diego green light to drop then? [15:33:44] 10Analytics: Check home/HDFS leftovers of rodolfovalentim - https://phabricator.wikimedia.org/T266467 (10diego) @elukey, yes. ` $ rm -R -f /* ` ;) [15:59:47] (03CR) 10Mforns: Adding quality alarms for mobile app data (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/633579 (https://phabricator.wikimedia.org/T257692) (owner: 10Nuria) [16:42:21] Hello all! I have two cloud-vps questions. [16:43:04] 1) Is the 'analytics' project still in use? If so someone needs to claim it on https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2020_Purge — and subscribe to the cloud-announce list so that I won't have to ask you on IRC next year :) [16:43:42] 2) Assuming the project is still in use… can I safely delete any VMs prefixed with k4 since that (I assume) refers to kate who hasn't worked here in ages? [16:52:26] 10Analytics, 10Product-Analytics: Analyze differences between checksum-based and revert-tag based reverts in mediawiki_history - https://phabricator.wikimedia.org/T266374 (10nettrom_WMF) @JAllemandou : Yes, and I'm expecting to see some checksum-based reverts not having the tag because the tag only checks the... [17:01:19] andrewbogott: deleted them, k there stood for kafka [17:01:31] haven't used them in a while [17:01:31] ah, ok [17:01:36] cloud-announce [17:01:50] can we put Internal team communication for the Analytics team on it ? [17:02:39] probably, it's just a standard mailing list [17:02:39] https://lists.wikimedia.org/mailman/listinfo/cloud-announce [17:05:15] 10Analytics, 10Analytics-Kanban, 10Privacy Engineering, 10Product-Analytics, and 3 others: Drop data from Prefupdate schema that is older than 90 days - https://phabricator.wikimedia.org/T250049 (10nettrom_WMF) @Milimetric : Thanks for clarifying that, and for your patience while I got back on this! I chat... [18:05:50] (03PS6) 10Nuria: Adding quality alarms for mobile app data [analytics/refinery] - 10https://gerrit.wikimedia.org/r/633579 (https://phabricator.wikimedia.org/T257692) [18:07:58] (03CR) 10Nuria: Adding quality alarms for mobile app data (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/633579 (https://phabricator.wikimedia.org/T257692) (owner: 10Nuria) [18:15:18] (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGTM!" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/633579 (https://phabricator.wikimedia.org/T257692) (owner: 10Nuria) [18:29:41] milimetric: you know more about mailing list stuff than me, can we add analytics-internal@lists.wikimedia.org to cloud-announce? [18:29:43] https://lists.wikimedia.org/mailman/listinfo/cloud-announce [18:32:18] * elukey afk! [18:36:13] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Add data quality alarm for mobile-app data - https://phabricator.wikimedia.org/T257692 (10Nuria) Code merged now, when the entropy counts are re run alarms for may18th will be resend. [18:47:15] (afk for a bit...time to get contacts!) [20:03:03] I have no idea, ottomata, I don't think so... but everyone can just subscribe, cloud announce is always interested [20:15:06] 10Analytics: Check data currently stored on thorium and drop what it is not needed anymore - https://phabricator.wikimedia.org/T265971 (10elukey) @milimetric the data to review (if you have time) is the one under https://analytics.wikimedia.org/published/datasets/archive/public-datasets/, especially: ` root@tho... [21:00:28] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Patch-For-Review: Make node-rdkafka an optional dependency of EventGate - https://phabricator.wikimedia.org/T266058 (10Ottomata) Ok, I did some work today to factory out our async Kafka factory wrapper from eventgate into its own library. This makes it... [21:02:30] ottomata: no rush but did you see my last comment on T262626 ? [21:02:31] T262626: Remove http.client_ip from EventGate default schema (again) - https://phabricator.wikimedia.org/T262626 [21:12:50] * razzi afk for a bit [21:13:41] cdanis: ah yes did see [21:14:10] no rush so i will respond later! we shoudl discuss and see where you want it [21:14:18] gotta go pick up my car from the shop rn, back in a bit [21:14:47] ok!