[03:38:16] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [07:38:16] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [09:37:06] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Enable HA for the mw-content-history-reconcile-enrich flink application - https://phabricator.wikimedia.org/T375176#10415006 (10tchin) [11:38:16] FIRING: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [12:06:38] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Research, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 03Discovery-Search (Current work): Low available space on Hadoop / HDFS - https://phabricator.wikimedia.org/T381707#10415329 (10BTullis) We have now added about 5% to the total capacity by a... [12:16:28] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Research, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 03Discovery-Search (Current work): Low available space on Hadoop / HDFS - https://phabricator.wikimedia.org/T381707#10415335 (10BTullis) [12:18:52] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Research, 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 03Discovery-Search (Current work): Low available space on Hadoop / HDFS - https://phabricator.wikimedia.org/T381707#10415339 (10BTullis) See {T382372} for the discussion with #dc-ops about a... [13:42:08] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10LPL Analytics, 06Product-Analytics: Invalid EventGate errors with content_translation_event 1.7.0 - https://phabricator.wikimedia.org/T382113#10415577 (10Ahoelzl) [13:43:18] !log Purging data rentained for more than 90 days after backfill is done [13:43:20] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:48:16] RESOLVED: HdfsCapacityRemainingPercent: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [13:48:34] \o/ --^ [14:14:47] {◕ ◡ ◕} [15:58:41] nice! [16:54:15] 10Data-Engineering (Q2 2024 October 1st - December 31th): Gitlab CI/CD Component for Blunderbuss - https://phabricator.wikimedia.org/T382348#10416529 (10BTullis) @amastilovic - referring back to the [[https://docs.google.com/document/d/1gytt1rzO5wO1IWmmrGKdfuwSlVvFyQ9pFK87wePUYFE/edit?tab=t.0#heading=h.8pacrt3kt... [17:22:50] 06Data-Engineering, 06Data Products, 06Traffic, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#10416668 (10Krinkle) [18:00:51] 06Data-Engineering, 06Growth-Team, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaEvents, and 7 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10416813 (10Krinkle) >>! In T355837#10147260, @Ottomata wrote: > Oh, it's POST? With a POST body?... [18:21:31] (03PS6) 10Mforns: Edit Geoeditors Daily Monthly to support Temp Account Changes [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1098083 (https://phabricator.wikimedia.org/T379728) (owner: 10Jennifer Ebe) [18:22:48] (03CR) 10Mforns: [V:03+2] Edit Geoeditors Daily Monthly to support Temp Account Changes (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1098083 (https://phabricator.wikimedia.org/T379728) (owner: 10Jennifer Ebe) [18:23:07] 06Data-Engineering, 06Growth-Team, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaEvents, and 7 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10416866 (10Ottomata) Thank you so much Timo for the details! > Punt new use cases (A: join pro... [18:34:54] (03PS5) 10Mforns: Add New Edit Hourly HQL for Temp Account Change [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084873 (https://phabricator.wikimedia.org/T377767) (owner: 10Jennifer Ebe) [18:35:04] (03CR) 10Mforns: Add New Edit Hourly HQL for Temp Account Change (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084873 (https://phabricator.wikimedia.org/T377767) (owner: 10Jennifer Ebe) [18:36:20] (03PS6) 10Mforns: Add New Edit Hourly HQL for Temp Account Change [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084873 (https://phabricator.wikimedia.org/T377767) (owner: 10Jennifer Ebe) [18:36:32] (03CR) 10Mforns: Add New Edit Hourly HQL for Temp Account Change (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084873 (https://phabricator.wikimedia.org/T377767) (owner: 10Jennifer Ebe) [18:36:55] (03CR) 10Mforns: Edit Geoeditors Daily Monthly to support Temp Account Changes (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1098083 (https://phabricator.wikimedia.org/T379728) (owner: 10Jennifer Ebe) [18:38:26] (03PS7) 10Mforns: Add New Edit Hourly HQL for Temp Account Change [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084873 (https://phabricator.wikimedia.org/T377767) (owner: 10Jennifer Ebe) [19:09:54] 06Data-Engineering, 06Growth-Team, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaEvents, and 7 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10416977 (10Ottomata) Now on to some technical comments / clarifications :) > Compatibility: Supp... [19:10:39] 06Data-Engineering, 06Growth-Team, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaEvents, and 7 others: Add Prometheus support to statsd.js via mw.track() - https://phabricator.wikimedia.org/T355837#10416983 (10Ottomata) > If the use case to join with data lake has a high priority, I suggest pull... [20:30:03] (03CR) 10Mforns: [V:03+2] Add New Edit Hourly HQL for Temp Account Change [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1084873 (https://phabricator.wikimedia.org/T377767) (owner: 10Jennifer Ebe) [20:59:10] (03CR) 10Mforns: Update Geoeditors Monthly to support Temp Accounts. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1092916 (https://phabricator.wikimedia.org/T379769) (owner: 10Snwachukwu) [20:59:28] (03CR) 10Mforns: [V:03+2 C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1092916 (https://phabricator.wikimedia.org/T379769) (owner: 10Snwachukwu) [21:01:50] (03CR) 10Mforns: [C:03+1] "LGTM! Let's test and then merge!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1099006 (https://phabricator.wikimedia.org/T379772) (owner: 10Snwachukwu)