[10:18:37] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Discovery-Search, 10Data-Platform-SRE (2025.02.10 - 2025.02.28), 13Patch-For-Review: Upload an image with flink-k8s-operator version that supports flink 1.20 - https://phabricator.wikimedia.org/T377137#10558924 (10brouberol) 05Open→03In progress... [13:14:58] 10Data-Engineering (Q3 2024 January 1st - March 31th): Update eventstreams Grafana Dashboards to fix GC metrics - https://phabricator.wikimedia.org/T386709 (10tchin) 03NEW [13:15:14] 06Data-Engineering, 10ChangeProp, 10observability, 10service-runner, 10Event-Platform: Upgrade prom-client in NodeJS service-runner and enable collectDefaultMetrics - https://phabricator.wikimedia.org/T350180#10559448 (10tchin) [13:15:14] 10Data-Engineering (Q3 2024 January 1st - March 31th): Update eventstreams Grafana Dashboards to fix GC metrics - https://phabricator.wikimedia.org/T386709#10559449 (10tchin) [13:26:00] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Discovery-Search, 10Data-Platform-SRE (2025.02.10 - 2025.02.28), 13Patch-For-Review: Upload an image with flink-k8s-operator version that supports flink 1.20 - https://phabricator.wikimedia.org/T377137#10559480 (10brouberol) 05In progress→03Re... [13:40:40] 10Data-Engineering (Q3 2024 January 1st - March 31th), 13Patch-For-Review: Publish Data Engineering maintained NodeJS packages to GitLab and use them in depender code - https://phabricator.wikimedia.org/T366612#10559564 (10tchin) [17:05:29] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Experimentation Lab, 06Product-Analytics: Draft of full process for instrumentation using new client libraries - https://phabricator.wikimedia.org/T275694#10560277 (10Mayakp.wiki) [17:32:09] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: Upgrade eventgate-wikimedia to node20 - https://phabricator.wikimedia.org/T383814#10560408 (10Ottomata) eventgate-analytics in codfw is upgraded to node20. I'll let this simmer for at least a day before proceeding i... [17:37:01] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Discovery-Search, 10Dumps 2.0, 10Data-Platform-SRE (2025.02.10 - 2025.02.28), 13Patch-For-Review: Add relevant kafka clusters to defined airflow connections in puppet - https://phabricator.wikimedia.org/T379676#10560416 (10xcollazo) >>! In T379676#... [17:57:01] 06Data-Engineering, 03Abstract Wikipedia Fix-It tasks, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 13Patch-For-Review: WikiLambda metrics: Repair the allowlist for mediawiki_product_metrics_wikilambda_api; backfill missing data as needed - https://phabricator.wikimedia.org/T384876#10560513 (10VirginiaPounds... [17:57:11] 06Data-Engineering, 03Abstract Wikipedia Fix-It tasks, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 13Patch-For-Review: WikiLambda metrics: Repair the allowlist for mediawiki_product_metrics_wikifunctions_ui; backfill missing data as needed - https://phabricator.wikimedia.org/T384531#10560514 (10VirginiaPoun... [17:58:47] (03PS1) 10Xcollazo: Add partition on source_table to data quality tables. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1120614 (https://phabricator.wikimedia.org/T386114) [18:15:10] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users ssh access and Kerberos identity for YLiou_WMF - https://phabricator.wikimedia.org/T385220#10560603 (10Ottomata) [18:15:41] 06Data-Engineering, 06Data-Engineering-Radar, 06Growth-Team, 10GrowthExperiments, and 4 others: mw.track: support for histogram metrics - https://phabricator.wikimedia.org/T383563#10560608 (10Ottomata) [18:19:08] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 06Campaigns-Product-Team, and 3 others: Convert `ce_participants.cep_private` from boolean to mwtinyint - https://phabricator.wikimedia.org/T383777#10560626 (10Ottomata) [18:19:32] 06Data-Engineering, 06Data-Engineering-Radar, 10Data-Services, 06DBA, 06Privacy Engineering: Create views for SecurePoll db tables in Toolforge replicas - https://phabricator.wikimedia.org/T381197#10560629 (10Ottomata) [18:25:47] 06Data-Engineering: [Maintenance] Reduce number of HDFS files - https://phabricator.wikimedia.org/T347975#10560663 (10Ottomata) @BTullis @JAllemandou what should we do with this task now? [18:26:21] 06Data-Engineering: Fix DPE alerts dashboard to work with Google Groups - https://phabricator.wikimedia.org/T365829#10560665 (10Ottomata) @Ahoelzl should we decline this task? Are we going to do this? [18:27:00] 06Data-Engineering: Update Airflow Developer Guide on WikiTech with instructions for using the DagProperties - https://phabricator.wikimedia.org/T365658#10560668 (10Ottomata) [18:27:38] 06Data-Engineering, 10Data Pipelines: Update Airflow Developer Guide on WikiTech with instructions for using the DagProperties - https://phabricator.wikimedia.org/T365658#10560669 (10Ottomata) [18:39:10] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10DPE-Data-Platform-related-Mediawiki-Content-data, 10Data-Platform (Data Platform Ops Week Working Group), 10Data-Platform-SRE (2025.02.10 - 2025.02.28), and 2 others: DAG failing due to failure to ac... - https://phabricator.wikimedia.org/T386114#10560698 [18:42:27] 10Data-Engineering-Roadmap, 10Dumps 2.0, 10Discovery-Search (2025.02.10 - 2025.02.28), 07Epic, 13Patch-For-Review: EPIC: Update flink jobs to support Flink 1.20 - https://phabricator.wikimedia.org/T376812#10560705 (10Ahoelzl) [18:47:07] 10Data-Engineering (Q3 2024 January 1st - March 31th), 03Abstract Wikipedia Fix-It tasks, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 13Patch-For-Review: WikiLambda metrics: Repair the allowlist for mediawiki_product_metrics_wikilambda_api; backfill missin... - https://phabricator.wikimedia.org/T384876#10560723 [18:47:11] 10Data-Engineering (Q3 2024 January 1st - March 31th), 03Abstract Wikipedia Fix-It tasks, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 13Patch-For-Review: WikiLambda metrics: Repair the allowlist for mediawiki_product_metrics_wikifunctions_ui; backfill miss... - https://phabricator.wikimedia.org/T384531#10560725 [18:50:47] 10Data-Engineering (Q3 2024 January 1st - March 31th), 06Release-Engineering-Team, 10GitLab (Administration, Settings & Policy): Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10560742 (10Ahoelzl) [18:50:54] 06Data-Engineering, 06Release-Engineering-Team, 10GitLab (Administration, Settings & Policy): Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10560744 (10Ahoelzl) [18:52:23] 06Data-Engineering, 10Data Pipelines, 10Data-Catalog: Upgrade to Spark 3.2 to support Spark lineage for Iceberg tables - https://phabricator.wikimedia.org/T378899#10560751 (10Ahoelzl) Need to clarify how this gets addressed with Hadoop 3 upgrade. [18:53:48] 06Data-Engineering, 10Dumps 2.0, 07Epic: Productionization of code to dump in XML - https://phabricator.wikimedia.org/T384382#10560753 (10Ottomata) [18:53:52] 10Data-Engineering-Roadmap, 10Dumps 2.0, 07Epic: Productionization of code to dump in XML - https://phabricator.wikimedia.org/T384382#10560756 (10Ahoelzl) [18:54:27] 06Data-Engineering, 07Sustainability: Move some analytics jobs to day time in Virginia - https://phabricator.wikimedia.org/T384166#10560758 (10Ottomata) [18:55:45] 10Data-Engineering (Q3 2024 January 1st - March 31th): [sqoop] Check if a sqoop needs changes when one mariadb field changes from int to smallint - https://phabricator.wikimedia.org/T383803#10560764 (10Ahoelzl) [18:55:51] 10Data-Engineering (Q3 2024 January 1st - March 31th): [sqoop] Check if a sqoop needs changes when one mariadb field changes from int to smallint - https://phabricator.wikimedia.org/T383803#10560765 (10Ahoelzl) a:03mforns [18:56:18] 10Data-Engineering (Q3 2024 January 1st - March 31th): Some search entries in wmf.webrequest have their query appended to their uri_path - https://phabricator.wikimedia.org/T383135#10560779 (10Ahoelzl) [18:56:47] 10Data-Engineering (Q3 2024 January 1st - March 31th): Some search entries in wmf.webrequest have their query appended to their uri_path - https://phabricator.wikimedia.org/T383135#10560781 (10Ahoelzl) a:03Antoine_Quhen [18:58:01] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Dumps-Generation: Wikimedia Downloads not complete - https://phabricator.wikimedia.org/T383030#10560799 (10Ottomata) [19:08:06] PROBLEM - Webrequests Varnishkafka log producer on cp5019 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:08:06] PROBLEM - statsv Varnishkafka log producer on cp5024 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:08:07] PROBLEM - Webrequests Varnishkafka log producer on cp5024 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:08:58] PROBLEM - Webrequests Varnishkafka log producer on cp3071 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:08:58] PROBLEM - statsv Varnishkafka log producer on cp3066 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:00] PROBLEM - Webrequests Varnishkafka log producer on cp3066 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:04] PROBLEM - statsv Varnishkafka log producer on cp5023 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:04] PROBLEM - Webrequests Varnishkafka log producer on cp5020 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:06] PROBLEM - Webrequests Varnishkafka log producer on cp5023 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:07] PROBLEM - statsv Varnishkafka log producer on cp5020 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:09] PROBLEM - statsv Varnishkafka log producer on cp5019 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:22] PROBLEM - Webrequests Varnishkafka log producer on cp3068 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:30] PROBLEM - statsv Varnishkafka log producer on cp3071 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:30] PROBLEM - Webrequests Varnishkafka log producer on cp3073 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:32] PROBLEM - statsv Varnishkafka log producer on cp3068 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:33] PROBLEM - statsv Varnishkafka log producer on cp3069 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:35] PROBLEM - Webrequests Varnishkafka log producer on cp3067 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:36] PROBLEM - statsv Varnishkafka log producer on cp3067 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:58] PROBLEM - statsv Varnishkafka log producer on cp3073 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:09:58] PROBLEM - Webrequests Varnishkafka log producer on cp3069 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:10:04] PROBLEM - statsv Varnishkafka log producer on cp5022 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:10:04] RECOVERY - Webrequests Varnishkafka log producer on cp5020 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:10:06] PROBLEM - Webrequests Varnishkafka log producer on cp5022 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:10:07] RECOVERY - statsv Varnishkafka log producer on cp5024 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:10:09] RECOVERY - Webrequests Varnishkafka log producer on cp5024 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:11:06] RECOVERY - statsv Varnishkafka log producer on cp5022 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:11:06] RECOVERY - Webrequests Varnishkafka log producer on cp5022 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:11:10] PROBLEM - Webrequests Varnishkafka log producer on cp5017 is CRITICAL: PROCS CRITICAL: 3 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:12:06] RECOVERY - Webrequests Varnishkafka log producer on cp5017 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:12:20] RECOVERY - Webrequests Varnishkafka log producer on cp3068 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:12:30] RECOVERY - statsv Varnishkafka log producer on cp3068 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:12:58] RECOVERY - Webrequests Varnishkafka log producer on cp3069 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:13:04] PROBLEM - Webrequests Varnishkafka log producer on cp5020 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:13:30] RECOVERY - Webrequests Varnishkafka log producer on cp3067 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:13:30] RECOVERY - statsv Varnishkafka log producer on cp3069 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:13:32] RECOVERY - statsv Varnishkafka log producer on cp3067 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:15:04] RECOVERY - Webrequests Varnishkafka log producer on cp5019 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:15:04] RECOVERY - statsv Varnishkafka log producer on cp5019 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:16:58] RECOVERY - statsv Varnishkafka log producer on cp3066 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:16:58] RECOVERY - Webrequests Varnishkafka log producer on cp3066 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:21:48] (03CR) 10Mforns: [C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114486 (https://phabricator.wikimedia.org/T384876) (owner: 10David Martin) [19:23:02] (03CR) 10Mforns: [V:03+2 C:03+2] Repair the allowlist for mediawiki_product_metrics_wikilambda_api [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114486 (https://phabricator.wikimedia.org/T384876) (owner: 10David Martin) [19:25:06] RECOVERY - statsv Varnishkafka log producer on cp5023 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:25:06] RECOVERY - Webrequests Varnishkafka log producer on cp5023 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:25:11] (03CR) 10Mforns: [V:03+2 C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1114485 (https://phabricator.wikimedia.org/T384531) (owner: 10David Martin) [19:29:04] RECOVERY - Webrequests Varnishkafka log producer on cp5020 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:29:04] RECOVERY - statsv Varnishkafka log producer on cp5020 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:33:30] RECOVERY - Webrequests Varnishkafka log producer on cp3073 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:33:40] 06Data-Engineering: Migrate analytics Airflow DAGs to k8s Airflow deployment - https://phabricator.wikimedia.org/T386282#10560997 (10amastilovic) > Another variation on this idea: we make a new airflow 'instance' in k8s. Not sure what to call it, but maybe foundational (meaning 'foundational datasets') or 'main'... [19:33:58] RECOVERY - statsv Varnishkafka log producer on cp3073 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:34:58] RECOVERY - Webrequests Varnishkafka log producer on cp3071 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:35:30] RECOVERY - statsv Varnishkafka log producer on cp3071 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [19:44:19] (03CR) 10Aleksandar Mastilovic: [C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1120614 (https://phabricator.wikimedia.org/T386114) (owner: 10Xcollazo) [20:02:20] (03PS1) 10Xcollazo: Delete dead code that used to generate wmf.wikidata_item_page_link. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1120647 (https://phabricator.wikimedia.org/T300023) [20:02:51] (03CR) 10Xcollazo: "CC @aquhen@wikimedia.org" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1120647 (https://phabricator.wikimedia.org/T300023) (owner: 10Xcollazo) [20:19:29] 06Data-Engineering: Migrate analytics Airflow DAGs to k8s Airflow deployment - https://phabricator.wikimedia.org/T386282#10561166 (10Ottomata) > If we're OK with this approach I will leave it up to you all to choose, all the approaches have pros and cons and yall will know best I think. > I also like the new n... [20:28:11] 06Data-Engineering: [eventstreams] Fix event detail yaml error - https://phabricator.wikimedia.org/T386750 (10mforns) 03NEW [20:36:48] 10Data-Engineering (Q3 2024 January 1st - March 31th): Provide a list of Phabricator tags relevant for Data Engineering - https://phabricator.wikimedia.org/T386752 (10Ahoelzl) 03NEW [20:42:40] (03CR) 10TChin: [C:03+2] Add partition on source_table to data quality tables. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1120614 (https://phabricator.wikimedia.org/T386114) (owner: 10Xcollazo) [21:01:42] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Growth-Structured-Tasks, 06Growth-Team, 10Image-Suggestions, and 6 others: wmf.wikidata_item_page_link and wmf.wikidata_entity snapshots stuck at 2025-01-20 - https://phabricator.wikimedia.org/T386255#10561314 (10xcollazo) I did a cursory check on th... [21:03:11] (03CR) 10Xcollazo: [V:03+2 C:03+2] Add partition on source_table to data quality tables. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1120614 (https://phabricator.wikimedia.org/T386114) (owner: 10Xcollazo) [21:30:08] 10Data-Engineering (Q3 2024 January 1st - March 31th): List out all migration candidates for mediawiki_content_history - https://phabricator.wikimedia.org/T386757 (10Ahoelzl) 03NEW [21:35:42] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10DPE-Data-Platform-related-Mediawiki-Content-data, 10Data-Platform (Data Platform Ops Week Working Group), 10Data-Platform-SRE (2025.02.10 - 2025.02.28), and 2 others: DAG failing due to failure to ac... - https://phabricator.wikimedia.org/T386114#10561391 [21:57:18] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10DPE-Data-Platform-related-Mediawiki-Content-data, 10Data-Platform (Data Platform Ops Week Working Group), 10Data-Platform-SRE (2025.02.10 - 2025.02.28), and 2 others: DAG failing due to failure to ac... - https://phabricator.wikimedia.org/T386114#10561463 [22:11:57] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Optimize XML Dump code to be able to handle wikis from simplewiki to enwiki - https://phabricator.wikimedia.org/T381016#10561483 (10Ahoelzl) [22:12:04] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Dumps 2.0, 13Patch-For-Review: Refactor code to use new table and column names - https://phabricator.wikimedia.org/T384385#10561484 (10Ahoelzl) [22:12:13] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Dumps 2.0, 13Patch-For-Review: Modify XML dumping code to be able to do 'partial' dumps - https://phabricator.wikimedia.org/T384383#10561485 (10Ahoelzl) [22:12:49] 10Data-Engineering (Q3 2024 January 1st - March 31th): List out all migration candidates for mediawiki_content_history - https://phabricator.wikimedia.org/T386757#10561488 (10Ahoelzl) This will inform KR 1.4 https://docs.google.com/document/d/1kGqINecJ9CQPdeVCiRsoc7-HRfgDhim5z1VHBZL6Zy0/edit?tab=t.19m2dicg3c8e [22:13:51] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10DPE-Data-Platform-related-Mediawiki-Content-data, 10Data-Platform (Data Platform Ops Week Working Group), 10Data-Platform-SRE (2025.02.10 - 2025.02.28), and 2 others: DAG failing due to failure to ac... - https://phabricator.wikimedia.org/T386114#10561493 [22:15:02] 10Data-Engineering (Q3 2024 January 1st - March 31th), 13Patch-For-Review: Timeout hive-metastore locks - https://phabricator.wikimedia.org/T365563#10561505 (10Ahoelzl) Related https://phabricator.wikimedia.org/T386114 [22:20:07] 10Data-Engineering (Q3 2024 January 1st - March 31th): Provide a list of Phabricator tags relevant for Data Engineering - https://phabricator.wikimedia.org/T386752#10561520 (10Ahoelzl) [22:28:15] 10Data-Engineering-Roadmap, 07Epic: All things DataHub - https://phabricator.wikimedia.org/T369756#10561567 (10Ahoelzl) [22:28:21] 10Data-Engineering-Roadmap, 07Epic: Data Platform Data Lineage - https://phabricator.wikimedia.org/T377789#10561569 (10Ahoelzl) [22:50:15] 10Data-Engineering (Q3 2024 January 1st - March 31th): Provide a list of Phabricator tags relevant for Data Engineering - https://phabricator.wikimedia.org/T386752#10561662 (10Ahoelzl) **Data Engineering** - Data Pipelines - Data-Catalog - Dumps-Generation - Dumps 2.0 (deprecated) - Event-Platform -... [22:50:45] 10Data-Engineering (Q3 2024 January 1st - March 31th): Provide a list of Phabricator tags relevant for Data Engineering - https://phabricator.wikimedia.org/T386752#10561665 (10Ahoelzl) @Ottomata can you check if this makes sense, needs re-grouping? [23:45:08] 10Analytics-Canonical-Data, 06Data-Engineering, 06Data-Engineering-Icebox, 06Movement-Insights: Automate the loading of canonical data tables to the Data Lake - https://phabricator.wikimedia.org/T339928#10561822 (10nshahquinn-wmf) [23:53:27] 06Data-Engineering, 06Java-Scala-Standardization: Resolve conflict between GitLab CI automated package deployment token variable names - https://phabricator.wikimedia.org/T386056#10561864 (10amastilovic) For sake of documentation, I wanted to note that the `CI_RELEASE_TOKEN` variable is not created automagical...