[00:31:02] 14Analytics, 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Wikistats: Wiki Stats: Problem Detecting (unique) Desktop Devices since August 2025? - https://phabricator.wikimedia.org/T425359#12585546 (10Hghani) Some observations around this: **de.wikipedia.org** If we look at the charts b... [00:34:31] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list for October 2026 - https://phabricator.wikimedia.org/T442978#12585552 (10GFontenelle_WMF) 05Open→03Resolved [00:51:01] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): growthbook: Attribute GrowthBook Presto queries from the Presto query log - https://phabricator.wikimedia.org/T445603 (10amastilovic) 03NEW [00:54:20] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): growthbook: Attribute GrowthBook Presto queries from the Presto query log - https://phabricator.wikimedia.org/T445603#12585626 (10amastilovic) == What GrowthBook tells Presto about its queries, from the deployment config and the source code == This i... [01:13:36] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): superset: Analyze the existing Superset event log and SQL Lab query history - https://phabricator.wikimedia.org/T442960#12585685 (10amastilovic) == What Superset records, from the deployment config and the source code == I don't have access to the me... [01:38:44] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): presto: Split resource groups into Superset dashboards, Superset SQL Lab and GrowthBook, and rebalance allocations - https://phabricator.wikimedia.org/T445606 (10amastilovic) 03NEW [03:33:21] FIRING: [4x] WebrequestPageviewHighKafkaConsumerLag: ... [03:33:21] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [03:58:21] FIRING: [3x] WebrequestPageviewHighKafkaConsumerLag: ... [03:58:21] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [06:31:29] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307#12586003 (10Marostegui) [07:58:36] FIRING: [2x] WebrequestPageviewHighKafkaConsumerLag: ... [07:58:36] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [08:01:02] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): KAPOW initial backfill - https://phabricator.wikimedia.org/T442233#12586134 (10mforns) [08:23:16] 14Analytics, 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Wikistats: Wiki Stats: Problem Detecting (unique) Desktop Devices since August 2025? - https://phabricator.wikimedia.org/T425359#12586160 (10Polimorph) Thank you very much @Hghani for this very helpful analysis and report helping... [09:32:22] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10DPE-Mediawiki-Content: refactor consistency_check module & lint repo - https://phabricator.wikimedia.org/T445318#12586360 (10APizzata-WMF) [09:32:52] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10DPE-Mediawiki-Content: refactor consistency_check module & lint repo - https://phabricator.wikimedia.org/T445318#12586361 (10APizzata-WMF) as discussed in https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-content-pipelines/-/merge_requ... [09:37:43] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Collaboration-Services, 10Phabricator, 13Patch-For-Review: data lake: phabricator events - https://phabricator.wikimedia.org/T434047#12586366 (10Antoine_Quhen) Thanks all for the input. This is where we are before merging and running full back... [09:44:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:44:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:44:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [09:49:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [09:49:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [09:49:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [10:08:21] FIRING: [2x] WebrequestPageviewHighKafkaConsumerLag: ... [10:08:21] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [10:10:44] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE: presto: Split resource groups into Superset dashboards, Superset SQL Lab and GrowthBook, and rebalance allocations - https://phabricator.wikimedia.org/T445606#12586439 (10BTullis) [10:11:17] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE: growthbook: Attribute GrowthBook Presto queries from the Presto query log - https://phabricator.wikimedia.org/T445603#12586441 (10BTullis) [10:16:51] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Data Warehouse: Measure interactive query performance for Superset and Growthbook - https://phabricator.wikimedia.org/T442959#12586462 (10BTullis) I'm tagging #data-platform-sre on all of these tickets, as I believe that we will be involved, at least... [10:30:54] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE: presto: Export per-resource-group metrics to Prometheus - https://phabricator.wikimedia.org/T442961#12586519 (10BTullis) [10:31:10] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE: superset: Use separate Presto connections for dashboards and SQL Lab - https://phabricator.wikimedia.org/T442963#12586525 (10BTullis) [10:31:41] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE: superset: Enable Superset application metrics (StatsdStatsLogger) - https://phabricator.wikimedia.org/T442964#12586535 (10BTullis) [10:32:18] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE: Data Warehouse: Privacy and retention review for query logs - https://phabricator.wikimedia.org/T442965#12586539 (10BTullis) [10:32:40] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE: presto: Add a query-completion event listener that sends events to Event Platform - https://phabricator.wikimedia.org/T442966#12586542 (10BTullis) [10:34:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [10:34:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [10:34:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [10:39:02] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work, 13Patch-For-Review: [Infra] Carry out end-user testing of spark on kubernetes - https://phabricator.wikimedia.org/T412925#12586577 (10Gehel) [10:39:03] 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 13Patch-For-Review: Upgrade to JupyterLab ≥ 4.2 - https://phabricator.wikimedia.org/T391905#12586579 (10Gehel) [10:39:09] 06Data-Engineering, 06Data-Engineering-Radar, 10Data Pipelines, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 13Patch-For-Review: Install jupyterhub separately from conda-analytics - https://phabricator.wikimedia.org/T321512#12586581 (10Gehel) [10:39:17] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Platform, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12586585 (10Gehel) [10:39:22] 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Platform-SRE (2026-11-20 - 2026-12-11): Jupyter spawner's list of Conda environments only updates when an environment is spawned - https://phabricator.wikimedia.org/T391894#12586591 (10Gehel) [10:39:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [10:39:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [10:39:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [10:39:43] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Define intake workflow for WMCS ingress logs - https://phabricator.wikimedia.org/T441140#12586602 (10Gehel) [10:39:49] 06Data-Engineering, 05acl*data-engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work, 13Patch-For-Review: Upgrade airflow-main to 3.3.x - https://phabricator.wikimedia.org/T435480#12586599 (10Gehel) [10:40:03] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE (2026-11-20 - 2026-12-11): Optimize enqueueing of refine_webrequest_hourly pipeline - https://phabricator.wikimedia.org/T419050#12586608 (10Gehel) [10:40:17] 06Data-Engineering, 06Data-Engineering-Radar, 06Privacy Engineering, 06Security-Team, and 3 others: Privacy review of x1 tables in preparation of adding them to wikireplicas - https://phabricator.wikimedia.org/T415219#12586614 (10Gehel) [10:40:23] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 4 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#12586612 (10Gehel) [10:40:56] 06Data-Engineering, 06Data-Engineering-Radar, 06cloud-services-team, 06Data-Persistence, and 4 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#12586624 (10Gehel) [10:41:38] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Wikimedia Enterprise, 10Wikimedia Enterprise - Content Integrity, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), and 2 others: Implement an Airflow operator for moving data from point A to B - https://phabricator.wikimedia.org/T405360#12586634 (... [10:42:25] 06Data-Engineering, 06Data-Platform-SRE, 10Test Kitchen, 07Essential-Work: Get schema browser in GrowthBook to work - https://phabricator.wikimedia.org/T440403#12586668 (10JAllemandou) My 2 cents on this. I think Presto doesn't understand the difference between a Hive table and an Iceberg at lookup time: `... [10:42:57] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work, 13Patch-For-Review: Deploy production and -next instances of the new table catalog on dse-k8s-eqiad - https://phabricator.wikimedia.org/T443248#12586678 (10Gehel) [10:43:03] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Create or import a Helm chart for Hive Metastore 4 - https://phabricator.wikimedia.org/T443250#12586680 (10Gehel) [10:43:09] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: [Tech evaluation] Table catalog options for the S3 data lake - https://phabricator.wikimedia.org/T435604#12586682 (10Gehel) [10:43:27] 06Data-Engineering, 06Data-Engineering-Icebox, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Public DataHub - https://phabricator.wikimedia.org/T366720#12586694 (10Gehel) [10:44:53] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11): Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12586744 (10Gehel) [10:45:03] 14Analytics-Radar, 06Data-Engineering, 10Kafka-Infrastructure, 06serviceops-radar, and 4 others: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12586742 (10Gehel) [10:47:05] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Plan access to the live Spark UI of applications on dse-k8s - https://phabricator.wikimedia.org/T443437#12586781 (10Gehel) [10:47:11] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Plan the Spark History Server alongside the S3 data lake - https://phabricator.wikimedia.org/T443434#12586782 (10Gehel) [10:47:17] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Plan the retrieval of Spark application logs on dse-k8s - https://phabricator.wikimedia.org/T443435#12586783 (10Gehel) [10:49:26] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work, 13Patch-For-Review: [Infra] Add a presto query logger - https://phabricator.wikimedia.org/T269832#12586818 (10Gehel) [10:49:34] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11): Investigate why a Spark YARN app continued after Airflow ended its try - https://phabricator.wikimedia.org/T443947#12586824 (10Gehel) [10:49:40] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Remove the spark_shuffle_3_3 and spark_shuffle_3_4 YARN aux services - https://phabricator.wikimedia.org/T440229#12586828 (10Gehel) [10:49:53] 06Data-Engineering, 10BetaFeatures, 10Data-Services, 06Privacy Engineering, and 5 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#12586830 (10Gehel) [11:08:21] RESOLVED: WebrequestPageviewHighKafkaConsumerLag: ... [11:08:21] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [11:14:51] FIRING: [2x] WebrequestPageviewHighKafkaConsumerLag: ... [11:14:57] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [11:19:51] RESOLVED: [2x] WebrequestPageviewHighKafkaConsumerLag: ... [11:19:51] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [11:28:21] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Citoid, 10CXServer, 10EventStreams, and 18 others: Migrate Event Platform to node26 - https://phabricator.wikimedia.org/T445574#12586921 (10aputhin) [11:31:47] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): data lake: server admin log (sal) - https://phabricator.wikimedia.org/T435417#12586935 (10Antoine_Quhen) Proposal: authenticate the producer at EventGate instead of signing each event. Related: T441140 (WMCS HAProxy logs). That ticket needs the same... [11:34:01] FIRING: MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [11:34:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [11:34:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [11:39:00] FIRING: [3x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [11:39:00] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [11:39:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [11:42:29] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work, 13Patch-For-Review: Define intake workflow for WMCS ingress logs - https://phabricator.wikimedia.org/T441140#12586992 (10Antoine_Quhen) @Volans @Ottomata we have the same need for... [11:44:01] FIRING: [6x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [11:44:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [11:44:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [11:54:01] FIRING: [7x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [11:54:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [11:54:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [11:59:01] FIRING: [12x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [11:59:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [11:59:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [12:09:01] FIRING: [12x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [12:09:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [12:09:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [12:14:01] FIRING: [12x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [12:14:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [12:14:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [13:20:35] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add cule_agent_id_time index to cu_changes, cuc_agent_id_time index to cu_log_event, and cupe_agent_id_time index to cu_private_event on WMF wikis - https://phabricator.wikimedia.org/T445307#12587257 (10Marostegui) [13:57:01] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: dbt: a dedicated dbt Airflow instance - https://phabricator.wikimedia.org/T435412#12587442 (10Gehel) [13:57:08] 06Data-Engineering, 06Wikimedia Enterprise, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Request for a dedicated Airflow instance for Commercial Partnerships (Enterprise) - https://phabricator.wikimedia.org/T440539#12587444 (10Gehel) [13:59:12] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11): Task Tries and Logs for Airflow DAGs sometimes unavailable - https://phabricator.wikimedia.org/T419162#12587492 (10Gehel) [14:00:02] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Superset "track job" button leads to broken URL - https://phabricator.wikimedia.org/T410149#12587519 (10Gehel) [14:00:18] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: ERROR AsyncEventQueue: Listener DatahubSparkListener threw an exception - https://phabricator.wikimedia.org/T400207#12587527 (10Gehel) [14:02:28] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: canary events (codfw partition) are not populated for dumps_v1 since 2026-11-11 hour=20 - https://phabricator.wikimedia.org/T444577#12587584 (10Gehel) [14:14:01] FIRING: [12x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [14:14:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [14:14:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [14:19:01] FIRING: [12x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [14:19:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [14:19:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [14:20:58] 06Data-Engineering: Data loss on eventgate-analytics and eventgate-analytics-external : 2026-11-19 to 2026-11-20 - https://phabricator.wikimedia.org/T445664 (10BTullis) 03NEW p:05Triage→03High [14:24:01] FIRING: [12x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [14:24:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [14:24:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [14:24:22] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Add lambda to spider pig - https://phabricator.wikimedia.org/T445665 (10AKhatun_WMF) 03NEW [14:29:16] FIRING: [12x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [14:29:16] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [14:29:16] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [14:34:01] FIRING: [10x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [14:34:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [14:34:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [14:34:36] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): KAPOW initial backfill - https://phabricator.wikimedia.org/T442233#12587883 (10mforns) [14:39:01] FIRING: [9x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [14:39:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [14:39:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [14:44:01] FIRING: [8x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [14:44:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [14:44:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [14:49:01] FIRING: [7x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [14:49:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [14:49:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [14:49:21] FIRING: WebrequestPageviewHighKafkaConsumerLag: ... [14:49:21] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [14:54:01] RESOLVED: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [14:54:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [14:54:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [14:54:21] FIRING: [2x] WebrequestPageviewHighKafkaConsumerLag: ... [14:54:21] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [14:57:28] (03CR) 10Joal: [C:03+1] hdfs: remove create_hdfs_usage_table.hql [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1372044 (https://phabricator.wikimedia.org/T438056) (owner: 10Xcollazo) [15:04:21] RESOLVED: [2x] WebrequestPageviewHighKafkaConsumerLag: ... [15:04:21] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [15:18:48] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Test Kitchen, 07Essential-Work: Consolidate Test Kitchen Automated Analytics MVP files in data lake - https://phabricator.wikimedia.org/T445156#12588181 (10xcollazo) 05In progress→03Resolved [15:19:42] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10AQS2.0, 10Data-Platform, 06Growth-Team, and 3 others: DE1.3.2: Implement "Pageviews for an editor’s edited pages" - https://phabricator.wikimedia.org/T445117#12588182 (10Snwachukwu) [15:32:34] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11): Yarn ResourceManager declared 8 NodeManagers LOST at the same time - https://phabricator.wikimedia.org/T445683 (10BTullis) 03NEW [15:32:49] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11): Yarn ResourceManager declared 8 NodeManagers LOST at the same time - https://phabricator.wikimedia.org/T445683#12588294 (10BTullis) p:05Triage→03High [15:55:04] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10EventStreams, 10Event-Platform, 07Technical-Debt: Migrate Event Platform to node26 - https://phabricator.wikimedia.org/T445574#12588415 (10tchin) [16:12:31] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11): Yarn ResourceManager declared 8 NodeManagers LOST at the same time - https://phabricator.wikimedia.org/T445683#12588492 (10BTullis) [16:18:48] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE, 07Epic, 13Patch-For-Review: Upgrade to Spark 3.5 - https://phabricator.wikimedia.org/T338057#12588521 (10xcollazo) [16:35:30] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: Make Spark 3.5.8 the default for new Airflow work. Existing use doesn't change behavior. - https://phabricator.wikimedia.org/T445689#12588562 (10xcollazo) [16:40:04] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): KAPOW initial backfill - https://phabricator.wikimedia.org/T442233#12588586 (10mforns) [17:08:57] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): refinery reorg: Phase 0 — Foundations for the Data Engineering monorepo - https://phabricator.wikimedia.org/T445692 (10amastilovic) 03NEW [17:11:30] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): refinery reorg: Choose a name for the new monorepo - https://phabricator.wikimedia.org/T445693 (10amastilovic) 03NEW [17:14:07] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): refinery reorg: Create the monorepo with its layout, boundary rule and conventions - https://phabricator.wikimedia.org/T445694 (10amastilovic) 03NEW [17:17:13] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): refinery reorg: Build the monorepo CI/CD skeleton with path rules - https://phabricator.wikimedia.org/T445697 (10amastilovic) 03NEW [17:19:48] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): refinery reorg: Deploy the monorepo to HDFS on merge via Blunderbuss - https://phabricator.wikimedia.org/T445698 (10amastilovic) 03NEW [18:34:23] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Add width bands to the mediarequests API - https://phabricator.wikimedia.org/T439420#12589159 (10Snwachukwu) This is feasible. The transcoded_(image|movie)_N already exists upstream in `wmf.mediarequest` table in the `transcoding` column. The pipelin... [19:16:21] (03CR) 10Snwachukwu: [C:03+1] hdfs: remove create_hdfs_usage_table.hql [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1372044 (https://phabricator.wikimedia.org/T438056) (owner: 10Xcollazo) [19:17:26] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: Make Spark 3.5.8 the default for new Airflow work. Existing use doesn't change behavior - https://phabricator.wikimedia.org/T445689#12589343 (10xcollazo) [19:19:35] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): KAPOW initial backfill - https://phabricator.wikimedia.org/T442233#12589352 (10mforns) [20:12:21] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Persistence-Design-Review: [Data Persistence Design Review] Thanks - https://phabricator.wikimedia.org/T440474#12589519 (10Snwachukwu) [20:13:07] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Persistence-Design-Review: [Data Persistence Design Review] Thanks - https://phabricator.wikimedia.org/T440474#12589521 (10Snwachukwu) [20:20:02] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10EventStreams, 10Event-Platform, 07Technical-Debt: Migrate Event Platform to node26 - https://phabricator.wikimedia.org/T445574#12589548 (10tchin) Hmm, `confluent-kafka-javascript` [[ https://github.com/confluentinc/confluent-kafka-javascript/i... [20:34:39] (03CR) 10Xcollazo: [V:03+2 C:03+2] hdfs: remove create_hdfs_usage_table.hql [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1372044 (https://phabricator.wikimedia.org/T438056) (owner: 10Xcollazo) [20:37:21] FIRING: [6x] WebrequestPageviewHighKafkaConsumerLag: ... [20:37:21] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [20:47:21] FIRING: [6x] WebrequestPageviewHighKafkaConsumerLag: ... [20:47:21] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [21:00:11] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Essential-Work, 13Patch-For-Review: Build an HDFS growth dashboard based on wmf.hdfs_usage - https://phabricator.wikimedia.org/T438056#12589681 (10xcollazo) Ran in prod removal of artifacts related to old Hive table `wmf.hdfs_usage`: ` $ hostna... [21:01:43] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Essential-Work, 13Patch-For-Review: Build an HDFS growth dashboard based on wmf.hdfs_usage - https://phabricator.wikimedia.org/T438056#12589695 (10xcollazo) [21:04:36] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Essential-Work, 13Patch-For-Review: Build an HDFS growth dashboard - https://phabricator.wikimedia.org/T438056#12589720 (10xcollazo) [21:07:21] FIRING: [4x] WebrequestPageviewHighKafkaConsumerLag: ... [21:07:21] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [21:07:39] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Task Tries and Logs for Airflow DAGs sometimes unavailable - https://phabricator.wikimedia.org/T419162#12589787 (10Gehel) [21:08:22] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Investigate why a Spark YARN app continued after Airflow ended its try - https://phabricator.wikimedia.org/T443947#12589796 (10Gehel) [21:09:02] 06Data-Engineering, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Enable STS and IAM on the codfw RADOS Gateway and register the OIDC identity providers - https://phabricator.wikimedia.org/T435590#12589804 (10Gehel) [21:10:09] 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Jupyter spawner's list of Conda environments only updates when an environment is spawned - https://phabricator.wikimedia.org/T391894#12589824 (10Gehel) [21:10:15] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Optimize enqueueing of refine_webrequest_hourly pipeline - https://phabricator.wikimedia.org/T419050#12589826 (10Gehel) [21:10:25] 06Data-Engineering, 10Data-Engineering-Jupyter, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), and 2 others: Upgrade to JupyterLab ≥ 4.2 - https://phabricator.wikimedia.org/T391905#12589825 (10Gehel) [21:22:06] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Essential-Work, 13Patch-For-Review: Build an HDFS growth dashboard - https://phabricator.wikimedia.org/T438056#12589885 (10xcollazo) All the work for this task is done. We have a new HDFS growth dashboard: https://superset.wikimedia.org/superset... [21:22:55] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): KAPOW initial backfill - https://phabricator.wikimedia.org/T442233#12589887 (10mforns) [21:37:21] RESOLVED: [2x] WebrequestPageviewHighKafkaConsumerLag: ... [21:37:21] High Kafka consumer lag for page_view in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/Webrequest_Pageview - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=webrequest-pageview&var-helm_release=production&var-operator_name=All&var-flink_job_name=page_view - https://alerts.wikimedia.org/?q=alertname%3DWebrequestPageviewHighKafkaConsumerLag [22:29:35] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Data Warehouse: Measure interactive query performance for Superset and Growthbook - https://phabricator.wikimedia.org/T442959#12590216 (10amastilovic) [22:51:12] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-11-20 - 2026-12-11), 07Essential-Work: Superset "track job" button leads to broken URL - https://phabricator.wikimedia.org/T410149#12590284 (10DLynch) I looked at this briefly: * Superset seems to have a `TRACKING_URL_TRANSFORMER` conf... [23:19:51] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: dbt_daily DAG exceeds its SLA - https://phabricator.wikimedia.org/T434307#12590395 (10amastilovic) [23:20:37] 06Data-Engineering, 06Data-Engineering-Radar: dbt_scheduled: Migrate product-analytics jobs from dbt_daily to scheduling groups - https://phabricator.wikimedia.org/T441958#12590404 (10amastilovic) [23:20:38] 06Data-Engineering, 06Data-Engineering-Radar, 06Movement-Insights (FY26-27 H1): dbt_scheduled: Migrate movement-insights account-registration jobs from dbt_monthly to scheduling groups - https://phabricator.wikimedia.org/T441957#12590405 (10amastilovic) [23:20:40] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: dbt_daily DAG exceeds its SLA - https://phabricator.wikimedia.org/T434307#12590402 (10amastilovic) [23:20:41] 06Data-Engineering, 06Data-Engineering-Radar, 06Movement-Insights (FY26-27 H1), 13Patch-For-Review: dbt_scheduled: Migrate movement-insights and page-trending jobs from legacy DAGs to scheduling groups - https://phabricator.wikimedia.org/T441959#12590403 (10amastilovic) [23:21:15] 06Data-Engineering, 06Data-Engineering-Radar, 06Movement-Insights (FY26-27 H1): dbt_scheduled: Migrate movement-insights account-registration jobs from dbt_monthly to scheduling groups - https://phabricator.wikimedia.org/T441957#12590409 (10amastilovic) 05Open→03In progress [23:21:45] 06Data-Engineering, 06Data-Engineering-Radar, 06Movement-Insights (FY26-27 H1), 13Patch-For-Review: dbt_scheduled: Migrate movement-insights and page-trending jobs from legacy DAGs to scheduling groups - https://phabricator.wikimedia.org/T441959#12590411 (10amastilovic) 05In progress→03Resolved [23:22:07] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: dbt_daily DAG exceeds its SLA - https://phabricator.wikimedia.org/T434307#12590413 (10amastilovic)