[02:07:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:07:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:07:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:07:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:35:44] 06Data-Engineering, 10ContentTranslation, 06LPL Hypothesis, 10MediaWiki-extensions-EventLogging, and 3 others: CI error: mediawiki.base/track trackError: unexpected "{\"exception\":{},\"module\":\"mw.cx.eventlogging\",\"source\":\"module-execute\"}" - https://phabricator.wikimedia.org/T413202#11590318 (10Ni... [09:19:01] 06Data-Engineering, 06Product-Analytics: [Request] Sample of refined webrequest table in data lake - https://phabricator.wikimedia.org/T413952#11590569 (10KCVelaga_WMF) @Milimetric > If this is high priority, let us know? I'd say medium. It isn't blocking any work. Later in the quarter or next, would be help... [09:22:12] 06Data-Engineering, 06Product-Analytics: [Request] Sample of refined webrequest table in data lake - https://phabricator.wikimedia.org/T413952#11590573 (10JAllemandou) I have created a job dowloading `webrequest_sampled` onto HDFS to batch-ingest the data into druid. An option could be to create a `webrequest_... [10:07:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [10:07:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [10:39:20] 06Data-Engineering, 10Data-Engineering-Wikistats, 10Pageviews-Anomaly: Sudden traffic increase on 1 November 2025 - https://phabricator.wikimedia.org/T412655#11590735 (10GGoncalves-WMF) @Hghani points out (on a first look) that this can be related to T413027. That ticket also hints at some signals we can inc... [10:43:19] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights (FY25-26 H2): dbt repository structure (Milestone 3) - https://phabricator.wikimedia.org/T416672 (10JMonton-WMF) 03NEW [10:43:26] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights (FY25-26 H2): dbt repository structure (Milestone 3) - https://phabricator.wikimedia.org/T416672#11590775 (10JMonton-WMF) [10:48:17] 06Data-Engineering, 10ContentTranslation, 06LPL Hypothesis, 10MediaWiki-extensions-EventLogging, and 3 others: CI error: mediawiki.base/track trackError: unexpected "{\"exception\":{},\"module\":\"mw.cx.eventlogging\",\"source\":\"module-execute\"}" - https://phabricator.wikimedia.org/T413202#11590812 (... [10:58:01] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights (FY25-26 H2): dbt repository structure (Milestone 3) - https://phabricator.wikimedia.org/T416672#11590836 (10JMonton-WMF) [11:11:11] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07Epic: dbt DPE work - https://phabricator.wikimedia.org/T416679 (10JMonton-WMF) 03NEW [11:12:13] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights (FY25-26 H2): dbt repository structure (Milestone 3) - https://phabricator.wikimedia.org/T416672#11590909 (10JMonton-WMF) [11:12:14] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07Epic: dbt DPE work - https://phabricator.wikimedia.org/T416679#11590910 (10JMonton-WMF) [11:14:56] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights (FY25-26 H2): dbt repository structure (Milestone 3) - https://phabricator.wikimedia.org/T416672#11590913 (10JMonton-WMF) [12:43:03] 06Data-Engineering, 06Machine-Learning-Team, 10Event-Platform: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11591172 (10gkyziridis) Hey, I am working on this, I think that I have finished the implementation for publishing... [13:39:48] 06Data-Engineering: [Data Quality] root cause analysis, pipeline improvement analysis image suggestion pipeline failure - https://phabricator.wikimedia.org/T351167#11591387 (10Aklapper) [13:40:28] 06Data-Engineering: [Data Quality] Define persona and user stories for system and data monitoring and alerting - https://phabricator.wikimedia.org/T349454#11591390 (10Aklapper) a:05Ahoelzl→03None @Ahoelzl: Removing task assignee as this open task has been assigned for more than two years - See the email sent... [13:40:29] 06Data-Engineering: [Data Quality] root cause analysis, pipeline improvement analysis image suggestion pipeline failure - https://phabricator.wikimedia.org/T351167#11591392 (10Aklapper) a:05Ahoelzl→03None @Ahoelzl: Removing task assignee as this open task has been assigned for more than two years - See the e... [14:07:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [14:07:52] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [14:30:02] !log Test Kitchen edge-unique experiments (poll 98542) - adds: none; removes: test-context-edit-color; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:30:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:46:37] 06Data-Engineering, 10Dumps-Generation, 10Wikidata, 10Wikidata data dumps: Feature request: Export Wikidata JSON as JSONL - https://phabricator.wikimedia.org/T414266#11591703 (10Lydia_Pintscher) [15:03:52] 06Data-Engineering, 06Data-Persistence, 10MediaWiki-extensions-OATHAuth, 06Security-Team, and 3 others: New database table for tracking WebAuthn userHandle values (oathauth_user_handles) - https://phabricator.wikimedia.org/T416544#11591749 (10JAllemandou) >>! In T416544#11588802, @Ahoelzl wrote: > @JAllema... [15:47:37] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:47:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:52:37] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:52:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:55:30] 06Data-Engineering, 06Test Kitchen: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#11592049 (10xcollazo) I think this will also require having a new service user for ExpP. [15:55:58] 06Data-Engineering, 06Data-Platform-SRE, 06Test Kitchen: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#11592052 (10xcollazo) [16:17:46] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights (FY25-26 H2): dbt repository structure (Milestone 3) - https://phabricator.wikimedia.org/T416672#11592116 (10JMonton-WMF) There are many different ways of configuring a //dbt// project to work with multiple teams and projects, here i... [16:42:37] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:42:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:47:37] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [16:47:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [16:55:39] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): OpsWeek: Bump memory of refine job for product_metrics.web_base_with_ip to avoid recent OOMs - https://phabricator.wikimedia.org/T416719 (10xcollazo) 03NEW [16:56:50] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): OpsWeek: Bump memory of refine job for product_metrics.web_base_with_ip to avoid recent OOMs - https://phabricator.wikimedia.org/T416719#11592272 (10xcollazo) [16:56:57] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): OpsWeek: Bump memory of refine job for product_metrics.web_base_with_ip to avoid recent OOMs - https://phabricator.wikimedia.org/T416719#11592273 (10xcollazo) [17:46:43] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: OpsWeek: Bump memory of refine job for product_metrics.web_base_with_ip to avoid recent OOMs - https://phabricator.wikimedia.org/T416719#11592362 (10xcollazo) Related: https://wikitech.wikimedia.org/wiki/Test_Kitchen/Incident_reports... [18:15:24] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: OpsWeek: Bump memory of refine job for product_metrics.web_base_with_ip to avoid recent OOMs - https://phabricator.wikimedia.org/T416719#11592496 (10xcollazo) Data backfilled via: ` analytics@an-launcher1003:/home/xcollazo$ hostname... [18:22:44] 06Data-Engineering: Requesting Kerberos access for hmonroy - https://phabricator.wikimedia.org/T416729 (10HMonroy) 03NEW [18:22:51] 06Data-Engineering: Requesting Kerberos access for hmonroy - https://phabricator.wikimedia.org/T416729#11592523 (10HMonroy) [18:44:26] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: OpsWeek: Bump memory of refine job for product_metrics.web_base_with_ip to avoid recent OOMs - https://phabricator.wikimedia.org/T416719#11592570 (10tchin) [19:13:50] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: OpsWeek: Bump memory of refine job for product_metrics.web_base_with_ip to avoid recent OOMs - https://phabricator.wikimedia.org/T416719#11592673 (10xcollazo) >>! In T416719#11592665, @gerritbot wrote: > Change #1237540 **merged** by... [19:43:06] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights (FY25-26 H2): dbt repository structure (Milestone 3) - https://phabricator.wikimedia.org/T416672#11592762 (10Mayakp.wiki) Thank you @JMonton-WMF ! for all this info. this is a very nice and clear proposal. all of this makes sense to... [19:52:29] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07Epic: dbt DPE work - https://phabricator.wikimedia.org/T416679#11592808 (10Mayakp.wiki) [19:52:39] 10Data-Engineering-Roadmap, 06Data-Platform-SRE, 06Movement-Insights, 07Epic, 13Patch-For-Review: Provide a dbt-core development environment and production setup in the data-platform - https://phabricator.wikimedia.org/T406764#11592809 (10Mayakp.wiki) [20:47:37] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [20:47:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:01:05] 06Data-Engineering, 06Data-Persistence, 10MediaWiki-extensions-OATHAuth, 06Security-Team, and 3 others: New database table for tracking WebAuthn userHandle values (oathauth_user_handles) - https://phabricator.wikimedia.org/T416544#11592971 (10Ladsgroup) If it's waiting on DBA signoff, to make it clear that... [21:10:19] (03Abandoned) 10AKhatun: Add hql files for new growthbook pipeline [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1237249 (https://phabricator.wikimedia.org/T414105) (owner: 10AKhatun) [21:24:12] 06Data-Engineering, 06Data-Platform-SRE, 06Test Kitchen: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#11593028 (10amastilovic) On DPE side I believe we need to cover the following items in order to support this new instance: - Implement necessary changes to the `air... [22:23:18] 06Data-Engineering, 06DBA: Move Mostcategories computation to Hadoop - https://phabricator.wikimedia.org/T413362#11593177 (10Zabe) https://analytics.wikimedia.org/published/datasets/querypage/MostCategories/ looks promising [23:19:42] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): [Iceberg Migration] Extend Iceberg table maintenance mechanism to support multiple Airflow instances - https://phabricator.wikimedia.org/T373693#11593265 (10xcollazo)