[01:22:37] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [01:22:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:02:37] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:02:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:12:37] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [02:12:37] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:12:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [06:12:53] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [06:23:35] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11594892 (10Marostegui) [07:54:13] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786#11594959 (10Marostegui) [08:33:29] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Reduce main Airflow DB size and consider splitting heavy workloads into separate instances - https://phabricator.wikimedia.org/T411992#11595034 (10Antoine_Quhen) 05Resolved→03Open [08:43:05] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Reduce main Airflow DB size and consider splitting heavy workloads into separate instances - https://phabricator.wikimedia.org/T411992#11595070 (10Antoine_Quhen) search and analytics_test are running on small DB instances with no fre... [09:48:54] 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11595502 (10elukey) a:03elukey [10:12:52] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [10:12:53] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [12:28:27] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10CheckUser, 06Product Safety and Integrity: Changes to the cuc_agent column in the cu_changes table - https://phabricator.wikimedia.org/T361210#11596347 (10Dreamy_Jazz) @Snwachukwu is this task complete? We would like to progressively stop writing to... [12:46:34] 06Data-Engineering, 06Data-Platform-SRE, 06Java-Scala-Standardization, 06Discovery-Search (2026.02.02 - 2026.02.27), and 2 others: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies ... - https://phabricator.wikimedia.org/T367405#11596410 [13:30:26] 06Data-Engineering, 06Data-Platform-SRE, 06Java-Scala-Standardization, 06Discovery-Search (2026.02.02 - 2026.02.27), and 2 others: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies ... - https://phabricator.wikimedia.org/T367405#11596602 [13:55:49] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights (FY25-26 H2): dbt repository structure (Milestone 3) - https://phabricator.wikimedia.org/T416672#11596735 (10GGoncalves-WMF) This is great, thanks for the detailed proposal @JMonton-WMF ! I think this generally looks good as well. A... [14:12:53] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [14:12:53] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [14:19:31] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10CheckUser, 06Product Safety and Integrity: Changes to the cuc_agent column in the cu_changes table - https://phabricator.wikimedia.org/T361210#11596804 (10Snwachukwu) The patch is ready but not yet deployed. I would be deployed with our deployment t... [14:21:59] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10CheckUser, 06Product Safety and Integrity: Changes to the cuc_agent column in the cu_changes table - https://phabricator.wikimedia.org/T361210#11596808 (10Dreamy_Jazz) >>! In T361210#11596804, @Snwachukwu wrote: > The patch is ready but not yet depl... [14:25:32] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights (FY25-26 H2): dbt repository structure (Milestone 3) - https://phabricator.wikimedia.org/T416672#11596819 (10Ottomata) Really nice write up! Thank you so much for putting this in writing in phabricator! My only comment: let's avoid... [14:28:22] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights (FY25-26 H2): dbt repository structure (Milestone 3) - https://phabricator.wikimedia.org/T416672#11596822 (10Ottomata) Peripherally relevant, in case you haven't seen: - https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Gui... [14:30:02] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10DPE-Mediawiki-Content: Use wmf.mediawiki_history as baseline for slo completeness - https://phabricator.wikimedia.org/T416312#11596829 (10GGoncalves-WMF) In general, I think this is good to consider, though I'm not seeing a very strong priority signal... [14:30:11] !log Test Kitchen edge-unique experiments (poll 111396) - adds: none; removes: synth-aaa-test-mw-js, synth-aa-test-traffic-impact; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:30:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:36:54] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights (FY25-26 H2): dbt repository structure (Milestone 3) - https://phabricator.wikimedia.org/T416672#11596861 (10Ottomata) Oh! Okay okay, I had more naming thoughts! ===== Project? > team1_project1_model1.sql If dbt already has a codi... [14:39:58] 06Data-Engineering, 06Data-Platform-SRE, 06Test Kitchen: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#11596867 (10mpopov) [14:40:30] 06Data-Engineering, 06Data-Platform-SRE, 06Test Kitchen: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#11596873 (10mpopov) [14:40:48] 06Data-Engineering, 06Data-Platform-SRE, 06Test Kitchen: Airflow instance for Experiment Platform - https://phabricator.wikimedia.org/T416709#11596874 (10mpopov) Thanks! I copied that into the acceptance criteria and updated `exp_platform` to `experiment_platform` [14:47:17] 06Data-Engineering, 06Data-Platform-SRE, 07Essential-Work: Simplify the datahub ingestion pipelines - https://phabricator.wikimedia.org/T402306#11596898 (10Ottomata) > I'm not sure how much Data Hub is useful these days. I think datahub is quite useful, especially when needing to send a link to someone to l... [14:54:01] 06Data-Engineering, 10Dumps-Generation: wikidatawiki fails dumps of the wbt_* tables, also lagging on XML Dumps - https://phabricator.wikimedia.org/T396125#11596926 (10xcollazo) >>! In T396125#11588880, @Milimetric wrote: > @xcollazo Is this obsolete? No, we continue to not dump the `wbt` tables as per T39612... [14:55:49] 06Data-Engineering, 06Machine-Learning-Team, 10Event-Platform: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11596956 (10Ottomata) > Andrew - what's the timeline on this for DE? Do we just track it in radar and advise or do... [14:59:31] 06Data-Engineering, 06Machine-Learning-Team, 10Event-Platform: Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11596969 (10Ottomata) @achou Do we know if we have any active consumers of `mediawiki.page_revert_risk_prediction_... [15:05:14] (03PS1) 10Xcollazo: MediawikiDumper: optionally repartition source data to avoid OOMs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1237923 (https://phabricator.wikimedia.org/T416642) [15:17:57] (03PS1) 10Aqu: Move bot detection pipeline into new repo [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1237928 (https://phabricator.wikimedia.org/T415874) [15:48:06] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07OKR-Work (WE1 FY2025-26): WE1.5.3 Productize Data for Monthly Active Moderator Actions - https://phabricator.wikimedia.org/T410940#11597195 (10AKhatun_WMF) a:05JMoore-WMF→03AKhatun_WMF [15:51:59] 06Data-Engineering, 10Dumps-Generation: wikidatawiki fails dumps of the wbt_* tables, also lagging on XML Dumps - https://phabricator.wikimedia.org/T396125#11597205 (10Milimetric) k, moving back for consideration with new information. [15:57:37] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): OpsWeek: Bump memory of refine job for product_metrics.web_base_with_ip to avoid recent OOMs - https://phabricator.wikimedia.org/T416719#11597239 (10xcollazo) Many more hourly refine jobs have failed since we opened this ticket. Same root cause. Examples... [16:01:10] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Movement-Insights (FY25-26 H2): dbt repository structure (Milestone 3) - https://phabricator.wikimedia.org/T416672#11597296 (10JMonton-WMF) > Let me make sure I understand: when we say "subfolders by projects", these are not distinct dbt projects righ... [16:09:04] 06Data-Engineering: [Data Quality] Define persona and user stories for system and data monitoring and alerting - https://phabricator.wikimedia.org/T349454#11597338 (10Ottomata) 05Open→03Declined Being bold and declining. Please reopen if incorrect. [16:22:46] 06Data-Engineering: [Data Quality] root cause analysis, pipeline improvement analysis image suggestion pipeline failure - https://phabricator.wikimedia.org/T351167#11597460 (10Ottomata) 05Open→03Declined Declining, please reopen if incorrect. [16:31:28] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: Get "edits hourly" on a daily basis - https://phabricator.wikimedia.org/T231938#11597540 (10Ottomata) →14Duplicate dup:03T358877 [16:31:30] 06Data-Engineering, 10DPE-Mediawiki-Content, 07Epic: Dumps 2.0 Phase II: Production intermediate table milestone - https://phabricator.wikimedia.org/T358877#11597542 (10Ottomata) [16:32:04] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: Get "edits hourly" on a daily basis - https://phabricator.wikimedia.org/T231938#11597549 (10Ottomata) 05Duplicate→03Resolved Marking as duplicate of T358877. edits + wikitext data is now available hourly and reconciled daily in the... [16:36:23] 06Data-Engineering, 10ChangeProp, 10EventStreams, 10iPoid-Service, and 17 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11597592 (10Jgiannelos) [16:45:12] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Reader Growth Team, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog, and 2 others: Add page_id and namespace to X-Analytics header in Mobile App requests (2025 remake) - https://phabricator.wikimedia.org/T409358#11597649 (10Ottomata)... [16:45:43] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Reader Growth Team, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog, and 2 others: Add page_id and namespace to X-Analytics header in Mobile App requests (2025 remake) - https://phabricator.wikimedia.org/T409358#11597650 (10Ottomata)... [16:48:47] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Research: Content history dataset issues - https://phabricator.wikimedia.org/T415311#11597687 (10APizzata-WMF) [16:50:48] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Content-Transform-Team, 06Reader Growth Team, 06Wikipedia-Android-App-Backlog, and 3 others: Add page_id and namespace to X-Analytics header in Mobile App requests (2025 remake) - https://phabricator.wikimedia.org/T409358#11597701 (10Ottomata) Oh,... [16:56:39] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Content-Transform-Team, 06Reader Growth Team, 06Wikipedia-Android-App-Backlog, and 4 others: Add page_id and namespace to X-Analytics header in Mobile App requests (2025 remake) - https://phabricator.wikimedia.org/T409358#11597761 (10Jgiannelos) I... [18:03:58] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10MediaWiki-extensions-WikimediaEvents, 07Essential-Work, and 3 others: Update name in EventLogging Extension and WikimediaEvents Extension - https://phabricator.wikimedia.org/T407904#11598298 (10Sfaci) [18:06:31] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Clean up artifacts.yaml - https://phabricator.wikimedia.org/T405379#11598326 (10Snwachukwu) @xcollazo When we move all the jobs to the latest, how do we plan to make sure that all the jobs are always updated whenever there is a latest artifact? This is on... [18:12:53] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [18:12:53] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [20:35:53] 06Data-Engineering, 06Data-Engineering-Icebox, 10Research-engineering, 06Research-Freezer, 10Event-Platform: Productionized Edit Types - https://phabricator.wikimedia.org/T351225#11598933 (10Ottomata) [20:36:35] 06Data-Engineering, 06Research: Incremental HTML wiki content dataset to support "Who are moderators" - https://phabricator.wikimedia.org/T380874#11598936 (10Ottomata) [20:47:29] <_Gerges_> Can a member of the data platform engineering team review this request https://meta.wikimedia.org/wiki/Steward_requests/Global_permissions#apihighlimits_for_Gerges [21:28:22] 06Data-Engineering, 06Research: Incremental HTML wiki content dataset to support "Who are moderators" - https://phabricator.wikimedia.org/T380874#11599204 (10Ottomata) [21:28:26] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 07OKR-Work (WE1 FY2025-26): WE1.5.3 Productize Data for Monthly Active Moderator Actions - https://phabricator.wikimedia.org/T410940#11599205 (10Ottomata) [21:30:28] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: OpsWeek: wikidatawiki fails to export due to OOM on partitionByXMLFileBoundaries - https://phabricator.wikimedia.org/T416642#11599230 (10xcollazo) >>! In T416642#11596991, @gerritbot wrote: > Change #1237923 had a related patch set u... [21:35:41] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10MediaWiki-extensions-WikimediaEvents, 07Essential-Work, and 3 others: Update name in EventLogging Extension and WikimediaEvents Extension - https://phabricator.wikimedia.org/T407904#11599242 (10Sfaci) [21:57:07] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: OpsWeek: wikidatawiki fails to export due to OOM on partitionByXMLFileBoundaries - https://phabricator.wikimedia.org/T416642#11599301 (10xcollazo) Moving successful test run to production, as there is no need to recalculate it via an... [22:12:33] 06Data-Engineering, 06Movement-Insights: Investigate and repair pageviews and unique devices spike starting in Nov 2025 - https://phabricator.wikimedia.org/T416933 (10GGoncalves-WMF) 03NEW [22:12:53] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [22:12:53] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [22:46:58] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): File BAR for Cursor and Claude Code - https://phabricator.wikimedia.org/T416937 (10Ahoelzl) 03NEW [23:06:55] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Adapt Sqoop for imagelinks schema changes - https://phabricator.wikimedia.org/T416481#11599594 (10Ahoelzl) a:03Snwachukwu