[08:44:47] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12500316 (10APizzata-WMF) it worked, I will play with it! [10:12:35] 06Data-Engineering, 06DBA, 10MediaWiki-extensions-WikimediaAntiAbuse, 06Product Safety and Integrity, 07Schema-change: Create two new tables for AbuseReview - https://phabricator.wikimedia.org/T443206 (10MPostoronca-WMF) 03NEW [10:19:16] 06Data-Engineering, 06DBA, 10MediaWiki-extensions-WikimediaAntiAbuse, 06Product Safety and Integrity, 07Schema-change: Create two new tables for AbuseReview - https://phabricator.wikimedia.org/T443206#12500781 (10MPostoronca-WMF) [11:22:22] aqu o/ Not sure if you hang our here really :D [12:33:50] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10SRE-SLO: page_change SLO windows - https://phabricator.wikimedia.org/T438054#12501581 (10APizzata-WMF) Hey @RLazarus I have managed to extract daily data points for the metric and will apply the same methodology applied in the [[ https://gitlab.wi... [12:34:14] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12501580 (10KartikMistry) Note that the Recommendation-API node version is no longer used in Produ... [12:38:41] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12501603 (10KartikMistry) [12:39:13] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12501607 (10KartikMistry) [12:39:36] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12501610 (10APizzata-WMF) Tried following the documentation you linked but the event service (as the doc states should be: `Filter by name = event`... [13:24:51] 06Data-Engineering, 06Data-Engineering-Radar, 10EntitySchema, 10JsonConfig, and 16 others: Add namespace descriptions for Special:NamespaceInfo in WMF-deployed extensions - https://phabricator.wikimedia.org/T373070#12501836 (10MHorsey-WMF) [13:27:50] 06Data-Engineering, 06DBA, 10MediaWiki-extensions-WikimediaAntiAbuse, 06Product Safety and Integrity, 07Schema-change: Create two new tables for AbuseReview - https://phabricator.wikimedia.org/T443206#12501843 (10MPostoronca-WMF) [13:33:57] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): KAPOW initial backfill - https://phabricator.wikimedia.org/T442233#12501862 (10mforns) [14:01:56] 06Data-Engineering, 06DBA, 10MediaWiki-extensions-WikimediaAntiAbuse, 06Product Safety and Integrity, 07Schema-change: Create two new tables for AbuseReview - https://phabricator.wikimedia.org/T443206#12502115 (10MPostoronca-WMF) [14:09:18] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Documentation: Complete Documentation of Article Feature Counts - https://phabricator.wikimedia.org/T436717#12502145 (10GGoncalves-WMF) Looks great, thank you! Two minor and non-blocking suggestions for the User Documentation part. > The response... [14:21:56] 06Data-Engineering, 06DBA, 10MediaWiki-extensions-WikimediaAntiAbuse, 06Product Safety and Integrity, 07Schema-change: Create two new tables for AbuseReview - https://phabricator.wikimedia.org/T443206#12502218 (10MPostoronca-WMF) [14:32:11] (03PS1) 10Aqu: dev-metrics: comment out Spark 4 dt bounds [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1361460 (https://phabricator.wikimedia.org/T433696) [14:40:30] 06Data-Engineering, 06Data-Platform-SRE: Deploy production and -next instances of the new table catalog on dse-k8s-eqiad - https://phabricator.wikimedia.org/T443248 (10BTullis) 03NEW [14:43:18] 06Data-Engineering, 06Data-Platform-SRE: Build a container image for Hive Metastore 4.2.1 - https://phabricator.wikimedia.org/T443249 (10BTullis) 03NEW [14:45:03] 06Data-Engineering, 06Data-Platform-SRE: Create or import a Helm chart for Hive Metastore 4 - https://phabricator.wikimedia.org/T443250 (10BTullis) 03NEW [14:45:17] 06Data-Engineering, 06Data-Platform-SRE: Create or import a Helm chart for Hive Metastore 4 - https://phabricator.wikimedia.org/T443250#12502411 (10BTullis) p:05Triage→03High [14:45:30] 06Data-Engineering, 06Data-Platform-SRE: Deploy production and -next instances of the new table catalog on dse-k8s-eqiad - https://phabricator.wikimedia.org/T443248#12502418 (10BTullis) p:05Triage→03High [14:45:42] 06Data-Engineering, 06Data-Platform-SRE: Build a container image for Hive Metastore 4.2.1 - https://phabricator.wikimedia.org/T443249#12502423 (10BTullis) p:05Triage→03High [14:47:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [14:47:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [14:53:26] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Dumps-Generation, 13Patch-For-Review: mediawiki-content-pipelines: Gate the File Exports on the ingest of the monthly reconcile - https://phabricator.wikimedia.org/T442415#12502465 (10xcollazo) Confirming that the change is WAD in production, an... [15:15:44] (03PS2) 10Aqu: dev-metrics: comment out Spark 4 dt bounds [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1361460 (https://phabricator.wikimedia.org/T433696) [15:16:46] (03CR) 10Aqu: [V:03+2 C:03+2] "follow up of 1348792" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1361460 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [15:20:02] 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review, 06Product-Analytics (Kanban): Switch to user_agents_info and deprecate user_agent_compliance_classified_hourly - https://phabricator.wikimedia.org/T433992#12502752 (10Ahoelzl) [15:27:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:27:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:32:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [15:32:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [15:51:33] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: dbt scheduler: Implement support for "backfill scheduling groups" - https://phabricator.wikimedia.org/T442623#12503065 (10amastilovic) [15:52:31] 06Data-Engineering, 06DBA, 10MediaWiki-extensions-WikimediaAntiAbuse, 06Product Safety and Integrity, and 2 others: Create two new tables for AbuseReview - https://phabricator.wikimedia.org/T443206#12503078 (10MPostoronca-WMF) [16:01:29] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): data lake: git commit history - https://phabricator.wikimedia.org/T433696#12503166 (10Antoine_Quhen) 05Open→03Resolved [16:36:55] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): data lake: git commit history - https://phabricator.wikimedia.org/T433696#12503465 (10Antoine_Quhen) 05Resolved→03Open [16:39:23] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Event-Platform, 13Patch-For-Review: eventlogging_EditAttemptStep sometimes misses dt field - https://phabricator.wikimedia.org/T442089#12503483 (10Ahoelzl) a:05amastilovic→03AKhatun_WMF [16:42:25] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Data Warehouse: Measure interactive query performance for Superset and Growthbook - https://phabricator.wikimedia.org/T442959#12503527 (10amastilovic) a:03amastilovic [16:42:48] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): superset: Analyze the existing Superset event log and SQL Lab query history - https://phabricator.wikimedia.org/T442960#12503535 (10amastilovic) a:03amastilovic [16:43:09] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): presto: Export per-resource-group metrics to Prometheus - https://phabricator.wikimedia.org/T442961#12503541 (10amastilovic) a:03amastilovic [16:43:32] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): superset: Use separate Presto connections for dashboards and SQL Lab - https://phabricator.wikimedia.org/T442963#12503545 (10amastilovic) a:03amastilovic [16:43:47] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): superset: Enable Superset application metrics (StatsdStatsLogger) - https://phabricator.wikimedia.org/T442964#12503551 (10amastilovic) a:03amastilovic [16:44:05] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Data Warehouse: Privacy and retention review for query logs - https://phabricator.wikimedia.org/T442965#12503554 (10amastilovic) a:03amastilovic [16:44:24] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): presto: Add a query-completion event listener that sends events to Event Platform - https://phabricator.wikimedia.org/T442966#12503558 (10amastilovic) a:03amastilovic [16:44:46] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Data Warehouse: Build a dbt mart and Superset dashboard for interactive query performance - https://phabricator.wikimedia.org/T442967#12503561 (10amastilovic) a:03amastilovic [16:46:20] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Optimize performance of dbt_daily upload_bandwidth_hourly task - https://phabricator.wikimedia.org/T439016#12503580 (10mforns) [16:46:22] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Epic: Epic: Migrate uses of webrequest to webrequest_v2 - https://phabricator.wikimedia.org/T441283#12503579 (10mforns) [16:48:34] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Put together the KAPOW deep dive - https://phabricator.wikimedia.org/T443273 (10mforns) 03NEW [16:48:57] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): User Acceptance tests in staging - https://phabricator.wikimedia.org/T437670#12503606 (10Eevans) >>! In T437670#12485630, @Ottomata wrote: >> ❓Cancelling a API call should return CANCELLED. It looked like LAC still computed for the full 10 seconds and... [16:49:21] 06Data-Engineering, 06DBA, 10MediaWiki-extensions-WikimediaAntiAbuse, 06Product Safety and Integrity, and 2 others: Create two new tables for AbuseReview - https://phabricator.wikimedia.org/T443206#12503619 (10MPostoronca-WMF) [16:50:34] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Load Google Search Console social account impressions and clicks data into Data Lake - https://phabricator.wikimedia.org/T436710#12503627 (10Ahoelzl) 05Open→03Declined [17:36:35] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12504018 (10mpopov) Approving @cfoster's membership in `analytics-product-users` Also should be added as primary approver with me... [17:53:05] 06Data-Engineering, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Conda-Analytics environments are prone to dependency conflicts and installation errors - https://phabricator.wikimedia.org/T423067#12504093 (10BTullis) 05Open→03Resolved I'm going to be bold and resolve this ticket, since I feel that we... [17:53:15] 06Data-Engineering, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Conda-Analytics environments are prone to dependency conflicts and installation errors - https://phabricator.wikimedia.org/T423067#12504096 (10BTullis) [18:21:57] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12504347 (10Ottomata) @APizzata-WMF and I deployed GuaranteedProducer and time_to_persist metric to eventgate-analytics. It does indeed look like... [19:17:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:17:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:22:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [19:22:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [19:24:05] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): User Acceptance tests in staging - https://phabricator.wikimedia.org/T437670#12505013 (10Ottomata) {T443331} [19:50:53] 06Data-Engineering: Airflow jobs cannot access `druid-public-coordinator.svc.eqiad.wmnet:8090` - https://phabricator.wikimedia.org/T443347 (10AKhatun_WMF) 03NEW [20:11:33] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): User Acceptance tests in staging - https://phabricator.wikimedia.org/T437670#12505464 (10AKhatun_WMF) TY! [20:41:51] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Test Kitchen, 07Essential-Work: Improve web instrument event data data lake management - https://phabricator.wikimedia.org/T429385#12505709 (10mpopov) Just noting that there is another short term solution I'll be working on since a longer term g... [21:17:36] FIRING: [2x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [21:17:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [21:39:02] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12506019 (10Ottomata) Well, 3 options: 3. just increase CPU resources and live with it. [23:47:36] RESOLVED: MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [23:47:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [23:55:01] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): IR2.1.2d: Feature count data product [Attribution API] - https://phabricator.wikimedia.org/T432604#12506534 (10AKhatun_WMF)