[02:56:14] 06Data-Engineering, 10BetaFeatures, 06cloud-services-team, 10Data-Services, and 5 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#12021914 (10aranyap) Hi @SD0001 , is there any possible way for this data to be correlated back to an individ... [04:52:49] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12021987 (10RKemper) >>! In T427951#12013159, @JMonton-WMF wrote: > Hi @RKemper, thanks for the det... [06:09:21] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12022059 (10RKemper) >>! In T427951#12020119, @AKhatun_WMF wrote: > rc0 is not running anywhere and... [06:23:57] 06Data-Engineering, 10Dumps-Generation: XML dumps does not re-load the config for depooled databases - https://phabricator.wikimedia.org/T429282 (10Marostegui) 03NEW [06:24:05] 06Data-Engineering, 06Data-Persistence, 10Dumps-Generation: XML dumps does not re-load the config for depooled databases - https://phabricator.wikimedia.org/T429282#12022094 (10Marostegui) [06:27:13] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE, 06Traffic: Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12022104 (10ayounsi) We discussed the HTTP Proxy vs. urldownloader during the I/F meeting and you can go ahead... [07:05:35] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: mw_content_reconcile_mw_content_history_monthly failed on rerun - https://phabricator.wikimedia.org/T428999#12022193 (10JAllemandou) I don't know which one is better between: * avoid emitting the field when the page_title is empty... [07:50:22] 06Data-Engineering: event_sanitized.serversideaccountcreation reports users that actually don't exist - https://phabricator.wikimedia.org/T429288 (10Urbanecm_WMF) 03NEW [07:50:53] 06Data-Engineering, 10FR-Tech-Analytics, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Epic, 13Patch-For-Review: Add FR Tech s3 secrets to airflow vars - https://phabricator.wikimedia.org/T429048#12022283 (10brouberol) [07:50:55] 06Data-Engineering, 10FR-Tech-Analytics, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Epic, 13Patch-For-Review: Add FR Tech s3 secrets to airflow vars - https://phabricator.wikimedia.org/T429048#12022284 (10brouberol) 05Open→03In progress [07:54:52] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Create API and User-Agent compliance related tables under wmf_traffic - https://phabricator.wikimedia.org/T427840#12022295 (10KCVelaga_WMF) 05Open→03Resolved p:05Triage→03High a:03JAllemandou [07:55:26] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Quality verification for mediawiki_history_incremental_v1 using Iceberg time travel - https://phabricator.wikimedia.org/T425734#12022301 (10APizzata-WMF) After a conversation with @JAllemandou we decided to start lookin... [08:16:07] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content, 13Patch-For-Review: Missing/inconsistent page_redirect_target field for redirects in Mediawiki content current v1 dumps - https://phabricator.wikimedia.org/T400632#12022350 (10APizzata-WMF) @MGerlach if you can confirm the improv... [09:51:04] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: mw_content_reconcile_mw_content_history_monthly failed on rerun - https://phabricator.wikimedia.org/T428999#12022794 (10APizzata-WMF) After the discussion with @JAllemandou we decided to filter out these redirects when the page_tit... [09:51:49] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Platform, 10DPE-MediaWiki-Incremental-History: Inconsistent counts of global account registrations in analytics datasets - https://phabricator.wikimedia.org/T429061#12022801 (10APizzata-WMF) [09:55:17] 06Data-Engineering, 10BetaFeatures, 06cloud-services-team, 10Data-Services, and 5 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#12022802 (10SD0001) The table doesn't have any user identifiers, so I don't see a way. [10:37:37] 06Data-Engineering, 06Data-Persistence, 10Dumps-Generation: XML dumps does not re-load the config for depooled databases - https://phabricator.wikimedia.org/T429282#12022934 (10Marostegui) [10:38:10] 06Data-Engineering, 06Data-Persistence, 10Dumps-Generation: XML dumps does not re-load the config for depooled databases - https://phabricator.wikimedia.org/T429282#12022936 (10Marostegui) This is partially blocking our migration to Debian Trixie of our external store clusters. [11:33:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:50:13] 06Data-Engineering, 06Data-Persistence, 10Dumps-Generation: XML dumps does not re-load the config for depooled databases - https://phabricator.wikimedia.org/T429282#12023283 (10Marostegui) ` 10:49 marostegui@cumin1003: dbctl commit (dc=all): 'Depool es1037 T429118', diff saved to https://phabricator.wikimedi... [11:50:26] 06Data-Engineering, 06Data-Persistence, 10Dumps-Generation: XML dumps does not re-load the config for depooled databases - https://phabricator.wikimedia.org/T429282#12023284 (10Marostegui) p:05Triage→03High [12:03:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-analytics-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-analytics-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [12:27:24] FYI, dse-k8s-worker1009 is down since almost six days [12:50:15] FIRING: HdfsRpcQueueLength: RPC queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue/latency - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength [12:55:15] RESOLVED: HdfsRpcQueueLength: RPC queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue/latency - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength [13:14:20] (03PS1) 10Joal: Update Incremental MWH with a fix [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1302824 (https://phabricator.wikimedia.org/T428928) [13:16:54] (03PS1) 10Joal: Update Incremental MWH schema readability (no-op) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1302825 (https://phabricator.wikimedia.org/T428928) [13:24:23] 06Data-Engineering, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Delete orphan EventLogging topic `eventlogging_HomepageModulet` - https://phabricator.wikimedia.org/T429017#12023779 (10Gehel) [13:24:35] 06Data-Engineering, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Delete orphan EventLogging topic `eventlogging_HomepageModulet` - https://phabricator.wikimedia.org/T429017#12023780 (10Gehel) p:05Triage→03Low [13:25:25] 06Data-Engineering, 10Kafka-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Incident Severity 3, 07Wikimedia-Incident: staging.webrequest.page_view.dev0 taking up most space on kafka-jumbo - https://phabricator.wikimedia.org/T429088#12023786 (10Gehel) [13:28:53] (03CR) 10CI reject: [V:04-1] Update Incremental MWH with a fix [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1302824 (https://phabricator.wikimedia.org/T428928) (owner: 10Joal) [13:36:44] (03PS1) 10Seanleong-wmde: Script to gather metrics for Recent Changes in pilot Wikis with labels. [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1302847 (https://phabricator.wikimedia.org/T426384) [13:37:07] (03CR) 10CI reject: [V:04-1] Script to gather metrics for Recent Changes in pilot Wikis with labels. [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1302847 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [13:37:10] (03CR) 10A-pizzata: [C:03+1] "LGTM" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1302825 (https://phabricator.wikimedia.org/T428928) (owner: 10Joal) [13:40:42] (03PS2) 10Seanleong-wmde: Script to gather metrics for Recent Changes in pilot Wikis with labels. [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1302847 (https://phabricator.wikimedia.org/T426384) [13:43:00] (03CR) 10Nicholusmuwonge: [C:03+2] Script to gather metrics for Recent Changes in pilot Wikis with labels. [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1302847 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [13:44:03] (03Merged) 10jenkins-bot: Script to gather metrics for Recent Changes in pilot Wikis with labels. [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1302847 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [13:44:27] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Product-Analytics: dbt-jobs backfill: PP3 API hourly and known clients aggregate jobs - https://phabricator.wikimedia.org/T429341 (10KCVelaga_WMF) 03NEW [13:44:40] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Product-Analytics: dbt-jobs backfill: PP3 API hourly and known clients aggregate jobs - https://phabricator.wikimedia.org/T429341#12023869 (10KCVelaga_WMF) p:05Triage→03High [13:51:27] (03PS1) 10Seanleong-wmde: Script to gather metrics for Recent Changes in pilot Wikis. [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1302863 (https://phabricator.wikimedia.org/T426384) [13:57:29] (03CR) 10Awight: [C:03+2] Script to gather metrics for Recent Changes in pilot Wikis. [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1302863 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [13:58:01] (03Merged) 10jenkins-bot: Script to gather metrics for Recent Changes in pilot Wikis. [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1302863 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [13:58:41] (03PS1) 10Seanleong-wmde: Script to gather metrics for Recent Changes in pilot Wikis with labels. [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1302873 (https://phabricator.wikimedia.org/T426384) [13:59:05] (03CR) 10Awight: [C:03+2] "Deploying!" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1302873 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [13:59:41] (03Merged) 10jenkins-bot: Script to gather metrics for Recent Changes in pilot Wikis with labels. [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1302873 (https://phabricator.wikimedia.org/T426384) (owner: 10Seanleong-wmde) [14:13:01] FIRING: [6x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [14:13:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [14:13:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [14:30:09] !log Test Kitchen experiment (poll 9490) - adds: none; removes: share-highlight; fields: none - TK tips at https://w.wiki/FwuD [14:30:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:32:00] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Platform, 10DPE-MediaWiki-Incremental-History: Inconsistent counts of global account registrations in analytics datasets - https://phabricator.wikimedia.org/T429061#12024164 (10APizzata-WMF) > maybe you have ideas about the difference between MWH and MWU... [14:42:16] 06Data-Engineering, 10FR-Tech-Analytics, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Epic: Add FR Tech s3 secrets to airflow vars - https://phabricator.wikimedia.org/T429048#12024242 (10brouberol) Confirmed that a DAG can access the fr-tech minio endpoint after adding secrets and the SSL CA certificat... [14:42:39] 06Data-Engineering, 10FR-Tech-Analytics, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Epic: Add FR Tech s3 secrets to airflow vars - https://phabricator.wikimedia.org/T429048#12024244 (10brouberol) 05In progress→03Resolved [14:43:01] FIRING: [6x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [14:43:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [14:43:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [14:43:15] FIRING: [6x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [14:43:16] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [14:43:16] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [14:59:11] 06Data-Engineering: Track mobile app and api account registration separately - https://phabricator.wikimedia.org/T429211#12024334 (10Ottomata) Assuming this is about permanent user account registration: If MW gave us some indication of how the user account is being created, we could add this to {T423952}. [15:02:51] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): WE5.3.3b: Contributor Count Per Page [Attribution API] - https://phabricator.wikimedia.org/T426316#12024351 (10AKhatun_WMF) [15:08:55] 06Data-Engineering, 06Data-Persistence, 10Dumps-Generation: XML dumps does not re-load the config for depooled databases - https://phabricator.wikimedia.org/T429282#12024383 (10Marostegui) @Ahoelzl is this something you can help us get prioritised? Thank you! [15:15:56] (03CR) 10A-pizzata: "missing trailing coma and replied to the commented questions" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1302824 (https://phabricator.wikimedia.org/T428928) (owner: 10Joal) [15:17:06] !log Test Kitchen experiment (poll 9770) - adds: cite-footnote-content-interaction-experiment; removes: none; fields: none - TK tips at https://w.wiki/FwuD [15:17:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:31:19] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Create mediawiki.user_change event stream - https://phabricator.wikimedia.org/T423952#12024612 (10Ottomata) [15:31:51] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Platform, 10DPE-MediaWiki-Incremental-History: Inconsistent counts of global account registrations in analytics datasets - https://phabricator.wikimedia.org/T429061#12024621 (10Ottomata) FYI: {T423952}. `event.mediawiki_user_change_dev0` has data now. [15:45:20] 06Data-Engineering, 10BetaFeatures, 06cloud-services-team, 10Data-Services, and 5 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#12024691 (10aranyap) Thank you @SD0001 . The Product Safety & Integrity team has reviewed this table and is c... [15:53:01] FIRING: [6x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [15:53:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [15:53:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [15:58:01] FIRING: [6x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [15:58:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [15:58:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [16:08:38] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: PyFlink - Enable partition discovery - https://phabricator.wikimedia.org/T429369 (10JMonton-WMF) 03NEW [16:42:37] 06Data-Engineering: Master's Thesis Proposal: Contributing to Wikimedia's Data Platform - https://phabricator.wikimedia.org/T428674#12025141 (10JVanderhoop-WMF) Hi @JaimeAvaloss, thanks for reaching out! Unfortunately we on the experiment platform side don't have the capacity right now, so I'll remove our projec... [17:00:25] 06Data-Engineering, 10DNS, 06SRE, 07Kubernetes: 10.67.28.73 reverse DNS showing 2(SERVFAIL) - https://phabricator.wikimedia.org/T428573#12025310 (10BCornwall) [17:20:42] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Platform, 10DPE-MediaWiki-Incremental-History: Inconsistent counts of global account registrations in analytics datasets - https://phabricator.wikimedia.org/T429061#12025402 (10nshahquinn-wmf) 05Open→03Invalid >>! In T429061#12024164, @APizzata-W... [17:25:59] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for Bliviero - https://phabricator.wikimedia.org/T428815#12025437 (10BCornwall) Please don't be afraid to ask if there's any other access issue :) [17:42:41] 06Data-Engineering, 06cloud-services-team, 10Data-Services, 06Privacy Engineering, 13Patch-For-Review: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#12025675 (10SD0001) Hi @aranyap, similar to T402145, could you review and approve this one too? For reference, this is... [17:46:44] 06Data-Engineering, 10Test Kitchen, 07Essential-Work: Improve instrument event data data lake management - https://phabricator.wikimedia.org/T429385#12025734 (10mpopov) [17:49:36] 06Data-Engineering, 10Test Kitchen, 07Essential-Work: Improve instrument event data data lake management - https://phabricator.wikimedia.org/T429385#12025764 (10mpopov) Funny enough, @JAllemandou and I both independently arrived at this proposal and it was a delight to find ourselves totally in-sync on this. [17:58:01] FIRING: [6x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [17:58:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [17:58:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [18:03:01] FIRING: [6x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [18:03:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [18:03:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [18:03:20] 06Data-Engineering: Refine the standard suite of Test Kitchen metrics to Iceberg tables. - https://phabricator.wikimedia.org/T419418#12025854 (10mpopov) 05Open→03Invalid Closing this in favor of {T429385} (instrument-produced data would go into `timestamp` and `instrument_name`-partitioned Iceberg table)... [18:03:57] 06Data-Engineering, 10Test Kitchen, 07Essential-Work: Improve instrument event data data lake management - https://phabricator.wikimedia.org/T429385#12025867 (10Ottomata) Q: Would {T417176} with {T377600} be another alternative? [18:27:56] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Upgrade eventstreams and eventstreams-internal to node24 (or node22) - https://phabricator.wikimedia.org/T420257#12025926 (10Ahoelzl) p:05Triage→03High a:03tchin [18:38:01] FIRING: [3x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [18:38:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [18:38:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [18:43:01] RESOLVED: [3x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [18:43:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [18:43:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [19:13:43] 06Data-Engineering, 10Test Kitchen, 07Essential-Work: Improve instrument event data data lake management - https://phabricator.wikimedia.org/T429385#12026173 (10mpopov) Maybe/possibly. Depends on how custom that custom partitioning can be. I guess we could have an `event_iceberg.product_metrics_web_base` ta... [20:17:49] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Upgrade eventstreams and eventstreams-internal to node24 (or node22) - https://phabricator.wikimedia.org/T420257#12026560 (10Ahoelzl) Scheduled for next up work in DE. [20:31:55] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Request for backfill of webrequest with is_api_request and ip_provenance columns - https://phabricator.wikimedia.org/T427474#12026589 (10Ahoelzl) @JAllemandou @KCVelaga_WMF can you provide an update on the progress? [20:32:22] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Request for backfill of webrequest with is_api_request and ip_provenance columns - https://phabricator.wikimedia.org/T427474#12026593 (10Ahoelzl) a:03KCVelaga_WMF [20:32:35] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Request for backfill of webrequest with is_api_request and ip_provenance columns - https://phabricator.wikimedia.org/T427474#12026594 (10Ahoelzl) [20:36:32] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Check Editor Counts - https://phabricator.wikimedia.org/T427548#12026622 (10Ahoelzl) @AKhatun_WMF do we have any sign off criteria and timeline for the evaluation? @HCoplin-WMF who is driving the eval effort on your side? cc @Bmueller [20:37:31] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: mediawiki.page_change.v1 event - Add user first_registration_dt field - https://phabricator.wikimedia.org/T426998#12026629 (10Ahoelzl) p:05Triage→03High a:03Ottomata [20:44:20] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Iceberg 1.6.1 bug makes SELECTs fail due to vectorized read path being the default - https://phabricator.wikimedia.org/T426801#12026661 (10Ahoelzl) a:03APizzata-WMF [20:45:07] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Iceberg 1.6.1 bug makes SELECTs fail due to vectorized read path being the default - https://phabricator.wikimedia.org/T426801#12026664 (10Ahoelzl) Moving this to v2, Q1. @APizzata-WMF can you update the title? [21:08:07] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: mediawiki.page_change.v1 event - add a 'new revision created' field - https://phabricator.wikimedia.org/T409464#12026736 (10Ahoelzl) a:03Ottomata [21:08:26] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: mediawiki.page_change.v1 event - add a 'new revision created' field - https://phabricator.wikimedia.org/T409464#12026739 (10Ahoelzl) @Ottomata MWH inc v2? [21:10:48] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): [SPIKE] Evaluate Astronomer Cosmos for dbt scheduling & external dependency handling in Airflow - https://phabricator.wikimedia.org/T425038#12026750 (10Ahoelzl) p:05Triage→03Medium a:03amastilovic [21:12:06] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: mediawiki.page_change event - add access_method & platform details - https://phabricator.wikimedia.org/T424887#12026757 (10Ahoelzl) a:03Ottomata [21:13:19] 06Data-Engineering: Airflow devenv cannot run SkeinOperator - https://phabricator.wikimedia.org/T429410 (10AKhatun_WMF) 03NEW [23:09:18] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Include pageview_actor in Turnilo - https://phabricator.wikimedia.org/T424235#12027201 (10Ahoelzl) @Hghani can you help us understand priority? @CDanis would this something helpful to your work? [23:09:57] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Include pageview_actor in Turnilo - https://phabricator.wikimedia.org/T424235#12027202 (10Ahoelzl) a:03JAllemandou [23:14:17] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Backfill commonswiki and enwiki HTML for latest HTML when non-existent in event.mediawiki_page_html_content_change_v1 - https://phabricator.wikimedia.org/T426347#12027213 (10Ahoelzl) @dr0ptp4kt please help prioritize and if you need help... [23:14:58] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Backfill commonswiki and enwiki HTML for latest HTML when non-existent in event.mediawiki_page_html_content_change_v1 - https://phabricator.wikimedia.org/T426347#12027228 (10Ahoelzl) a:03dr0ptp4kt [23:15:44] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Emit comprehensive mediawiki user block change information in an event stream - https://phabricator.wikimedia.org/T424685#12027235 (10Ahoelzl) a:03Ottomata [23:17:55] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Investigate oversized driver logs for `mediawiki_history_denormalize` - https://phabricator.wikimedia.org/T424358#12027241 (10Ahoelzl) a:03APizzata-WMF [23:18:50] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Unbalanced partitions in eqiad.mediawiki.content_history_reconcile.v1 topic - https://phabricator.wikimedia.org/T420359#12027254 (10Ahoelzl) a:03JMonton-WMF [23:19:31] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Reconcile Flink Job: Too many warnings - https://phabricator.wikimedia.org/T420353#12027256 (10Ahoelzl) a:03JMonton-WMF [23:20:43] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Analyze size distribution of wiki page html - https://phabricator.wikimedia.org/T419495#12027258 (10Ahoelzl) p:05Triage→03Medium a:03Ottomata [23:21:35] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Analyze size distribution of wiki page html - https://phabricator.wikimedia.org/T419495#12027261 (10Ahoelzl) 05Open→03Resolved [23:22:06] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): The revision_seconds_to_identity_revert field in wmf.mediawiki_history has sometimes negative values - https://phabricator.wikimedia.org/T419267#12027263 (10Ahoelzl) a:03JAllemandou [23:24:54] 06Data-Engineering, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Task Tries and Logs for Airflow DAGs sometimes unavailable - https://phabricator.wikimedia.org/T419162#12027265 (10Ahoelzl) [23:25:25] 06Data-Engineering, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Task Tries and Logs for Airflow DAGs sometimes unavailable - https://phabricator.wikimedia.org/T419162#12027268 (10Ahoelzl) @Gehel looks like this is on the SRE side, correct? [23:26:06] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Optimize enqueueing of refine_webrequest_hourly pipeline - https://phabricator.wikimedia.org/T419050#12027269 (10Ahoelzl) a:03Antoine_Quhen [23:31:56] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 07OKR-Work (WE1 FY2025-26): [Spike] Adding access_method metadata to moderator action event streams - https://phabricator.wikimedia.org/T419019#12027273 (10Ahoelzl) @CMyrick-WMF do you need more input from DE? [23:32:07] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 07OKR-Work (WE1 FY2025-26): [Spike] Adding access_method metadata to moderator action event streams - https://phabricator.wikimedia.org/T419019#12027274 (10Ahoelzl) a:03Ahoelzl [23:33:49] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): haproxykafka and varnishkafka sent different uri_paths - https://phabricator.wikimedia.org/T418767#12027280 (10Ahoelzl) @Milimetric sounds like this should be prioritized and maybe handed over? [23:34:44] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Create a data product of IP range to owner/provenance label - https://phabricator.wikimedia.org/T418466#12027281 (10Ahoelzl) a:03GGoncalves-WMF [23:35:25] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Create a data product of IP range to owner/provenance label - https://phabricator.wikimedia.org/T418466#12027295 (10Ahoelzl) @GGoncalves-WMF assigned to you for prioritization. [23:40:36] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): [OpsWeek] Testing on airflow-devenvs can generate false alerts such as SLO misses - https://phabricator.wikimedia.org/T416596#12027306 (10Ahoelzl) a:03Ahoelzl [23:41:07] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Secret management on airflow for the automated transfer of (public) datasets from stats infra --> WME AWS - https://phabricator.wikimedia.org/T415208#12027308 (10Ahoelzl) a:03Snwachukwu [23:41:29] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Secret management on airflow for the automated transfer of (public) datasets from stats infra --> WME AWS - https://phabricator.wikimedia.org/T415208#12027309 (10Ahoelzl) @Htriedman @Snwachukwu is this resolved? [23:41:59] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 07Essential-Work: Refine generates very large XCOM values - https://phabricator.wikimedia.org/T414953#12027310 (10Ahoelzl) a:03Antoine_Quhen [23:42:22] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 07Essential-Work: Refine generates very large XCOM values - https://phabricator.wikimedia.org/T414953#12027311 (10Ahoelzl) @Antoine_Quhen is this still relevant? [23:42:44] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Analyze and optimize Airflow Postgres backend performance - https://phabricator.wikimedia.org/T411990#12027312 (10Ahoelzl) a:03Antoine_Quhen [23:43:00] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Analyze and optimize Airflow Postgres backend performance - https://phabricator.wikimedia.org/T411990#12027315 (10Ahoelzl) @Antoine_Quhen is this still relevant? [23:43:31] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Airflow main performance instance optimization - https://phabricator.wikimedia.org/T411988#12027316 (10Ahoelzl) a:03Antoine_Quhen [23:43:40] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Airflow main performance instance optimization - https://phabricator.wikimedia.org/T411988#12027318 (10Ahoelzl) @Antoine_Quhen assigning to you for tracking [23:44:34] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Data-Engineering-Wikistats, 06Movement-Insights: NEW FEATURE REQUEST: Temp Accounts on Wikistats - https://phabricator.wikimedia.org/T410796#12027334 (10Ahoelzl) @Milimetric can you help assess? [23:44:45] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Engineering-Wikistats, 06Movement-Insights: NEW FEATURE REQUEST: Temp Accounts on Wikistats - https://phabricator.wikimedia.org/T410796#12027335 (10Ahoelzl) a:03Ahoelzl [23:45:07] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data-Engineering-Wikistats: The stat site show INVALID on Cantonese Wikipedia. - https://phabricator.wikimedia.org/T411938#12027337 (10Ahoelzl) a:03Ahoelzl [23:45:54] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Migrate Sqoop jobs to Airflow - https://phabricator.wikimedia.org/T409514#12027339 (10Ahoelzl) a:03amastilovic [23:46:32] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Migrate Sqoop jobs to Airflow - https://phabricator.wikimedia.org/T409514#12027342 (10Ahoelzl) We got some new motivation for that: https://phabricator.wikimedia.org/T425385 [23:47:03] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Fix iceberg table location in hive metastore - https://phabricator.wikimedia.org/T408939#12027344 (10Ahoelzl) a:03JAllemandou [23:47:42] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data Pipelines, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Essential-Work: Airflow dynamic task mapping logs mix up when, on rerun, an id is mapped to a different map_index_template - https://phabricator.wikimedia.org/T408802#12027346 (10Aho... [23:47:51] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Data Pipelines, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Essential-Work: Airflow dynamic task mapping logs mix up when, on rerun, an id is mapped to a different map_index_template - https://phabricator.wikimedia.org/T408802#12027348 (10Aho... [23:48:38] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE, 06SRE: Move Druid realtime configuration out of Refinery into standalone repo on GitLab - https://phabricator.wikimedia.org/T407994#12027349 (10Ahoelzl) a:03amastilovic [23:48:58] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Do not set WMF-Last-Access cookie when Sec-Fetch-Dest is not 'document' - https://phabricator.wikimedia.org/T403897#12027362 (10Ahoelzl) a:03Ottomata [23:49:37] 06Data-Engineering, 06Java-Scala-Standardization, 07Essential-Work: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514#12027363 (10Ahoelzl) [23:50:07] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): refine_to_hive dag optimizations - https://phabricator.wikimedia.org/T392668#12027365 (10Ahoelzl) a:03Antoine_Quhen [23:50:45] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): refine_to_hive dag optimizations - https://phabricator.wikimedia.org/T392668#12027368 (10Ahoelzl) @Antoine_Quhen can you re-assess priority? [23:51:07] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): [Refine Simplification] Remove Schema Merging in Refine Process by Enforcing Backward Compatibility - https://phabricator.wikimedia.org/T381072#12027369 (10Ahoelzl) a:03Antoine_Quhen [23:51:19] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): [Refine Simplification] Remove Schema Merging in Refine Process by Enforcing Backward Compatibility - https://phabricator.wikimedia.org/T381072#12027371 (10Ahoelzl) @Antoine_Quhen status? [23:52:20] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Observability-Metrics: [Data Quality] Sending Apache Spark metrics to PushGateway - https://phabricator.wikimedia.org/T297231#12027373 (10Ahoelzl) a:03JAllemandou