[01:02:00] FIRING: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [01:02:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [01:02:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [01:22:01] RESOLVED: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [01:22:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [01:22:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [03:07:01] 06Data-Engineering, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 07Epic, 13Patch-For-Review: Upgrade Spark to a version with long term Iceberg support - https://phabricator.wikimedia.org/T338057#11920431 (10nshahquinn-wmf) I'm very excited this is happening! @BTullis and @xcollazo, thank you for working... [09:15:47] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635#11920763 (10FCeratto-WMF) [09:35:04] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Schema and Stream for "webrequest.page_view" - https://phabricator.wikimedia.org/T426091#11920829 (10JMonton-WMF) [09:36:09] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Audit and fix observability (logging and metrics) for pyflink jobs - https://phabricator.wikimedia.org/T418996#11920833 (10JMonton-WMF) Not sure if this is the best ticket to report it, but we should log this somewhe... [09:36:59] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Audit and fix observability (logging and metrics) for pyflink jobs - https://phabricator.wikimedia.org/T418996#11920854 (10JMonton-WMF) Maybe we can close this ticket, we have decent monitoring of these pipelines. We... [09:39:09] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Streaming HTML & Edit Types - productionization checklist - https://phabricator.wikimedia.org/T423920#11920861 (10JMonton-WMF) [09:43:26] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Product-Analytics, 06Product Safety and Integrity (Sprint lily-of-the-valley (May 4 - May 22)): Backfill via AirFlow for time_to_revert_bad_faith_edits - https://phabricator.wikimedia.org/T425526#11920872 (10Tchanders) [09:57:36] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Streaming HTML & Edit Types - productionization checklist - https://phabricator.wikimedia.org/T423920#11920934 (10JMonton-WMF) We have to delete the following topics in Kafka Jumbo: ` codfw.mediawiki.page_html_cont... [11:09:02] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 05MW-1.47-notes (1.47.0-wmf.2; 2026-05-12), 13Patch-For-Review: Event schemas - mediawiki user entity should be wiki aware - https://phabricator.wikimedia.org/T426198#11921226 (10Ottomata) @Dream... [12:57:46] 06Data-Engineering, 10Data-Engineering-Wikistats, 10Pageviews-Anomaly: Sudden traffic increase on 1 November 2025 - https://phabricator.wikimedia.org/T412655#11921465 (10Oesjaar) Hi there! Our views for April was back to normal. Regards! [12:59:48] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): WE5.3.3b: Contributor Count Per Page [Attribution API] - https://phabricator.wikimedia.org/T426316 (10AKhatun_WMF) 03NEW [14:44:48] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635#11921975 (10FCeratto-WMF) [15:07:51] 06Data-Engineering, 06Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 07Essential-Work, and 2 others: Migrate "WikiLambda API" instrument to use the Test Kitchen SDK - https://phabricator.wikimedia.org/T415254#11922061 (10KReid-WMF) 05Open→03Resolved [15:11:41] 06Data-Engineering, 06Data-Engineering-Radar, 06Growth-Team, 10MediaWiki-extensions-WikimediaEvents, and 4 others: Could not hoist data into experiment.subject_id for event - https://phabricator.wikimedia.org/T421152#11922097 (10KReid-WMF) [15:32:36] 06Data-Engineering, 10EventStreams: Support namespace_prefix and page_title_noprefix - https://phabricator.wikimedia.org/T426268#11922231 (10Ottomata) Hm, I'm not opposed! Quick name bikeshed: - How about just `namespace` (since we already have the id as namespace_id) or uh `namespace_name`? I think MW calls... [15:48:08] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Backfill datasets affected by Nov 2025 automated traffic incident - https://phabricator.wikimedia.org/T421735#11922343 (10mforns) [15:58:51] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Add new API rate limiting fields from webrequest_logs to Turnilo view - https://phabricator.wikimedia.org/T419736#11922370 (10JTweed-WMF) This is amazing to have in place, thank you so much! [16:03:36] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Add new API rate limiting fields from webrequest_logs to Turnilo view - https://phabricator.wikimedia.org/T419736#11922384 (10HCoplin-WMF) +1 to what @JTweed-WMF said! This is great and so much easier & more convenient to use -- thank you so much for respond... [16:12:52] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Update dbt-jobs README.md and dbt Wikitech documentation with Troubleshooting section - https://phabricator.wikimedia.org/T426329 (10amastilovic) 03NEW [16:13:28] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Update dbt-jobs README.md and dbt Wikitech documentation with Troubleshooting sections - https://phabricator.wikimedia.org/T426329#11922435 (10amastilovic) [16:13:36] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 07Epic, 13Patch-For-Review: Incremental MediaWiki History - https://phabricator.wikimedia.org/T424350#11922437 (10xcollazo) 05Open→03In progress [16:16:24] (03PS6) 10Xcollazo: Add MWHistoryDeltaWriter and MWHistorySnapshotMerger to refinery-job-35 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1284858 (https://phabricator.wikimedia.org/T425729) [16:19:14] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Streaming HTML & Edit Types - productionization checklist - https://phabricator.wikimedia.org/T423920#11922490 (10AKhatun_WMF) Want to add ` eqiad.mw_page_edit_type_enrich.error ` [16:20:37] 06Data-Engineering, 10EventStreams: Support namespace_prefix and page_title_noprefix - https://phabricator.wikimedia.org/T426268#11922511 (10dr0ptp4kt) [16:20:41] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11922512 (10dr0ptp4kt) [16:21:41] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Implement MERGE INTO writers for mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T425729#11922524 (10xcollazo) **End-to-end testing on stat1011.eqiad.wmnet** against production data (Spark 3.5.8, YA... [16:22:19] 06Data-Engineering, 10EventStreams: Support namespace_prefix and page_title_noprefix - https://phabricator.wikimedia.org/T426268#11922526 (10dr0ptp4kt) [16:22:20] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11922527 (10dr0ptp4kt) [16:22:54] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Streaming HTML & Edit Types - productionization checklist - https://phabricator.wikimedia.org/T423920#11922535 (10AKhatun_WMF) We should also get rid of the hive tables for dev and rc0 versions. Can we just drop tabl... [16:25:15] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Streaming HTML & Edit Types - productionization checklist - https://phabricator.wikimedia.org/T423920#11922549 (10Ottomata) > Can we just drop tables? Do we also need to cleanup the hdfs files? Since these are exter... [16:31:57] (03PS1) 10AKhatun: Add mw feature counts table for event sanitization [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1287443 (https://phabricator.wikimedia.org/T423920) [16:35:41] (03CR) 10Ottomata: [C:03+1] Add mw feature counts table for event sanitization [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1287443 (https://phabricator.wikimedia.org/T423920) (owner: 10AKhatun) [17:01:27] 06Data-Engineering, 06DBA, 10DiscussionTools, 07Schema-change-in-production: Drop DiscussionTools database tables from wikis which don't need them - https://phabricator.wikimedia.org/T426341 (10Dreamy_Jazz) 03NEW [17:01:35] 06Data-Engineering, 06DBA, 10DiscussionTools, 07Schema-change-in-production: Drop DiscussionTools database tables from wikis which don't need them - https://phabricator.wikimedia.org/T426341#11922801 (10Dreamy_Jazz) [17:08:46] 06Data-Engineering, 10EventStreams: Support namespace_prefix and page_title_noprefix - https://phabricator.wikimedia.org/T426268#11922836 (10dr0ptp4kt) `namespace_name` is usually the English canonical version. Maybe do both of these? - `namespace_name_canonical` - `namespace_name_local` If you think it... [17:11:22] 06Data-Engineering, 06Wikimedia Enterprise: WME Pageviews DAG for HDFS to S3 Transfer - https://phabricator.wikimedia.org/T426017#11922852 (10JArguello-WMF) [17:12:02] 06Data-Engineering, 10EventStreams: Support new fields: (1) namespace textual prefix and (2) page title without prefix - https://phabricator.wikimedia.org/T426268#11922854 (10dr0ptp4kt) [17:12:33] 06Data-Engineering, 06Wikimedia Enterprise: PageViews S3 Data Transfer MR [Enterprise] - https://phabricator.wikimedia.org/T425543#11922856 (10JArguello-WMF) [17:13:11] 06Data-Engineering, 10EventStreams: Support new fields: (1) namespace textual prefix and (2) page title without prefix - https://phabricator.wikimedia.org/T426268#11922857 (10dr0ptp4kt) [17:13:17] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Research, 10Event-Platform, 13Patch-For-Review: Event stream with latest revision HTML & parent revision HTML diff - https://phabricator.wikimedia.org/T360794#11922858 (10dr0ptp4kt) [17:15:06] 06Data-Engineering, 10EventStreams: Support new fields: (1) namespace textual prefix and (2) page title without prefix - https://phabricator.wikimedia.org/T426268#11922864 (10dr0ptp4kt) (Post message send thought: `namespace_name` could also simply be the canonical one, and then we could let `namespace_name_lo... [17:20:21] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Incremental MWH - MediaWiki event data source improvements - https://phabricator.wikimedia.org/T423935#11922874 (10Ottomata) [17:21:00] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: mediawiki.page_change.v1 event - add a page namespace_is_content field - https://phabricator.wikimedia.org/T409462#11922876 (10Ottomata) [17:23:19] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: mediawiki.page_change.v1 - add revision.editor.first_edit_dt field - https://phabricator.wikimedia.org/T425029#11922879 (10Ottomata) a:03tchin I'm not sure if we should only add this in page_change... [17:24:31] (03CR) 10AKhatun: [C:03+2] Add mw feature counts table for event sanitization [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1287443 (https://phabricator.wikimedia.org/T423920) (owner: 10AKhatun) [17:24:40] (03CR) 10AKhatun: [V:03+2 C:03+2] Add mw feature counts table for event sanitization [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1287443 (https://phabricator.wikimedia.org/T423920) (owner: 10AKhatun) [17:25:14] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: mediawiki.page_change.v1 - add revision.editor.first_edit_dt field - https://phabricator.wikimedia.org/T425029#11922898 (10Ottomata) p:05Triage→03High [17:28:44] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11922906 (10Dzahn) 05In progress→03Stalled [18:04:07] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Architectural design agreement: Incremental MediaWiki History - https://phabricator.wikimedia.org/T424359#11922993 (10xcollazo) [18:12:56] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Backfill commonswiki and enwiki HTML for latest HTML when non-existent in event.mediawiki_page_html_content_change_v1 - https://phabricator.wikimedia.org/T426347 (10dr0ptp4kt) 03NEW [18:16:02] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Backfill commonswiki and enwiki HTML for latest HTML when non-existent in event.mediawiki_page_html_content_change_v1 - https://phabricator.wikimedia.org/T426347#11923029 (10dr0ptp4kt) [18:44:19] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Backfill commonswiki and enwiki HTML for latest HTML when non-existent in event.mediawiki_page_html_content_change_v1 - https://phabricator.wikimedia.org/T426347#11923079 (10Ottomata) What you want is a mediawiki_html_content_current like... [18:53:59] !log Test Kitchen edge-unique experiments (poll 239209) - adds: none; removes: synth-aa-ncs-1; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [18:54:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:39:08] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Add log_id to wmf.mediawiki_history - https://phabricator.wikimedia.org/T425986#11923202 (10xcollazo) Verification run for this ticket's `event_log_id`, but also for T425735's `event_user_is_cross_... [20:47:07] !log Test Kitchen edge-unique experiments (poll 239544) - adds: none; removes: none; fields: logged-out-retention-round11 - xLab/MPIC/TK tips at https://w.wiki/FwuD [20:47:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:48:37] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Backfill commonswiki and enwiki HTML for latest HTML when non-existent in event.mediawiki_page_html_content_change_v1 - https://phabricator.wikimedia.org/T426347#11923384 (10dr0ptp4kt) >>! In T426347#11923079, @Ottomata wrote: > What you... [20:59:35] !log Test Kitchen edge-unique experiments (poll 239581) - adds: none; removes: none; fields: logged-out-retention-round11 - xLab/MPIC/TK tips at https://w.wiki/FwuD [20:59:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:01:25] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform: Backfill commonswiki and enwiki HTML for latest HTML when non-existent in event.mediawiki_page_html_content_change_v1 - https://phabricator.wikimedia.org/T426347#11923427 (10dr0ptp4kt) [23:16:58] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Improve dbt-jobs GitLab CI/CD - https://phabricator.wikimedia.org/T426361 (10amastilovic) 03NEW [23:20:29] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Improve dbt-jobs GitLab CI/CD - https://phabricator.wikimedia.org/T426361#11923893 (10amastilovic)