[03:18:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [05:14:40] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10933494 (10Marostegui) [05:22:00] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10933495 (10Marostegui) [05:28:19] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10933500 (10Marostegui) [07:18:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [08:03:07] 06Data-Engineering-Radar, 10QuickSurveys, 06WMDE-TechWish: QuickSurveys should show an error when response is blocked - https://phabricator.wikimedia.org/T256463#10933656 (10phuedx) [08:28:04] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 07Technical-Debt: Integration tests should exercise code with and without EventStreamConfig loaded - https://phabricator.wikimedia.org/T344239#10933723 (10phuedx) [10:46:37] 06Data-Engineering, 06Data-Engineering-Icebox, 06SRE Observability, 10Data-Platform-SRE (2025.06.13 - 2025.07.04): [Data Platform] Install a Prometheus connector for Presto, pointed at thanos-query - https://phabricator.wikimedia.org/T347430#10934184 (10BTullis) This is now working, so we can query prometh... [11:13:27] 06Data-Engineering, 06Data-Engineering-Icebox, 06SRE Observability, 10Data-Platform-SRE (2025.06.13 - 2025.07.04): [Data Platform] Install a Prometheus connector for Presto, pointed at thanos-query - https://phabricator.wikimedia.org/T347430#10934231 (10BTullis) I have now added the thanos connector to Sup... [11:18:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:20:08] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Dumps-Generation: wikidatawiki fails dumps of the wbt_* tables, also lagging on XML Dumps - https://phabricator.wikimedia.org/T396125#10934469 (10xcollazo) Partial XML Dump for `2025-06-20` also failed to dump the `wbt` tables: https://dumps.wikimedia.org/w... [14:24:04] 06Data-Engineering, 06Data-Engineering-Radar, 06Experimentation Lab, 10MediaWiki-extensions-EventLogging, 07Technical-Debt: Remove deprecated $logger parameter from EventLogging::submit() - https://phabricator.wikimedia.org/T376410#10934648 (10phuedx) [14:41:51] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Create Airflow pool for bursty MW Content Pipelines tasks - https://phabricator.wikimedia.org/T397333#10934708 (10xcollazo) 05Open→03In progress p:05Triage→03High [14:56:57] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Data-Quality, 10DPE-Mediawiki-Content: Duplicate rows wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T397525 (10xcollazo) 03NEW [14:57:05] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Data-Quality, 10DPE-Mediawiki-Content: Duplicate rows wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T397525#10934765 (10xcollazo) p:05Triage→03High [15:02:25] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 13Patch-For-Review, 07Technical-Debt: Update EventLogging to use the new HookContainer/HookRunner system - https://phabricator.wikimedia.org/T346540#10934786 (10phuedx) [15:18:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [17:14:12] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Dumps-Generation: wikidatawiki fails dumps of the wbt_* tables, also lagging on XML Dumps - https://phabricator.wikimedia.org/T396125#10935198 (10BTullis) I had a quick look at this. It seems that the tables aren't showing up in the list of tables than need... [17:24:53] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Data-Quality, 10DPE-Mediawiki-Content: Duplicate rows wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T397525#10935214 (10xcollazo) [[ https://airflow.wikimedia.org/dags/mw_content_merge_changes_to_mw_content_current_daily... [19:05:22] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Data-Quality, 10DPE-Mediawiki-Content: Duplicate rows wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T397525#10935336 (10xcollazo) As per distribution in description above, there are no issues on the big wikis (`enwiki`,... [19:18:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [19:43:07] 06Data-Engineering, 06Data-Engineering-Radar, 10CheckUser, 06Trust and Safety Product Team, and 2 others: Add '*_actor_ip_hex_time' indexes to 'cu_changes', 'cu_log_event', and 'cu_private_event' - https://phabricator.wikimedia.org/T395683#10935443 (10Dreamy_Jazz) [20:51:57] 06Data-Engineering, 06Traffic, 10MediaWiki-Platform-Team (Radar): Add ismobile attribute to X-Analytics header - https://phabricator.wikimedia.org/T390924#10935663 (10Krinkle) 05Open→03Resolved This is now live. ` $ curl -I 'https://en.m.wikipedia.org/wiki/Main_Page' -H 'X-Wikimedia-Debug: 1' … x-an... [20:52:15] 10Data-Engineering (Q4 2025 April 1st - June 30th): Analyze impact for webrequest and unique devices pipelines to derive access_method without m-dot domain - https://phabricator.wikimedia.org/T389696#10935668 (10Krinkle) >>! In T390924#10935663, @Krinkle wrote: > This is now live. > > ` > $ curl -I 'https://en.... [21:51:53] 10Data-Engineering (Q4 2025 April 1st - June 30th): Analyze impact for webrequest and unique devices pipelines to derive access_method without m-dot domain - https://phabricator.wikimedia.org/T389696#10935799 (10mforns) Awesome! [22:33:30] (03PS1) 10Andrea Denisse: graphite: Delete deprecated TemplateData metrics [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/1162101 (https://phabricator.wikimedia.org/T359271) [22:35:44] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Data-Quality, 10DPE-Mediawiki-Content: Duplicate rows wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T397525#10935860 (10xcollazo) Success: ` analytics@an-launcher1002:/home/xcollazo$ screen -S xcollazo_mw_content_history... [22:39:24] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Data-Quality, 10DPE-Mediawiki-Content: Duplicate rows wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T397525#10935862 (10xcollazo) Verification that issue is gone: ` # verification after hot fix spark.sql(""" SELECT count... [23:18:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem