[03:18:30] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [05:32:56] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10936946 (10Marostegui) [05:33:12] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10936947 (10Marostegui) [07:18:30] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [07:18:46] (03CR) 10Jforrester: [C:03+1] "LGTM." [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/1162101 (https://phabricator.wikimedia.org/T359271) (owner: 10Andrea Denisse) [07:20:00] !log remove an-conf100[1-3] from the zookeeper analytics cluster T374922 [07:20:02] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:20:03] T374922: Bring an-conf100[4-6] into service to replace an-conf100[1-3] - https://phabricator.wikimedia.org/T374922 [07:23:48] !log roll restart zookeeper analytics cluster to pickup removed hosts T374922 [07:23:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:41:28] !log restart 'hadoop-hdfs-zkfc.service' on an-master1004 T374922 [07:41:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:41:31] T374922: Bring an-conf100[4-6] into service to replace an-conf100[1-3] - https://phabricator.wikimedia.org/T374922 [07:43:33] !log restart 'hadoop-hdfs-zkfc.service' on an-master1003 T374922 [07:43:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:14:53] 06Data-Engineering, 06Data-Engineering-Radar, 06Infrastructure-Foundations, 06SRE, 10Data-Platform-SRE (2025.06.13 - 2025.07.04): Rebuild Spark images with Bookworm / bullseye-backports deprecation - https://phabricator.wikimedia.org/T390139#10937359 (10MoritzMuehlenhoff) The removal of bullseye-backport... [08:20:18] !log applying views fix to an-redacteddb1001 for T397508 [08:20:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:23:02] 06Data-Engineering, 06Data-Engineering-Icebox, 06SRE Observability, 10Data-Platform-SRE (2025.06.13 - 2025.07.04): [Data Platform] Install a Prometheus connector for Presto, pointed at thanos-query - https://phabricator.wikimedia.org/T347430#10937387 (10BTullis) 05Open→03Resolved [08:47:00] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10937465 (10Marostegui) [10:11:08] 06Data-Engineering, 06Experimentation Lab, 10Experimentation Lab Roadmap, 06Growth-Team, and 4 others: [EPIC] Deprecate EventLogging::logEvent() - https://phabricator.wikimedia.org/T318263#10937698 (10phuedx) [11:18:30] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:18:30] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:41:04] 06Data-Engineering, 10AbuseFilter, 10Data-Services, 06Privacy Engineering, and 6 others: Wiki replicas contain the hit count for private and protected filters - https://phabricator.wikimedia.org/T397508#10938830 (10sbassett) [16:25:59] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10939118 (10Marostegui) [16:27:27] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10939134 (10Marostegui) 05Open→03Resolved All done [17:40:51] 06Data-Engineering, 06Data-Engineering-Radar, 06Product-Analytics: Present "Notebooks in Airflow" solution to PA and discuss ownership of different steps - https://phabricator.wikimedia.org/T325181#10939574 (10nshahquinn-wmf) 05Open→03Declined As far as I know, there is no current plan to do {T322532... [18:40:51] 06Data-Engineering, 06Data-Engineering-Icebox, 06SRE Observability, 10Data-Platform-SRE (2025.06.13 - 2025.07.04): [Data Platform] Install a Prometheus connector for Presto, pointed at thanos-query - https://phabricator.wikimedia.org/T347430#10939712 (10nshahquinn-wmf) Thanks for the explanation, @btul... [19:35:40] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Data-Quality, 10DPE-Mediawiki-Content: Duplicate rows wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T397525#10939817 (10xcollazo) The issue seems related to the parallel ingestion of visibility and page moves events. Her... [23:18:30] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [23:34:24] (03CR) 10Cwhite: [C:03+1] "LGTM thanks!" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/1162101 (https://phabricator.wikimedia.org/T359271) (owner: 10Andrea Denisse)