[03:18:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [05:20:50] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10930445 (10Marostegui) [05:33:53] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10930453 (10Marostegui) [07:18:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [08:01:15] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 10ci-test-error (WMF-deployed Build Failure), 10Event-Platform, and 2 others: phpunit\integration\PageChangeEmissionTest::testPageMove with data set "Valid move with redirect" ('... - https://phabricator.wikimedia.org/T397087#10930761 [08:15:15] 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: Investigate Airflow scheduler performance on main deployment - https://phabricator.wikimedia.org/T396686#10930951 (10Antoine_Quhen) I've added a `pod counts` graph here: https://grafana.wikimedia.org/d/ca9c0221-4a0d-4833-865b-f14a3e813c97/... [09:13:06] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 10ci-test-error (WMF-deployed Build Failure), 10Event-Platform, and 2 others: phpunit\integration\PageChangeEmissionTest::testPageMove with data set "Valid move with redirect" ('... - https://phabricator.wikimedia.org/T397087#10931251 [09:22:20] 06Data-Engineering, 06Data-Platform-SRE: Enable async queries for Superset with Celery - https://phabricator.wikimedia.org/T397338#10931268 (10JAllemandou) +1 for per-database config so that we only enable asynchronous querying for Spark compute-engine. About `spark-thrift-server` vs `Apache kyuubi`, I don't... [09:29:51] 06Data-Engineering, 06Data-Platform-SRE: Enable async queries for Superset with Celery - https://phabricator.wikimedia.org/T397338#10931294 (10GGoncalves-WMF) Excellent discussion, thank you both. I won't have an informed opinion until I interview analysts over the next 2-3 weeks, but my initial impression se... [10:56:15] 06Data-Engineering, 06Data-Engineering-Icebox, 06SRE Observability, 10Data-Platform-SRE (2025.06.13 - 2025.07.04), 13Patch-For-Review: [Data Platform] Install a Prometheus connector for Presto, pointed at thanos-query - https://phabricator.wikimedia.org/T347430#10931568 (10BTullis) >>! In T347430#1091756... [11:18:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [11:48:38] 06Data-Engineering, 10MediaWiki-extensions-WikimediaEvents, 13Patch-For-Review, 07PHP 8.1 support, 07Wikimedia-production-error: PHP Deprecated: preg_split(): Passing null to parameter #2 ($subject) of type string is deprecated - https://phabricator.wikimedia.org/T388812#10931723 (10Aklapper) Still seen... [12:09:57] 06Data-Engineering, 06Data-Engineering-Icebox, 06SRE Observability, 10Data-Platform-SRE (2025.06.13 - 2025.07.04), 13Patch-For-Review: [Data Platform] Install a Prometheus connector for Presto, pointed at thanos-query - https://phabricator.wikimedia.org/T347430#10931857 (10fgiunchedi) Thank you for the d... [12:26:22] 06Data-Engineering, 06Data-Platform-SRE: Airflow UI sometimes shows no response for a DAG run task with many mapped tasks - https://phabricator.wikimedia.org/T381479#10931905 (10BTullis) Let's take this out of the current milestone, as we are not currently looking into it. I haven't spotted it recently, but I... [15:13:24] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10932517 (10Marostegui) [15:18:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [15:47:57] 10Data-Engineering (Q4 2025 April 1st - June 30th): Investigate mw-page-content-change memory alerts - https://phabricator.wikimedia.org/T397336#10932604 (10gmodena) I don't see anything unusual with this job. It might just be that 1GB of memory is on the low end. Bumping to 1.5 and will keep monitoring. [16:06:38] 06Data-Engineering, 10Data-Platform-SRE (2025.06.13 - 2025.07.04): Request for dedicated Airflow instance for WME - https://phabricator.wikimedia.org/T396672#10932657 (10BTullis) >>! In T396672#10925243, @HShaikh wrote: > Hey ben, So for the need for new instance was looking at it from a future isolation persp... [18:08:00] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 10ci-test-error (WMF-deployed Build Failure), 10Event-Platform, and 2 others: phpunit\integration\PageChangeEmissionTest::testPageMove with data set "Valid move with redirect" ('... - https://phabricator.wikimedia.org/T397087#10932928 [18:47:37] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 10ci-test-error (WMF-deployed Build Failure), 10Event-Platform, and 2 others: phpunit\integration\PageChangeEmissionTest::testPageMove with data set "Valid move with redirect" (... - https://phabricator.wikimedia.org/T397087#10933025 [19:18:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [23:10:18] 06Data-Engineering, 10ContentTranslation, 10MediaWiki-extensions-EventLogging, 10Metrics Platform, 10MW-1.45-notes (1.45.0-wmf.7; 2025-06-24): Update WMF-deployed extensions to use mw.config checks instead of manual m-dot URL hacks - https://phabricator.wikimedia.org/T390923#10933315 (10Krinkle) 05O... [23:11:30] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Remove m-dot subdomain, serve mobile and desktop variants through the same URL - https://phabricator.wikimedia.org/T214998#10933319 (10Krinkle) [23:18:29] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem