[01:14:20] 10Data-Engineering (Q4 2025 April 1st - June 30th): Investigate Airflow scheduler performance on main deployment - https://phabricator.wikimedia.org/T396686 (10amastilovic) 03NEW [01:33:30] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [05:33:30] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [06:07:02] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10907584 (10Marostegui) [07:30:33] 06Data-Engineering, 06SRE: WE 5.4 FY 25/26: Improve automata detection at the edge and pass it to the refinery pipeline - https://phabricator.wikimedia.org/T396562#10907844 (10JAllemandou) This idea is great, thank you @Joe for filling this. I concur with the idea that when the frontend computes some sort of... [07:48:42] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06cloud-services-team, 10Cloud-VPS, 07IPv6: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468#10907888 (10JAllemandou) [07:49:44] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06cloud-services-team, 10Cloud-VPS, 07IPv6: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468#10907889 (10JAllemandou) 05Open→03Resolved Sorry I forgot to follow up. The boxes are ticked, the code is live, I'm resolvi... [08:37:40] 06Data-Engineering, 10Data-Services: Create a view for existencelinks table - https://phabricator.wikimedia.org/T394898#10908068 (10Tacsipacsi) Combined with {T395366}, this is very bad and urgent. Useful and previously-accessible data is currently **COMPLETELY** inaccessible for anyone without an NDA signed:... [09:23:38] 06Data-Engineering: AlertLintProblem - https://phabricator.wikimedia.org/T395539#10908323 (10phaultfinder) [09:33:30] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [12:53:55] 06Data-Engineering, 06Data-Engineering-Icebox, 06SRE Observability, 10Data-Platform-SRE (2025.05.24 - 2025.06.13): [Data Platform] Install a Prometheus connector for Presto, pointed at thanos-query - https://phabricator.wikimedia.org/T347430#10908972 (10fgiunchedi) Thank you for working on this -- for my o... [13:34:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:49:59] 06Data-Engineering, 10EventStreams: SSE events from offline DC topics - https://phabricator.wikimedia.org/T396564#10909238 (10Gehel) To add some context, this is problematic when consuming the [[ https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Streaming_Updater/Public_Update_Stream | Wikidata RDF st... [14:38:31] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Movement-Insights, 06Traffic: NEW BUG REPORT: Investigate rise in May 2025 Reader metrics - https://phabricator.wikimedia.org/T395934#10909406 (10OSefu-WMF) [15:51:57] 10Quarry, 10cloud-services-team (FY2024/2025-Q3-Q4): Improve Quarry's observability - https://phabricator.wikimedia.org/T396770 (10taavi) 03NEW p:05Triage→03High [15:52:22] 10Quarry, 10cloud-services-team (FY2024/2025-Q3-Q4): Improve Quarry's observability - https://phabricator.wikimedia.org/T396770#10909953 (10taavi) a:03taavi [15:53:59] 10Quarry, 10cloud-services-team (FY2024/2025-Q3-Q4): Deploy prometheus-redis-exporter - https://phabricator.wikimedia.org/T396771 (10taavi) 03NEW [15:54:06] 10Quarry, 10cloud-services-team (FY2024/2025-Q3-Q4): Deploy prometheus-redis-exporter - https://phabricator.wikimedia.org/T396771#10909972 (10taavi) p:05Triage→03High [15:54:16] 10Quarry, 10cloud-services-team (FY2024/2025-Q3-Q4): Deploy prometheus-redis-exporter - https://phabricator.wikimedia.org/T396771#10909975 (10taavi) a:03taavi [15:57:00] 10Quarry, 10cloud-services-team (FY2024/2025-Q3-Q4): quarry is leaking tmp files - https://phabricator.wikimedia.org/T395237#10909983 (10taavi) a:03taavi I am planning to "fix" this by disabling the Excel export feature. [16:00:48] 10Quarry: worker nodes issue with garbage collection - https://phabricator.wikimedia.org/T375997#10910003 (10taavi) I suspect this is a duplicate of T395237 so merging this there. [16:00:56] 10Quarry: worker nodes issue with garbage collection - https://phabricator.wikimedia.org/T375997#10910006 (10taavi) →14Duplicate dup:03T395237 [16:01:00] 10Quarry, 10cloud-services-team (FY2024/2025-Q3-Q4): quarry is leaking tmp files - https://phabricator.wikimedia.org/T395237#10910008 (10taavi) [16:01:46] 10Quarry: Setup an easy way to have Quarry dump information / results on a wiki page - https://phabricator.wikimedia.org/T137179#10910011 (10taavi) 05Open→03Invalid I'm boldly declining this, as there's already a bot that does essentially the same thing. [16:02:00] 06Data-Engineering, 06SRE: WE 5.4 FY 25/26: Improve automata detection at the edge and pass it to the refinery pipeline - https://phabricator.wikimedia.org/T396562#10910014 (10CDanis) >>! In T396562#10907844, @JAllemandou wrote: > This idea is great, thank you @Joe for filling this. > > I concur with the idea... [16:35:04] 10Data-Engineering (Q4 2025 April 1st - June 30th): Investigate Airflow scheduler performance on main deployment - https://phabricator.wikimedia.org/T396686#10910211 (10BTullis) I have just applied this change: [Bump up the CPU and RAM resources for airflow related namespaces](https://gerrit.wikimedia.org/r/c/op... [16:47:12] 10Quarry, 10cloud-services-team (FY2024/2025-Q3-Q4): Fix Quarry's Redis pod exiting causing frequent outages - https://phabricator.wikimedia.org/T396785 (10taavi) 03NEW p:05Triage→03High [17:34:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [18:04:57] 10Quarry: Setup an easy way to have Quarry dump information / results on a wiki page - https://phabricator.wikimedia.org/T137179#10910587 (10Stevietheman) See also the [[ https://en.wikipedia.org/wiki/Template:Database_report | Database report ]] template in the English Wikipedia. It's a brilliant way of usi... [18:33:15] 10Data-Engineering (Q4 2025 April 1st - June 30th): Investigate Airflow scheduler performance on main deployment - https://phabricator.wikimedia.org/T396686#10910714 (10amastilovic) [21:25:33] 10Data-Engineering (Q4 2025 April 1st - June 30th): Investigate Airflow scheduler performance on main deployment - https://phabricator.wikimedia.org/T396686#10911157 (10amastilovic) No change in scheduler utilization after bumping up the CPU and RAM resources: {F62306502} [21:34:27] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [22:23:47] 06Data-Engineering, 13Patch-For-Review: Update the sqoop configuration for mediawiki to obtain linktarget from the production replicas, instead of wikireplicas - https://phabricator.wikimedia.org/T352879#10911329 (10Zabe) >>! In T352879#10911325, @gerritbot wrote: > Change #1156466 had a related patch set...