[00:43:35] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [00:56:49] 06Data-Engineering, 10observability, 10Observability-Metrics, 10Event-Platform: Produce MediaWiki client emitted operational metrics into Event Platform, allowing them to be queried in the WMF Data Lake with SQL - https://phabricator.wikimedia.org/T390328#10886238 (10Ottomata) Hm! Yeah maybe! Do you know... [04:43:35] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [08:43:36] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [09:47:04] gmodena: the mysterious out of memory I talked about was due to fork(), a memory allocation for a copy of the entire calling process is made to the kernel which would refuse it if there is not enough memory [09:47:09] even if that is copy on write [09:47:35] so if a container has 4G, a process uses 2G and does a fork, that is an attempt toallocate 2G [09:48:09] that can be disabled via a kernel setting. I wrote a summary at https://phabricator.wikimedia.org/T219114#5084302 [09:48:19] but your issue is certainly a different one :-] [10:53:20] 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team, 06serviceops: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10887218 (10Clement_Goubert) [11:28:54] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, 06Privacy Engineering: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#10887329 (10fnegri) [11:29:25] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, 06Privacy Engineering: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#10887330 (10fnegri) [11:34:10] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, 06Privacy Engineering: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#10887344 (10fnegri) @VirginiaPoundstone I updated the task description with more context. I will also ask in the user-c... [11:35:08] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, 10Data-Platform-SRE (2025.05.24 - 2025.06.13): Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10887347 (10fnegri) Does anybody have a specific use case that needs to access t... [12:43:35] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [12:59:47] 06Data-Engineering, 10Data Pipelines: Provide an easy way for MediaWiki to fetch aggregate statistics from the data lake - https://phabricator.wikimedia.org/T341649#10887591 (10Seddon) [14:32:36] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 2 others: Testing the domain event refactoring with production data - https://phabricator.wikimedia.org/T394899#10887885 (10gmodena) [14:45:21] 06Data-Engineering, 10Dumps-Generation: wikidatawiki fails dumps of the wbt_* tables - https://phabricator.wikimedia.org/T396125 (10xcollazo) 03NEW [15:01:11] 06Data-Engineering, 06Data-Engineering-Radar, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user for "Dena WMDE" - https://phabricator.wikimedia.org/T396129 (10dena) 03NEW [15:05:11] 10Analytics-Canonical-Data, 06Data-Engineering, 06Data-Engineering-Icebox, 06Movement-Insights: Automatically update the canonical data tables - https://phabricator.wikimedia.org/T339928#10888051 (10mpopov) May I recommend splitting this into two sub-tasks? One for countries, one for wikis. [15:05:18] 06Data-Engineering, 06Data-Engineering-Radar, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user for "Dena WMDE" - https://phabricator.wikimedia.org/T396129#10888054 (10WMDE-leszek) I approve this request on WMDE's end. Account already in `nde` and `wmde` gr... [15:05:49] 06Data-Engineering, 06Data-Engineering-Radar, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-user for "Dena WMDE" - https://phabricator.wikimedia.org/T396129#10888056 (10WMDE-leszek) [15:07:19] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130 (10Dreamy_Jazz) 03NEW [15:53:12] 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: Unify Airflow's datasets.yaml config files across instances - https://phabricator.wikimedia.org/T383931#10888231 (10amastilovic) [15:57:59] 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: Unify Airflow's datasets.yaml config files across instances - https://phabricator.wikimedia.org/T383931#10888256 (10amastilovic) As per @mforns [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1282#note_... [16:43:51] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [17:55:59] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add afl_ip_hex column and afl_var_dump_timestamp index to abuse_filter_log - https://phabricator.wikimedia.org/T396130#10888597 (10Marostegui) a:03Marostegui [17:57:28] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10MediaWiki-DomainEvents, 07Epic, 10Event-Platform, and 2 others: Testing the domain event refactoring with production data - https://phabricator.wikimedia.org/T394899#10888603 (10gmodena) Good so far: - field-by-field showed that the payloads generated... [18:23:27] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop afl_patrolled_by from abuse_filter_log in production - https://phabricator.wikimedia.org/T391056#10888669 (10FCeratto-WMF) 05Open→03In progress [18:41:09] 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Investigate reasons for remaining inconsistencies - https://phabricator.wikimedia.org/T385112#10888711 (10xcollazo) On T385112#10875545, we found that we now have ~2M inconsistencies, most of them related to visibility changes. We kno... [20:25:23] 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: Unify Airflow's datasets.yaml config files across instances - https://phabricator.wikimedia.org/T383931#10888958 (10amastilovic) In order to alleviate any potential synchronization issues between the new Helm deployment and the `airflow-da... [20:47:37] FIRING: [2x] AlertLintProblem: Linting problems found for HaproxyKafkaDeliveryErrors - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [21:00:33] 07Analytics-Data-Problem, 06Data-Engineering, 06Movement-Insights: NEW BUG REPORT <"Domain" field issue: some domains have trailing dots> - https://phabricator.wikimedia.org/T395963#10889075 (10VirginiaPoundstone) @BBlack and @KOfori any chance this is related to Varnish upgrade? @Ahoelzl please evaluate f... [21:24:34] 07Analytics-Data-Problem, 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Movement-Insights: NEW BUG REPORT <"Domain" field issue: some domains have trailing dots> - https://phabricator.wikimedia.org/T395963#10889220 (10Ahoelzl) [21:31:05] 07Analytics-Data-Problem, 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Movement-Insights: NEW BUG REPORT <"Domain" field issue: some domains have trailing dots> - https://phabricator.wikimedia.org/T395963#10889226 (10Ahoelzl) @mforns please take a look