[00:24:14] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10function-evaluator, 10Wikifunctions, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 13Patch-For-Review: Function Evaluator log data loss due to ECS nonconforming fields - https://phabricator.wikimedia.org/T383448#10510815 (10colewhite) 05In progr... [00:28:26] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10function-evaluator, 10Wikifunctions, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 13Patch-For-Review: Function Evaluator log data loss due to ECS nonconforming fields - https://phabricator.wikimedia.org/T383448#10510844 (10ecarg) Thank you! And... [06:26:49] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592#10511082 (10Marostegui) [08:43:01] 06Data-Engineering, 10Observability-Logging, 06SRE, 13Patch-For-Review: Add x-analytics nocookie=1 and x-tls-sess to webrequest-sampled-live stream - https://phabricator.wikimedia.org/T383900#10511213 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is done! Fields show up in the kafka topic... [10:43:17] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Dumps 2.0 (Kanban Board): Implement alerting for wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T384962#10511418 (10phuedx) [11:25:31] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592#10511491 (10Marostegui) [18:43:43] 06Data-Engineering, 06Experimentation Lab, 06Web-Team: WebClientError events have version in unexpected format - https://phabricator.wikimedia.org/T383275#10513049 (10Jdlrobson-WMF) p:05Medium→03Low The events with polluted fields are no longer showing up therefore no longer need manual filtering so drop... [21:37:59] 10Data-Engineering (Q3 2024 January 1st - March 31th): HDFS capacity needs HTML dumps - https://phabricator.wikimedia.org/T384099#10513803 (10Ahoelzl) [21:40:26] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list January 2025 - https://phabricator.wikimedia.org/T384259#10513808 (10mforns) Reviewed the change, merged and deployed! [21:40:50] 10Data-Engineering (Q3 2024 January 1st - March 31th): HDFS capacity needs HTML dumps - https://phabricator.wikimedia.org/T384099#10513811 (10Ahoelzl) As discussed with research team stakeholders, there is currently no blocking dependency on the availability of a productionized HTML dumps table. Hence there won'... [21:41:32] 10Data-Engineering (Q3 2024 January 1st - March 31th): HDFS capacity needs HTML dumps - https://phabricator.wikimedia.org/T384099#10513815 (10Ahoelzl) [21:41:34] 10Data-Engineering (Q3 2024 January 1st - March 31th), 10Dumps 2.0 (Kanban Board): Calculate rough HDFS storage requirements for wmf_content.mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T383816#10513816 (10Ahoelzl) [21:53:28] 10Data-Engineering (Q3 2024 January 1st - March 31th): HDFS capacity needs data engineering and platform users - https://phabricator.wikimedia.org/T384100#10513870 (10Ahoelzl) After the 2024-12-20 cleanup we are consistently in the 23-29% buffer capacity range. https://grafana.wikimedia.org/d/000000585/hadoop?or... [21:58:36] 06Data-Engineering, 06Data-Platform-SRE, 07Epic: HDFS capacity needs FY24/25 - https://phabricator.wikimedia.org/T384098#10513878 (10Ahoelzl) Proposal for Q3 and Q4, all non-replicated: **HTML dumps:** not relevant yet (but would be 80-230TB) **Dumps 2:** plus 140TB (leveraging merg-on-read optimization) **... [23:31:45] 10Analytics-Canonical-Data, 06Movement-Insights: Null fields in canonical data are uploaded as empty strings - https://phabricator.wikimedia.org/T355847#10514192 (10nshahquinn-wmf)