[01:04:18] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Migrate Sqoop jobs to Airflow - https://phabricator.wikimedia.org/T409514 (10amastilovic) 03NEW [03:02:03] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Migrate Sqoop jobs to Airflow - https://phabricator.wikimedia.org/T409514#11352303 (10amastilovic) [06:35:17] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10AbuseFilter, 06DBA, 07Schema-change-in-production: Drop the afl_ip column and the afl_ip_timestamp index from the abuse_filter_log table - https://phabricator.wikimedia.org/T407997#11352442 (10Marostegui) [10:45:17] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 06DBA, 06Product Safety and Integrity, and 2 others: Add the sic_url_identifier to cusi_case on WMF wikis - https://phabricator.wikimedia.org/T409539 (10Dreamy_Jazz) 03NEW [10:45:32] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 06DBA, 06Product Safety and Integrity, and 2 others: Add the sic_url_identifier column to the cusi_case table on WMF wikis - https://phabricator.wikimedia.org/T409539#11353002 (10Dreamy_Jazz) [10:45:53] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 06DBA, 06Product Safety and Integrity, 07Schema-change-in-production: Add the sic_url_identifier column to the cusi_case table on WMF wikis - https://phabricator.wikimedia.org/T409539#11353003 (10Dreamy_Jazz) [10:47:20] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 06DBA, 06Product Safety and Integrity, 07Schema-change-in-production: Add the sic_url_identifier column to the cusi_case table on WMF wikis - https://phabricator.wikimedia.org/T409539#11353006 (10Dreamy_Jazz) I think the table should be small enoug... [10:49:08] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 06DBA, 06Product Safety and Integrity, 07Schema-change-in-production: Add the sic_url_identifier column to the cusi_case table on WMF wikis - https://phabricator.wikimedia.org/T409539#11353016 (10Dreamy_Jazz) [11:31:44] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07OKR-Work: Create a dbt Docker container - https://phabricator.wikimedia.org/T406636#11353127 (10JMonton-WMF) We have changed the approach, rather than using `uv` installed manually, we are using the... [11:33:35] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 06DBA, 06Product Safety and Integrity, 07Schema-change-in-production: Add the sic_url_identifier column to the cusi_case table on WMF wikis - https://phabricator.wikimedia.org/T409539#11353130 (10Marostegui) a:03Marostegui [11:36:15] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Explore a local dbt environment setup (independent from Conda) - https://phabricator.wikimedia.org/T409054#11353134 (10JMonton-WMF) Added some documentation in a [[ https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/4/diffs#8... [13:27:18] 06Data-Engineering, 10CampaignEvents, 06DBA, 06Connection-Team (Connection-Current-Sprint), 07Schema-change-in-production: Apply ce_address cleanup schema changes in production (x1) - https://phabricator.wikimedia.org/T409101#11353384 (10Daimona) 05Stalled→03Open I think this is good to go now, a tra... [13:31:15] 06Data-Engineering, 06Data-Engineering-Radar, 06cloud-services-team, 06Data-Persistence, and 3 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#11353414 (10Gehel) [13:31:27] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Do performance testing of a big Hadoop Table hosted by Ceph - https://phabricator.wikimedia.org/T381416#11353420 (10Gehel) [13:31:37] 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, 06serviceops, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11353424 (10Gehel) [13:32:36] 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), and 2 others: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315#11353441 (10Gehel) [13:32:41] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Movement-Insights, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Epic, 07Essential-Work: Create example dbt models using Iceberg - https://phabricator.wikimedia.org/T408687#11353446 (10Gehel) [13:35:09] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 3 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11353512 (10Gehel) [13:36:01] 06Data-Engineering, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: ERROR AsyncEventQueue: Listener DatahubSparkListener threw an exception - https://phabricator.wikimedia.org/T400207#11353547 (10Gehel) [13:36:36] 06Data-Engineering, 10BetaFeatures, 06cloud-services-team, 10Data-Services, and 2 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11353553 (10Gehel) [13:36:52] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Blunderbuss: Move Hadoop/HDFS XML configuration into Helm deployment chart - https://phabricator.wikimedia.org/T402323#11353562 (10Gehel) [13:37:32] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Move the dumps_v1 DAGs from the Airflow test_k8s instance to the main instance - https://phabricator.wikimedia.org/T404084#11353568 (10Gehel) [13:38:28] 06Data-Engineering, 10Technical-blog-posts, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Write a blog post about the recent Airflow migration to Kubernetes - https://phabricator.wikimedia.org/T393603#11353590 (10Gehel) [13:38:50] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Provide an access to MaxMind GeoIP in DSE K8S pods - https://phabricator.wikimedia.org/T405509#11353597 (10Gehel) [13:39:00] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Wikimedia Enterprise, 10Wikimedia Enterprise - Content Integrity, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Implement an Airflow operator for moving data from point A t... - https://phabricator.wikimedia.org/T405360#11353594 [13:39:35] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07OKR-Work: Set up a working, usable dbt installation on stat boxes - https://phabricator.wikimedia.org/T406634#11353609 (10Gehel) [13:39:41] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07OKR-Work: Create a dbt Docker container - https://phabricator.wikimedia.org/T406636#11353613 (10Gehel) [13:39:55] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 3 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11353612 (10Gehel) [13:46:51] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 06DBA, 06Product Safety and Integrity, 07Schema-change-in-production: Add the sic_url_identifier column to the cusi_case table on WMF wikis - https://phabricator.wikimedia.org/T409539#11353676 (10Marostegui) 05Open→03Resolved Done ` enwiki... [14:02:40] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 06DBA, 06Product Safety and Integrity, 07Schema-change-in-production: Add the sic_url_identifier column to the cusi_case table on WMF wikis - https://phabricator.wikimedia.org/T409539#11353734 (10kostajh) Thank you! [14:09:27] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 06DBA, 06Product Safety and Integrity, 07Schema-change-in-production: Add the sic_url_identifier column to the cusi_case table on WMF wikis - https://phabricator.wikimedia.org/T409539#11353767 (10Dreamy_Jazz) Thanks for applying this quickly [15:20:42] 06Data-Engineering, 06cloud-services-team, 06Data-Persistence, 10Data-Services, and 3 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11354035 (10taavi) [15:41:38] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): enwiki file export failed due to OOM - https://phabricator.wikimedia.org/T409565 (10xcollazo) 03NEW [15:43:02] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): enwiki file export failed due to OOM - https://phabricator.wikimedia.org/T409565#11354152 (10xcollazo) Current setting: ` max_partition_size="10240", # This is MBs, thus 10 GB partitions, to generate ~512 MB files. ` Will rerun at 90% = 10240 *.9 = 9... [16:28:23] (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1202718 (https://phabricator.wikimedia.org/T408178) (owner: 10Phuedx) [16:36:27] 06Data-Engineering, 10CirrusSearch, 06Data-Platform-SRE, 10DPE-Mediawiki-Content, and 4 others: Source the CirrusSearch index dumps from hadoop instead of a MW maintenance script - https://phabricator.wikimedia.org/T366248#11354373 (10EBernhardson) In the communication we went with promising dumps through... [17:00:59] !log Test Kitchen edge-unique experiments (poll 4689) - adds: none; removes: none; fields: fy2025-26-we3.1-image-browsing-ab-test, hcaptcha-on-french-wikipedia, xlab-mw-module-loaded-v2 - xLab/MPIC/TK tips at https://w.wiki/FwuD [17:01:02] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:10:18] ^^ this was just that we are now omitting mdot subdomains from the emitted config [17:23:17] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): enwiki file export failed due to OOM - https://phabricator.wikimedia.org/T409565#11354540 (10xcollazo) Failure point passed with `max_partition_size="9216"`. Will put together a patch. Probably best to apply this to all huge wikis and avoid this failu... [18:10:48] 10Data-Engineering-Roadmap, 07Epic, 07OKR-Work: Analyze JA3N data and generate JA3N-UA table - https://phabricator.wikimedia.org/T409577 (10mforns) 03NEW [18:34:02] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 10ConfirmEdit (CAPTCHA extension), 06Data-Persistence, and 3 others: Add columns to store associated log ID or revision ID that caused a signal to match - https://phabricator.wikimedia.org/T409093#11354660 (10Dreamy_Jazz) [18:34:32] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 10ConfirmEdit (CAPTCHA extension), 06Data-Persistence, and 3 others: Add columns to store associated log ID or revision ID that caused a signal to match - https://phabricator.wikimedia.org/T409093#11354661 (10Dreamy_Jazz) I'll go with `trigger_id` an... [18:54:26] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 10ConfirmEdit (CAPTCHA extension), 06Data-Persistence, and 4 others: Add columns to store associated log ID or revision ID that caused a signal to match - https://phabricator.wikimedia.org/T409093#11354744 (10Dreamy_Jazz) [18:55:49] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 10ConfirmEdit (CAPTCHA extension), 06Data-Persistence, and 4 others: Add columns to store associated log ID or revision ID that caused a signal to match - https://phabricator.wikimedia.org/T409093#11354746 (10Dreamy_Jazz) >>! In T409093#11351255, @La... [18:57:09] 10Data-Engineering-Roadmap, 07Epic, 07OKR-Work: Productionize JA3N-UA table to improve bot detection - https://phabricator.wikimedia.org/T409584 (10mforns) 03NEW [20:30:02] (03PS1) 10Xcollazo: Fix bug MW Dumper in which vertical bars ( `|` ) were not being honored. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1203124 (https://phabricator.wikimedia.org/T407649) [20:44:11] (03PS1) 10Snwachukwu: Fix Duplicate Pageview metrics records in data quality tables. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1203129 [20:50:31] (03CR) 10Aleksandar Mastilovic: Fix Duplicate Pageview metrics records in data quality tables. (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1203129 (owner: 10Snwachukwu) [22:21:42] (03CR) 10Zabe: "Hi! Thank you. This can get merged now imo." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1199521 (https://phabricator.wikimedia.org/T309738) (owner: 10Zabe) [23:20:04] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Review and productionize the WME differential privacy data set - https://phabricator.wikimedia.org/T409601 (10Ahoelzl) 03NEW