[00:32:13] 14Analytics-Radar, 06Data-Engineering, 06Data-Engineering-Icebox, 06Editing-team, and 4 others: EditPage save hooks pass an entire `EditPage` object - https://phabricator.wikimedia.org/T251588#12513916 (10matmarex) >I did consider that initially, but I wasn't sure whether that's a good idea, as there won't... [02:01:35] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Platform, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12514079 (10Ottomata) > Do we keep Refine and... [04:18:11] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Documentation: Complete Documentation of Article Feature Counts - https://phabricator.wikimedia.org/T436717#12514150 (10AKhatun_WMF) Love the simplified Caveats section. Changes made: https://wikitech.wikimedia.org/w/index.php?title=Revision_Featu... [08:40:22] 06Data-Engineering, 13Patch-Needs-Improvement: HiveExtensions.convertToSchema does not properly convert arrays of structs - https://phabricator.wikimedia.org/T259924#12514466 (10dcausse) @Ahoelzl for the search this limitation is very annoying because we can't change most of our current schema (https://schema.... [09:30:49] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12514598 (10APizzata-WMF) [[ http://grafana-rw.wikimedia.org/d/ZB39Izmnz/eventgate?from=now-30m&to=now&timezone=utc&var-service=eventgate-main&var-... [09:36:50] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 10MediaWiki-extensions-WikimediaAntiAbuse, and 3 others: Create two new tables for AbuseReview - https://phabricator.wikimedia.org/T443206#12514617 (10kostajh) [09:52:06] 06Data-Engineering, 06Data-Platform-SRE: Enable Ceph S3 locations for Hive Metastore tables - https://phabricator.wikimedia.org/T425673#12514631 (10BTullis) a:05BTullis→03None Removing myself as assignee, for now. The analytics-test-hive metastore is still S3-enabled, but I can either undo it or promote a... [10:34:01] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Test Kitchen, 07Essential-Work: Improve web instrument event data data lake management - https://phabricator.wikimedia.org/T429385#12514779 (10GGoncalves-WMF) It's probably unreasonable to have Test Kitchen //users// develop individual pipelines... [10:51:09] (03CR) 10Andrew McAllister (WMDE): [C:03+1] "praise: Thanks for jumping in and sending these deletions along, @addshorewiki@gmail.com! Confirming that these helpers are no longer used" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1362229 (owner: 10Addshore) [10:52:30] (03CR) 10Andrew McAllister (WMDE): [C:03+1] "praise: Thanks for this, @addshorewiki@gmail.com! Makes total sense to remove :)" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1362230 (owner: 10Addshore) [11:33:03] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 10MediaWiki-extensions-WikimediaAntiAbuse, and 3 others: Create two new tables for AbuseReview - https://phabricator.wikimedia.org/T443206#12515023 (10Ladsgroup) As long as it's on x1 and stays there, it's fine. But: - Please catalog the table - Please... [11:58:20] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE: Airflow jobs cannot access `druid-public-coordinator.svc.eqiad.wmnet:8090` - https://phabricator.wikimedia.org/T443347#12515146 (10brouberol) @BTullis took words right out of my mouth :) [12:40:44] (03CR) 10WMDE-leszek: [C:03+2] Remove unused database and output helpers [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1362229 (owner: 10Addshore) [12:41:44] (03CR) 10WMDE-leszek: [C:03+2] Remove obsolete 2015 Graphite backfill script [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1362230 (owner: 10Addshore) [13:09:08] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 05FY2025-26 KR 5.1, 06MediaWiki-Core-Platform-Team (Radar), 07OKR-Work, 13Patch-For-Review: redioscope: periodically publish top clients to the data lake - https://phabricator.wikimedia.org/T424823#12515476 (10pmiazga) [13:10:32] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 05FY2025-26 KR 5.1, 06MediaWiki-Core-Platform-Team (Radar), 07OKR-Work, 13Patch-For-Review: redioscope: periodically publish top clients to the data lake - https://phabricator.wikimedia.org/T424823#12515485 (10pmiazga) Moved out of Core-Platfo... [13:19:40] 06Data-Engineering, 06Data-Engineering-Radar, 10Data-Platform, 06Data-Platform-SRE, 06Wikimedia Enterprise: Allow "free tier" access to Enterprise snapshots from WMF Analytics cluster - https://phabricator.wikimedia.org/T441382#12515519 (10Tobi_WMDE_SW) @xcollazo Here's the latest instance of our issue:... [13:31:59] 06Data-Engineering, 06Data-Engineering-Radar, 10Data-Platform, 06Data-Platform-SRE, 06Wikimedia Enterprise: Allow "free tier" access to Enterprise snapshots from WMF Analytics cluster - https://phabricator.wikimedia.org/T441382#12515589 (10awight) Yes we're interested in "free tier" to specifically mean... [13:56:23] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Platform, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12515768 (10JAllemandou) >>! In T437472#12514... [14:07:17] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968#12515828 (10Ottomata) Sweet! I copied a couple of panels from the JobQueue Job dashboard to add a heatmap of latencies in each bucket, and also to... [14:22:59] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Platform, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12515900 (10Ottomata) No objection to focusin... [14:24:50] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Emit comprehensive mediawiki user block change information in an event stream - https://phabricator.wikimedia.org/T424685#12515902 (10Ottomata) @JMonton-WMF I haven't clo... [14:40:10] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Emit comprehensive mediawiki user block change information in an event stream - https://phabricator.wikimedia.org/T424685#12516056 (10JMonton-WMF) I'm not sure if I under... [14:40:32] 06Data-Engineering, 06Data-Platform-SRE: Plan the Spark History Server alongside the S3 data lake - https://phabricator.wikimedia.org/T443434#12516067 (10xcollazo) > Spark 4.2.0 added the ability for the History Server to read event logs from several directories Nice! That would be ideal! > The History Serve... [14:46:46] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): refinery: RFC on reorganizing Data Engineering repositories as part of the GitLab migration - https://phabricator.wikimedia.org/T441257#12516098 (10mforns) I like the monorepo idea for data processing code: HQL, Scala, Pyspark & static files. (OPTION... [15:01:51] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Data-Platform, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Epic, 07Essential-Work: [Tech evaluation] Table management options for the S3 data lake - https://phabricator.wikimedia.org/T437472#12516284 (10JAllemandou) No objection on wait... [16:31:36] (03CR) 10A-pizzata: [C:03+2] "as agreed with @kcvelaga@wikimedia.org, I am proceeding to merge" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1357549 (https://phabricator.wikimedia.org/T433992) (owner: 10KCVelaga) [16:31:39] (03CR) 10A-pizzata: [V:03+2 C:03+2] Update User-Agent compliance hourly and Top non-compliance UA metrics to use user_agents_info. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1357549 (https://phabricator.wikimedia.org/T433992) (owner: 10KCVelaga) [16:41:58] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Create or import a Helm chart for Hive Metastore 4 - https://phabricator.wikimedia.org/T443250#12517360 (10Gehel) [16:41:59] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Deploy production and -next instances of the new table catalog on dse-k8s-eqiad - https://phabricator.wikimedia.org/T443248#12517361 (10Gehel) [16:42:01] 06Data-Engineering, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Plan access to the live Spark UI of applications on dse-k8s - https://phabricator.wikimedia.org/T443437#12517362 (10Gehel) [16:42:09] 06Data-Engineering, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Plan the retrieval of Spark application logs on dse-k8s - https://phabricator.wikimedia.org/T443435#12517363 (10Gehel) [16:42:20] 06Data-Engineering, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Plan the Spark History Server alongside the S3 data lake - https://phabricator.wikimedia.org/T443434#12517364 (10Gehel) [16:43:05] 06Data-Engineering, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): dbt: a dedicated dbt Airflow instance - https://phabricator.wikimedia.org/T435412#12517373 (10Gehel) [16:43:11] 06Data-Engineering, 06Data-Engineering-Icebox, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Public DataHub - https://phabricator.wikimedia.org/T366720#12517374 (10Gehel) [16:43:17] 06Data-Engineering, 06Wikimedia Enterprise, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Request for a dedicated Airflow instance for Commercial Partnerships (Enterprise) - https://phabricator.wikimedia.org/T440539#12517372 (10Gehel) [16:43:41] 06Data-Engineering, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Plan the Spark History Server alongside the S3 data lake - https://phabricator.wikimedia.org/T443434#12517383 (10Gehel) [16:43:47] 06Data-Engineering, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Plan the retrieval of Spark application logs on dse-k8s - https://phabricator.wikimedia.org/T443435#12517384 (10Gehel) [16:43:57] 06Data-Engineering, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Plan access to the live Spark UI of applications on dse-k8s - https://phabricator.wikimedia.org/T443437#12517387 (10Gehel) [16:44:13] 06Data-Engineering, 06Data-Engineering-Icebox, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Public DataHub - https://phabricator.wikimedia.org/T366720#12517392 (10Gehel) p:05Triage→03Medium [16:44:45] 06Data-Engineering, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: dbt: a dedicated dbt Airflow instance - https://phabricator.wikimedia.org/T435412#12517405 (10Gehel) p:05Triage→03Medium [16:44:52] 06Data-Engineering, 06Wikimedia Enterprise, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Request for a dedicated Airflow instance for Commercial Partnerships (Enterprise) - https://phabricator.wikimedia.org/T440539#12517402 (10Gehel) p:05Triage→03Medium [16:47:43] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Create or import a Helm chart for Hive Metastore 4 - https://phabricator.wikimedia.org/T443250#12517439 (10Gehel) [16:47:47] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Deploy production and -next instances of the new table catalog on dse-k8s-eqiad - https://phabricator.wikimedia.org/T443248#12517440 (10Gehel) [16:48:20] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Define intake workflow for WMCS ingress logs - https://phabricator.wikimedia.org/T441140#12517442 (10Gehel) [16:52:51] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Define intake workflow for WMCS ingress logs - https://phabricator.wikimedia.org/T441140#12517504 (10Gehel) [17:00:01] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10DPE-MediaWiki-Incremental-History, 10Event-Platform, 13Patch-For-Review: Emit comprehensive mediawiki user block change information in an event stream - https://phabricator.wikimedia.org/T424685#12517622 (10JMonton-WMF) And I have a doubt abou... [17:44:41] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10ChangeProp, 10Citoid, 10CXServer, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12518207 (10dduvall) [17:50:20] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 05MW-1.47-notes (1.47.0-wmf.23; 2026-10-06), 05MW-1.48-notes (1.48.0-wmf.4; 2026-11-10), 13Patch-For-Review: Linked Artifact Cache - Automated MediaWiki JobQueue prefetch - https://phabricator.wikimedia.org/T436695#12518283 (10Ottomata) @AKhatun... [18:26:51] (03PS2) 10Xcollazo: hdfs: Add wmf_data_ops.hdfs_usage_v2 table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1360963 (https://phabricator.wikimedia.org/T438056) [18:26:52] (03PS1) 10Xcollazo: hdfs: Add hdfs_usage_rollup_v1 and hdfs_capacity_v1 tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1363408 (https://phabricator.wikimedia.org/T438056) [18:36:36] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Essential-Work, 13Patch-For-Review: Build an HDFS growth dashboard based on wmf.hdfs_usage - https://phabricator.wikimedia.org/T438056#12518546 (10xcollazo) **Status update (2026-11-04)** All changes are in review. Review order: # https://gerr... [19:36:37] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Add IP provenance and Special page base name to webrequest_v2 - https://phabricator.wikimedia.org/T441010#12518833 (10mforns) >I'm a little confused by the "provenance" part. The X-Analytics page says wprov essentially captures a URL parameter to trac... [19:39:07] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Add Special page base name and wprov fields to webrequest_v2 - https://phabricator.wikimedia.org/T441010#12518835 (10mforns) [19:40:10] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): Add Special page base name and wprov fields to webrequest_v2 - https://phabricator.wikimedia.org/T441010#12518837 (10mforns) [19:42:47] 06Data-Engineering: Add IP provenance to webrequest_v2 - https://phabricator.wikimedia.org/T443686 (10mforns) 03NEW [19:43:47] 06Data-Engineering: Add IP provenance to webrequest_v2 - https://phabricator.wikimedia.org/T443686#12518865 (10mforns) [19:43:53] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 07Epic: Epic: Migrate uses of webrequest to webrequest_v2 - https://phabricator.wikimedia.org/T441283#12518866 (10mforns) [20:02:28] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): data lake: git commit history - https://phabricator.wikimedia.org/T433696#12518935 (10Ottomata) > Here is a first version of the Iceberg table: wmf_dev_metrics.git_interaction Cool! @Antoine_Quhen could you add `wmf_dev_metrics` to the list of databa... [20:57:38] 06Data-Engineering, 06Data-Engineering-Radar, 10ChangeProp, 10EventStreams, and 5 others: Update change-prop to node22+ - https://phabricator.wikimedia.org/T442195#12519350 (10ArielGlenn) 05Open→03Resolved I have been monitoring the changeprop and jobqueue dashboards this morning and they look good... [21:03:40] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Citoid, 10CXServer, 10EventStreams, and 21 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12519364 (10Jdforrester-WMF) [21:12:45] 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 06MediaWiki-API-Platform-Team, 10Event-Platform, 13Patch-For-Review: mediawiki.page_change.v1 - adapt page change kind model to MediaWiki's PageUpdateCauses. - https://phabricator.wikimedia.org/T430588#12519417 (10Ottomata) Update: added a `effe...