[07:31:53] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add temporary account columns to site_stats - https://phabricator.wikimedia.org/T432200 (10kostajh) 03NEW [07:31:57] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add temporary account columns to site_stats - https://phabricator.wikimedia.org/T432200#12122570 (10kostajh) [07:46:42] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add temporary account columns to site_stats - https://phabricator.wikimedia.org/T432200#12122585 (10Dreamy_Jazz) [07:47:37] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Add temporary account columns to site_stats - https://phabricator.wikimedia.org/T432200#12122586 (10Dreamy_Jazz) [08:36:20] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Review `pageview` and `pageview.trending.relative` Kafka topic sizes. - https://phabricator.wikimedia.org/T432204 (10JMonton-WMF) 03NEW [08:38:05] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Review `pageview` and `pageview.trending.relative` Kafka topic sizes. - https://phabricator.wikimedia.org/T432204#12122731 (10JMonton-WMF) I think adding `cleanup.policy = compact,delete` to `pageview.trending.relative.v1` is a quick... [08:44:39] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Review `pageview` and `pageview.trending.relative` Kafka topic sizes. - https://phabricator.wikimedia.org/T432204#12122753 (10brouberol) > First question to answer would be, can we afford having 2 new topics with around 30TB each? I'm... [08:46:09] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Review `pageview` and `pageview.trending.relative` Kafka topic sizes. - https://phabricator.wikimedia.org/T432204#12122757 (10brouberol) ` brouberol@kafka-jumbo1010:~$ sudo -E kafka configs --alter --entity-type topics --entity-name e... [08:46:14] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Review `pageview` and `pageview.trending.relative` Kafka topic sizes. - https://phabricator.wikimedia.org/T432204#12122758 (10brouberol) [08:47:32] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Review `pageview` and `pageview.trending.relative` Kafka topic sizes. - https://phabricator.wikimedia.org/T432204#12122766 (10JMonton-WMF) Thanks! I'll start the `pageview.trending.relative.v1` application now and monitorize it during... [08:52:23] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Review `pageview` and `pageview.trending.relative` Kafka topic sizes. - https://phabricator.wikimedia.org/T432204#12122801 (10JMonton-WMF) We could reduce the size of events on `pageview.v1` by removing many fields. I don't think many... [08:53:16] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Review `pageview` and `pageview.trending.relative` Kafka topic sizes. - https://phabricator.wikimedia.org/T432204#12122805 (10JMonton-WMF) [08:59:00] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Review `pageview` and `pageview.trending.relative` Kafka topic sizes. - https://phabricator.wikimedia.org/T432204#12122837 (10JMonton-WMF) [09:13:01] 06Data-Engineering, 10CirrusSearch, 06Discovery-Search, 10Event-Platform: EventRowSerializer does not support schema upgrades - https://phabricator.wikimedia.org/T432207 (10dcausse) 03NEW [09:14:12] 06Data-Engineering, 10CirrusSearch, 06Discovery-Search (2026.06.01 - 2026.07.03), 10Event-Platform: EventRowSerializer does not support schema upgrades - https://phabricator.wikimedia.org/T432207#12122889 (10dcausse) [09:14:22] 06Data-Engineering, 10CirrusSearch, 06Discovery-Search (2026.06.01 - 2026.07.03), 10Event-Platform: EventRowSerializer does not support schema upgrades - https://phabricator.wikimedia.org/T432207#12122895 (10dcausse) a:03dcausse [09:23:34] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Review `pageview` and `pageview.trending.relative` Kafka topic sizes. - https://phabricator.wikimedia.org/T432204#12122958 (10JMonton-WMF) I believe `zstd` compression cannot be enabled if we don't upgrade the Gobblin Kafka client: ht... [09:30:03] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [09:30:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.pageview.trending.relative.v1&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [09:38:44] 06Data-Engineering, 10Event-Platform: EventBus - move event producers to a different extension - https://phabricator.wikimedia.org/T432050#12123041 (10phuedx) How about the WikimediaEvents or WikimediaCustomizations extensions? Also, if EventBus is to be a standalone library for producing Event Platform event... [10:00:04] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [10:00:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.pageview.trending.relative.v1&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [11:00:04] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [11:00:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.pageview.trending.relative.v1&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [11:04:06] 06Data-Engineering, 10MediaWiki-Change-tagging, 06Product Safety and Integrity, 05DE4.1 Platform capacity: Remove private tags from mediawiki-history public dumps - https://phabricator.wikimedia.org/T431524#12123416 (10kostajh) [11:04:19] 06Data-Engineering, 10MediaWiki-Change-tagging, 06Product Safety and Integrity, 05DE4.1 Platform capacity: Update Sqoop to get `change_tag` and `change_tag_def` from production replicas - https://phabricator.wikimedia.org/T431523#12123419 (10kostajh) [11:12:56] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Review `pageview` and `pageview.trending.relative` Kafka topic sizes. - https://phabricator.wikimedia.org/T432204#12123488 (10JMonton-WMF) [11:30:03] RESOLVED: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [11:30:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.pageview.trending.relative.v1&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [11:31:03] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [11:31:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.pageview.trending.relative.v1&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [11:36:04] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [11:36:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.pageview.trending.relative.v1&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [12:01:04] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [12:01:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.pageview.trending.relative.v1&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [12:31:04] RESOLVED: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [12:31:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.pageview.trending.relative.v1&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [12:31:07] 06Data-Engineering, 06Data-Engineering-Radar, 06Privacy Engineering, 06Security-Team, and 2 others: Privacy review of x1 tables in preparation of adding them to wikireplicas - https://phabricator.wikimedia.org/T415219#12123882 (10Daimona) [12:32:03] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [12:32:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.pageview.trending.relative.v1&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [12:38:26] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Iceberg 1.6.1 vectorized Parquet reader crashes Spark 3.5 SELECTs - https://phabricator.wikimedia.org/T426801#12123940 (10JAllemandou) 05Open→03Declined We have two working solutions to overcome this problem... [12:40:03] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 07Epic: Incremental MediaWiki History - v2 - https://phabricator.wikimedia.org/T432025#12123966 (10JAllemandou) [12:43:58] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Quality verification for mediawiki_history_incremental_v1 using Iceberg time travel - https://phabricator.wikimedia.org/T425734#12123994 (10JAllemandou) [12:54:12] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): wikidatawiki.dump_remaining never completes in time - https://phabricator.wikimedia.org/T431267#12124042 (10brouberol) The partial dump of wikidatawiki has completed on its own! While not ideal, this still gets us to a bet... [12:54:31] 06Data-Engineering, 10Dumps-Generation, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): wikidatawiki.dump_remaining never completes in time - https://phabricator.wikimedia.org/T431267#12124046 (10brouberol) 05In progress→03Resolved [12:54:54] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): analytics-refinery-source repo fails post-merge builds, releases succeed - https://phabricator.wikimedia.org/T426369#12124049 (10JAllemandou) [12:56:26] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Incremental MWH - MediaWiki event data source improvements - https://phabricator.wikimedia.org/T423935#12124054 (10JAllemandou) @Ottomata : We keep this task under the Inc-MWH Phase II umbrella f... [13:00:01] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 07Epic: Incremental MediaWiki History - v2 - https://phabricator.wikimedia.org/T432025#12124065 (10JAllemandou) [13:00:55] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Add user status details for cross-wiki users in MWH - https://phabricator.wikimedia.org/T428018#12124072 (10JAllemandou) We won't work on this on Phase II. Moving to next phase parent task. [13:01:23] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Add user status details for cross-wiki users in MWH - https://phabricator.wikimedia.org/T428018#12124081 (10JAllemandou) [13:01:25] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 07Epic: Incremental MediaWiki History - v2 - https://phabricator.wikimedia.org/T432025#12124082 (10JAllemandou) [13:02:55] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Rework how mediawiki-history differentiates fake page-create from real ones - https://phabricator.wikimedia.org/T264791#12124088 (10JAllemandou) This requires deep analysis on MWH data and algorithm, and should be part of the work for v2, not Phase... [13:03:09] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox: Rework how mediawiki-history differentiates fake page-create from real ones - https://phabricator.wikimedia.org/T264791#12124103 (10JAllemandou) [13:03:10] 06Data-Engineering, 10DPE-MediaWiki-Incremental-History, 07Epic: Incremental MediaWiki History - v2 - https://phabricator.wikimedia.org/T432025#12124104 (10JAllemandou) [13:07:31] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History: Add source='snapshot+events' for incrementally-enriched snapshot rows - https://phabricator.wikimedia.org/T428969#12124124 (10JAllemandou) [13:07:36] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10DPE-MediaWiki-Incremental-History, 07Epic, 13Patch-For-Review: Incremental MediaWiki History Phase I - https://phabricator.wikimedia.org/T424350#12124126 (10JAllemandou) [13:07:52] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): MWH `page_is_deleted` and `revision_is_deleted_py_page_deletion` are not consistent - https://phabricator.wikimedia.org/T430543#12124130 (10JAllemandou) [13:16:25] (03CR) 10Joal: "Nit" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1305857 (https://phabricator.wikimedia.org/T427457) (owner: 10Aqu) [14:57:48] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 05Metrics-Sprint-2026-2027, 13Patch-For-Review: Create a new PostgreSQL cluster database with read-only from Superset and write access from Spark - https://phabricator.wikimedia.org/T432104#12124788 (10... [15:02:04] RESOLVED: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [15:02:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.pageview.trending.relative.v1&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [15:06:48] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Review `pageview` and `pageview.trending.relative` Kafka topic sizes. - https://phabricator.wikimedia.org/T432204#12124828 (10JMonton-WMF) [15:07:57] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Review `pageview` and `pageview.trending.relative` Kafka topic sizes. - https://phabricator.wikimedia.org/T432204#12124836 (10JMonton-WMF) I changed a couple of configs. https://gitlab.wikimedia.org/repos/data-en... [18:29:04] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [18:29:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.pageview.trending.relative.v1&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [18:59:04] RESOLVED: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [18:59:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.pageview.trending.relative.v1&viewPanel=24 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [21:04:05] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 05Metrics-Sprint-2026-2027: Metrics Superset dashboard access for all WMF - https://phabricator.wikimedia.org/T431697#12126132 (10BTullis) This ticket has a lot in common with {T258962}, which is roughly... [23:45:57] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): dbt-jobs backfill: all base models for moderator actions - https://phabricator.wikimedia.org/T429995#12126499 (10amastilovic) Backfill complete