[00:48:02] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11827851 (10Ottomata) Ah! It also [[ https://grafana.wikimedia.org/goto/cfj7pldl0ge80d?orgId=1 | slowly container OOMs... [06:44:01] 06Data-Engineering, 06Data-Engineering-Radar, 06cloud-services-team, 10Data-Services, 06DBA: Re-run maintainviews on all clouddb* and an-redacteddb1001.eqiad.wmnet - https://phabricator.wikimedia.org/T422459#11828202 (10Marostegui) [07:10:45] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Pipeline - Performance improvements - https://phabricator.wikimedia.org/T422928#11828227 (10JMeybohm) >>! In T422928#11824753, @Ottomata wrote: >> Avoid envoy retries (there's a header) > > Quickly reading {T3... [07:44:54] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Pipeline - Performance improvements - https://phabricator.wikimedia.org/T422928#11828331 (10dcausse) >>! In T422928#11828227, @JMeybohm wrote: > > From what I recall there are certain limitations on the headers... [08:25:02] 06Data-Engineering, 06Data-Platform-SRE: Conda-Analytics environments stuck with very outdated packages - https://phabricator.wikimedia.org/T423052#11828451 (10nshahquinn-wmf) [08:51:25] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 10Event-Platform, 07good first task: Flink base image should not install into system python environment - https://phabricator.wikimedia.org/T418525#11828529 (10atsuko) 05Open→03In progress Looking at `docker-pkg` [09:09:57] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Dumps-Generation: Data missing from en.wiktionary.org February 2026 "MediaWiki Content File Exports" compared to "XML Database dump" - https://phabricator.wikimedia.org/T417596#11828597 (10xcollazo) @JeffDoozan once the 2026-05-01 File Export for enwikti... [09:13:48] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10WMF-JobQueue, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 10Event-Platform: Kafka-topics broken in beta: "zookeeper is not a recognized option" - https://phabricator.wikimedia.org/T422842#11828609 (10brouberol) a:03brouberol [09:13:51] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10WMF-JobQueue, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 10Event-Platform: Kafka-topics broken in beta: "zookeeper is not a recognized option" - https://phabricator.wikimedia.org/T422842#11828612 (10brouberol) 05Open→03In progress [09:13:53] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10WMF-JobQueue, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 10Event-Platform: Kafka-topics broken in beta: "zookeeper is not a recognized option" - https://phabricator.wikimedia.org/T422842#11828611 (10brouberol) [09:26:51] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Add dbt base model for Wikipedia moderator actions metrics - https://phabricator.wikimedia.org/T423565#11828697 (10xcollazo) [10:07:30] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10WMF-JobQueue, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), and 2 others: Kafka-topics broken in beta: "zookeeper is not a recognized option" - https://phabricator.wikimedia.org/T422842#11828772 (10brouberol) Fixed on kafka-test, which was also migrated... [11:08:59] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Blunderbuss doesn't replace the whole destination folder in HDFS - https://phabricator.wikimedia.org/T423573 (10amastilovic) 03NEW [11:21:09] (03PS1) 10Milimetric: Release v2.11.2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1272662 (https://phabricator.wikimedia.org/T333716) [11:22:38] (03CR) 10Milimetric: [C:03+2] "Add link to Country Protection list and deploy latest translations." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1272662 (https://phabricator.wikimedia.org/T333716) (owner: 10Milimetric) [11:23:44] (03Merged) 10jenkins-bot: Release v2.11.2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1272662 (https://phabricator.wikimedia.org/T333716) (owner: 10Milimetric) [12:02:22] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 07Essential-Work: Provide an access to MaxMind GeoIP in DSE K8S pods - https://phabricator.wikimedia.org/T405509#11829105 (10BTullis) I decided to add a boolean to enable the GeoIP access per instance, with a comme... [12:50:14] 06Data-Engineering, 10Event-Platform: mediawiki.page_change.v1 event - Add revision is revert field - https://phabricator.wikimedia.org/T423583 (10Ottomata) 03NEW [12:51:55] 06Data-Engineering, 06Data-Platform-SRE, 06Traffic, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11829232 (10BTullis) [12:55:35] 06Data-Engineering, 10Data-Engineering-Wikistats: Bring a basic version of Wikistats v1 back online - https://phabricator.wikimedia.org/T423585#11829265 (10Milimetric) [13:12:58] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11829489 (10Ottomata) Since OOMs don't seem to be related to `async_enabled=false` or the v1.50.0 release I'm going to... [13:47:36] !log Test Kitchen mw-user experiment (poll 119153) - adds: ab-test-email-confirmation-banner; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [13:47:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:17:52] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11829798 (10AnnieKim_WMDE) @Martyn.ranyard I need to use jupyter notebooks to conduct data analysis of mobile-first communities for... [14:30:01] !log Test Kitchen mw-user experiment (poll 119279) - adds: none; removes: growthexperiments-revise-tone; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:30:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:21:41] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10WMF-JobQueue, 10Event-Platform: Jobs are not being processed in beta, April 2026 edition - https://phabricator.wikimedia.org/T423615 (10Daimona) 03NEW [15:22:47] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10WMF-JobQueue, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 10Event-Platform: Kafka-topics broken in beta: "zookeeper is not a recognized option" - https://phabricator.wikimedia.org/T422842#11830331 (10Daimona) >>! In T422842#11828772, @brouberol wrot... [15:25:11] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 10WMF-JobQueue, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 10Event-Platform: Kafka-topics broken in beta: "zookeeper is not a recognized option" - https://phabricator.wikimedia.org/T422842#11830343 (10brouberol) 05In progress→03Resolved [15:28:23] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Engineering-Radar, 06Product-Analytics: Creating a Spark session causes a torrent of log spam - https://phabricator.wikimedia.org/T315024#11830357 (10xcollazo) a:03xcollazo [16:37:16] 06Data-Engineering, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 10Event-Platform, 07good first task, 13Patch-For-Review: Flink base image should not install into system python environment - https://phabricator.wikimedia.org/T418525#11830744 (10atsuko) [[ https://gerrit.wikimedia.org/r/1272726 | Test p... [18:22:00] 06Data-Engineering, 10Beta-Cluster-Infrastructure, 06MW-Interfaces-Team, 10WMF-JobQueue, 10Event-Platform: Jobs are not being processed in beta, April 2026 edition - https://phabricator.wikimedia.org/T423615#11831127 (10bd808) The cert I see from outside the cluster for https://en.wikipedia.beta.wmcloud.... [19:16:21] 06Data-Engineering, 10Data-Engineering-Wikistats, 06Wikimedia Enterprise, 07Russian-Sites: "Active editors by country" doesn't display numbers for Belarus, Kazakhstan, Russia - https://phabricator.wikimedia.org/T333716#11831365 (10Iniquity) >>! In T333716#11829013, @Milimetric wrote: > I got a chance to ad... [19:19:35] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11831370 (10Ottomata) Status report! - I fixed the process function metrics. I accidentally renamed them. - I think s... [19:21:04] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Enrichment - Tuning & Backfilling configuration - https://phabricator.wikimedia.org/T421216#11831372 (10Ottomata) > I fixed the process function metrics. I accidentally renamed them. Errr, I thought I fixed them... [19:22:41] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: HTML Pipeline - Performance improvements - https://phabricator.wikimedia.org/T422928#11831373 (10Ottomata) [19:37:48] 06Data-Engineering, 06Data-Engineering-Radar, 06Commons, 06Data-Persistence, and 5 others: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys) - https://phabricator.wikimedia.org/T28741#11831436 (10Zabe) We will start removing the old tables from wikireplicas on 28... [19:42:23] 06Data-Engineering, 06Data-Engineering-Radar, 06Commons, 06Data-Persistence, and 5 others: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys) - https://phabricator.wikimedia.org/T28741#11831470 (10Ladsgroup) Thank you <3 [19:43:09] 06Data-Engineering, 10Data Pipelines, 10Wikibase (3rd party installations), 10Wikidata, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: Pipeline for Moving Data Between Wikibase Instances and Wikidata - https://phabricator.wikimedia.org/T423333#11831471 (10Superraptor123) Beginning to roughly outli... [20:35:41] 06Data-Engineering, 06Data-Engineering-Radar, 06Commons, 06Data-Persistence, and 5 others: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys) - https://phabricator.wikimedia.org/T28741#11831634 (10Zabe)