[00:04:36] RESOLVED: [3x] MediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag: ... [00:04:36] High Kafka consumer lag for mw_content_history_reconcile_enrich in eqiad - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-content-history-reconcile-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_content_history_reconcile_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiContentHistoryReconcileEnrichHighKafkaConsumerLag [02:33:26] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12012935 (10RKemper) a:03RKemper Starting on this [03:23:35] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12012980 (10RKemper) @JMonton-WMF Okay, all 20 topics are deleted (each verified empty beforehand).... [06:21:15] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 10Event-Platform: Delete some unused development topics on Kafka Jumbo - https://phabricator.wikimedia.org/T427951#12013159 (10JMonton-WMF) Hi @RKemper, thanks for the detailed explanation! (1) I'm checking it but... [09:37:45] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: mw_content_reconcile_mw_content_history_monthly failed on rerun - https://phabricator.wikimedia.org/T428999#12013647 (10APizzata-WMF) error found in the logs of `application_1778834239566_465645` (wiki `zh_min_nanwiki`): ` error:... [10:00:57] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: mw_content_reconcile_mw_content_history_monthly failed on rerun - https://phabricator.wikimedia.org/T428999#12013719 (10xcollazo) I suspect this one is `zh_min_nanwiki ` bad data. Suggested follow up checks: Are there other insta... [10:50:23] (03CR) 10Xcollazo: [C:03+2] MWHistoryDeltaWriter: fix CAST format bug and stale comment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1300876 (https://phabricator.wikimedia.org/T428503) (owner: 10Xcollazo) [10:57:27] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#12013846 (10xcollazo) **Schema update — what changed since the original spec** Since the spec we... [11:00:47] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#12013875 (10xcollazo) **Current schema for `mediawiki_history_incremental_v1` (full reference)**... [11:03:56] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#12013880 (10xcollazo) 05Open→03Resolved (I'm now boldly closing this phase I task. There... [11:06:25] (03Merged) 10jenkins-bot: MWHistoryDeltaWriter: fix CAST format bug and stale comment [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1300876 (https://phabricator.wikimedia.org/T428503) (owner: 10Xcollazo) [11:10:31] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Add row_update_dt watermark column to mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T428503#12013899 (10xcollazo) (https://gerrit.wikimedia.org/r/c/analytics/refinery/source/... [11:13:57] !log Test Kitchen edge-unique experiments (poll 1) - adds: logged-out-retention-round12, share-highlight, we-1-8-mobile-account-menu, account-creation-reading-list-cta, logged-out-retention-round13, attribution-research-2026-06-run, logged-out-retention-round15, ncs-cors-test, logged-out-retention-round14; removes: none; fields: none - TK tips at https://w.wiki/FwuD [11:14:01] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:14:17] !log Test Kitchen edge-unique experiments (poll 2) - adds: logged-out-retention-round12, share-highlight, we-1-8-mobile-account-menu, account-creation-reading-list-cta, logged-out-retention-round13, attribution-research-2026-06-run, logged-out-retention-round15, ncs-cors-test, logged-out-retention-round14; removes: none; fields: none - TK tips at https://w.wiki/FwuD [11:14:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:14:37] !log Test Kitchen edge-unique experiments (poll 3) - adds: logged-out-retention-round12, share-highlight, we-1-8-mobile-account-menu, account-creation-reading-list-cta, logged-out-retention-round13, attribution-research-2026-06-run, logged-out-retention-round15, ncs-cors-test, logged-out-retention-round14; removes: none; fields: none - TK tips at https://w.wiki/FwuD [11:14:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:14:57] !log Test Kitchen edge-unique experiments (poll 4) - adds: logged-out-retention-round12, share-highlight, we-1-8-mobile-account-menu, account-creation-reading-list-cta, logged-out-retention-round13, attribution-research-2026-06-run, logged-out-retention-round15, ncs-cors-test, logged-out-retention-round14; removes: none; fields: none - TK tips at https://w.wiki/FwuD [11:15:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:40:42] 06Data-Engineering: Delete orphan EventLogging topic `eventlogging_HomepageModulet` - https://phabricator.wikimedia.org/T429017 (10Sgs) 03NEW [11:53:21] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-Mediawiki-Content: mw_content_reconcile_mw_content_history_monthly failed on rerun - https://phabricator.wikimedia.org/T428999#12014074 (10APizzata-WMF) Thanks for the suggestions @xcollazo! > Are there other instances of empty string page_titles on... [12:39:47] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Add row_update_dt watermark column to mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T428503#12014184 (10xcollazo) a:05xcollazo→03APizzata-WMF [12:53:07] 06Data-Engineering, 06Data-Platform-SRE: Delete orphan EventLogging topic `eventlogging_HomepageModulet` - https://phabricator.wikimedia.org/T429017#12014198 (10Ottomata) Hi TY! There are many kafka topics that are typos or are unused. Cleaning them up never hurts, but in practice it doesn't matter very mu... [13:01:15] !log Test Kitchen experiment (poll 1) - adds: incident_reporting_system_interaction, fy25-26-we-1-7-8-suggestion-mode-beta, we-1-10-articleguidance-v1, logged-out-retention-round15, attribution-research-2026-06-run, logged-out-retention-round14, ncs-cors-test, growthexperiments-revise-tone, logged-out-retention-round12, share-highlight, account-creation-reading-list-cta, we-1-8-mobile-account-menu, logged-out-retention-round13; remov [13:01:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:16:20] !log Test Kitchen experiment (poll 1) - adds: we-1-8-mobile-account-menu, incident_reporting_system_interaction, growthexperiments-revise-tone, fy25-26-we-1-7-8-suggestion-mode-beta, we-1-10-articleguidance-v1, logged-out-retention-round12, attribution-research-2026-06-run, share-highlight, ncs-cors-test, logged-out-retention-round14, logged-out-retention-round15, logged-out-retention-round13, acc... [13:16:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:30:06] !log Test Kitchen experiment (poll 441) - adds: none; removes: logged-out-retention-round12; fields: none - TK tips at https://w.wiki/FwuD [14:30:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:33:21] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Add Wikitech documentation for mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T428824#12014525 (10xcollazo) A first iteration of docs now available at https://wikitech.wikimedia.org/wiki/Data_Platf... [14:36:20] !log Test Kitchen experiment (poll 1) - adds: logged-out-retention-round15, share-highlight, account-creation-reading-list-cta, we-1-10-articleguidance-v1, incident_reporting_system_interaction, attribution-research-2026-06-run, we-1-8-mobile-account-menu, growthexperiments-revise-tone, logged-out-retention-round14, ncs-cors-test, logged-out-retention-round13, fy25-26-we-1-7-8-suggestion-mode-beta... [14:36:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:47:34] !log Test Kitchen experiment (poll 68) - adds: we-1-8-account-creation-form-v2; removes: none; fields: none - TK tips at https://w.wiki/FwuD [14:47:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:57:50] (03PS1) 10Xcollazo: incremental mediawiki history: Add ingestion pipeline diagram [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1301381 (https://phabricator.wikimedia.org/T428824) [14:59:11] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Add Wikitech documentation for mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T428824#12014607 (10xcollazo) Associated brief diagram of the pipeline now at https://gerrit.wiki... [15:01:50] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Add Wikitech documentation for mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T428824#12014612 (10xcollazo) a:05xcollazo→03APizzata-WMF This is as far as I can get this be... [15:03:54] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE, 06Traffic: Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12014617 (10cmooney) @Gehel thanks for the info. Does anyone have any answers to the remaining questions fr... [15:11:56] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10PageViewInfo, 06ServiceOps new, 10ServiceOps-SharedInfra, 13Patch-For-Review: Migrate PageViewInfo calls away from rest-gateway - https://phabricator.wikimedia.org/T411771#12014631 (10tchin) (Added the page-analytics port to [[ https://wikitech.wik... [15:57:42] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE, 06Traffic: Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12014769 (10Gehel) Ben should be able to help answer those other questions on Monday. [17:58:46] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE, 06Traffic: Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12015117 (10BTullis) Here are some answers to the best of my current knowledge. > * What IP/fqdn are you conne... [18:02:17] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE, 06Traffic: Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12015124 (10CDanis) >>! In T427457#12010832, @Gehel wrote: >> * Why use urldownloader and not the HTTP proxies... [18:10:15] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE, 06Traffic: Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12015131 (10BTullis) >>! In T427457#11984157, @Gehel wrote: > @Ahoelzl has access to provision a service accou... [18:16:42] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Platform-SRE, 06Traffic: Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12015134 (10BTullis) >! In T427457#12015123, @CDanis wrote: > I was actually probably wrong about this. The u... [19:13:44] 06Data-Engineering, 06Data-Platform-SRE, 10FR-Tech-Analytics, 07Epic: Add FR Tech s3 secrets to airflow vars - https://phabricator.wikimedia.org/T429048 (10AStein-WMF) 03NEW [19:58:01] 06Data-Engineering, 06Data-Engineering-Radar, 10Event-Platform: Replace getInvocationCount() in EventBus tests to support PHPUnit 10 - https://phabricator.wikimedia.org/T426505#12015335 (10Umherirrender) 05Open→03Resolved [21:15:46] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats: Wikimedia Statistics page-views-by-country no-data - https://phabricator.wikimedia.org/T426935#12015467 (10nshahquinn-wmf) >>! In T426935#11961873, @Skif.res wrote: > As per https://foundation.wikimedia.org/wiki/Legal:Wikimedia_Foundation_Country... [21:25:42] 06Data-Engineering, 06Data-Platform-SRE, 10FR-Tech-Analytics, 07Epic: Add FR Tech s3 secrets to airflow vars - https://phabricator.wikimedia.org/T429048#12015473 (10brouberol) a:03brouberol [21:26:45] 06Data-Engineering, 06Data-Platform-SRE, 10FR-Tech-Analytics, 07Epic: Add FR Tech s3 secrets to airflow vars - https://phabricator.wikimedia.org/T429048#12015474 (10brouberol) p:05Triage→03High