[00:09:16] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#11929101 (10nshahquinn-wmf) Thank you @xcollazo for all your work on this design and for welcomin... [00:20:49] (03PS1) 10Seanleong-wmde: Script to gather metrics for Recent Changes in pilot Wikis. [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1288258 (https://phabricator.wikimedia.org/T426384) [01:01:00] FIRING: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [01:01:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [01:01:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [01:06:00] RESOLVED: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [01:06:00] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [01:06:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [02:14:41] FIRING: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [02:14:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [02:14:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [02:15:00] FIRING: MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [02:15:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [02:15:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [02:19:41] RESOLVED: MediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag: ... [02:19:41] High Kafka consumer lag for mw_page_html_content_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Enrichment#Alerting - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-content-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_content_change_enrich - ... [02:19:41] https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlContentChangeEnrichHighKafkaConsumerLag [02:20:00] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [02:20:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [02:20:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [02:30:01] FIRING: [4x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [02:30:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [02:30:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [03:00:01] RESOLVED: [2x] MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag: ... [03:00:01] High Kafka consumer lag for mw_page_html_feature_counts_change_enrich in eqiad - https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/HTML_Feature_Counts_Enrichment#Alerting - ... [03:00:01] https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus/k8s-dse&var-namespace=mw-page-html-feature-counts-change-enrich&var-helm_release=production&var-operator_name=All&var-flink_job_name=mw_page_html_feature_counts_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag [07:27:33] (03PS1) 10KCVelaga: Determine API requests in webrequest refine job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1288444 (https://phabricator.wikimedia.org/T419522) [07:30:41] 07Analytics-Data-Problem: Netherlands (NL) absent from country_project_page flat files since 2023-11-09 - https://phabricator.wikimedia.org/T426559 (10Effeietsanders) 03NEW [07:33:50] 07Analytics-Data-Problem: Netherlands (NL) absent from country_project_page flat files since 2023-11-09 - https://phabricator.wikimedia.org/T426559#11929431 (10Effeietsanders) [07:45:35] (03PS4) 10A-pizzata: change create table for mediawiki_content to become private [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1285337 [08:17:58] (03PS1) 10Joal: Determine API requests in webrequest refine job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1288444 (https://phabricator.wikimedia.org/T419522) (owner: 10KCVelaga) [08:17:58] (03CR) 10Joal: [C:03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1288444 (https://phabricator.wikimedia.org/T419522) (owner: 10KCVelaga) [08:24:15] 07Analytics-Data-Problem: Netherlands (NL) absent from country_project_page flat files since 2023-11-09 - https://phabricator.wikimedia.org/T426559#11929579 (10HakanIST) I looked into this a bit. Two things I noticed: 1. The [[https://dev.maxmind.com/geoip/release-notes | MaxMind GeoNames diff report for Februa... [08:31:18] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Streaming HTML & Edit Types - productionization checklist - https://phabricator.wikimedia.org/T423920#11929619 (10JMonton-WMF) [08:52:49] !log Test Kitchen edge-unique experiments (poll 10966) - adds: synth-aa-ncs-1; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [08:52:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:47:24] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#11930199 (10diego) Hi @Isaac You are right, the report didn't have specific numbers comparing th... [09:52:44] !log Test Kitchen mw-user experiment (poll 11144) - adds: none; removes: none; fields: incident_reporting_system_interaction - xLab/MPIC/TK tips at https://w.wiki/FwuD [09:52:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:55:15] !log Test Kitchen edge-unique experiments (poll 11152) - adds: none; removes: logged-out-retention-round9, logged-out-retention-round10; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [09:55:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:06:00] !log Test Kitchen edge-unique experiments (poll 11184) - adds: none; removes: none; fields: synth-aa-ncs-1 - xLab/MPIC/TK tips at https://w.wiki/FwuD [10:06:01] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:07:00] !log Test Kitchen edge-unique experiments (poll 11187) - adds: logged-out-retention-round9; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [10:07:01] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:27:19] (03PS2) 10KCVelaga: Determine API requests in webrequest refine job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1288444 (https://phabricator.wikimedia.org/T419522) [10:34:21] 06Data-Engineering, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 07Epic, 13Patch-For-Review: Upgrade Spark to a version with long term Iceberg support - https://phabricator.wikimedia.org/T338057#11930375 (10BTullis) [10:36:41] 06Data-Engineering, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 07Epic, 13Patch-For-Review: Upgrade Spark to a version with long term Iceberg support - https://phabricator.wikimedia.org/T338057#11930395 (10BTullis) [10:40:40] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635#11930399 (10FCeratto-WMF) [10:44:17] 06Data-Engineering, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 07Epic, 13Patch-For-Review: Upgrade Spark to a version with long term Iceberg support - https://phabricator.wikimedia.org/T338057#11930404 (10BTullis) I have now pushed out `conda-analytics-next` and the new `profile::hadoop::spark35` to pr... [10:56:40] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Deploy new Spark version on production - https://phabricator.wikimedia.org/T354737#11930451 (10BTullis) 05Open→03Resolved a:03BTullis We have decided to install Spark 3.5.8 by using a `conda-analytics-next`... [11:11:07] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE, 10DPE-Mediawiki-Content: Test if an existing conda environment with Spark 3.1.2 clients works fine with Spark 3.5.3 - https://phabricator.wikimedia.org/T380417#11930485 (10BTullis) We have decided to use a `conda-analytics-next` environment... [11:11:18] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE, 10DPE-Mediawiki-Content: Test if an existing conda environment with Spark 3.1.2 clients works fine with Spark 3.5.3 - https://phabricator.wikimedia.org/T380417#11930486 (10BTullis) [11:11:29] 06Data-Engineering, 06Data-Engineering-Radar, 10DPE-Mediawiki-Content, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Test if an existing conda environment with Spark 3.1.2 clients works fine with Spark 3.5.3 - https://phabricator.wikimedia.org/T380417#11930487 (10BTullis) [11:11:43] 06Data-Engineering, 06Data-Engineering-Radar, 10DPE-Mediawiki-Content, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Test if an existing conda environment with Spark 3.1.2 clients works fine with Spark 3.5.3 - https://phabricator.wikimedia.org/T380417#11930488 (10BTullis) 05Open→03Resolved a:03BT... [11:41:42] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 07Schema-change-in-production: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635#11930555 (10FCeratto-WMF) [12:16:48] 06Data-Engineering, 06Data-Engineering-Radar, 10Event-Platform, 06Machine-Learning-Team (Q4 FY2025-26): Add Multilingual RevertRisk predictions to mediawiki.page_revert_risk_prediction_change - https://phabricator.wikimedia.org/T415892#11930739 (10gkyziridis) Hey @Ottomata, I cannot see results in`event_s... [12:21:08] 06Data-Engineering, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 07Epic, 13Patch-For-Review: Upgrade Spark to a version with long term Iceberg support - https://phabricator.wikimedia.org/T338057#11930760 (10BTullis) >>! In T338057#11920431, @nshahquinn-wmf wrote: > I'm very excited this is happening! @BT... [12:24:17] !log Test Kitchen edge-unique experiments (poll 11596) - adds: account-creation-reading-list-cta; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [12:24:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:32:54] (03Abandoned) 10Btullis: Update Spark to version 3.5.3 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1093393 (https://phabricator.wikimedia.org/T338057) (owner: 10Btullis) [12:37:38] Starting build #60 for job analytics-refinery-maven-release [13:06:56] Yippee, build fixed! [13:06:56] Project analytics-refinery-maven-release build #60: 09FIXED in 29 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/60/ [13:11:13] (03CR) 10Xcollazo: [C:03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1285337 (owner: 10A-pizzata) [13:16:36] Starting build #40 for job analytics-refinery-update-jars [13:18:42] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.3.14 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1288853 [13:18:45] Project analytics-refinery-update-jars build #40: 09SUCCESS in 2 min 9 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/40/ [13:24:41] (03CR) 10TChin: [C:03+2] Add refinery-source jars for v0.3.14 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1288853 (owner: 10Maven-release-user) [13:34:59] 06Data-Engineering, 10EventStreams: Support new fields: (1) namespace textual prefix and (2) page title without prefix - https://phabricator.wikimedia.org/T426268#11931189 (10Isaac) no strong feelings -- it's true that e.g., mediawiki history drops the namespace prefix and so that seems to be the standard we'v... [13:40:27] 06Data-Engineering, 06Data-Platform-SRE, 06Java-Scala-Standardization, 06Discovery-Search (2026.05.04 - 2026.05.29), and 2 others: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies ... - https://phabricator.wikimedia.org/T367405#11931211 [13:41:41] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Image-Suggestions, 06Discovery-Search (2026.05.04 - 2026.05.29): ALIS data pipeline produced too many suggestions - https://phabricator.wikimedia.org/T423238#11931219 (10pfischer) [13:43:07] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Add log_id to wmf.mediawiki_history - https://phabricator.wikimedia.org/T425986#11931244 (10xcollazo) Table validation: First fix metastore: ` spark-sql (default)> show partitions mediawiki_histor... [13:44:17] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Add log_id to wmf.mediawiki_history - https://phabricator.wikimedia.org/T425986#11931271 (10xcollazo) [13:54:24] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Engineering-Radar, 06ServiceOps new, 10ServiceOps-Services-Oids, and 2 others: Make eventstreams-internal available to WMF staff without an ssh tunnel - https://phabricator.wikimedia.org/T348763#11931323 (10atsuko) Created the namespaces, unbloc... [13:55:13] !log Deployed refinery using scap, then deployed onto hdfs [13:55:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:30:09] !log Test Kitchen edge-unique experiments (poll 11971) - adds: none; removes: logged-out-retention-round8, mobile-page-previews; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [14:30:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:33:15] 06Data-Engineering, 07Sustainability: Move some analytics jobs to day time in Virginia - https://phabricator.wikimedia.org/T384166#11931522 (10Ladsgroup) Another idea: Maybe already implemented. A lot of my queries for my work are done on webrequest raw because most the derivative tables won't work for my case... [14:47:44] 06Data-Engineering: Consider classifying www.wikipedia.org from "internal" referrer - https://phabricator.wikimedia.org/T422584#11931619 (10Krinkle) [15:18:24] 06Data-Engineering, 07Sustainability: Move some analytics jobs to day time in Virginia - https://phabricator.wikimedia.org/T384166#11931809 (10BTullis) >>! In T384166#11931522, @Ladsgroup wrote: > Another idea: Maybe already implemented. A lot of my queries for my work are done on webrequest raw because most t... [16:04:55] 06Data-Engineering, 07Sustainability: Move some analytics jobs to day time in Virginia - https://phabricator.wikimedia.org/T384166#11932122 (10Ladsgroup) oh that's a clever trick. I tried it on a query and it went from 235s to 175s which is nice (granted, the data might have gotten to cache, etc. so it's not 1... [16:49:42] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Add log_id to wmf.mediawiki_history - https://phabricator.wikimedia.org/T425986#11932459 (10xcollazo) Steps ran in prod to get the new schema: ` $ hostname -f an-launcher1003.eqiad.wmnet $ sudo -... [17:00:24] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Data-Engineering-Radar, 06Product-Analytics: Creating a Spark session causes a torrent of log spam - https://phabricator.wikimedia.org/T315024#11932542 (10nshahquinn-wmf) 05Open→03Resolved Thank you @xcollazo! [17:07:45] 07Analytics-Data-Problem: Netherlands (NL) absent from country_project_page flat files since 2023-11-09 - https://phabricator.wikimedia.org/T426559#11932608 (10Effeietsanders) (initially posted on the wrong ticket) The same scan confirmed that no other countries showed similar behavior across all countries. Most... [17:11:18] 06Data-Engineering, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Kanban Board), 07OKR-Work, 13Patch-For-Review: redioscope: periodically publish top clients to the data lake - https://phabricator.wikimedia.org/T424823#11932617 (10daniel) Thank you for looking into this @Ottomata! I can use kcat, and I... [18:49:20] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Add log_id to wmf.mediawiki_history - https://phabricator.wikimedia.org/T425986#11932946 (10xcollazo) Deleted the `DagProperties` of the following DAGs to pickup the new refiniery-source artifact:... [18:52:42] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Add log_id to wmf.mediawiki_history - https://phabricator.wikimedia.org/T425986#11932955 (10xcollazo) Now [[ https://airflow.wikimedia.org/dags/mediawiki_history_denormalize/grid?dag_run_id=schedul... [20:39:35] 06Data-Engineering, 06Stewards-and-global-tools, 10Event-Platform, 05MW-1.47-notes (1.47.0-wmf.2; 2026-05-12), and 2 others: TypeError: MediaWiki\Extension\EventBus\HookHandlers\MediaWiki\UserChangeHooks::calculateUserEffectiveGroups(): Argument #1 ($user... - https://phabricator.wikimedia.org/T426185#11933440