[02:45:07] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:50:07] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [02:58:07] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:03:07] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [03:21:53] 06Data-Engineering, 10Research-engineering, 13Patch-For-Review: Kubernetes setup for similarity search - https://phabricator.wikimedia.org/T420379#11730949 (10fkaelin) [08:04:45] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Data-Platform-SRE (2026-03-06 - 2026-03-27): Analyze SQL queries generating metrics - https://phabricator.wikimedia.org/T420434#11731327 (10JAllemandou) > Hi @JAllemandou , the metrics have been developed in accordance to the new Contributor measureme... [08:24:20] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Growth-Team, 10Image-Suggestions: Section Image Suggestions no longer available? - https://phabricator.wikimedia.org/T420244#11731373 (10APizzata-WMF) Everything went smooth, [[ https://airflow-platform-eng.wikimedia.org/dags/check_bad_parsing/grid?... [10:37:18] (03PS1) 10A-pizzata: change mapper-weight for localuser [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256302 [10:50:04] (03PS2) 10A-pizzata: change mapper-weight for localuser [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256302 [10:52:43] (03PS3) 10A-pizzata: change mapper-weight for localuser [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256302 [11:14:03] (03CR) 10Tarrow: [C:03+1] Split active_user_changes.sql into user/temp account versions and run both [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1243177 (https://phabricator.wikimedia.org/T416680) (owner: 10Andrew McAllister (WMDE)) [11:14:20] (03CR) 10Lucas Werkmeister (WMDE): [C:03+2] Split active_user_changes.sql into user/temp account versions and run both [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1243177 (https://phabricator.wikimedia.org/T416680) (owner: 10Andrew McAllister (WMDE)) [11:14:47] (03Merged) 10jenkins-bot: Split active_user_changes.sql into user/temp account versions and run both [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1243177 (https://phabricator.wikimedia.org/T416680) (owner: 10Andrew McAllister (WMDE)) [11:16:01] (03CR) 10Andrew McAllister (WMDE): [C:03+1] "I was expecting us to have a conversation about this 😊 I had strict SQL guidelines in a prior role that I've brought over to the Analytics" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1243177 (https://phabricator.wikimedia.org/T416680) (owner: 10Andrew McAllister (WMDE)) [11:17:12] (03PS1) 10Andrew McAllister (WMDE): Split active_user_changes.sql into user/temp account versions and run both [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1256323 (https://phabricator.wikimedia.org/T416680) [11:17:30] (03CR) 10Andrew McAllister (WMDE): [C:03+2] Split active_user_changes.sql into user/temp account versions and run both [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1256323 (https://phabricator.wikimedia.org/T416680) (owner: 10Andrew McAllister (WMDE)) [11:17:59] (03Merged) 10jenkins-bot: Split active_user_changes.sql into user/temp account versions and run both [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1256323 (https://phabricator.wikimedia.org/T416680) (owner: 10Andrew McAllister (WMDE)) [12:54:49] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 06Test Kitchen: Logged in reader retention logging - https://phabricator.wikimedia.org/T420621#11732069 (10JMoore-WMF) yes 100% sampling to start is good and we should cover as many wikis as feasible [13:14:15] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 06DBA, and 4 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11732158 (10MLechvien-WMF) Can we target the Monday (Mar 30th) or Tue (Mar 31st)? From calendars both @Ladsgroup and @Marostegui shou... [13:19:32] (03CR) 10Joal: [C:03+1] change mapper-weight for localuser [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256302 (owner: 10A-pizzata) [13:35:13] (03CR) 10Snwachukwu: Extend mediarequest Cassandra loads with poster/plays for video-requests API (035 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1250005 (https://phabricator.wikimedia.org/T415202) (owner: 10Snwachukwu) [13:55:35] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10AQS2.0, 13Patch-For-Review: Alter AQS Cassandra tables in support of video plays endpoints - https://phabricator.wikimedia.org/T420008#11732260 (10Snwachukwu) Thanks for creating the tables in staging @Eevans. > be sure to put in place some samp... [14:07:27] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 06DBA, and 4 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11732313 (10Ladsgroup) My suggestion has been to do it gradually across different wikis so it can't be put to one date. Second thing... [14:17:00] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 06DBA, and 4 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11732368 (10Raine) >>! In T419980#11732313, @Ladsgroup wrote: > My suggestion has been to do it gradually across different wikis. Th... [14:17:55] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team: CentralAuth's localuser table contains many nulls and duplicate mappings - https://phabricator.wikimedia.org/T411116#11732369 (10APizzata-WMF) >The file I am talking about is only made up... [14:19:19] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 06DBA, and 4 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11732373 (10Raine) >>! In T419980#11732313, @Ladsgroup wrote: > [...] the script needs to populate the new table [...], I'm not sure... [14:19:45] (03CR) 10Xcollazo: [C:03+1] change mapper-weight for localuser (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256302 (owner: 10A-pizzata) [14:23:44] (03CR) 10Xcollazo: [C:03+1] "@apizzata@wikimedia.org: to get the auto link to phab, we need to change:" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256302 (owner: 10A-pizzata) [14:37:03] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): enwiki File Export failed for 2026-03-01 - https://phabricator.wikimedia.org/T419291#11732445 (10xcollazo) File export successful. All files now exposed publicly at https://dumps.wikimedia.org/other/mediawiki_content_history/enwiki/2026-03-01/xml/bzip2/.... [14:55:10] (03PS1) 10A-pizzata: change mapper-weight for localuser [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256402 [15:00:12] (03Abandoned) 10A-pizzata: change mapper-weight for localuser [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256402 (owner: 10A-pizzata) [15:07:22] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 06DBA, and 4 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11732573 (10Ladsgroup) >>! In T419980#11732368, @Raine wrote: >>>! In T419980#11732313, @Ladsgroup wrote: >> My suggestion has been t... [15:16:47] (03PS4) 10A-pizzata: change mapper-weight for localuser, fix mr comment [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256302 [15:18:28] (03PS5) 10A-pizzata: change mapper-weight for localuser, fix mr comment [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256302 (https://phabricator.wikimedia.org/T411116) [15:19:20] (03CR) 10A-pizzata: change mapper-weight for localuser, fix mr comment (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256302 (https://phabricator.wikimedia.org/T411116) (owner: 10A-pizzata) [15:19:28] (03CR) 10Xcollazo: [C:03+2] change mapper-weight for localuser, fix mr comment [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256302 (https://phabricator.wikimedia.org/T411116) (owner: 10A-pizzata) [15:19:30] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 06DBA, and 4 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11732643 (10Raine) >>! In T419980#11732573, @Ladsgroup wrote: >>>! In T419980#11732368, @Raine wrote: >>>>! In T419980#11732313, @Lad... [15:20:22] (03CR) 10Xcollazo: [C:03+2] change mapper-weight for localuser, fix mr comment (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256302 (https://phabricator.wikimedia.org/T411116) (owner: 10A-pizzata) [15:24:35] 06Data-Engineering, 10Data-Platform, 06Moderator-Tools-Team, 10PersonalDashboard, 06Product-Analytics (Kanban): Personal Dashboard Instrumentation Superset Dashboard - https://phabricator.wikimedia.org/T412137#11732650 (10Samwalton9-WMF) 05Open→03Resolved Thanks Megan, this is all looking good! T... [15:28:35] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Monthly reconcile continues to emit a really large amount of events after user_id changes - https://phabricator.wikimedia.org/T419055#11732672 (10xcollazo) >>! In T419055#11729676, @xcollazo wrote: > `spark_process_reconciliation_eve... [15:49:34] (03PS1) 10Jforrester: pageviews/allowlist: Add Abstract Wikipedia [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256413 (https://phabricator.wikimedia.org/T420615) [15:52:56] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 10Wikidata, 10Wikidata-Query-Service, 13Patch-For-Review: Add a --output-dir argument to wikibase rdf and json dumps - https://phabricator.wikimedia.org/T401296#11732772 (10xcollazo) >>! In T401296#11703564, @CodeReviewBot wrote: > xcollazo opened h... [16:28:43] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Improvements to local dev environment for Airflow - https://phabricator.wikimedia.org/T420752 (10xcollazo) 03NEW [17:01:18] 06Data-Engineering, 06Data-Engineering-Radar, 06Test Kitchen, 10Wikidata, 10Wikidata Analytics (Kanban): Add rcshowwikidata property to the existing PrefUpdate instrumentation for wmf_raw.mediawiki_user_properties - https://phabricator.wikimedia.org/T418246#11733116 (10AndrewTavis_WMDE) 05Open→03Stall... [17:01:48] 06Data-Engineering, 06Data-Engineering-Radar, 06Test Kitchen, 10Wikidata, 10Wikidata Analytics (Kanban): Add rcshowwikidata property to the existing PrefUpdate instrumentation for wmf_raw.mediawiki_user_properties - https://phabricator.wikimedia.org/T418246#11733117 (10AndrewTavis_WMDE) Stalling for now... [17:10:02] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Implement list of JA3N-JA4H pairs to be tagged as automated into the bot detection pipeline - https://phabricator.wikimedia.org/T420412#11733160 (10mforns) OK, will switch to an Iceberg table and put it in wmf_traffic! Thanks for the feedback. The valid_u... [17:16:12] (03CR) 10Mforns: [C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1250005 (https://phabricator.wikimedia.org/T415202) (owner: 10Snwachukwu) [18:02:06] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Implement list of JA3N-JA4H pairs to be tagged as automated into the bot detection pipeline - https://phabricator.wikimedia.org/T420412#11733480 (10GGoncalves-WMF) To be explicit, @OSefu-WMF agreed that RDS can periodically audit the list and make decisio... [18:43:52] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Improvements to local dev environment for Airflow - https://phabricator.wikimedia.org/T420752#11733605 (10xcollazo) ## Make local development compatible with git worktrees ### Problem Running `make` commands (`lint`, `test`, `ci`)... [18:49:08] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Improvements to local dev environment for Airflow - https://phabricator.wikimedia.org/T420752#11733621 (10xcollazo) [19:20:13] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th), 13Patch-For-Review: Improvements to local dev environment for Airflow - https://phabricator.wikimedia.org/T420752#11733747 (10xcollazo) [19:58:10] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Visualizing inconsistencies and reconciles via Superset - https://phabricator.wikimedia.org/T420787 (10xcollazo) 03NEW [20:32:30] 06Data-Engineering, 06Data-Engineering-Radar, 10FR-Tech-Analytics, 06Data-Platform-SRE (2026-03-06 - 2026-03-27), 13Patch-For-Review: Create FR Tech Airflow instance - https://phabricator.wikimedia.org/T417213#11733934 (10bking) [20:33:26] (03PS5) 10Snwachukwu: Extend mediarequest Cassandra loads with poster/plays for video-requests API [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1250005 (https://phabricator.wikimedia.org/T415202) [20:54:44] !log Test Kitchen edge-unique experiments (poll 5116) - adds: we-1-8-account-creation-form-v1; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [20:54:46] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:44:38] (03CR) 10Mforns: [C:03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1256413 (https://phabricator.wikimedia.org/T420615) (owner: 10Jforrester) [22:10:33] 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Schedule three new monthly DBT models for Movement Insights - https://phabricator.wikimedia.org/T420069#11734169 (10Mayakp.wiki) Update: we tried this today and the test failed. Discovered a bug where the same dbt model run works when run manually but not... [22:22:16] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Persistence, 06DBA, and 4 others: ICU 72 upgrade: `categorylinks` table swap - https://phabricator.wikimedia.org/T419980#11734256 (10Ladsgroup) s3 wikis with exception of ruwikinews I think would be good.