[00:37:52] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Define and document the earliest start_date for dbt DAGs - https://phabricator.wikimedia.org/T438276#12372228 (10amastilovic) > Ideally, if a model owner forgets to specify start_date, this would be caught at MR review time or, failing that, cause a fail... [02:26:53] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Define and document the earliest start_date for dbt DAGs - https://phabricator.wikimedia.org/T438276#12372343 (10nshahquinn-wmf) >>! In T438276#12372228, @amastilovic wrote: > `permit_large_backfill` is possible, but I wonder if the utility of it is grea... [05:55:39] 06Data-Engineering, 10Kafka-Infrastructure, 06SRE, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Monitor kafka-configurator runs and pending changes - https://phabricator.wikimedia.org/T439506 (10RKemper) 03NEW [06:01:28] 14Analytics-Radar, 06Data-Engineering, 10Kafka-Infrastructure, 06serviceops-radar, and 5 others: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12372533 (10RKemper) [07:28:10] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA: Move SpecialUnusedTemplates computation to Hadoop - https://phabricator.wikimedia.org/T422443#12372702 (10JAllemandou) I have checked the job, and the problem is deeper than just memory resources. The job was suffering from data-skew and Spark couldn't opt... [07:28:58] (03PS1) 10Joal: Add spark_stage_metrics table creation script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345860 (https://phabricator.wikimedia.org/T297231) [07:34:55] (03PS1) 10Joal: Update querypage queries with join optimizations [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345865 (https://phabricator.wikimedia.org/T422443) [07:36:32] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 13Patch-For-Review: Move SpecialUnusedTemplates computation to Hadoop - https://phabricator.wikimedia.org/T422443#12372720 (10JAllemandou) One mopre note: I have paused the airflow job to prevent alerts to be sent while testing. Let's not forget to unpau... [08:02:41] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Observability-Metrics, 13Patch-For-Review: [Infra] Mechanism to get Apache Spark operational metrics - https://phabricator.wikimedia.org/T297231#12372783 (10JAllemandou) Good progress to report here. Reading existing EventLogs is a way better idea... [08:13:53] (03CR) 10Joal: "Tested on the cluster!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1344687 (https://phabricator.wikimedia.org/T297231) (owner: 10Joal) [08:17:37] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Include v2 dumps into mirrors rsync configuration - https://phabricator.wikimedia.org/T437463#12372843 (10GGoncalves-WMF) Excellent analysis, thank you! I'd like to see @HCoplin-WMF 's thoughts but my answers are: > Do you agree with rule 2 (cutoff dat... [08:41:05] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Define and document the earliest start_date for dbt DAGs - https://phabricator.wikimedia.org/T438276#12372930 (10GGoncalves-WMF) Sounds good to me too! [09:03:03] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Observability-Metrics, 13Patch-For-Review: [Infra] Mechanism to get Apache Spark operational metrics - https://phabricator.wikimedia.org/T297231#12373012 (10JAllemandou) Had a chat with @Antoine_Quhen and he had great suggestions: * Add some job c... [09:17:44] (03CR) 10Aqu: [V:03+2] Declare new table Google Sitedata Aggregate [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345123 (https://phabricator.wikimedia.org/T436999) (owner: 10Aqu) [09:26:59] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Implement alerts and create runbooks - https://phabricator.wikimedia.org/T438384#12373122 (10Clement_Goubert) It may be interesting for you to get these alerts during the early stages of your lambda so you can investigate possible m... [11:17:18] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Migrate cleanup jobs for snapshot datasets from systemd timers to Airflow - https://phabricator.wikimedia.org/T411999#12373785 (10JAllemandou) If I don't mistake, the repo you pointed don't have the `refinery` code you reference. If we decide to put the... [12:18:05] (03CR) 10A-pizzata: "small nits" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345215 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [12:44:11] (03PS2) 10Aqu: Add weekly change_tag raw table creation HQL [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345215 (https://phabricator.wikimedia.org/T437961) [12:46:06] (03CR) 10Aqu: Add weekly change_tag raw table creation HQL (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345215 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [13:03:14] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: [Tech evaluation] Table catalog options for the S3 data lake - https://phabricator.wikimedia.org/T435604#12374192 (10JAllemandou) One of the key decision regarding moving this forward is whether or not we wish the new data l... [13:10:02] 06Data-Engineering, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover, 10Event-Platform: DC Switchover - MW requests to EventGate cause long queuing waiting time - https://phabricator.wikimedia.org/T439552#12374234 (10CDanis) [13:24:40] 06Data-Engineering, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover, 10Event-Platform: DC Switchover - MW requests to EventGate cause long queuing waiting time - https://phabricator.wikimedia.org/T439552#12374341 (10hnowlan) Just to note as an initial follow-up post outage the time... [13:26:42] (03CR) 10A-pizzata: [C:03+2] Add weekly change_tag raw table creation HQL [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345215 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [13:34:42] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Migrate cleanup jobs for snapshot datasets from systemd timers to Airflow - https://phabricator.wikimedia.org/T411999#12374392 (10dcausse) >>! In T411999#12373785, @JAllemandou wrote: > If I don't mistake, the repo you pointed don't have the `refinery` c... [13:45:49] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Observability-Metrics, 13Patch-For-Review: [Infra] Mechanism to get Apache Spark operational metrics - https://phabricator.wikimedia.org/T297231#12374446 (10xcollazo) > and insert them into an iceberg table. Can we please estimate how big this tab... [13:54:32] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Observability-Metrics, 13Patch-For-Review: [Infra] Mechanism to get Apache Spark operational metrics - https://phabricator.wikimedia.org/T297231#12374495 (10JAllemandou) >>! In T297231#12374446, @xcollazo wrote: >> and insert them into an iceberg t... [14:10:49] (03CR) 10Aqu: [V:03+1 C:03+1] Add the mediawiki_private_actor_weekly table. It stores each weekly snapshot in a day partition under tables/weekly. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1343956 (https://phabricator.wikimedia.org/T437961) (owner: 10A-pizzata) [14:21:24] (03CR) 10Xcollazo: "Lots of comments, but just because I am really excited about this work 😊" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345860 (https://phabricator.wikimedia.org/T297231) (owner: 10Joal) [14:21:34] 06Data-Engineering, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover, and 2 others: DC Switchover - MW requests to EventGate cause long queuing waiting time - https://phabricator.wikimedia.org/T439552#12374694 (10MLechvien-WMF) [14:21:46] 06Data-Engineering, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover, and 2 others: DC Switchover - MW requests to EventGate cause long queuing waiting time - https://phabricator.wikimedia.org/T439552#12374695 (10phuedx) Per https://gerrit.wikimedia.org/g/operations/mediawiki-config/+... [14:26:15] 06Data-Engineering, 10Research-engineering: dse-k8s: Create a `research` namespace for development workloads - https://phabricator.wikimedia.org/T439563 (10fkaelin) 03NEW [14:30:09] 06Data-Engineering, 10Research-engineering: dse-k8s: DPE SRE support to deploy a first Research service as a template - https://phabricator.wikimedia.org/T439565 (10fkaelin) 03NEW [14:34:42] 06Data-Engineering, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: [Tech evaluation] Table catalog options for the S3 data lake - https://phabricator.wikimedia.org/T435604#12374836 (10Ottomata) > if "must be able to carry existing Hive-style tables as-is" is a hard requirement I hope it is... [14:40:46] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06Commons, 06Data-Persistence, 07OKR-Work: Data Lake - sqoop of commons link tables needs to adapt to move to dedicated MariaDB cluster - https://phabricator.wikimedia.org/T431476#12374858 (10Milimetric) FYI: with this deployed I updated the `analy... [14:49:25] 06Data-Engineering, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover, and 2 others: DC Switchover - MW requests to EventGate cause long queuing waiting time - https://phabricator.wikimedia.org/T439552#12374915 (10Ottomata) > Since there aren't any EventGate owners to consult, we shoul... [14:58:40] 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Matomo: Enable β€œAIAgents” core plugin - https://phabricator.wikimedia.org/T438862#12374966 (10bking) 05In progressβ†’03Resolved @SCampos-WMF A s Ben mentioned, we won't be able to do this un... [15:28:30] (03CR) 10Xcollazo: Add spark_stage_metrics table creation script (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345860 (https://phabricator.wikimedia.org/T297231) (owner: 10Joal) [15:49:55] (03CR) 10Zabe: [C:03+1] "LGTM" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345865 (https://phabricator.wikimedia.org/T422443) (owner: 10Joal) [16:56:35] (03PS1) 10Snwachukwu: Update Changelog for v0.3.28 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1346074 [16:58:22] (03CR) 10Xcollazo: [C:03+1] Update Changelog for v0.3.28 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1346074 (owner: 10Snwachukwu) [16:58:53] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Update Changelog for v0.3.28 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1346074 (owner: 10Snwachukwu) [17:03:12] Starting build #75 for job analytics-refinery-maven-release [17:34:32] Project analytics-refinery-maven-release build #75: 09SUCCESS in 31 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/75/ [17:37:32] Starting build #42 for job analytics-refinery-update-jars [17:39:44] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.3.28 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1346095 [17:39:45] Project analytics-refinery-update-jars build #42: 09SUCCESS in 2 min 12 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/42/ [17:43:41] !log Deployed refinery-source using jenkins as part of weekly deployment train [17:43:43] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:45:12] (03CR) 10Snwachukwu: [C:03+2] Add refinery-source jars for v0.3.28 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1346095 (owner: 10Maven-release-user) [17:45:14] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Add refinery-source jars for v0.3.28 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1346095 (owner: 10Maven-release-user) [17:53:13] !log Deploying Refinery as part of weekly deployment train [17:53:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:56:04] (03CR) 10Xcollazo: "Bunch of comments below for your consideration." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1344687 (https://phabricator.wikimedia.org/T297231) (owner: 10Joal) [18:03:06] (03CR) 10Xcollazo: [C:03+1] Update querypage queries with join optimizations [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1345865 (https://phabricator.wikimedia.org/T422443) (owner: 10Joal) [18:05:25] !log Deployed Refinery as part of weekly deployment train [18:05:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:13:43] 06Data-Engineering, 10Wmfdata-Python, 06Movement-Insights (FY26-27 H1): Release Wmfdata 3.0 - https://phabricator.wikimedia.org/T439613 (10nshahquinn-wmf) 03NEW [19:26:47] (03PS1) 10Kengkong: Added MapChart population per capita for editors and viewers [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/1346120 (https://phabricator.wikimedia.org/T424227) [19:33:44] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats, 13Patch-For-Review: Country per capita metrics in wikistats & Analytics API - https://phabricator.wikimedia.org/T424227#12376604 (10Kengkong1) Hi everyone. I have added functionality as close to as outlined as i can. Am here for comments or que... [19:37:16] 14Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: mediawiki_history_incremental_v1: schema specification for stakeholder review - https://phabricator.wikimedia.org/T425573#12376614 (10CMyrick-WMF) > @Pablo @CMyrick-WMF mediawiki.page_change.v1 event stream (and Hiv... [19:46:26] 06Data-Engineering, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover, and 2 others: DC Switchover - MW requests to EventGate cause long queuing waiting time - https://phabricator.wikimedia.org/T439552#12376657 (10jasmine_) [20:01:19] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06SRE Observability, 13Patch-For-Review: Finalize and/or standardize metrics - https://phabricator.wikimedia.org/T438383#12376740 (10AKhatun_WMF) @Ottomata: The current list of metrics we expose in afc lambds ([observability.py](https://gitlab.wikime... [20:06:55] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Add airflow-dags test to check if all dbt_scheduled YAML dependencies exist in datasets.yaml - https://phabricator.wikimedia.org/T439631 (10amastilovic) 03NEW [20:13:45] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 06SRE Observability, 13Patch-For-Review: Finalize and/or standardize metrics - https://phabricator.wikimedia.org/T438383#12376823 (10Ottomata) Thank you! Didn't review in depth but only 1 comment: For the generic metrics, let's check in with @hnowl... [20:30:08] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: MediawikiDumper: Reduce the memory that the huge wiki exports reserve - https://phabricator.wikimedia.org/T438245#12376863 (10xcollazo) >>! In T438245#12376855, @CodeReviewBot wrote: > xcollazo **merged** https://gitlab.wikimedia.or... [20:52:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [20:57:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:06:43] 06Data-Engineering: dbt: a dedicated dbt Airflow instance - https://phabricator.wikimedia.org/T435412#12377064 (10Ahoelzl) a:05amastilovicβ†’03None [21:06:57] 06Data-Engineering: Refactor pingback reports pipelines using dbt - https://phabricator.wikimedia.org/T418190#12377066 (10Ahoelzl) a:05amastilovicβ†’03None [21:08:06] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:10:22] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Dbt lint runs break if done by more than one user on a host - https://phabricator.wikimedia.org/T434460#12377075 (10Ahoelzl) Example: problem can occurs when running dbt compile under `sudo -u analytics` and then running jobs as user. [21:11:59] 06Data-Engineering, 07Essential-Work: Blunderbuss: Move Hadoop/HDFS XML configuration into Helm deployment chart - https://phabricator.wikimedia.org/T402323#12377077 (10Ahoelzl) a:05amastilovicβ†’03None [21:13:06] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=000000026&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [21:13:32] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Create a custom DBT materialization macro - https://phabricator.wikimedia.org/T419310#12377083 (10Ahoelzl) 05Openβ†’03Declined Not needed atm. Microbatching might be the path forward. [21:13:47] 06Data-Engineering: Refactor pingback analytics pipeline - https://phabricator.wikimedia.org/T415283#12377086 (10Ahoelzl) a:05amastilovicβ†’03None [22:06:28] 06Data-Engineering, 06Data-Platform-SRE: Conda-Analytics environments stuck with very outdated packages - https://phabricator.wikimedia.org/T423052#12377234 (10nshahquinn-wmf) [22:06:37] 06Data-Engineering, 06Data-Platform-SRE: Conda-Analytics environments stuck with very outdated packages - https://phabricator.wikimedia.org/T423052#12377235 (10nshahquinn-wmf) [22:13:40] 06Data-Engineering, 06Data-Platform-SRE: Conda-Analytics environments are prone to dependency conflicts and installation errors - https://phabricator.wikimedia.org/T423067#12377251 (10nshahquinn-wmf) [22:28:30] 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Cloud-VPS (Debian Bullseye Deprecation), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12377278 (10tchin) `http://stream.wikimedia.beta.wmcloud.org` and `http://str... [23:41:17] 06Data-Engineering, 06Data-Engineering-Radar, 06Commons, 06Data-Persistence, and 5 others: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys) - https://phabricator.wikimedia.org/T28741#12377472 (10TheDJ)