[00:37:18] (03CR) 10Xcollazo: Add DDL for mediawiki_history_incremental_v1 Iceberg table (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1287959 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [02:52:35] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Implement MERGE INTO writers for mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T425729#11943446 (10xcollazo) Added `event_entity = 'page'` support to `MWHistoryDeltaWriter` (... [02:53:06] (03PS1) 10Xcollazo: Add event_entity='page' to MWHistoryDeltaWriter (MERGE 5) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290172 (https://phabricator.wikimedia.org/T425729) [06:53:35] dse-k8s-etcd1001 will be down for a bit, the underlying Ganeti node is being decommissioned and it temporarily needs to go back to DRBD to migrate the VM away [07:35:13] 06Data-Engineering, 10MediaWiki-User-management, 06Stewards-and-global-tools, 10Event-Platform, and 4 others: userrights-interwiki fails with server error - https://phabricator.wikimedia.org/T426832#11943669 (10mszwarc) 05In progress→03Resolved >>! In T426832#11942842, @Xaosflux wrote: > FYI: Just... [07:37:31] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats: Wikimedia Statistics page-views-by-country no-data - https://phabricator.wikimedia.org/T426935 (10Skif.res) 03NEW [07:43:12] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats: Wikimedia Statistics page-views-by-country no-data - https://phabricator.wikimedia.org/T426935#11943689 (10Skif.res) [08:31:05] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Image-Suggestions: ALIS data pipeline produced too many suggestions - https://phabricator.wikimedia.org/T423238#11943863 (10dcausse) I believe this is done? Recent runs produced a reasonable amount of suggestions: {F83116366} If there are no objections I... [08:35:50] (03CR) 10Joal: Add DDL for mediawiki_history_incremental_v1 Iceberg table (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1287959 (https://phabricator.wikimedia.org/T425729) (owner: 10Xcollazo) [08:43:40] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06Traffic, 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11943911 (10JAllemandou) Summarizing a [[ https://wikimedia.slack.com/archives/C05RHK7PS6Q/p1779213929117539 | slack thread ]] from @mforns he... [08:44:02] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11943913 (10AnnieKim_WMDE) In discussing what I need in order to do my work, @AndrewTavis_WMDE indicated tha... [08:45:30] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Image-Suggestions: ALIS data pipeline produced too many suggestions - https://phabricator.wikimedia.org/T423238#11943921 (10APizzata-WMF) >If there are no objections I'll reduce the rate-limit back to 20 evt/sec from the 60 we set when the suggestions co... [08:50:05] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11943959 (10SLyngshede-WMF) @Dzahn if you verified the ssh key out of band, then we can just restore your pa... [08:52:45] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11943962 (10SLyngshede-WMF) [08:53:14] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11943963 (10SLyngshede-WMF) @Ottomata already approved. [09:03:25] (03CR) 10A-pizzata: [C:03+1] "LGTM, +1 given the open discussion on the 90 days." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290064 (https://phabricator.wikimedia.org/T425573) (owner: 10Xcollazo) [09:05:29] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Schema and Stream for "webrequest.page_view" - https://phabricator.wikimedia.org/T426091#11943989 (10JMonton-WMF) [09:05:51] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Schema and Stream for "webrequest_frontend_text" - https://phabricator.wikimedia.org/T426092#11943991 (10JMonton-WMF) [09:35:02] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Iceberg 1.2.1 JAR seems to clash with 1.6.1 on Spark 3.5 executor classpath - https://phabricator.wikimedia.org/T426801#11944048 (10JAllemandou) I think we have hit : https://github.com/apache/iceberg/issues/521 Our sch... [09:39:19] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats: Wikimedia Statistics page-views-by-country no-data - https://phabricator.wikimedia.org/T426935#11944070 (10Skif.res) [09:40:36] 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats: Wikimedia Statistics page-views-by-country no-data - https://phabricator.wikimedia.org/T426935#11944073 (10Skif.res) [10:23:14] 06Data-Engineering, 06Data-Engineering-Radar, 06Growth-Team, 10MediaWiki-extensions-WikimediaEvents, and 4 others: Could not hoist data into experiment.subject_id for event - https://phabricator.wikimedia.org/T421152#11944153 (10phuedx) We've just run our first non-cache-splitting experiment where we only... [12:59:19] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Incremental MWH - MediaWiki event data source improvements - https://phabricator.wikimedia.org/T423935#11944747 (10Ottomata) [13:18:48] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Iceberg 1.2.1 JAR seems to clash with 1.6.1 on Spark 3.5 executor classpath - https://phabricator.wikimedia.org/T426801#11944848 (10APizzata-WMF) > It'd be interesting to check without those fields to validate. I can gi... [15:06:08] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10Event-Platform, 13Patch-For-Review: Schema and Stream for "webrequest_frontend_text" - https://phabricator.wikimedia.org/T426092#11945397 (10Krinkle) [15:19:28] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11945476 (10Dzahn) I have not confirmed the key out-of-band yet. [15:22:09] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11945491 (10Dzahn) @AnnieKim_WMDE Could you please send an email directly from your WMDE account to us (dzah... [15:28:45] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Iceberg 1.2.1 JAR seems to clash with 1.6.1 on Spark 3.5 executor classpath - https://phabricator.wikimedia.org/T426801#11945520 (10APizzata-WMF) Tested and got the same result. repro: ` CREATE TABLE your_db.mediawiki... [15:49:33] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11945608 (10AnnieKim_WMDE) Done! [15:53:03] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Load Google Search Console data into the Data Lake - https://phabricator.wikimedia.org/T420996#11945626 (10Ahoelzl) Thanks @nshahquinn-wmf for double-checking. You are right the first statement regarding supposedly dropped long-tail queries is wrong (Google... [16:04:00] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Iceberg 1.2.1 JAR seems to clash with 1.6.1 on Spark 3.5 executor classpath - https://phabricator.wikimedia.org/T426801#11945665 (10JAllemandou) Ack! Thanks for testing @APizzata-WMF :) [16:13:51] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11945704 (10Dzahn) @SLyngshede-WMF Have you received it? Not sure I have. [16:15:02] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11945707 (10Dzahn) restored the abandoned patched and rebasing it to move forward with... [16:44:11] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History: Iceberg 1.2.1 JAR seems to clash with 1.6.1 on Spark 3.5 executor classpath - https://phabricator.wikimedia.org/T426801#11945752 (10xcollazo) Ideally we would try with latest Iceberg 1.11.0 to see if we can repro there,... [17:30:00] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Fix mediarequest_top_files Dag Failures - https://phabricator.wikimedia.org/T426983 (10Snwachukwu) 03NEW [17:31:27] (03PS1) 10Snwachukwu: Optimize Cassandra load mediarequest_top_files to avoid OOM. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1290844 (https://phabricator.wikimedia.org/T426983) [18:13:58] (03CR) 10Ottomata: [C:03+1] "I don't know enough or have enough time to do a thorough review, but go for it!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1290844 (https://phabricator.wikimedia.org/T426983) (owner: 10Snwachukwu) [18:26:20] 06Data-Engineering, 06Data-Engineering-Radar, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11946072 (10Dzahn) Hi @AnnieKim_WMDE for some reason I have not received the mail. Can... [18:27:16] 06Data-Engineering, 10EventStreams: Support new fields: (1) namespace textual prefix and (2) page title without prefix - https://phabricator.wikimedia.org/T426268#11946076 (10Ottomata) `page_title_unprefixed` ? I like either that or `page_title_key` I think. [18:47:10] 06Data-Engineering, 10EventStreams: Support new fields: (1) namespace textual prefix and (2) page title without prefix - https://phabricator.wikimedia.org/T426268#11946106 (10dr0ptp4kt) Oooh...maybe `page_title_unnamespaced` or `page_title_denamespaced` would get it accurately (even if it isn't technically de-... [19:37:11] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11946252 (10xcollazo) >>! In T424355#11944814, @gerritbot wrote: > Change #1285335 **merged... [19:39:12] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: mediawiki.page_change.v1 - Add user first_registration_dt field - https://phabricator.wikimedia.org/T426998 (10Ottomata) 03NEW [19:39:33] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: Incremental MWH - MediaWiki event data source improvements - https://phabricator.wikimedia.org/T423935#11946264 (10Ottomata) [19:39:39] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 10Event-Platform: mediawiki.page_change.v1 event - Add user first_registration_dt field - https://phabricator.wikimedia.org/T426998#11946267 (10Ottomata) [20:07:59] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Accelerate sqoop landing for MediaWiki History private tables - https://phabricator.wikimedia.org/T424355#11946324 (10APizzata-WMF) Yes, tomorrow I will validate everything on `an-launcher` and mer... [21:12:11] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Improve DbtSkeinOperator usability - https://phabricator.wikimedia.org/T426911#11946475 (10amastilovic) [21:59:52] !log Test Kitchen edge-unique experiments (poll 26177) - adds: none; removes: we-1-8-account-creation-form-v1; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [21:59:54] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:18:10] (03PS1) 10Xcollazo: Add event_entity='user' to MWHistoryDeltaWriter (MERGE 6) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1290945 (https://phabricator.wikimedia.org/T425729) [22:36:42] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 10DPE-MediaWiki-Incremental-History, 13Patch-For-Review: Implement MERGE INTO writers for mediawiki_history_incremental_v1 - https://phabricator.wikimedia.org/T425729#11946848 (10xcollazo) MERGE 6 (`event_entity='user'`) is implemented and cluster-tested... [23:02:48] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Add support for the initialization SQL file to SparkSqlOperator in Airflow - https://phabricator.wikimedia.org/T426111#11946921 (10amastilovic) Closing this task since the init file won't be very useful to us - SparkSQL doesn't support user-defined `MACRO`s. [23:02:54] 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Add support for the initialization SQL file to SparkSqlOperator in Airflow - https://phabricator.wikimedia.org/T426111#11946922 (10amastilovic) 05Open→03Declined [23:17:41] (03PS4) 10Xcollazo: Add DDL for mediawiki_history_incremental_v1 Iceberg table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1287959 (https://phabricator.wikimedia.org/T425729)