[00:35:30] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07OKR-Work: Set up a working, usable dbt installation on stat boxes - https://phabricator.wikimedia.org/T406634#11386761 (10Ahoelzl) [00:35:42] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07OKR-Work: Set up a working, usable dbt installation on stat boxes - https://phabricator.wikimedia.org/T406634#11386762 (10Ahoelzl) [00:43:04] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07OKR-Work: Set up a working, usable dbt installation on stat boxes - https://phabricator.wikimedia.org/T406634#11386779 (10Ahoelzl) @BTullis with https://phabricator.wikimedia.org/T406766 being comple... [00:44:05] Test Kitchen edge-unique experiments (poll 1) - adds: fy2025-26-we3.1-image-browsing-ab-test, fy25-26-we-4-2-hcaptcha-editing, hcaptcha-on-french-wikipedia, image-browsing-enwiki; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [00:45:05] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: Use Hive/Spark timestamps in Refined event data - https://phabricator.wikimedia.org/T278467#11386781 (10nshahquinn-wmf) 05Open→03Declined >>! In T278467#8892637, @xcollazo wrote: > Agreed that the Hive tables can stay as they are, an... [02:03:13] Test Kitchen edge-unique experiments (poll 1) - adds: image-browsing-enwiki, hcaptcha-on-french-wikipedia, fy25-26-we-4-2-hcaptcha-editing, fy2025-26-we3.1-image-browsing-ab-test; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [02:03:23] Test Kitchen mw-user experiment (poll 1) - adds: growthexperiments-get-started-notification, we-3-3-4-reading-list-test1-en, we-3-3-4-reading-list-test1; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [02:14:48] !log Test Kitchen edge-unique experiments (poll 1) - adds: fy25-26-we-4-2-hcaptcha-editing, hcaptcha-on-french-wikipedia, fy2025-26-we3.1-image-browsing-ab-test, image-browsing-enwiki; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [02:14:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [02:14:58] !log Test Kitchen mw-user experiment (poll 1) - adds: we-3-3-4-reading-list-test1, we-3-3-4-reading-list-test1-en, growthexperiments-get-started-notification; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [02:15:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [02:34:18] 10Analytics-Canonical-Data, 06Movement-Insights, 06Product-Analytics: Create a structured list of Wikimedia projects' creation and closure dates - https://phabricator.wikimedia.org/T336999#11386841 (10nshahquinn-wmf) I just worked on a [Wikipedia 25](https://meta.wikimedia.org/wiki/Wikipedia_25)-related requ... [02:48:02] 06Data-Engineering, 06Data-Engineering-Icebox, 06Product-Analytics: Identify imported revisions in mediawiki_history - https://phabricator.wikimedia.org/T221482#11386856 (10nshahquinn-wmf) One additional point I've thought of: if you look in MediaWiki history and find that a group of revisions have the same... [08:34:32] (03CR) 10Snwachukwu: Add HQL for pageviews_per_editor and pageview_top_pages_per_editor (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1206879 (https://phabricator.wikimedia.org/T410289) (owner: 10Snwachukwu) [09:04:53] (03CR) 10Joal: Add HQL for pageviews_per_editor and pageview_top_pages_per_editor (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1206879 (https://phabricator.wikimedia.org/T410289) (owner: 10Snwachukwu) [10:09:39] (03CR) 10Joal: [C:03+2] "Merging for later deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1206823 (https://phabricator.wikimedia.org/T410378) (owner: 10Joal) [10:21:40] (03Merged) 10jenkins-bot: Update Refine-CLI job to not overwhelm metastore [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1206823 (https://phabricator.wikimedia.org/T410378) (owner: 10Joal) [11:04:25] (03PS1) 10Aqu: Add 0.3.11 in changelog [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1207135 [11:19:17] (03CR) 10Aqu: [C:03+2] Add 0.3.11 in changelog [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1207135 (owner: 10Aqu) [11:33:30] (03Merged) 10jenkins-bot: Add 0.3.11 in changelog [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1207135 (owner: 10Aqu) [11:34:32] Starting build #56 for job analytics-refinery-maven-release [11:58:44] Project analytics-refinery-maven-release build #56: 09SUCCESS in 24 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/56/ [12:18:53] 06Data-Engineering, 10CheckUser, 07Essential-Work, 06Product Safety and Integrity (Essential Work Sprint (Dec 15th - Jan 9th)), and 2 others: Drop the cupe_private column in the cu_private_event table - https://phabricator.wikimedia.org/T409710#11387784 (10Dreamy_Jazz) [13:35:06] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28): Provide a Spark thrift-server for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11388069 (10BTullis) I have a suggestion for how we might want to do this, but it relies on the spark driver... [13:50:15] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28): Provide a Spark thrift-server for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11388130 (10brouberol) Naïve question: do we gain something by running the thrift server as a sidecar vs run... [13:58:19] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Product-Analytics, 10Event-Platform: [Event Platform] Disable default collection of user agent for analytics streams - https://phabricator.wikimedia.org/T384964#11388175 (10A_smart_kitten) [13:59:58] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform, 13Patch-For-Review: mediawiki_event_enrichment should enrich all events for the page_content_change stream - https://phabricator.wikimedia.org/T408850#11388177 (10A_smart_kitten) [14:01:52] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Observability-Alerting, 10Event-Platform: EventgateProduceRateStop / EventGateProduceRateAnomaly alert should be active datacenter aware - https://phabricator.wikimedia.org/T405952#11388182 (10A_smart_kitten) [14:02:55] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform, 13Patch-For-Review: mediawiki_event_enrichment - update default params and tests to use mediawiki/page_change 1.3.0 (latest) schema - https://phabricator.wikimedia.org/T407779#11388195 (10A_smart_kitten) [14:03:22] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 07Essential-Work, 06Movement-Insights (FY25-26 H1), 13Patch-For-Review: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11388198 (10A_smart_kitten) [14:15:22] 06Data-Engineering, 10Phabricator, 06Release-Engineering-Team (Doing 😎): Data Engineering's Herald rule (H126) might not currently react to any tasks - https://phabricator.wikimedia.org/T410248#11388233 (10A_smart_kitten) >>! In T410248#11386476, @A_smart_kitten wrote: > [...] to avoid automated actions... [14:17:20] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28): Provide a Spark thrift-server for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11388239 (10BTullis) >>! In T410017#11388130, @brouberol wrote: > Naïve question: do we gain something by ru... [14:20:49] 06Data-Engineering, 10DPE-Mediawiki-Content, 07Epic: Dumps 2.0 Phase III: Production level dumps (SDS 1.2) - https://phabricator.wikimedia.org/T366752#11388247 (10Aklapper) [14:22:49] 06Data-Engineering, 10Data-Engineering-Wikistats: Get visibility which pages are being heavily edited, plundered, which need patrolling - https://phabricator.wikimedia.org/T315196#11388259 (10Aklapper) [14:25:37] 06Data-Engineering, 10Phabricator, 06Release-Engineering-Team (Doing 😎): Data Engineering's Herald rule (H126) might not currently react to any tasks - https://phabricator.wikimedia.org/T410248#11388265 (10Aklapper) Oh, good catch. Edited accordingly after another look at https://phabricator.wikimedia.or... [15:07:05] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 07Epic: Dumps 2.0 Phase III: Production level dumps (SDS 1.2) - https://phabricator.wikimedia.org/T366752#11388501 (10Aklapper) [15:24:02] !log Test Kitchen edge-unique experiments (poll 2354) - adds: logged-out-retention-round2; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [15:24:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:42:52] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Provide a Spark thrift-server for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11388781 (10JAllemandou) If we run the dbt-spark job in k8s, no need for a thrift server... [15:43:30] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Provide a Spark thrift-server for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11388782 (10JAllemandou) The other option not using a Thrift-server not k8s is to make d... [15:44:08] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Provide a Spark thrift-server for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11388784 (10BTullis) Maybe we can get benefit from implementing both approaches. Here i... [15:46:09] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Provide a Spark thrift-server for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11388800 (10JAllemandou) Also, if using spark-thrift, I'm not sure at all it would run i... [15:50:55] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Provide a Spark thrift-server for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11388813 (10BTullis) >>! In T410017#11388780, @JAllemandou wrote: > If we run the dbt-sp... [15:55:49] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Provide a Spark thrift-server for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11388822 (10BTullis) >>! In T410017#11388800, @JAllemandou wrote: > Also, if using spark... [16:21:09] 06Data-Engineering, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Sustainability (Incident Followup): Add monitoring / alerting on the number of MySQL queries done by Hive - https://phabricator.wikimedia.org/T410528#11388939 (10Gehel) [16:28:54] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work: Provide a Spark thrift-server for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11388957 (10JAllemandou) >>! In T410017#11388822, @BTullis wrote: >>>! In T410017#113888... [17:25:02] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rc_type from recentchanges in wmf production - https://phabricator.wikimedia.org/T410531 (10Zabe) 03NEW [17:25:19] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rc_type from recentchanges in wmf production - https://phabricator.wikimedia.org/T410531#11389101 (10Zabe) 05Open→03Stalled Need to wait until T408273 is done. [18:31:31] 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop rc_type from recentchanges in wmf production - https://phabricator.wikimedia.org/T410531#11389347 (10Marostegui) a:03Marostegui @Zabe please let me know when the above is done so I can proceed [18:53:35] dr0ptp4kt: you should get a wikimedia-affiliated bot cloak for wmftkbot: https://meta.wikimedia.org/wiki/IRC/Cloaks [19:10:37] 06Data-Engineering: mediawiki_history_reduced.check_mediawiki_history_reduced error ratio hit - https://phabricator.wikimedia.org/T406525#11389590 (10Ahoelzl) @amastilovic is this blocking any Global Editor Metrics work, specifically the daily updates going forward? [20:12:40] 06Data-Engineering: Make blunderbuss synchronize artifacts for the test-cluster - https://phabricator.wikimedia.org/T410557 (10JAllemandou) 03NEW [21:40:14] 06Data-Engineering, 10CheckUser-SuggestedInvestigations, 10ConfirmEdit (CAPTCHA extension), 06Data-Persistence, and 4 others: Add columns to store associated log ID or revision ID that caused a signal to match - https://phabricator.wikimedia.org/T409093#11390092 (10Dreamy_Jazz) [23:07:58] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Implement Mediawiki Content History SLO monitoring and alerting - https://phabricator.wikimedia.org/T410579 (10Ahoelzl) 03NEW [23:15:00] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Reduce `refine_to_hive_hourly` airflow task number - https://phabricator.wikimedia.org/T380856#11390459 (10Ahoelzl) @Antoine_Quhen is this related to the high number of requests to the Hive metastore? [23:25:05] thx taavi . lemme see if w.mopbot will let me make a request on behalf of the other handle [23:30:19] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Create data-steward-alerts@wikimedia.org google group - https://phabricator.wikimedia.org/T410580 (10Ahoelzl) 03NEW [23:30:39] looks like it'll let me do that. i'll stop the bot for a moment so i can auth on the side and send it the command to request the cloak [23:35:45] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07OKR-Work, 13Patch-For-Review: Provide a Spark thrift-server for dbt with Airflow - https://phabricator.wikimedia.org/T410017#11390490 (10Ahoelzl) [23:36:24] cloak submitted, will resume bot shortly. [23:43:04] !log Test Kitchen edge-unique experiments (poll 1) - adds: fy25-26-we-4-2-hcaptcha-editing, logged-out-retention-round2, image-browsing-enwiki, hcaptcha-on-french-wikipedia, fy2025-26-we3.1-image-browsing-ab-test; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [23:43:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [23:43:14] !log Test Kitchen mw-user experiment (poll 1) - adds: we-3-3-4-reading-list-test1, growthexperiments-get-started-notification, we-3-3-4-reading-list-test1-en; removes: none; fields: none - xLab/MPIC/TK tips at https://w.wiki/FwuD [23:43:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [23:46:01] ^ just the bot starting back up ( "(poll 1)" )