[00:13:47] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Data Pipelines, 10Data-Catalog: Integrate Spark with DataHub with lineage (Data-Engineering) - https://phabricator.wikimedia.org/T306896#10603623 (10Ahoelzl) 05Open→03Resolved p:05Triage→03High [09:17:04] dse-k8s-etcd1001 will temporarily be switched to DRBD, latencies will go up a bit [09:38:32] and dse-k8s-etcd1001 is back to normal [10:11:27] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#10604443 (10ayounsi) So what about: * turnilo full dimensions - 1 months * turnilo sanisitzed/reduced - 12 mo... [10:13:37] 10Data-Engineering (Q3 2025 January 1st - March 31th), 06Infrastructure-Foundations, 10netops: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#10604451 (10JAllemandou) >>! In T387839#10604443, @ayounsi wrote: > So what about: > * turnilo full dimension... [10:22:55] 10Data-Engineering-Roadmap, 06Discovery-Search, 10DPE-Mediawiki-Content, 07Epic, 13Patch-For-Review: EPIC: Update flink jobs to support Flink 1.20 - https://phabricator.wikimedia.org/T376812#10604474 (10Gehel) [10:23:14] 14Analytics, 06Data-Engineering, 06Data-Engineering-Icebox, 06Discovery-Search, and 4 others: [EPIC] Expose rdf-streaming-updater.mutation content through EventStreams - https://phabricator.wikimedia.org/T294133#10604484 (10Gehel) [11:09:02] moritzm: Many thanks. What was the reason for switching it to DRBD, out of interest? Was it for Ganeti maintenance? [11:12:21] yeah, the Ganeti node where the etcd node was running will be reimaged to Bookworm and DRBD is needed to move the VM off to a new node (given these are currently not redundant) [11:23:44] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 07Epic: DomainEvents - Broadcasting and receiving cross-process events - https://phabricator.wikimedia.org/T379935#10604713 (10daniel) >>! In T379935#10603265, @Ottomata wrote: > And then, BusDomainEventS... [11:33:49] moritzm: Ack, thanks. I only wondered because I think that we are due to reimage dse-k8s-etcd1001 today, but it's just a coincidence. [14:00:56] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content, 10Dumps-Generation: commonswiki duymp stuck for 20250301 - https://phabricator.wikimedia.org/T387992 (10xcollazo) 03NEW [14:14:13] 10Data-Engineering (Q3 2025 January 1st - March 31th): Enable Spark data lineage for all Airflow instances - https://phabricator.wikimedia.org/T386862#10605390 (10Ahoelzl) [14:15:12] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content, 10Dumps-Generation: commonswiki duymp stuck for 20250301 - https://phabricator.wikimedia.org/T387992#10605403 (10xcollazo) Killed it similarly as in T362454#9711173. Will wait a bit for `systemd` to rerun automatically since we... [14:21:13] 10Data-Engineering (Q3 2025 January 1st - March 31th): Enable Spark data lineage for all Airflow instances - https://phabricator.wikimedia.org/T386862#10605415 (10Ahoelzl) [14:25:03] !log draining and depooling dse-k8s-ctrl1001 ready for reimage to bookworm and containerd for T377875 [14:28:26] hmmh looks like stashbot is having some issues in the channel [14:32:38] !log reimaging dse-k8s-ctrl1001 [15:07:38] 06Data-Engineering, 10Data-Services: Create views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10605729 (10joanna_borun) [15:08:24] 06Data-Engineering: Create views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10605730 (10fnegri) [15:24:14] !log draining and depooling dse-k8s-ctrl1002 ready for reimage to bookworm and containerd for T377875 [15:24:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:24:17] T377875: Migrate dse-k8s cluster from docker to containerd - https://phabricator.wikimedia.org/T377875 [15:25:27] !log reimaging dse-k8s-ctrl1002 [15:25:28] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:03:50] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Remove .m. subdomain, serve mobile and desktop variants through the same URL - https://phabricator.wikimedia.org/T214998#10606045 (10Krinkle) [16:19:26] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Remove .m. subdomain, serve mobile and desktop variants through the same URL - https://phabricator.wikimedia.org/T214998#10606183 (10Jdlrobson-WMF) >>! In T214998#10600094, @Peter wrote: > I've been looking into the data we get... [16:23:39] 06Data-Engineering, 10MediaWiki-extensions-General, 07Documentation, 10Event-Platform: Update code comment links to Meta-Wiki schemas to new event platform - https://phabricator.wikimedia.org/T371305#10606241 (10Pppery) [17:05:23] 06Data-Engineering, 06Machine-Learning-Team, 06Research, 10Event-Platform: Emit revision revert risk scores as a stream and expose in EventStreams - https://phabricator.wikimedia.org/T326179#10606508 (10Ottomata) [17:05:26] 06Data-Engineering, 06Machine-Learning-Team, 06Research, 10Event-Platform: Emit revision revert risk scores as a stream and expose in EventStreams API - https://phabricator.wikimedia.org/T326179#10606512 (10Ottomata) [17:16:35] 10Data-Engineering (Q3 2025 January 1st - March 31th), 07Essential-Work: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10606581 (10mforns) Hey all! Here's a second version of the dashboard with: - Deduplicated multipart range requests (206) - UserAgent filter - Self identi... [17:44:55] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content: Estimate effort for migrating wmf.wikidata_entity to the new mediawiki content pipelines - https://phabricator.wikimedia.org/T388040 (10xcollazo) 03NEW [17:47:07] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content: Estimate effort for migrating wmf.wikidata_entity to the new mediawiki content pipelines - https://phabricator.wikimedia.org/T388040#10606767 (10xcollazo) [17:47:08] 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 07Epic: Support downstream users in adopting mediawiki_content_history_v1 - https://phabricator.wikimedia.org/T387021#10606768 (10xcollazo) [17:51:34] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content: Estimate effort for migrating wmf.wikidata_entity to the new mediawiki content pipelines - https://phabricator.wikimedia.org/T388040#10606781 (10xcollazo) [18:11:41] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content: Estimate effort for migrating wmf.wikidata_entity to the new mediawiki content pipelines - https://phabricator.wikimedia.org/T388040#10606895 (10Ottomata) > could be generated from the new wmf_content.mediawiki_content_history_v1 t... [18:29:12] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Remove .m. subdomain, serve mobile and desktop variants through the same URL - https://phabricator.wikimedia.org/T214998#10607012 (10Krinkle) I've written up my analysis and proposal at: https://www.mediawiki.org/wiki/Requests_f... [19:05:31] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 10Charts (Sprint 17), 07Schema-change-in-production: Deploy patch-gjlw_namespace_text.sql on x1.commonswiki for JsonConfig - https://phabricator.wikimedia.org/T385917#10607173 (10bvibber) >>! In T385917#10580894, @Ladsgroup wrote: > Once the patch is m... [19:06:02] 06Data-Engineering, 06Data-Engineering-Radar, 06DBA, 10Charts (Sprint 17), 07Schema-change-in-production: Deploy patch-gjlw_namespace_text.sql on x1.commonswiki for JsonConfig - https://phabricator.wikimedia.org/T385917#10607174 (10bvibber) [19:08:14] 10Analytics-Canonical-Data, 06Data-Engineering, 06Data-Engineering-Icebox, 06Movement-Insights: Automate the loading of canonical data tables to the Data Lake - https://phabricator.wikimedia.org/T339928#10607179 (10Milimetric) FYI: this icebox task: T241741 was trying to do the same thing quite a while ago... [19:10:52] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content: Estimate effort for migrating wmf.wikidata_entity to the new mediawiki content pipelines - https://phabricator.wikimedia.org/T388040#10607193 (10xcollazo) >This would give you the 'stable' wikibase 'rendered' entity content, rather... [19:30:08] 10Analytics-Canonical-Data, 06Data-Engineering, 06Data-Engineering-Icebox, 06Movement-Insights: Automate the loading of canonical data tables to the Data Lake - https://phabricator.wikimedia.org/T339928#10607249 (10Ottomata) Should we close one of these tasks as a duplicate? [19:37:44] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content: Estimate effort for migrating wmf.wikidata_entity to the new mediawiki content pipelines - https://phabricator.wikimedia.org/T388040#10607267 (10Ottomata) Hm! Interesting. Not sure I'm following. (1) is a 'metadata' table...meanin... [19:46:16] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10MediaWiki-DomainEvents, 06MW-Interfaces-Team, 07Epic: DomainEvents - Broadcasting and receiving cross-process events - https://phabricator.wikimedia.org/T379935#10607303 (10Ottomata) Hm, interesting. I'm about at 70% understanding how all this fits... [20:12:56] 10Data-Engineering (Q3 2025 January 1st - March 31th), 07Essential-Work: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10607401 (10HCoplin-WMF) Fabulous!! Thank you for the quick turnaround. I also got access approved yesterday, so I will take a deeper look this afternoon. [20:25:32] 10Data-Engineering (Q3 2025 January 1st - March 31th), 07Essential-Work: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10607421 (10HCoplin-WMF) First impressions are wonderful! I do have a couple of immediate follow up questions, if you wouldn't mind clarifying: 1. Coul... [20:52:54] 10Data-Engineering (Q3 2025 January 1st - March 31th): Fix service-utils metrics routing naming discrepancy - https://phabricator.wikimedia.org/T387824#10607474 (10tchin) Reconstructing the path using only the `req` object does work, but only if the params belong to the local router. So basically it doesn't work... [22:37:38] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content, 06Research: A dataset sensor should work indepent of airflow instance - https://phabricator.wikimedia.org/T386973#10607772 (10amastilovic) @xcollazo that is correct, yes. [22:46:22] 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Commons-Impact-Metrics, 13Patch-For-Review: [CIM] Skewed ranking with the top Editors monthly API - https://phabricator.wikimedia.org/T370470#10607782 (10EChukwukere-WMF) Test status: //**QA PASS**// APIs Tested and ranking has been confirmed to be in... [22:48:01] 10Data-Engineering (Q3 2025 January 1st - March 31th), 07Essential-Work: Analyze Dumps Usage Through Apache Logs - https://phabricator.wikimedia.org/T383175#10607785 (10Ahoelzl) Great work @mforns ! Happy you haven access @HCoplin-WMF now. Regarding your follow up questions: 1. the chart helps understand wha... [22:55:45] 06Data-Engineering, 10Commons-Impact-Metrics: [Automation] Add test to check for ranking order of all the Top series APIs in CIM - https://phabricator.wikimedia.org/T388063#10607814 (10EChukwukere-WMF) [23:34:20] 06Data-Engineering: 19 new wikis missing from mediawiki_history - https://phabricator.wikimedia.org/T386649#10608098 (10nshahquinn-wmf) Since this task was filed, `satwiktionary` and `sylwiki` have also been created, so I expect those also need to be added. [23:41:36] 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Remove .m. subdomain, serve mobile and desktop variants through the same URL - https://phabricator.wikimedia.org/T214998#10608111 (10Jdlrobson-WMF) > As part of my analysis at T214998#10551073, I went through much of the long ta... [23:57:15] 10Analytics-Canonical-Data, 06Data-Engineering, 06Data-Engineering-Icebox, 06Movement-Insights: Automate the loading of canonical data tables to the Data Lake - https://phabricator.wikimedia.org/T339928#10608184 (10nshahquinn-wmf) [23:57:18] 06Data-Engineering, 06Data-Engineering-Icebox, 10Data Pipelines, 06Movement-Insights: Keep canonical_data.wikis updated - https://phabricator.wikimedia.org/T241741#10608187 (10nshahquinn-wmf) →14Duplicate dup:03T339928