[07:59:01] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Enable HA for the mw-dump-rev-content-reconcile-enrich flink application - https://phabricator.wikimedia.org/T375176#10399757 (10gmodena) @brouberol to confirm: these changes now require that helmfiles f... [08:00:31] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Enable HA for the mw-dump-rev-content-reconcile-enrich flink application - https://phabricator.wikimedia.org/T375176#10399761 (10brouberol) @gmodena ah! I think we're missing a simple value. Let me whip... [08:03:56] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Enable HA for the mw-dump-rev-content-reconcile-enrich flink application - https://phabricator.wikimedia.org/T375176#10399768 (10brouberol) Just to confirm, this is the part of the template that is relev... [08:22:17] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Enable HA for the mw-dump-rev-content-reconcile-enrich flink application - https://phabricator.wikimedia.org/T375176#10399781 (10brouberol) With [mw-content-history-reconcile-enrich: add missing s3 confi... [08:22:54] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Enable HA for the mw-dump-rev-content-reconcile-enrich flink application - https://phabricator.wikimedia.org/T375176#10399782 (10gmodena) >>! In T375176#10379784, @tchin wrote: > @gmodena We don't, altho... [08:24:05] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Enable HA for the mw-dump-rev-content-reconcile-enrich flink application - https://phabricator.wikimedia.org/T375176#10399783 (10brouberol) Yep, that's fine. I've already updated the secrets and submitte... [08:25:15] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Enable HA for the mw-dump-rev-content-reconcile-enrich flink application - https://phabricator.wikimedia.org/T375176#10399784 (10gmodena) >>! In T375176#10399781, @brouberol wrote: > With [mw-content-his... [08:55:33] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board), 13Patch-For-Review: Enable HA for the mw-dump-rev-content-reconcile-enrich flink application - https://phabricator.wikimedia.org/T375176#10399866 (10brouberol) It's because the [patch](https://gerrit.wikimedia.org/r/c/opera... [10:33:21] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Dumps 2.0 (Kanban Board): [Data Quality] Update data_quality schemas to be compatible with Iceberg tables - https://phabricator.wikimedia.org/T356866#10400034 (10gmodena) a:05gmodena→03None [12:14:06] (03PS5) 10Peter Fischer: Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 [12:15:53] (03CR) 10Peter Fischer: "Thanks for the review, I adapted my CR accordingly." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (owner: 10Peter Fischer) [12:26:14] (03CR) 10Peter Fischer: Rewrite MediawikiDumper partitioning implementation (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (owner: 10Peter Fischer) [12:27:12] 06Data-Engineering, 10CampaignEvents, 06Data Products, 06Data-Persistence, and 3 others: Add "event_is_test_event" field to "campaign_events" table - https://phabricator.wikimedia.org/T381759#10400319 (10Ladsgroup) On design of the schema change, one different schema could be to have a dedicated table for... [13:25:44] (03PS2) 10Gehel: refactoring(RefineHelper): extract SparkSchemaLoader from RefineHelper [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1102386 [13:43:39] (03PS6) 10Xcollazo: Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (https://phabricator.wikimedia.org/T381016) (owner: 10Peter Fischer) [13:48:08] (03CR) 10Xcollazo: [C:03+1] "Great stuff Peter!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (https://phabricator.wikimedia.org/T381016) (owner: 10Peter Fischer) [14:32:54] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Product-Analytics, 13Patch-For-Review: [SPIKE] Experiment with approaches for a incremental updates of MediaWiki data in the Data Lake - https://phabricator.wikimedia.org/T370354#10400720 (10Ottomata) [17:49:17] 06Data-Engineering, 07Epic: 20241201 wikidatawiki xml dump not progressing - https://phabricator.wikimedia.org/T382084 (10Ottomata) 03NEW [17:49:33] 06Data-Engineering: 20241201 wikidatawiki xml dump not progressing - https://phabricator.wikimedia.org/T382084#10401359 (10Ottomata) [17:50:29] 06Data-Engineering: 20241201 wikidatawiki xml dump not progressing - https://phabricator.wikimedia.org/T382084#10401362 (10Ottomata) @xcollazo recommended we try to kill (and restart?) the job, as described in T376926#10218853. I'll try that now... [17:55:55] 06Data-Engineering: 20241201 wikidatawiki xml dump not progressing - https://phabricator.wikimedia.org/T382084#10401371 (10Ottomata) ` 17:45:21 [@snapshot1011:/home/otto] $ cd /mnt/dumpsdata/xmldatadumps/private/wikidatawiki 17:45:23 [@snapshot1011:/mnt/dumpsdata/xmldatadumps/private/wikidatawiki] $ cat lock_202... [17:57:19] (03PS6) 10Mforns: Update MediaWiki History to support Temp Accounts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1088232 (https://phabricator.wikimedia.org/T379230) [17:59:45] (03PS7) 10Mforns: Update MediaWiki History to support Temp Accounts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1088232 (https://phabricator.wikimedia.org/T379230) [19:00:56] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data Products, 06Data-Platform, 06Movement-Insights, 13Patch-For-Review: Backfill and recalculate unique devices data from July 2024 to present - https://phabricator.wikimedia.org/T378852#10401539 (10JAllemandou) Today's update: * `backfill_webr... [19:24:10] 06Data-Engineering, 10Phabricator: Add ahoezel, aotto and abaso to acl*phabricator project - https://phabricator.wikimedia.org/T382089 (10Ottomata) 03NEW [19:32:40] 06Data-Engineering: 20241201 wikidatawiki xml dump not progressing - https://phabricator.wikimedia.org/T382084#10401631 (10Ottomata) Discussed with Xabriel in [[ https://wikimedia.slack.com/archives/C07UBR61GPR/p1734031678254259?thread_ts=1734022280.343919&cid=C07UBR61GPR | slack ]]. I was worried about the fac... [19:56:39] (03PS7) 10Peter Fischer: Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 [20:11:12] (03CR) 10CI reject: [V:04-1] Rewrite MediawikiDumper partitioning implementation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (owner: 10Peter Fischer) [21:25:04] 06Data-Engineering, 10SRE-Access-Requests: Add otto to analytics-admins posix user group - https://phabricator.wikimedia.org/T382099 (10Ottomata) 03NEW [21:25:42] 06Data-Engineering, 10SRE-Access-Requests, 13Patch-For-Review: Add otto to analytics-admins posix user group - https://phabricator.wikimedia.org/T382099#10401795 (10Ottomata) I should already be in this group. Proceeding to add myself. [21:34:51] (03CR) 10Peter Fischer: "recheck" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1101892 (owner: 10Peter Fischer) [21:38:57] 10Data-Engineering (Q2 2024 October 1st - December 31th), 06Data Products, 06Data-Platform, 06Movement-Insights, 13Patch-For-Review: Backfill and recalculate unique devices data from July 2024 to present - https://phabricator.wikimedia.org/T378852#10401809 (10Ottomata) Looks like the current webrequest_a... [21:43:35] 06Data-Engineering, 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#10401813 (10Ottomata) Instructions for deploying [[ http... [21:47:08] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add otto to analytics-admins posix user group - https://phabricator.wikimedia.org/T382099#10401827 (10Ottomata) 05Open→03Resolved [21:47:55] 10Data-Engineering (Q2 2024 October 1st - December 31th), 10Data-Platform-SRE (2024.11.30 - 2024.12.20), 13Patch-For-Review: Deploy the HDFS synchronizer (blunderbuss) service to the dse-k8s cluster - https://phabricator.wikimedia.org/T371994#10401830 (10bking) @jelto see above, Puppet laid down the ferm rul... [21:49:16] 10Data-Engineering (Q2 2024 October 1st - December 31th): Replace service runner with a simplified library to better support metrics and debugging: service-utils - https://phabricator.wikimedia.org/T360924#10401847 (10Catrope)