[01:58:38] (03PS1) 10David Martin: Add wikilambda_zobject_join query to sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1041815 (https://phabricator.wikimedia.org/T363434) [09:36:27] 06Data-Engineering, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team-WIP, and 9 others: Upgrade push notifications to node18 - https://phabricator.wikimedia.org/T367272 (10Jgiannelos) 03NEW [09:37:05] 06Data-Engineering, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team-WIP, and 9 others: Upgrade push notifications to node18 - https://phabricator.wikimedia.org/T367272#9883510 (10Jgiannelos) a:03Jgiannelos [10:05:13] 06Data-Engineering, 10Metrics Platform Backlog, 10Data Products (Data Products Sprint 15): [MPIC] Analyse risk of potential performance issues with static approach to stream configuration - https://phabricator.wikimedia.org/T366627#9883590 (10phuedx) >>! In T366627#9876441, @Ottomata wrote: > @phuedx can you... [11:25:20] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Event Platform] Instrument EventBus with prometheus MW Statslib - https://phabricator.wikimedia.org/T363587#9883890 (10gmodena) a:03gmodena [12:28:55] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9884091 (10IKhitron) Well, for the last 21 hours no querry could run. So it's pretty bad. [12:53:55] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Movement-Insights, 07Epic: [Data Quality] Implement basic data quality metrics for Unique Devices datasets - https://phabricator.wikimedia.org/T357833#9884141 (10lbowmaker) [13:14:59] 06Data-Engineering, 10Metrics Platform Backlog, 10Data Products (Data Products Sprint 15): [MPIC] Analyse risk of potential performance issues with static approach to stream configuration - https://phabricator.wikimedia.org/T366627#9884235 (10Ottomata) > we should use pre-computed essential metrics with all... [13:31:51] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9884291 (10fnegri) > Ah well, looks like the deployment of https://github.com/toolforge/quarry/pull/40 didn't fully go through. The keys TOOLS_DB_USER and TOOLS_DB_PASSWORD are missing i... [13:47:09] 10Data-Engineering (Q4 2024 April 1st - June 30th): Migrate and re-deploy eventgate using new service runner - https://phabricator.wikimedia.org/T361768#9884377 (10Ottomata) [13:47:10] 10Data-Engineering (Q4 2024 April 1st - June 30th): Replace service runner with a simplified library to better support metrics and debugging - https://phabricator.wikimedia.org/T360924#9884378 (10Ottomata) [13:47:26] 10Data-Engineering (Q4 2024 April 1st - June 30th): Support metrics platform backend migration to new service runner - https://phabricator.wikimedia.org/T361770#9884379 (10Ottomata) [13:47:27] 10Data-Engineering (Q4 2024 April 1st - June 30th): Replace service runner with a simplified library to better support metrics and debugging - https://phabricator.wikimedia.org/T360924#9884380 (10Ottomata) [13:47:51] 10Data-Engineering (Q4 2024 April 1st - June 30th): Replace service runner with a simplified library to better support metrics and debugging - https://phabricator.wikimedia.org/T360924#9884382 (10Ottomata) [13:47:53] 10Data-Engineering (Q4 2024 April 1st - June 30th): Migrate and re-deploy eventstreams using new service runner - https://phabricator.wikimedia.org/T361769#9884381 (10Ottomata) [13:48:23] 10Data-Engineering (Q4 2024 April 1st - June 30th): Migrate and re-deploy eventgate using new service runner - https://phabricator.wikimedia.org/T361768#9884383 (10Ottomata) a:05Ottomata→03None [13:48:35] 10Data-Engineering (Q4 2024 April 1st - June 30th): Migrate and re-deploy eventstreams using new service runner - https://phabricator.wikimedia.org/T361769#9884385 (10Ottomata) a:05Ottomata→03None [13:49:27] 10Data-Engineering (Q4 2024 April 1st - June 30th): Replace service runner with a simplified library to better support metrics and debugging: service-utils - https://phabricator.wikimedia.org/T360924#9884387 (10Ottomata) [14:02:23] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 15), 10MediaWiki-Platform-Team (Radar): Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#9884485 (10Milimetric) a:03Milimetric [14:25:38] 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search, 07Epic: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315 (10Gehel) 03NEW [14:25:53] 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search, 07Epic: [Epic] Replace Archiva with Gitlab artifact repositories - https://phabricator.wikimedia.org/T367315#9884720 (10Gehel) [14:38:05] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9884857 (10Xaosflux) p:05Triage→03High Problem still occurring today and all recent requests have failed (see https://quarry.wmcloud.org/query/runs/all) [14:46:44] 10Quarry, 10cloud-services-team (FY2023/2024-Q3-Q4): [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9884949 (10fnegri) 05Open→03In progress a:05GTrang→03fnegri [15:10:35] 10Quarry, 10cloud-services-team (FY2023/2024-Q3-Q4): [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9885050 (10github-toolforge-bot) siddharthvp opened https://github.com/toolforge/quarry/pull/47 [15:11:05] 10Quarry, 10cloud-services-team (FY2023/2024-Q3-Q4): [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9885051 (10SD0001) >>! In T365374#9880226, @SD0001 wrote: > But this doesn't explain why the issue occurs when querying the wiki replicas. F... [15:12:56] 10Quarry, 10cloud-services-team (FY2023/2024-Q3-Q4): [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9885069 (10fnegri) https://github.com/toolforge/quarry/pull/46 and https://github.com/toolforge/quarry/pull/47 should probably fix the issue,... [15:16:32] 10Quarry, 10cloud-services-team (FY2023/2024-Q3-Q4): [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9885089 (10SD0001) I'm not sure either about how to deploy in the new k8s-based setup. However, `/home/rook/quarry` seems to be the git check... [15:18:49] 10Quarry, 10cloud-services-team (FY2023/2024-Q3-Q4): [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9885105 (10fnegri) I'm confused because that directory has a checkout of a non-main branch: ` root@quarry-bastion:/home/rook/quarry# sudo -u... [16:48:13] 06Data-Engineering, 06Abstract Wikipedia team, 10MediaWiki-extensions-EventLogging, 10Metrics Platform Backlog, and 2 others: wikimedia/metrics-platform 4.1.0 in composer.json causes errors - https://phabricator.wikimedia.org/T367338#9885685 (10vaughnwalters) a:03vaughnwalters [16:57:54] 06Data-Engineering, 06Abstract Wikipedia team, 10MediaWiki-extensions-EventLogging, 10Metrics Platform Backlog, and 2 others: wikimedia/metrics-platform 4.1.0 in composer.json causes errors - https://phabricator.wikimedia.org/T367338#9885741 (10vaughnwalters) p:05Triage→03Unbreak! [16:58:59] 06Data-Engineering, 06Abstract Wikipedia team, 10MediaWiki-extensions-EventLogging, 10Metrics Platform Backlog, and 2 others: wikimedia/metrics-platform 4.1.0 in composer.json causes errors - https://phabricator.wikimedia.org/T367338#9885747 (10DMartin-WMF) Thanks, @vaughnwalters ! Note that updating to v... [16:59:53] 06Data-Engineering, 06Abstract Wikipedia team, 10MediaWiki-extensions-EventLogging, 10Metrics Platform Backlog, and 2 others: wikimedia/metrics-platform 4.1.0 in composer.json causes errors - https://phabricator.wikimedia.org/T367338#9885746 (10vaughnwalters) per @DMartin-WMF - the versioning issue is also... [17:00:23] 06Data-Engineering-Radar, 06Data Products, 10MediaWiki-extensions-EventLogging, 10Metrics Platform Backlog, 07Technical-Debt: Non-deterministic unit test "streamInSample() - session sampling resets" - https://phabricator.wikimedia.org/T304379#9885748 (10phuedx) 05Open→03Resolved a:03phuedx AIUI... [17:07:31] 06Data-Engineering, 06Abstract Wikipedia team, 10MediaWiki-extensions-EventLogging, 10Metrics Platform Backlog, and 2 others: wikimedia/metrics-platform 4.1.0 in composer.json causes errors - https://phabricator.wikimedia.org/T367338#9885775 (10phuedx) Well this is terrifying 😱 v4.1.0 definitely includes... [17:24:47] 06Data-Engineering, 06Abstract Wikipedia team, 10MediaWiki-extensions-EventLogging, 10Metrics Platform Backlog, and 2 others: wikimedia/metrics-platform 4.1.0 in composer.json causes errors - https://phabricator.wikimedia.org/T367338#9885870 (10Reedy) [17:34:17] 06Data-Engineering, 06Abstract Wikipedia team, 10MediaWiki-extensions-EventLogging, 10Metrics Platform Backlog, and 2 others: wikimedia/metrics-platform 4.1.0 in composer.json causes errors - https://phabricator.wikimedia.org/T367338#9885900 (10vaughnwalters) > Could you paste the output of `composer upda... [17:41:32] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10GitLab (Pipeline Services Migration🐤), 13Patch-For-Review: Migrate Data Engineering Pipelinelib repos to GitLab - https://phabricator.wikimedia.org/T344730#9885931 (10Snwachukwu) I drafted a [[ https://docs.google.com/document/d/1F8jUW... [17:52:30] 06Data-Engineering, 06Abstract Wikipedia team, 10MediaWiki-extensions-EventLogging, 10Metrics Platform Backlog, and 2 others: wikimedia/metrics-platform 4.1.0 in composer.json causes errors - https://phabricator.wikimedia.org/T367338#9885957 (10phuedx) I don't see `wikimedia/metrics-platform (4.1.0)` in th... [18:12:45] 10Quarry, 10cloud-services-team (FY2023/2024-Q3-Q4): [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9886036 (10Liz) Some of my queries are running okay now, others are just spinning their wheels, like they are running but they never conclude... [18:15:36] 06Data-Engineering, 06Abstract Wikipedia team, 10MediaWiki-extensions-EventLogging, 10Metrics Platform Backlog, and 2 others: wikimedia/metrics-platform 4.1.0 in composer.json causes errors - https://phabricator.wikimedia.org/T367338#9886063 (10vaughnwalters) >>! In T367338#9885957, @phuedx wrote: > I don'... [18:18:14] 07Analytics-Data-Problem: WMF-last-access cookie sets wrong domain for static assets on commons - https://phabricator.wikimedia.org/T367346 (10Bawolff) 03NEW [18:28:28] 10Quarry, 10cloud-services-team (FY2023/2024-Q3-Q4): [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9886138 (10KylieTastic) >>! In T365374#9886036, @Liz wrote: > Some of my queries are running okay now, others are just spinning their wheels,... [19:23:57] 10Quarry: query runs forever - https://phabricator.wikimedia.org/T366909#9886318 (10SD0001) This could be because of a worker going down. [19:24:18] 10Quarry: query runs forever - https://phabricator.wikimedia.org/T366909#9886320 (10SD0001) →14Duplicate dup:03T278583 [19:25:04] 10Quarry, 06cloud-services-team: Quarry should detect a dead worker and report something better than "running" forever - https://phabricator.wikimedia.org/T278583#9886322 (10SD0001) [19:32:58] 10Quarry, 07patch-welcome: Timer that counts up as the query is running - https://phabricator.wikimedia.org/T353690#9886357 (10SD0001) To implement this, we can record the start time as another attribute in the extra_info JSON blob in query_run table when the query status is set to running, and use JS client-s... [20:32:37] 10Quarry, 10Cloud-VPS: [bug] Lot of queries stuck in queued state for hours and days - https://phabricator.wikimedia.org/T365136#9886632 (10SD0001) No, but the PR fixes the UI so that the actual error shows up. [20:39:07] 10Quarry: Unbreak Quarry query killer - https://phabricator.wikimedia.org/T367363 (10SD0001) 03NEW [20:52:12] 10Quarry: Unbreak Quarry query killer - https://phabricator.wikimedia.org/T367363#9886698 (10SD0001) https://github.com/toolforge/quarry/pull/48 [21:06:24] !log performing rolling reboot of analytics workers: `ryankemper@cumin2002:~$ sudo -E cookbook sre.hadoop.reboot-workers analytics` [21:06:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:27:23] 06Data-Engineering, 06Tech-Docs-Team, 05Goal: Redesign Data Platform docs on Wikitech - https://phabricator.wikimedia.org/T350911#9886785 (10TBurmeister) Decision from 12 June working session (see https://wikitech.wikimedia.org/wiki/Data_Engineering/TOC and [[ https://phabricator.wikimedia.org/T350913#93908... [21:30:34] 06Data-Engineering, 06Tech-Docs-Team, 05Goal: Redesign Data Platform docs on Wikitech - https://phabricator.wikimedia.org/T350911#9886790 (10Ottomata) Heroic! <3 [22:43:06] 10Data-Engineering (Q4 2024 April 1st - June 30th), 06Release-Engineering-Team, 07Spike: [Developer Experience] [SPIKE] Investigate process to automate deployment of folders and artifacts to HDFS - https://phabricator.wikimedia.org/T360968#9886891 (10Ottomata) [22:43:07] 06Data-Engineering, 10Data Pipelines: Add support for repository artifacts in Airflow - https://phabricator.wikimedia.org/T322690#9886892 (10Ottomata) [22:44:25] 06Data-Engineering, 10Data Pipelines: Add support for repository artifacts in Airflow - https://phabricator.wikimedia.org/T322690#9886895 (10Ottomata) [22:44:27] 06Data-Engineering, 14Data-Engineering-Kanban, 10Data Pipelines: [Airflow] Automate sync'ing archiva packages to HDFS - https://phabricator.wikimedia.org/T294024#9886896 (10Ottomata)