[07:54:35] Good morning folks - btullis, would you have a minute for me? I have a question regarding the `wikishared` database connection in superset [07:56:36] actually, I found the answer myself btullis - sorry for the ping :) [08:30:15] 06Data-Engineering, 06Data-Platform-SRE, 10Event-Platform: mw-page-content-change-enrich flink app is missing in k8s staging - https://phabricator.wikimedia.org/T367116#9878404 (10gmodena) [...] > Somehow, the Flink app stopped in staging. We don't alert on stuff in staging, so we never noticed. > This isn't... [08:53:55] 06Data-Engineering, 06Data-Platform-SRE, 10Event-Platform: mw-page-content-change-enrich flink app is missing in k8s staging - https://phabricator.wikimedia.org/T367116#9878490 (10gmodena) >>! In T367116#9878404, @gmodena wrote: [...] > Looking into a manual restart. Will ping SRE if we need privileged acces... [10:21:08] Hello peeps, can someone confirm that docker-registry.wikimedia.org/wikimedia/analytics-datahub is effectively deprecated and I can delete it? It doesn't look like it appears anywhere in deployment-charts, and the last build seems to be early 2022, so I'm pretty confident it is, but I'd rather not delete it without confirmation. Cheers :) [11:07:32] claime: Yes, you can go ahead and delete that. Thanks for checking. [11:07:39] cool thx [11:07:44] yw [11:08:29] !log decommissioning stat1004 for T367147 [11:08:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:08:33] T367147: decommission stat100[4-7].eqiad.wmnet - https://phabricator.wikimedia.org/T367147 [11:18:13] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Event Platform] Instrument EventBus with prometheus MW Statslib - https://phabricator.wikimedia.org/T363587#9879108 (10lbowmaker) [11:18:38] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Dumps 2.0 (Kanban Board): [Data Quality] Update data_quality schemas to be compatible with Iceberg tables - https://phabricator.wikimedia.org/T356866#9879106 (10lbowmaker) [11:18:49] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Data Quality] [SPIKE] Can we identify indicators to inform an SLO for event emission and intake? - https://phabricator.wikimedia.org/T345195#9879104 (10lbowmaker) [11:40:52] !log decommissioning stat1005 for T367147 [12:26:33] so long stat1004! [12:26:35] * elukey is sad [12:57:54] elukey: Sorry about that. Did you have a soft spot for stat1004 :-) [12:59:12] I'm sure we can have dcops send you its name tag :D [12:59:25] !log decommissioning stat1006 for T367147 [12:59:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:59:29] T367147: decommission stat100[4-7].eqiad.wmnet - https://phabricator.wikimedia.org/T367147 [13:01:14] 06Data-Engineering, 10SRE-Access-Requests: Requesting access to Kubernetes deployment for ebysans - https://phabricator.wikimedia.org/T367173 (10Ottomata) 03NEW [13:01:19] yeah I always used it since my early days in Analytics [13:01:21] * elukey sighs [13:02:22] 06Data-Engineering, 10SRE-Access-Requests: Requesting access to Kubernetes deployment for ebysans - https://phabricator.wikimedia.org/T367173#9879592 (10Ottomata) @Snwachukwu needs this to finish {T344730} [13:11:18] FYI, dse-k8s-etcd1001 will go down to a few minutes due to a reboot of the underlying Ganeti node [13:34:57] 10Data-Engineering (Q4 2024 April 1st - June 30th), 07Spike: [SPIKE] Evaluate and document solutions for table-management tooling - https://phabricator.wikimedia.org/T360969#9879760 (10Ottomata) [13:37:25] 10Quarry, 10Data-Services, 10cloud-services-team (FY2023/2024-Q3-Q4), 13Patch-For-Review: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9879766 (10KCVelaga_WMF) @fnegri I am curious about the status of this task, and especially the su... [13:41:06] !log decommissioning stat1007 for T367147 [13:41:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:41:09] T367147: decommission stat100[4-7].eqiad.wmnet - https://phabricator.wikimedia.org/T367147 [13:43:40] 10Quarry, 10Data-Services, 10cloud-services-team (FY2023/2024-Q3-Q4), 13Patch-For-Review: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9879789 (10fnegri) @KCVelaga_WMF it's not working but I'm struggling to understand why. I had a qu... [13:46:48] 10Quarry, 10Data-Services, 10cloud-services-team (FY2023/2024-Q3-Q4), 13Patch-For-Review: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9879841 (10fnegri) We could also try with Superset in the meantime, maybe that will be easier. I w... [13:48:12] 10Quarry, 10Data-Services, 10cloud-services-team (FY2023/2024-Q3-Q4), 13Patch-For-Review: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9879844 (10KCVelaga_WMF) @fnegri thanks for the update! [14:03:36] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Refine Refactoring] Integrate Refine workflow configuration into ESC - https://phabricator.wikimedia.org/T367134#9879964 (10Ottomata) Can we connect this to an (epic?) parent task(s) and subscribe some more folks from DE? Thank you! [14:22:19] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9880054 (10SD0001) 05Resolved→03Open Many failures from today. (Could it be a coincidence that it's occurring only on the days someone is trying to test T348407?) [14:29:24] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9880096 (10fnegri) I suspect they are related yes. Maybe Quarry is trying to connect to another database but using ToolsDB credentials, or viceversa. [14:34:09] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to Kubernetes deployment for ebysans - https://phabricator.wikimedia.org/T367173#9880114 (10Ahoelzl) Approved. [14:34:47] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to Kubernetes deployment for ebysans - https://phabricator.wikimedia.org/T367173#9880116 (10ttaylor) Approved in place of @thcipriani while he is on vacation [14:50:23] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to Kubernetes deployment for ebysans - https://phabricator.wikimedia.org/T367173#9880169 (10Snwachukwu) [14:51:01] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to Kubernetes deployment for ebysans - https://phabricator.wikimedia.org/T367173#9880179 (10Snwachukwu) [16:15:09] !log performing rolling reboot of an-test-worker cluster [16:15:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:12:48] (03PS1) 10Aleksandar Mastilovic: Update changelog.md [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1041712 [17:13:45] (03CR) 10TChin: [C:03+2] Update changelog.md [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1041712 (owner: 10Aleksandar Mastilovic) [17:14:13] (03CR) 10Aleksandar Mastilovic: [V:03+2] Update changelog.md [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1041712 (owner: 10Aleksandar Mastilovic) [17:15:09] Starting build #8 for job analytics-refinery-maven-release [17:41:23] Project analytics-refinery-maven-release build #8: 09SUCCESS in 26 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/8/ [17:44:01] Starting build #8 for job analytics-refinery-update-jars [17:44:53] 10Data-Engineering (Q4 2024 April 1st - June 30th), 06Release-Engineering-Team, 07Spike: [Developer Experience] [SPIKE] Investigate process to automate deployment of folders and artifacts to HDFS - https://phabricator.wikimedia.org/T360968#9881281 (10mforns) I agree. I think we should support `current` as we... [17:46:16] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.42 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1041727 [17:46:17] Project analytics-refinery-update-jars build #8: 09SUCCESS in 2 min 15 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/8/ [17:48:18] (03CR) 10Aleksandar Mastilovic: [C:03+2] Add refinery-source jars for v0.2.42 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1041727 (owner: 10Maven-release-user) [17:48:29] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to Kubernetes deployment for ebysans - https://phabricator.wikimedia.org/T367173#9881285 (10herron) 05Open→03Resolved a:03herron The patch to provision this access has been merged, and will be fully propagated... [17:48:43] (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] Add refinery-source jars for v0.2.42 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1041727 (owner: 10Maven-release-user) [17:49:34] !log Deployed refinery-source using jenkins [17:49:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:57:35] 06Data-Engineering, 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), and 2 others: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#9881346 (10Ottomata) A way to test: ` curl -H 'X-Wikimedia-Debu... [18:03:51] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to Kubernetes deployment for ebysans - https://phabricator.wikimedia.org/T367173#9881401 (10Ottomata) Thank you! [18:16:19] (03CR) 10Mforns: "Also looks good to me!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/958518 (https://phabricator.wikimedia.org/T345446) (owner: 10Fabian Kaelin) [18:50:23] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to Kubernetes deployment for ebysans - https://phabricator.wikimedia.org/T367173#9881782 (10Snwachukwu) Thanks! [19:31:29] 06Data-Engineering, 10Metrics Platform Backlog, 10Data Products (Data Products Sprint 15): [MPIC] Analyse risk of potential performance issues with static approach to stream configuration - https://phabricator.wikimedia.org/T366627#9881968 (10mpopov) @MNeisler just caught me up on this. I just want to share... [19:40:07] 06Data-Engineering, 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), and 2 others: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#9882007 (10Ottomata) Manually tested the code move today. Looks... [19:54:00] 06Data-Engineering, 06Content-Transform-Team-WIP, 10MediaWiki-extensions-Linter, 10Parsoid-Read-Views, and 3 others: Commons page fatals when accessed in parsoid read views mode - https://phabricator.wikimedia.org/T364229#9882060 (10ABreault-WMF) 05Open→03Resolved Witnessed in T366408#9866167 [20:15:51] 06Data-Engineering, 10Data Products (Data Products Sprint 14), 10Web-Team-Backlog (FY2023-24 Q4 Sprint 5): Follow-Up Ticket for QA: Validate Sample Rate Adjustments - https://phabricator.wikimedia.org/T365489#9882104 (10SToyofuku-WMF) 05Open→03Resolved Value continues to trend down, gonna mark this a... [20:25:55] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9882152 (10apaskulin) [21:09:44] 10Quarry: [bug] Trouble running Quarry queries - https://phabricator.wikimedia.org/T367237 (10Liz) 03NEW [21:12:26] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9882298 (10JJMC89) [21:12:27] 10Quarry: [bug] Trouble running Quarry queries - https://phabricator.wikimedia.org/T367237#9882296 (10JJMC89) →14Duplicate dup:03T365374 [22:05:00] 06Data-Engineering, 10Data Products (Data Products Sprint 14), 10Web-Team-Backlog (FY2023-24 Q4 Sprint 5): Follow-Up Ticket for QA: Validate Sample Rate Adjustments - https://phabricator.wikimedia.org/T365489#9882501 (10KSarabia-WMF) @SToyofuku-WMF Thank you for the due diligence! [22:51:14] 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9882558 (10Liz) Sorry to post a duplicate notice. I couldn't find this one when I wanted to post about this bug.