[12:49:57] 10DBA, 10Patch-For-Review, 10Upstream, 10cloud-services-team (Kanban): Reimage labsdb1011 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T249188 (10Marostegui) After 5 days importing data, labsdb1011 has imported 2.8TB out of around 6.5T, so it is going to take ages to import + replicate. I... [12:55:07] db1092 looks like it is being worked hard [12:56:21] Another slow query from what I can see [12:56:24] I will need to kill it [12:56:28] looks like a maintenance script [12:57:07] Ah no, it is not [12:57:32] db1092 is a new host in s8, was placed there on Friday and I forgot to copy the query killer events, and hence that query has not been killed [12:57:42] Going to kill it and add it to a list of queries to optimize [13:02:47] The fact that the host only lags 1-2 seconds doesn't help to get it depooled by MW [13:02:51] Anyways, it is fixed now addshore [13:06:19] I just realised that db1092 is actually not new, but for some reason didn't have the events there (the new host I was talking about is db1091 which is in another section anyways) [13:08:00] Interesting, it was the only host without events [13:09:35] addshore: created https://phabricator.wikimedia.org/T254688 to follow up that query [13:11:47] 10DBA: Check that all core hosts have events installed and enabled - https://phabricator.wikimedia.org/T254689 (10Marostegui) [13:11:56] And created https://phabricator.wikimedia.org/T254689 to double check all the core hosts [13:14:01] 10DBA: Check that all core hosts have events installed and enabled - https://phabricator.wikimedia.org/T254689 (10Marostegui) [13:15:10] 10DBA: Check that all core hosts have events installed and enabled - https://phabricator.wikimedia.org/T254689 (10Marostegui) [13:16:12] 10DBA: Check that all core hosts have events installed and enabled - https://phabricator.wikimedia.org/T254689 (10Marostegui) [13:16:24] 10DBA: Check that all core hosts have events installed and enabled - https://phabricator.wikimedia.org/T254689 (10Marostegui) [13:16:37] 10DBA: Check that all core hosts have events installed and enabled - https://phabricator.wikimedia.org/T254689 (10Marostegui) [13:18:00] 10DBA: Check that all core hosts have events installed and enabled - https://phabricator.wikimedia.org/T254689 (10Marostegui) Everything checked and they all have apart from db1092 in s8 and es1 which doesn't have them. I will install them on Monday to be consistent with the rest of sections. [13:18:17] 10DBA: Check that all core hosts have events installed and enabled - https://phabricator.wikimedia.org/T254689 (10Marostegui) [13:36:04] Thanks marostegui ! Yeahz we have some grafana alarms setup in API response times which is what flagged it [16:05:16] addshore: I would love to be added to those if they don't send many false positives!