[01:55:45] springle: are you here? [01:56:01] * YuviPanda is here to ask about https://phabricator.wikimedia.org/T104476#1487225 [02:00:53] YuviPanda: hi [02:00:56] * springle looks [02:01:10] springle: hi! [02:01:27] I commented with more info [02:05:06] YuviPanda: jynu.s created 'labsdbadmin'@'10.64.37.6' and you connected from 10.64.37.7. Which is correct? [02:05:31] springle: I think that was labstore1001 and I'm now on labstore1002? [02:05:42] (we had to fialover after catstrophic failure) [02:05:59] springle: so I'd say both, but for now I need .7 [02:06:05] so we need both? [02:06:08] ok [02:08:31] YuviPanda: try now [02:08:35] springle: doing [02:09:51] springle: > SELECT command denied to user 'labsdbadmin'@'10.64.37.7' for table 'user' [02:09:58] I'm doing a select from mysql.user to see if the user already exists. [02:11:41] now? [02:14:15] springle: checking [02:16:08] springle: wonderful, that works! is this for all the 4 labsdbs? [02:16:36] 1, 2, 3, 5 [02:16:38] * springle suddenly remembers there is a fourth labsdb :) [02:16:57] springle: :D [02:17:05] there's actually supposed to be a fifth (1004) but not yet [02:17:47] looks like labsdbadmin was not added to labsdb1005 at all. so it should be? [02:17:52] springle: yes [02:17:54] should be [02:18:44] ok. i'll comment all this on the ticket so jynu.s can update whatever puppet stuff he did [02:18:52] springle: ok! [02:18:53] which i havn't looked at, at all [02:21:16] springle: afaict there was the passwords in the private repo, and nothing outside of that? [02:22:12] well the prod boxes at least have grants puppetized into scripts. hopefully here too [02:22:17] but idk yet [02:22:19] ah, I'm unaware of that [02:32:28] springle: wonderful! the new script works! [03:18:11] \o/ [10:47:34] so, db1035 are 3 different issues: [10:49:56] 1) database was repooled too quickly. The problem is that I am unsure if it can be done much slower. Everithing is related to the number of objects, which makes them slower to cache, etc. This created swapping that it is slowly going away. [10:51:08] 2) There is lag, I think, but not 100% sure related to UPDATE /* Wikibase\Client\Usage\Sql\EntityUsageTable::touchUsages. I saw one of those take 5 seconds, but I cannot reproduce on EXPLAIN [10:52:52] 3) There is an underlying job execution problem on connection (probably creating connection avalanches). This is generalized, but it shows more prominently in case of failover/restart [13:24:06] as you may see on the SAL, I have changed db1069:3311 to be a child of db1051 (vslow, dump) so I can run pt-table-sync without affecting production [13:57:50] So I am currently syncronizing on a screen session on iron db1051 enwiki.templatelinks to db1069. I expect this to take days, but at least it does not require labs downtime [14:12:57] ^expect more select ranges than average at those dbs. Context: T106470 [23:37:08] Hey jynus, how easy is it for you to pull up statistics on how often a table has been queried recently? [23:38:37] or springle