[00:09:07] is there anything I should do to get eyes on https://phabricator.wikimedia.org/T440801 (https://phabricator.wikimedia.org/T440801)? [07:17:46] Hi, Is it expected that random tools got their replica credentials rotated yesterday around 16:18? I had some tool accounts change and others not, which unfortunately broke some things as these have not previously changed [19:15:39] Has something happened with the DNS side of things on WMCS? [19:16:26] My DB VM is configured to accept credentials from my other host machines based on their hostnames, but when my web host tries to connect, the DB is only seeing the IP and thus refusing access. [19:20:05] I've tried rebooting both VMs (cyberbot-db-01) and (cyberbot-web-iabot-01) [19:23:10] hm, rDNS looks okay from Toolforge AFAICT (`host cyberbot-db-01.cyberbot.eqiad1.wikimedia.cloud` ⇒ IP, `host 172.16.3.20` ⇒ hostname) [19:40:30] Yea, the IP looks wrong too. [19:41:10] From cyberbot-web-iabot-01 to cyberbot-db-01 I get Access denied for user 's51059'@'10.64.151.2'. cyberbot-db-01 is what's seeing this IP. [20:27:42] Cyberpower678: something was changed recently in response to an incident, which may be related to what you are experiencing. If so, the right set of eyes should see this first thing in the morning in Europe and respond back - thank you for your patience!! [23:14:40] my Trove database has been stuck resizing for a few days now so my tool is down rn, and unfortunately I can't really do anything about fixing it myself. [23:14:51] https://phabricator.wikimedia.org/T440801 (https://phabricator.wikimedia.org/T440801) [23:18:35] ^ if there's anything complex involved in fixing that issue I'm happy to move the tool back to using my ToolsDB database for now, it shouldn't cause any problems since I had only just migrated over to Trove before I hit this issue