[14:56:32] bd808: what is your level of rando toolforge network failures since Sunday? Still keeping steady? [15:46:59] andrewbogott: things are much, much better. gitlab-account-approval is only showing 8 complete failures in the last ~7 days. [16:14:16] great! I'm hoping that replacing the wonky cloudgw host has resolved the issue but we're going to replace the other on as well for consistency and then I'll see how Integration fares without the special recursor [16:14:18] thanks for checking [17:34:09] toolsdb update: the primary looks mostly stable, so I didn't want to touch it before the weekend, I will do the mariadb minor version upgrade on monday [17:34:28] the replica is still alternating between working correctly and getting stuck for a few hours [17:34:42] I did some more investigation and wrote the results in T386240 [17:34:43] T386240: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2025-02-12 - https://phabricator.wikimedia.org/T386240 [17:41:14] thanks for the update dhinus