[08:37:03] jynus: about designated new master for S3 I have a doubt [if you didn't arrived yet to my email in the backlog :-) ] [08:38:43] yeah, saw it [08:38:56] in theory, we will work only with the new servers [08:39:14] also for vslow/dump and the other more "expensive" roles? [08:39:58] those will lag no matter what, so we can have a smaller one, if it is needed [08:40:42] that is a yet-unresolved-problem: https://phabricator.wikimedia.org/T124681 [08:41:10] no need for special servers on s3 much due to being smaller tables [08:41:23] dump/vslow can be an issue(?) [08:42:07] those are the kind of questions why we do not decommision servers right away [08:42:22] and we could keep them around for some time just in case [08:42:47] yes, on that I totally agree [08:42:57] however, IF that was an issue, we may want to have other servers, between db1050 and db1073 preciselly for that [08:43:32] so that was my best bet, but I do not really have a perfect argument [08:44:06] the other option is that servers with with low load may be consolidated with several shards [08:44:42] on machines of the same power of the others, just 2 for all shards, make sense too [08:45:19] in theory, those 3 machines have way more power (and performance) than all the existent ones [08:45:33] I just cannot predict potental problems [08:45:53] but please share your opinion, this is more than debatable [08:47:08] so, from what I can see from tendril I'm pretty confident that 1 of the new slave can take all the load in s3, and that is required if we have only 3 of them and one is the master [08:47:09] e.g. do you have an alternative option or are you just doubting because I told you preciselly the opposite on the other shards? :-) [08:47:21] yep [08:47:59] the main issue is the automatic depooling [08:48:09] less servers, less ha [08:48:10] I'm just wondering if on the application side, given that the developers know that there are dedicated hosts for specific roles, maybe they will end up doing some heavy query that might affect production queries [08:48:17] if they run on the same slave [08:48:44] I know that usually the queries pass through your revision [08:48:52] for that, maybe keeping an aditional slave? [08:49:10] even if it lags a bit? [08:49:36] problem is that also there is not mechanism to say "it is ok for this server to lag" [08:49:57] and again we go to the same problem as usual, lack of flexibility on load balancing for mediawiki [08:49:57] could be an option, I like too the idea of the 2 slaves with all shards dedicated to all this sort of stuff [08:50:08] yep [08:50:11] yeah, always back to square one [08:50:12] that would be ideal [08:50:25] but again, that is something we do not have now [08:50:55] it just "feels bad" to have large servers with enough capacity (unlike the other shards) [08:50:58] and not use them [08:51:29] in particular, s3 had replication issues in the past due to concurrency [08:52:23] so, my bet would be- use the newer ones, should the worst happen, we can go back to a more conservative distribution [08:52:54] I think we could go ahead with db1075 and have a plan B with one of the older [08:52:59] yes [08:53:10] oh sorry, press enter before reading the backlog :) [08:53:11] look, if it does not work, totally my fault [08:53:37] if it increases the performance 5x, as I think it would, my fault too :-) [08:53:50] rotfl, not "fault" in that case ;) [08:54:06] we can do some things before hand, like increasing the current weight to test it [08:54:37] I forgot about doing that before I left (but it was risky then) [08:55:09] do you want to have them for some hours and I do s3 on the afternoon? [08:55:28] it's the same for me [08:56:07] actually, I would do it the other way round, but whatever works better for your schedule [08:56:25] restart first, then (slowly) put them into more load [08:57:27] you want to restart all of them? I was planning only the designated master [08:58:15] could make sense, so they have already the new certs among them [08:58:16] yes, that [08:58:28] ah, true [08:58:51] if at any time, probably now is the place [08:58:52] so we have one shard less during the switchover to do :D [08:59:03] *moment [08:59:12] as you wish, what is the current state? [08:59:18] of the others, I mean? [08:59:26] is only this left? [08:59:43] yes, S4 just waiting the buffer pool to replenish before repooling [09:00:05] then, *if* you have time, go ahead [09:00:22] ok, I'll do it. [09:00:28] as you know, the main issue is that we cannot just do everthing for tuesday [09:00:43] I'll move a bit the slaves around to make codfw slave of db1075 for s3 [09:00:45] and during the switchover [09:01:18] we can keep later the master around for a proper content check [09:01:33] yep, about that I wanted your opinion on which days you would like me working next week [09:01:39] when you might need more hands [09:02:10] well, tuesday evening or wednesday morning [09:02:13] probably [09:02:33] I think it is scheduled at 18h or so [09:02:41] (have to check) [09:03:09] <_joe_> actually I think it's scheduled for 4 PM [09:03:18] UTC isn't it? [09:03:31] 19 April at 14:00 UTC from the blog post [09:03:33] that is why I assumed 18 CEST [09:03:36] ah! [09:03:46] so I got it 2 hours shifted [09:03:47] and switchback on Thursday, 21 April at 14:00 UTC [09:04:03] so the same for that, evening or morning, whatever works better [09:04:26] probably the swithover will be more fun [09:04:40] I have no problem to work 3 days and if really needed I can do probably more [09:04:45] switch-back, I meant [09:05:26] so depending if you need a hand on monday to finish stuff or not, I can do tue-wed-thu or whatever other combination [09:05:37] worse case scenario, a) replication breakage due to data drift b) queries not working due to different query planner [09:05:38] you think is better [09:06:17] I think thursday and friday will be more important [09:07:13] ok for those 2, pick another :) [09:32:27] jynus: Wikibase client usage tracking database changes are live on all wikis now... look at the update rates ;) [09:33:04] should I be worried? or happy :-)? [09:33:18] jynus: happy! [09:33:38] Thu 23:38:24 volans| hoo: just saw, you're not here unfortunately, if it's all working as before that's awesome! [09:33:39] Happy :D [09:34:13] do you have a one-line summary? [09:34:25] volans: It took me a lot of research and thinking (and alos testing) to get this right... but I think it's ok :) [09:34:38] I am totally disconnected [09:34:51] still reading old phab tickets [09:35:05] jynus: We found a way to implement the wikibase usage tracking without having to use (and updated) the eu_touched field at all (compared to on every page edit/ purge/ ...) [09:35:38] https://phabricator.wikimedia.org/T122429 that ticket made me look into that (and what you said in San Francisco) [09:37:08] the what I know, the how is what intrigues me [09:37:31] lol https://phabricator.wikimedia.org/T125838#2209090 [09:37:57] so, do you believe me when I said "this is worse db problem we have now?" [09:38:05] Yes, indeed [09:38:16] similar declines can be found on other shards [09:38:32] I do not know how you track it usage now? [09:38:33] on commons it's huge also (but we already deployed it there on Tuesday) [09:38:34] yeah, that's way I said, if all is working as before is awesome :) because if I didn't know about the change I'll be quite worried about that drop [09:38:53] jynus: As described here: https://phabricator.wikimedia.org/T124737 [09:39:06] in a nutshell: Like pagelinks etc. [09:39:23] but we have some added complexity because our data is multilingual [09:40:07] We still (potentially) do an unhealthy amount of delete and inserts, but that's far from what we did before [09:41:05] do you have actuall commit links, I think I still do not understand the previous and new behaviour [09:41:36] Is it this? https://gerrit.wikimedia.org/r/#/c/278045/ [09:41:46] Indeed [09:44:14] "Behaviour on edit:" in the RfC ticket (https://phabricator.wikimedia.org/T124737) is probably what's interesting for you [09:44:21] just read over the class names [09:45:00] That's the old behavior [09:45:17] Ah right, we also have docs in Wikibase: https://gerrit.wikimedia.org/r/#/c/278045/5/docs/usagetracking.wiki [09:45:33] so you delete and insert instead of large updates? [09:45:58] Not instead, we did that in the past as well [09:46:10] ah, I think I get it [09:46:21] so that didn't change (it probably even reduced because I have smarter diffing of what needs to be changed) [09:46:28] so it is more of a "bug", the previous method was not so good [09:46:45] It was a to complicated design trying to solve to many things at once [09:46:47] I am saying this because I have a proposal [09:46:49] but that didn't fly [09:47:01] to change the current *links tables [09:47:20] which technically, you are already using [09:47:54] once everthing is tested, etc, feel free to send a drop column alter [09:48:06] of course, it will have very little precedence [09:48:50] Already opened a ticket for that... not high on my list, but that's a nice little task to do on an evening... doesn't need so much thinking [09:49:14] *links updates are the other cause of issues and I think I can solve those [09:49:24] with a title table [09:49:36] plus normalization [09:49:45] interesting, do you have a ticket for that w/ details? [09:49:50] I'm really interested in that [09:49:52] no, it is only on my mind [09:50:03] and I need to gather evidence first [09:50:17] but the idea (on topic of the channel, not wikidata related) [09:50:37] would be that links tables now store the target of the links as text [09:50:57] mainly vecause we do not want to handle problems if the target moves [09:51:05] the links are to titles, not to ids [09:51:09] Indeed, yes [09:51:25] title, namespace pairs, actually [09:51:29] however, that means that there are millions of Template:GPL links [09:51:45] sorry [09:52:08] as in 'Template', 'CC-BY-SA' [09:52:19] it is an example, it may not be exatly like this [09:52:40] but only only templatelinks, on externallinks, pagelinks, imagelinks, etc. [09:53:10] the idea is to have a title table (becaming a strong entity) [09:53:35] and add only the ids of the title (not the page, which can be moved) [09:53:49] the joins would be as fast (PK lookups) [09:54:09] but we would reduce 10x the storage needed? [09:54:32] and that is 10x times better latency (even more if tables now fit into memory) [09:54:39] ah, so a title (pk, page ns, page title, page id) table? [09:54:55] the relation between title and page is not clear [09:55:04] page id could be null [09:55:07] I guess [09:55:07] I have to study it [09:55:19] in that case, maybe it should be an id on page [09:55:23] yeah, that as well, depending on when you apply normalization :/ [09:55:26] that part is not clear [09:55:39] even if there is no relation, it would be ok [09:56:03] the problem is how much code should be rewritten [09:56:18] handling unreferenced titles, etc. [09:56:24] are the gains real? [09:56:28] needs testing [09:56:36] when done, I will write an RFC [09:56:48] Adopting MediaWiki and all extensions to that will be a multi year effort [09:56:52] painful [09:56:53] :( [09:56:59] it is ok [09:57:13] we could literally do a view to fake that and do it slowly [09:57:42] or keep the old table around [09:57:58] but unlike normalizing the users [09:58:25] which is I think more complicated, I think 300GB tables have better ROI [09:59:33] if we can modify core and keep backwards compatibility, it would have been worth it [10:00:08] hence the investigation, and then the RFC [10:00:33] Writing it down and pointing people to it (aaron, tim, ...) is probably a good idea [10:00:55] also, even if it is not practical [10:01:10] I want to stop people from doing the same [10:01:41] aka storing a string for a field with just a few values [10:01:51] Jaime, do you know anything about our CI automation? [10:02:17] hm... maybe such an id => entity id (string) table would be of value for Wikibase client's as well [10:02:48] people that do not know much about mysql think that joins are expensive [10:03:11] when in many cases, storing duplicate strings is many times worse [10:03:29] joining to a table using PK is O(0) [10:03:56] literally, not even log(n) (due to PK lookup optimization on InnoDB) [10:04:39] while my estimation is that we can save hundreds of GB in space [10:05:07] volans, question worries me, probably I wouls say no, but maybe more than you so I can help? [10:05:22] seems stuck, but I cannot see any job pending on jenkins, could be Zuul [10:05:39] there are no phplint on my last CR and the one +2 is pending the automerge [10:06:04] CI is handled by Releng, try on that channel or -ops? [10:06:34] ops already tried without luck, I'll check releng [10:06:40] Yeah, we should have kept that in mind when designing these tables [10:07:00] I'll see whether I can do that when redoing Wikidata (repo) tables [10:07:01] is it a server issue? [10:07:21] because in that case, restart button is the solution, no need to contact them [10:08:43] Anyway, I should head off... I hope you appreciate the changes and I'll consult you when I get to redesigning the other Wikibase tables [10:08:49] not that I'm aware off, I can ssh to gallium, and zuul-gearman.py status works,but I don't know which job/worker to check [10:10:34] then that is beyond me, releng it the right place [10:54:45] jynus: FYI on s3 I'm moving db1075 under db1027, then db2018 under db1075 and then db1075 back under db1038 [10:55:31] sounds good, to maintain SSL, right? [10:56:24] yes, to have db1075 ready to become s3 master during the switchover [10:56:30] +1 [10:56:32] and codfw replicating from it [11:00:24] strange, the semisync [11:00:52] I was expecting that on some servers and a few times per hour, not so many times [11:01:07] at 1ms? [11:01:14] it is 1? [11:01:24] it should be 1000 [11:01:35] on s1 and s3 yes, the others default at 10000 [11:01:45] https://phabricator.wikimedia.org/T131753 [11:01:58] well, that is 10 seconds, right? [11:02:21] yes, too much [11:02:29] brb, 3 min [11:03:34] we have a config issue there [11:14:44] Are you sure about this? T132539 [11:14:44] T132539: Dumps for misc triggers unnecessary pages - https://phabricator.wikimedia.org/T132539 [11:25:52] yes, my findings for semi-sync are the same, for s3 I changed it to 100ms when it was logging like crazy the other day [11:26:23] because s3 has only 1 semi-sync slave and it was having a bit more load that probably triggered all the failing at 1ms [11:26:43] why 1 ms, maybe me or someone else meant 1 second at some point? [11:27:09] I thought the same, someone didn't check the documentation I think because 1ms is really to small [11:27:26] athough that timeout could be added to any transaction before acking the commit [11:27:30] in a proper config, that would be very low, but realisticly we probably cannot guarantee more than 1 second [11:27:33] less in the future [11:27:34] so I would try to keep it small [11:27:55] starting from 1000 and lowering over time checking the increase of the counters [11:28:00] +1 [11:28:13] optimistically down to something around 10~50ms probably [11:28:26] as a goal, yes [11:28:33] sadly that is mostly out of our control [11:28:42] except for ROW/hardware/etc. [11:29:48] 1 seconds should not be more than a few edits, so not that a big compromise [11:30:25] true and we should have an homogeneous number of slave with semi-sync enabled, s3 has only one for example [11:30:46] and not using semisinc could be worse than having a low timeout [11:30:54] (not using it because it timeouts) [11:31:04] when we'll have the 3 servers setup probably just both of the slaves should have it [11:31:10] yeah [11:31:35] I mentioned that that needs a review, and then like a plan, but we should block it after the swithover [11:31:59] then enforce it on proper config and monitor it [11:32:06] yes [11:32:16] I still need to audit codfw, hopefully today [11:32:34] do not give that huge priority [11:32:45] it is very dynamic after all [11:32:50] ok [11:32:56] for T132539 what was your question? [11:32:56] T132539: Dumps for misc triggers unnecessary pages - https://phabricator.wikimedia.org/T132539 [11:32:57] from enabling the plugin to changing config [11:33:17] the lag, is it recurring? [11:33:35] every week [11:33:49] I remember having issues with the stats gathering, never with the backups [11:33:58] which may also happen at the same time [11:34:10] when the crontab runs, because I think that the backup stops the replica, but the nagios check with heartbeat checks the real time [11:34:13] lag [11:34:51] maybe there is an interaction between the backups and the long running queries [11:35:06] we don't get the pages, it's night here, by I was wake up by aNag on my phone and then checked with Yuvi and he confirmed me he got pages for the last 3 weeks [11:35:21] and he said nothing! [11:35:36] both m3 and x1, I guess the others not because are smaller and don't reach the alarm threshold [11:35:45] well, to be fair, phab host had issues for a month due to migrations [11:36:02] so understandable (remember I had to disable those for a while) [11:36:26] mmm, I have to check those, I only checked the one for production [11:36:41] it does mysqldump --single-transaction --quick --dump-slave=2 --include-master-host-port [11:36:56] check /a/dumps-misc on dbstore1001 root's crontab [11:38:11] I said in the task that the replica stopped because it was NULL and then started by itself after that [11:38:29] mmm [11:38:37] needs testing [11:38:41] and I don't see any stop slave start slave i nthe script [11:38:47] could be the --single-transaction? [11:39:02] in theory, it should only run SHOW SLAVE STATUS [11:40:04] at first I though that stopped the replica to be sure to have coherent static data [11:40:23] it could be, but not during the whole copy [11:40:49] maybe it is version-dependent [11:41:13] could be, the issue is basically split in two [11:41:32] is correct that the replica is stopped? if not, fix it, if yes, icinga should not alarm :) [11:41:56] no, it should not stop, and even if it stops, it should not alarm [11:42:13] the lag should be the only alarm [11:42:17] the page is for the lag [11:42:25] then there is an issue [11:42:37] and I guess that started when you did the hearthbeat stuff [11:42:48] because icinga lag is calculated also if the replica is stopped [11:42:50] but a stopped replication should not alarm [11:43:01] could it be load related? [11:43:41] needs research, it wouldn't be the first time that MariaDB's SHOW SLAVE STATUS shows NULL values incorrectly [11:44:22] why do you think it is the backup, and not the statistics? [11:45:18] because I think I saw only the export there at the time, my bad to not have put a full processlist in the phab let me see if by any chance I have it in my editor [11:45:49] I will check it, not a big deal- x1 should go to dbstore at some point [11:46:06] and phab secondary server is not a big deal [11:46:21] and we could evend drop the alarm [11:46:33] but I want to know the actual cause first [11:47:17] 2016-04-06 03:06:35 icinga-wm| PROBLEM - MariaDB Slave Lag: m3 on db1048 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 301.71 seconds [11:47:20] 2016-04-13 03:08:16 icinga-wm| PROBLEM - MariaDB Slave Lag: m3 on db1048 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 390.36 seconds [11:47:23] from irc logs [11:49:07] good news is that we have such HA, that we do not realize about error [11:49:09] *S [11:49:35] (server is depooled for x1, and not important for m3) [11:52:38] so the only query that I saw was the dump, user root, I checked the IP and pointed me to dbstore1001 [11:53:08] | 3983797 | root | 10.64.48.17:52809 | phabricator_metamta | Query | 558 | Sending data | SELECT /*!40001 SQL_NO_CACHE */ * FROM `metamta_mail` | 0.000 [11:54:23] I have also a show slave status if needed [11:54:37] that has Seconds_Behind_Master: NULL [11:58:40] * volans going to get some lunch, he'll continue with TLS on s3 just after it [12:07:33] I noticed that a few mw* servers have mysql-server-5.5 installed and listening on localhost: mw1022, mw1023, mw1024, mw1025, mw1114 and mw1163. does anyone why and what this is used for? [12:10:58] * volans lunch delayed a bit [12:11:26] moritzm: I have no idea, checked very quickly mw1024 and has no database apart default ones [12:11:37] which puppet role is installing it? [12:15:27] I can't find it in any role, my suspicion is that this is some manual debugging cruft [12:16:06] those are all canary? [12:18:20] all except mw1163, yes [12:20:00] mmmh, then could be related to that, we can see if Jaime know something about it otherwise probably better checking with ori and guys in -perf. My 2 cents given that I have no clue [12:20:24] sorry for not being helpful [12:27:53] thanks, it's nothing urgent anyway, just something that made me wonder during upgrades [12:28:53] of course, it's quite strange, good catch [12:34:12] * volans lunch for real now [12:59:23] no, I investigated those when told by joe, no reasons that I can know [14:12:42] jynus: I have some spare time (waiting that buffer pools of s3 slaves replenishes, will take a while with those big servers) [14:12:50] I was thinking to reimage db2047 [14:12:57] but let me know if you have anything more urgent [14:14:45] let me see in one second what is the list of important things left for next week [14:15:29] but I would say either that or the missing table on dbstore200 something [14:15:52] also, did you check the depooled es2* server? [14:16:03] the one that crashed? that would be more important [14:16:08] 2019? [14:16:12] maybe [14:16:36] https://phabricator.wikimedia.org/T130702 ? [14:16:42] by uptime, yes [14:16:53] yep [14:17:08] es is more important than the rest, I think [14:17:24] ok, I'll take that [14:18:36] I think that creating an hash of the rows before and after that time on all wikis would be enough [14:18:59] my biggest fear is missing revisions [14:19:31] I remember recording the timestamp preciselly for that [14:20:49] it's in the task [14:39:08] jynus: given that the blobs_cluster25 tables don't have any time, should I get the IDs around that time for each wiki from the core dbs? [14:41:35] oh, yes [14:41:53] you can actually check the ids, just if you are interested [14:42:14] the revision/text table should give you the actual ids on es* [14:48:41] I'll take a look [15:22:21] mmh jynus but is the revision.rev_text_id the one that corresponds to blobs_cluster25.blob_id? [15:25:14] I think revision stores a text_id, that stores a blob_id [15:25:20] (text) [15:25:27] I do not remember well [15:26:26] but if it is taking too much time, just ignore it, I was just suggesting that for your own research [15:26:44] ok, because the text table has only old_id, old_text, old_flags, so I ignored it [15:26:51] given the prefix :) [15:27:19] ok :-) [15:27:32] actually is useful to get the IDs around that timestamp, otherwise I can get only count(*) where id < FIXED ID [15:27:58] quite heavy queries and if they mismatch I still need to do the other approach to find it [15:28:07] at some point we should cut the "cluster" [15:28:18] and start writing to a different table [15:29:08] I would like at some point fit the current table on the "old" servers to have really 12TB free [15:33:11] goodbye blobs_cluster25, welcome blobs_cluster26 [15:35:52] FYI after 1h30m the new super powerful DBs have loaded only 130GB of buffer pool, and I don't think there is a variable to tell MySQL to use more resources for it to speed it up IIRC [15:36:39] is true that we could put it back in production in case of emergency with half buffer pool, but I would like a way to tell MySQL use all the resources to restore the buffer pool [15:37:17] are you saying that it has slowed down, or that it is terribly slow due to the size? [15:38:01] "I would like a way to tell MySQL use all the resources to restore the buffer pool" looks like a mysql feature request [15:38:16] ehehe I knew you would say that :D [15:38:20] I'll open it [15:38:23] I created another one [15:38:28] similar saying [15:38:49] "do not trash the buffer pool it mysql fails before loading it fully" [15:39:02] it happened to me that mysql fails to start [15:39:22] but with out config, if that happens, and the file has started to load, it is overwritten [15:39:36] which basically empties it [15:39:44] s/out/our/ [15:39:57] they are quicker than the previous ones, but have a huge buffer pool hence slow, uses only 25% CPU and 1.2% I/O wait [15:40:09] that's bad [15:40:10] yep, that is by design [15:40:16] single thread, I suppose [15:40:19] I guess [15:40:29] even 1/4 of a thread :D [15:40:38] it is thought for full production usage after all [15:40:53] yes and I agree, but give the option of a quick restore [15:41:14] but to be fair, with a 150GB buffer pool loaded, we probably put it at 100% load and it wouls work well [15:41:33] in fact, 5.7, by default only loads like the 25% or so [15:41:45] in your case was innodb_buffer_pool_dump_at_shutdown that triggered the overwrite I guess [15:41:50] yes [15:42:04] I mentioned on the bug that technically is my fault [15:42:04] yes they changed the default innodb_buffer_pool_dump_pct in MySQL, not in Maria [15:42:31] because we could disable while it is loading, but it is too sophisticated right now [15:42:39] we have worse problems [15:42:55] and having 400 GB buffer pool is not one of them :-) [15:43:21] but that is basically why I usually have 4-5 tickets at the same time [15:43:30] eheheh [15:44:37] this is interestinghttps://gerrit.wikimedia.org/r/#/c/193834/4 [15:45:03] we should drop the mariadb disk warning and set the generic one to 10-15% [15:45:36] make sense [15:45:58] and if you missed elukey email, now the puppet compiler should work for changes in the submodule too [15:46:06] yep, saw that [15:46:12] I am slowly catching up [15:46:23] didn't tested it yet, I'll try it at next change [15:48:48] I will need to test all replication paths, as they do not get shown on tendril, I do not trust my circular replications [15:49:22] not a huge deal, as datacenter will be independent, but better check than "oh, I forgot" [15:49:51] true but all but s2 will need to be done after the switchover [15:49:58] ? [15:50:08] what is different about s2? [15:50:24] we will replicate back to the designated master, that now is replicating from the maste [15:50:36] while s2 will not change master [15:50:47] yes, but I setup the circular replication already, just in case [15:50:53] I saw it [15:51:04] we can stop it, but I prefer to start with it enabled [15:51:05] but from the current masters [15:51:23] then stop it once we make sure we are not rolling back [15:51:26] yep [15:51:29] I though that one way to do it could be: [15:51:40] yes, I understand [15:51:41] before the switchover move all the slaves under the designated master [15:52:17] but you want to have a path to rollback without changing master [15:52:20] I prefer to compromise that, because I cannot trust 100% the 10 masters [15:52:25] yep [15:52:34] it should be only 1 minute [15:52:38] or a few [15:53:18] but I do not trust what would be a 11 server simultanoeus failover [16:21:19] I have a small errand to do, bbl to repool s3 DBs