[06:00:34] <arnaudb>	 looks like s5 went allright 
[07:27:28] <volans>	 good morning, I see that the switchdc prepare cookbook was run tonight and AFAICT it went ok beside T375144 Do you have any feedback or things you would like modified?
[07:27:29] <stashbot>	 T375144: ROW-based replicas broke with cleaned up heartbeat tables after setting up circular replication - https://phabricator.wikimedia.org/T375144
[09:00:22] <jynus>	 volans: I have 2 immediate requests that should be easy
[09:00:46] <volans>	 sure, either fire away or open task, whatever is easier
[09:01:09] <jynus>	 can the script/is it reasonable to log per section on phabricator?
[09:01:21] <jynus>	 it would be typical that we do 1 section, then we check
[09:01:26] <volans>	 sure, I thought you migth ask for it :)
[09:01:33] <jynus>	 and it would be useful to see it on phab too
[09:01:40] <jynus>	 the other is the order
[09:01:47] <jynus>	 I think it does s1 s2 s3
[09:01:47] <volans>	 inverted?
[09:02:10] <jynus>	 not sure if inverted, as es or s8 are I would say second tier
[09:02:10] <volans>	 it does in this order 
[09:02:11] <volans>	 https://doc.wikimedia.org/spicerack/master/api/spicerack.mysql_legacy.html#spicerack.mysql_legacy.CORE_SECTIONS
[09:02:39] <jynus>	 does it keep the configured order?
[09:02:56] <jynus>	 I guess so
[09:03:01] <volans>	 yes, it goes through the list, so if I invert it it would got es7, es6....
[09:04:01] <jynus>	 I would do something like s6 s5 s2 s7 s3 s8 s4 s1
[09:04:51] <jynus>	 let me get the list from the order that mysql upgrades happen
[09:05:07] <jynus>	 the reasoning is: "how hard it is to recover from backups if something wen't wrong"
[09:05:14] <volans>	 makes sense
[09:05:19] <jynus>	 and with s1 being between 1/3 and 1/2 of the visits
[09:05:24] <jynus>	 it is very impacting
[09:05:25] <volans>	 if we think is the best way we can change the constant directly in spicerack
[09:05:40] <volans>	 so anything that loops through sections will get them in the safest order
[09:05:55] <jynus>	 that's ok, if you can do like a sample patch to where it is
[09:06:04] <jynus>	 and then we can agree the right order
[09:06:43] <jynus>	 with the dbas
[09:06:51] <jynus>	 those are more usability patches
[09:06:58] <jynus>	 but I think they will want it
[09:07:18] <jynus>	 the important one is to mitigate the ROW replication issues
[09:07:35] <jynus>	 T375144
[09:07:36] <stashbot>	 T375144: ROW-based replicas broke with cleaned up heartbeat tables after setting up circular replication - https://phabricator.wikimedia.org/T375144
[09:08:21] <jynus>	 (note the other also applies to the switchover, if something breaks, we prefer it breaking on s6 first, as it is trivial to fix)
[09:08:28] <jynus>	 *order
[09:10:30] <volans>	 jynus: starting point for discussion https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1074111
[09:10:47] <jynus>	 volans: thanks
[09:11:19] <jynus>	 that's exactly what I wanted, the order to be discussed
[09:11:56] <jynus>	 we may actually in the future use something else to read those
[09:12:09] <jynus>	 as we have a valid_sections global puppet config
[09:13:03] <jynus>	 and I am sure the logging will take some more time, but it shouldn't be too hard (?)
[09:13:14] <volans>	 nah, few minutes
[09:13:23] <jynus>	 the ROW will be more complex because I am not sure how to fix it logically
[09:13:51] <jynus>	 as what failed there wasn't a script logic, as I understood it
[09:13:54] <jynus>	 but our practices
[09:15:23] <jynus>	 the issue coming from ROW not having a replace command
[09:15:39] <jynus>	 so it translates it to either insert or update row
[09:16:01] <volans>	 yeah
[09:16:10] <jynus>	 most likely update
[09:16:26] <jynus>	 but the row doesn't exist on the other replica set
[09:16:50] <jynus>	 so either we keep it clean and either delete from the original replica set or insert a fake one on the primary
[09:17:13] <jynus>	 or we keep row ones uncleaned all the time
[09:18:17] <jynus>	 the logic is not too crazy "if secondary master uses row and primary master doesn't have a row, insert it" but it doesn't feel right
[09:19:37] <jynus>	 arnaudb: I have a few asks to check before switchover too, regarding checking codfw to be prepared
[09:20:56] <jynus>	 volans: I don't know if you have any immediate thought about the row stuff, I would like to know if I am missing something obvious
[09:21:06] <arnaudb>	 sure
[09:25:09] <volans>	 jynus: do you have handy the query that broke the replication? I want to make sure I fully understand the problem
[09:25:14] <jynus>	 volans: there is one last thing
[09:25:29] <volans>	 I need to rewrite it in rust? :D
[09:25:36] <jynus>	 I think the automation should I have caught the issue
[09:25:46] <jynus>	 early
[09:26:13] <jynus>	 can you see if the automation checks that the primary master (eqiad for us) was replication yes/yes ?
[09:26:19] <jynus>	 maybe we missed that check
[09:26:24] <jynus>	 or it happened too early
[09:26:36] <arnaudb>	 volans: we're shifting to cobol
[09:26:38] <jynus>	 but ideally it should have caught there was an error
[09:27:10] <jynus>	 volans: I am filling in T375144
[09:27:11] <stashbot>	 T375144: ROW-based replicas broke with cleaned up heartbeat tables after setting up circular replication - https://phabricator.wikimedia.org/T375144
[09:27:12] <volans>	 what do you mean by "replication yes/yes" ?
[09:27:29] <jynus>	 replication sql running: Yes replication io running: Yes
[09:27:39] <jynus>	 the wording may be slightly different
[09:27:42] <jynus>	 in show slave status
[09:27:48] <volans>	 this before starting?
[09:27:53] <jynus>	 nope, after setup
[09:28:07] <jynus>	 I wonder if the 1 second wait was enough
[09:28:13] <jynus>	 as heartbeat happens every second
[09:28:22] <jynus>	 so it may have passed 1+ second since start
[09:28:31] <jynus>	 or we just didn't check that
[09:29:05] <volans>	 we could check that for N times with a small sleep
[09:29:32] <jynus>	 please check how it is the logic now
[09:29:33] <volans>	 after slave start we check that the values are the ones in this dict:
[09:29:36] <volans>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/refs/heads/master/cookbooks/sre/switchdc/databases/prepare.py#242
[09:29:44] <volans>	 see the 'expected' variable
[09:29:56] <volans>	 but that's for master_to
[09:30:00] <jynus>	 yeah, so it likely failed in 1+ second
[09:30:06] <jynus>	 se it wen't under the radar
[09:30:10] <volans>	 no we chefck only the target
[09:30:14] <volans>	 so codfw
[09:30:15] <jynus>	 or
[09:30:19] <jynus>	 it happened much later
[09:30:25] <jynus>	 after the start of heartbeat
[09:30:31] <jynus>	 so we may just need a later check
[09:30:33] <jynus>	 at the end
[09:30:46] <jynus>	 after pt-heartbeat-wikimedia starts at the end
[09:30:47] <volans>	 no no it checks master_to only, not master_from, you're right
[09:30:54] <volans>	 so I just add a check for that too
[09:30:56] <jynus>	 I see
[09:31:06] <jynus>	 so one last check at the end should be ok
[09:31:12] <volans>	 adding it
[09:31:27] <jynus>	 as we worried a lot about the secondary
[09:31:35] <jynus>	 because the primary didn't broke much
[09:31:41] <jynus>	 but it would have alerted us earlier
[09:31:47] <jynus>	 so that is for detection
[09:32:19] <volans>	 yeah, sorry about that, I don't think it was explicitely mentioned in the doc, but makes total sense to check that too
[09:32:41] <jynus>	 that's ok, if we knew it was going to fail, I had said it!
[09:32:53] <jynus>	 this was a surprise for everybody!
[09:33:54] <jynus>	 just make sure there is 1+ second between pt-heartbeat and the last check
[09:34:06] <volans>	 k
[09:34:15] <jynus>	 which I think it happens anywhay because of the start slave on the secondary
[09:34:45] <jynus>	 your check was there
[09:35:00] <jynus>	 but it failed on the following step, so we need to check again
[09:43:17] <jynus>	 volans: I have the timing- change master happened at 2024-09-18 21:27:14 
[09:43:33] <jynus>	 and error at 2024-09-18 21:27:17
[09:44:10] <jynus>	 so the original check wouldn't catch it
[09:44:30] <volans>	 should we add a sleep before the last checks?
[09:44:38] <volans>	 or repeat them N times
[09:44:58] <jynus>	 nah, we just need a last check after the last start and check
[09:45:10] <volans>	 ok
[09:45:24] <jynus>	 as it depends on when the other do their things
[09:55:22] <jynus>	 I filled in T375144 with all information
[09:55:23] <stashbot>	 T375144: ROW-based replicas broke with cleaned up heartbeat tables after setting up circular replication - https://phabricator.wikimedia.org/T375144
[09:55:32] <jynus>	 if you want to reference it for patches
[09:55:37] <arnaudb>	 <3
[09:56:11] <jynus>	 arnaudb: quick 5 1:1 to get you up to date and then we work individually?
[09:56:16] <jynus>	 5min
[09:56:39] <arnaudb>	 sure, I'll send you an url!
[09:57:45] <arnaudb>	 url sent :) hop-in whenever
[10:05:34] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on ms-be1058:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[10:20:54] <Emperor>	 That's still T372207 ; I'll extend the silence until after my vacation; we were expecting replacement hardware this Q, but it seems to have got delayed rather
[10:20:55] <stashbot>	 T372207: Disk (sdc) failed on ms-be1058 - https://phabricator.wikimedia.org/T372207
[10:25:30] <volans>	 patches sent, I've added both of you
[10:26:03] <volans>	 and for the sections order also am.ir and ma.nuel so anyone can chime in
[12:00:57] <jynus>	 volans: please be patient, as testing may take some time and we have more important tasks atm blocking the switchover + incident handling
[12:01:03] <jynus>	 thanks for the work, volans
[12:01:17] <jynus>	 it is very appreciated you are so open to improvements
[12:04:23] <volans>	 sure, no worries, just wanted to get them out before I forget the details :)
[13:02:21] <btullis>	 I have a patch to add a new fileset to bacula, if anyone would like to review it: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1074156
[13:03:58] <btullis>	 Overall size to bacula shouldn't change much, as we will be migrating postgresql databases from an-db1001 to this new system.
[13:10:12] <jynus>	 thanks for checking, normally I just +1, but it is nice to be aware for capacity planning
[13:11:46] <jynus>	 in this case I actually have some comments that may improve stuff
[13:12:00] <btullis>	 Great! All input welcome.
[13:16:19] <jynus>	 you have my +1, but I commented something to try to setup for the future
[13:17:28] <jynus>	 but you won't see me blocking adding more backups :-D
[13:36:35] <btullis>	 Cheers.
[14:22:56] <jynus>	 at your will, feel free to resolve T371351 and the improvement can go to the post-incident ticket or elsewhere.
[14:22:57] <stashbot>	 T371351: Automate the pre/post switchover tasks related to databases - https://phabricator.wikimedia.org/T371351
[15:02:19] <jynus>	 ^that was for volans
[15:15:28] <volans>	 sure thx
[15:18:40] <jynus>	 btullis: db1208 is warning that there is no backups, is it ready for an initial backup run already?
[16:20:54] <btullis>	 jynus: yes, but there will probably just be a single zero length file. 
[16:22:01] <jynus>	 ok, then I will let it run on it own tonight
[16:22:31] <jynus>	 (just FYI, it may alert if it gets 0 bytes backups)
[16:26:24] <btullis>	 OK, I may be able to get some content there before tonight's run.   Thanks. 
[16:41:35] <jynus>	 have a nice day, see you on monday
[16:43:06] <volans>	 o/
[16:45:39] <urandom>	 Empero.r: are you aware of ms-be1058?  I went to look at the puppet failure (as seen in my inbox), and the root cause seems to be a disk failure (sdc).  But the disk failure seems to be from Aug 10, and the puppet failure recent.
[16:45:47] <urandom>	 I didn't find an open ticket.
[17:12:31] <Emperor>	 urandom: see scroll, where I silenced the alert earlier. The DC team closed out the ticket for the dead disk (I think they have objectives around closing tickets); there are no spares and the system is due to be retired (was going to be this Q, but that looks less likely now).
[18:31:58] <urandom>	 auh, the silence expired...
[18:32:44] <urandom>	 I was most concerned with why we'd only just get the notification