[06:10:24] <arnaudb>	 got to admit that 10min~ to do a switchover from task creation to task resolve is quite cool <3
[06:28:07] <arnaudb>	 I think I triggered something close to the previous problematic situation on codfw/s3
[06:30:05] <arnaudb>	 not 100% sure though
[06:33:15] <arnaudb>	 I'll keep codfw/s3 the topology at an intermediate state (cc Amir1 jynus), lets talk about it a bit when you're around. db2190 move triggered the issue, I fixed it by disabling semi_sync on it (it was not moved under db2205), and doing the same thing on db2205 which is the candidate master
[06:36:55] <arnaudb>	 jynus: this time I backed up the diff: T374421 :p
[06:36:55] <stashbot>	 T374421: Switchover s3 master (db2209 -> db2205) - https://phabricator.wikimedia.org/T374421
[06:37:12] <arnaudb>	 was done yesterday but was not different this morning
[08:16:31] <jynus>	 rpl_semi_sync_master_ is off on the new one
[08:16:43] <jynus>	 it shouldn't be off, that probably caused issues
[08:17:15] <jynus>	 or, it should be off for clients too, and then enabled
[08:17:34] <arnaudb>	 ack, will disable for the replication clients then
[08:17:34] <jynus>	 is db2205 stable? is it replicating?
[08:17:40] <arnaudb>	 it is satble/replicating indeed
[08:17:57] <jynus>	 and hasn't been killed, right?
[08:18:05] <arnaudb>	 I've cloned it fresh yesterday, running properly since
[08:18:29] <jynus>	 ok, so the issue clearly is the semisync stuff
[08:18:59] <jynus>	 you should disable it on db2205 (as a client)
[08:18:59] <arnaudb>	 it looks like it indeed, but I wasn't feeling confident enough to handle it solo :D
[08:19:17] <jynus>	 rpl_semi_sync_slave_en <-- this should be off
[08:19:23] <jynus>	 on the new one
[08:19:40] <jynus>	 so the idea of checking is preciselly to make it equal
[08:19:41] <arnaudb>	 @@rpl_semi_sync_slave_enabled: 0
[08:19:53] <arnaudb>	 on both then?
[08:20:15] <jynus>	 in reality I think it should be off for dc replication
[08:20:30] <arnaudb>	 so both, indeed
[08:20:42] <jynus>	 intra-dc replication, but that is something to decide with manuel
[08:20:50] <arnaudb>	 ah
[08:20:55] <arnaudb>	 this I won't tinker with
[08:21:08] <arnaudb>	 I'm only disabling it when its causing trouble 
[08:21:12] <jynus>	 but I belive that, as a rule
[08:21:21] <jynus>	 the workflow should be this:
[08:21:36] <jynus>	 apply new puppet role -> restart
[08:21:41] <jynus>	 and then run the procedure
[08:21:50] <jynus>	 that way they should have a closer config
[08:22:00] <jynus>	 or if not restart, apply it manually,
[08:22:13] <arnaudb>	 I believe this is what happens already
[08:22:45] <arnaudb>	 I'll check how this case differs 
[08:22:48] <jynus>	 well, but something must be failing, either on puppet or on restart, because that option should be the same
[08:22:59] <arnaudb>	 yeah clearly
[08:23:26] <jynus>	 e.g. maybe for the lastest verion something is slightly different, etc
[08:23:38] <jynus>	 for now, disable it
[08:23:53] <jynus>	 enable it as a master so clients can use it
[08:24:22] <jynus>	 on the clients that it has already below, make sure semisync is working
[08:24:30] <jynus>	 move pending clients
[08:24:40] <jynus>	 and we should be ok, unless it explodes again
[08:24:53] <arnaudb>	 I'll check/try, thanks for the hint
[08:25:14] <jynus>	 I think it must be that
[08:25:27] <jynus>	 as it is show replica status what fails, not show master status
[08:26:15] <jynus>	 then we should open a task to test what is the trigger and document it/automate it to avoid it
[08:27:04] <arnaudb>	 will backlink the rest of this to it as well, indeed
[08:55:39] <arnaudb>	 this is new: arnaudb@cumin1002:dbtools $ sudo dbctl --scope codfw section s3 set-master db2205                                                                                                                                    
[08:55:39] <arnaudb>	 Execution FAILED
[08:55:39] <arnaudb>	 Reported errors:
[08:55:39] <arnaudb>	 Section s3 has no master
[08:56:43] <arnaudb>	 db2209 (former replication source) has the proper config and db2205 also in puppet
[08:57:21] <jynus>	 is it on dbctl?, what does dbctl say about db2205? is it pooled?
[08:58:29] <arnaudb>	 ah 
[08:58:32] <arnaudb>	 it wasn't pooled
[08:59:20] <jynus>	 check also heartbeat
[08:59:35] <jynus>	 orchestrator is saying there is 4 minutes of lag on all of codfw
[08:59:43] <jynus>	 maybe it is an orch glitch
[08:59:48] <arnaudb>	 no no its normal
[08:59:53] <arnaudb>	 https://phabricator.wikimedia.org/T374421
[08:59:54] <jynus>	 ok
[09:00:03] <arnaudb>	 I'll get to the pt-heartbeat part soon 
[09:07:29] <jynus>	 s8 backup took 30 hours
[09:07:41] <jynus>	 very weird behaviour
[09:34:22] <arnaudb>	 we have this new alert set that is being merged: https://gerrit.wikimedia.org/r/c/operations/alerts/+/1053689
[09:35:48] <jynus>	 remember to move db2139 when s3 backups finish
[09:39:48] * arnaudb creates a task to avoid forgetting
[09:39:50] <arnaudb>	 thanks for the reminder
[10:51:36] <Amir1>	 I'm booting up
[11:18:12] <jynus>	 db1171 has been lagging for a few days now
[11:21:00] <jynus>	 s8 needs a larger buffer pool
[11:21:23] <jynus>	 will update the config and restart it when dump finishes
[11:31:07] <jynus>	 Amir1: will merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1072515 ok?
[11:31:29] <Amir1>	 one thing
[11:31:50] <jynus>	 let me know
[11:32:35] <Amir1>	 yeah, it's slowly recovering, it was because of the schema change 
[11:32:49] <jynus>	 oh, was there a schema change?
[11:32:57] <Amir1>	 yeah
[11:33:00] <jynus>	 then that would explain it
[11:33:02] <Amir1>	 https://phabricator.wikimedia.org/T367856
[11:33:07] <Amir1>	 a massive one
[11:33:11] <jynus>	 still I got a warning for "low buffer pool"
[11:33:14] <Amir1>	 the alter takes 48 hours
[11:33:18] <jynus>	 and given we have the resources
[11:33:35] <jynus>	 I want to add those, s8 needs way more than s7
[11:33:49] <jynus>	 there was a lot of unalocated memory on that host
[11:33:56] <Amir1>	 oh sure thing
[11:34:12] <jynus>	 can I copy your comment to the ticket
[11:34:30] <jynus>	 as that is the main explanation (I was worried there was somethign wrong with the host)
[11:34:46] <Amir1>	 I think I commented there already :D
[11:34:47] <jynus>	 but the patch I wanted to do it regardless of the explanation
[11:34:49] <jynus>	 thanks!
[11:35:01] <jynus>	 as it was a mitigation that would work no matter what
[11:35:28] <jynus>	 right now s8 dbs hardly fit on multiinstance hosts
[11:37:04] <jynus>	 So now I understand the need to remove those from mw core
[11:38:16] <jynus>	 the restart will also help after the poisoning of the b.p. by the schema change
[11:41:51] <Amir1>	 I'm working on getting s8 to a better shape https://www.wikidata.org/wiki/User:ASarabadani_(WMF)/Growth_of_databases_of_Wikidata
[11:42:02] <jynus>	 nah, no worries
[11:42:10] <jynus>	 we work with what we have
[11:44:25] <jynus>	 I will restart now db1171:s7, and s8 on the same host in ~1h, when the dumps there finish
[11:44:36] <jynus>	 to apply the buffer pool change
[11:45:22] <Amir1>	 sounds good to me
[12:47:10] <Amir1>	 I called dibs on the old s3 codfw master
[12:53:59] <arnaudb>	 ack, cancelling the --check, lmk when I can grab it :)
[13:09:18] <Amir1>	 sigh, nah replication to its child is broken
[13:10:05] <Amir1>	 backup source. I fix it
[13:18:47] <Amir1>	 arnaudb: I suggest not doing s3 old master, first db2139:3313 should move out
[13:19:13] <Amir1>	 (let me fix the mess I made because of it). It'll finish soon
[13:19:35] <arnaudb>	 ack I'll let s3 aside until your greenlight
[13:45:28] <Amir1>	 db2139:3313 is normal now but we shouldn't touch db2209 until db2139:3313 is moved out of db2209
[13:46:25] <arnaudb>	 I'll move it behind db2205
[13:57:39] <jynus>	 thanks
[14:08:18] <arnaudb>	 done Amir1 !
[14:08:30] <Amir1>	 Thanks!
[14:09:23] <jynus>	 they are doing the alert migration atm
[14:09:48] <jynus>	 so consider slowing down maintenance until they finish, unsure what the impact to observability could be
[14:10:57] <arnaudb>	 ack, thanks for the reminder
[15:04:38] <jynus>	 Last snapshot for s8 at eqiad (db1171) taken on 2024-09-12 13:13:24 is 1467 GiB, but the previous one was 1546 GiB, a change of -5.1 %
[15:42:47] <topranks>	 jynus: hey just catching up with your comments on task 
[15:43:04] <topranks>	 the request is for us to do ms-backup2002 and backup2011 asap is it?
[15:43:17] <topranks>	 that should be fine, we can possibly even do them now before the official start 
[15:43:18] <jynus>	 not in a rush
[15:43:24] <topranks>	 ok
[15:43:49] <jynus>	 just, dont keep me until late, so I can pool them back and finish my day :-D
[15:44:21] <topranks>	 urandom: are you able to depool ms-fe2012, moss-fe2002 & thanos-fe2003?
[15:45:47] <urandom>	 yup
[15:45:54] <topranks>	 super, thanks !
[15:46:37] <urandom>	 {{done}}
[16:00:19] <topranks>	 thanks :) 
[16:00:53] <topranks>	 jynus: ms-backup2002 and backup2011 have been moved, so you can start the backups on them again 
[16:00:57] <topranks>	 thaks for the help!
[16:05:01] <jynus>	 topranks: thanks to you for accomodating! So great!
[16:05:23] <jynus>	 about to restart backups 
[16:18:02] <topranks>	 urandom, arnaudb: rest of those are done if you want to repool things 
[16:18:04] <topranks>	 thanks :) 
[16:18:11] <arnaudb>	 🎉
[16:18:14] <arnaudb>	 thanks topranks 
[16:19:06] <urandom>	 topranks: awesome; thanks!
[16:20:01] <topranks>	 arnaudb: actually I forgot db2209
[16:20:07] <topranks>	 can we still do it?
[16:20:12] <arnaudb>	 next round?
[16:20:19] <arnaudb>	 I was on my way out :D
[16:20:28] <topranks>	 sure yeah let's do it Tuesday 
[16:20:38] <jynus>	 topranks: he had repooled it 
[16:20:45] <jynus>	 depool could take some time
[16:20:48] <topranks>	 all good it was my mistake 
[16:20:51] <topranks>	 yeah there is no problem 
[16:20:53] <topranks>	 thanks guys 
[22:11:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: systemd-timedated.service on backup2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[22:31:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: systemd-timedated.service on backup2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed