[08:45:05] <dhinus>	 we have an issue with tools-proxy-9 apparently, many tools are unresponsive
[08:51:43] <dhinus>	 I'm declaring an incident
[08:53:07] <dhinus>	 incident doc: https://docs.google.com/document/d/1CLY_iZyXDTyJEl4fKYeU1aRSNsheO9-TZcjyW9wFyEk/edit?tab=t.0
[08:53:28] <dhinus>	 dcaro if you're around you might be quicker than me at identifying the problem
[08:56:09] <dhinus>	 it looks like things are working now
[09:22:12] <dhinus>	 the incident is resolved. follow-up investigation at T399261
[09:22:14] <stashbot>	 T399261: Widespread instances down in project deployment-prep - https://phabricator.wikimedia.org/T399261
[09:22:28] <dhinus>	 wrong task :D
[09:22:35] <dhinus>	 correct task: T399281
[09:22:35] <stashbot>	 T399281: 2025-07-11 Toolforge tools not responding, proxy issue - https://phabricator.wikimedia.org/T399281
[10:08:32] <dhinus>	 there are some alerts in cloudcontrol1006: designate_floating_ip_ptr_records_updater.service is in failed status
[10:08:45] <dhinus>	 the logs shows "designateclient.exceptions.Unknown: timeout"
[10:14:00] <dhinus>	 I think there is some ongoing Ceph instability, the dashboard at https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health shows 2 OSDs out, 8 OSDs down
[10:22:11] <dhinus>	 and now "CephSlowOps: Ceph cluster in eqiad has 847 slow ops"
[10:23:12] <dhinus>	 toolforge is down again
[10:24:40] <dhinus>	 and back up
[10:27:44] <dhinus>	 and down again
[10:27:44] <dhinus>	 RESOLVED: CephSlowOp
[10:33:06] <dhinus>	 things look better again but Ceph is still in HEALTH_WARN
[10:33:23] <dhinus>	 I pasted the output of ceph health detail to T399281
[10:33:23] <stashbot>	 T399281: 2025-07-11 Toolforge tools not responding - https://phabricator.wikimedia.org/T399281
[10:38:09] <dhinus>	 there was another spike of Cloud VPS vms reporting down, but things are now back to normal
[10:39:08] <dhinus>	 I'm not understanding what is the root cause of the Ceph instability
[10:39:35] * dhinus lunch, back later
[10:51:26] <dhinus>	 topranks: can you check if you can see anything network-wise that could explain some Ceph instability since this morning?
[10:51:28] <dhinus>	 T399281
[10:51:29] <stashbot>	 T399281: 2025-07-11 Toolforge tools not responding - https://phabricator.wikimedia.org/T399281
[10:51:57] <dhinus>	 I'm especially puzzled by the SSH alerts on multiple cloudcephosd hosts
[10:52:10] * topranks looking 
[10:52:14] * dhinus lunch, for real this time :)
[10:55:05] <topranks>	 checking a basic graph I have based on the cloud host ping checks from previous I see a few issues for cloudceph nodes only 
[10:55:08] <topranks>	 https://grafana.wikimedia.org/goto/cSmIOesNg
[10:55:22] <topranks>	 which may be expected, but no general thing is jumping out 
[10:55:28] <topranks>	 I'll check a few more things 
[10:55:32] <topranks>	 it is still ongoing I take it?
[10:57:07] <dhinus>	 kind of 
[10:57:12] <dhinus>	 not right now
[10:57:22] <dhinus>	 but it's been going up and down a few times
[10:58:21] <dhinus>	 I pasted some graphs in the task
[10:58:50] <dhinus>	 a.ndrew was also doing some upgrades on those hosts yesterday so it might be related (or not)
[11:14:11] <topranks>	 yeah ok 
[11:14:22] <topranks>	 nothing jumping out in the graphs or logs that would suggest a network problem 
[11:14:42] <topranks>	 only thing in the graphs that looks different to normal is all the cloudrabbit* nodes are less spikey since about 8am this morning 
[11:14:45] <topranks>	 https://grafana.wikimedia.org/goto/-qYOFesNg
[11:15:04] <topranks>	 could be totally unrelated still all of them seeing decent level of traffic 
[11:17:53] <andrewbogott>	 I'm still half asleep and haven't read the backscroll, but my emails suggest that ceph pacific + bookworm + ceph traffic is a bad combination.
[11:18:34] <andrewbogott>	 So probably the fix for this is for me to downgrade those hosts back to bullseye. I can start doing that shortly, unless someone has a different diagnosis
[11:19:02] <andrewbogott>	 dhinus:  ^
[11:20:19] <andrewbogott>	 I don't think we can just depool the affected nodes pre-emptively without causing a whole lot of rebalancing which might make things worse
[11:23:14] <andrewbogott>	 The bookworm hosts are 1006-1008,1035-1037 <- does that correspond to the failures?
[11:43:06] <dhinus>	 andrewbogott: 1035-1037 seem to match
[11:43:36] <andrewbogott>	 yeah. I thought it was specifically the md1 rebuild but 1036 seems to have flapped many times during the night...
[11:43:46] <andrewbogott>	 and I only see log messages about md1 once.
[11:44:17] <andrewbogott>	 I'm already in process of rolling 1037 back to bullseye; do you want to help me dig in the logs on 1035/1036 and see what you can see?
[11:45:33] <andrewbogott>	 Oh, one other data point: I upgraded ssd fw on 1035-1037 but did /not/ upgrade firmware on 1006-1008 (although I did upgrade those three to bookworm as well).
[11:45:48] <andrewbogott>	 but I also upgraded ssd fw on 1038-1041
[11:46:09] <andrewbogott>	 So pretty sure that still points to bookworm unless you've seen issues on 1038-1041
[11:49:50] <andrewbogott>	 dhinus: on cloudcephosd1036, look at time stamp '2025-07-11T10:27:42.396143+00:00' and below. That's one of the storms beginning.
[11:50:07] <andrewbogott>	 It /looks/ like a network issue since there are kafka complaints at the same time as ceph complaints...
[11:52:26] <dhinus>	 I'm quite confused, they look like network issues to me, but the network could also be a consequence and not the root cause
[11:52:34] <andrewbogott>	 yeah
[11:52:57] <dhinus>	 good catch about md1, did not see that
[11:53:09] <andrewbogott>	 If it is a 'network' issue it would still have to be caused by system misbehavior, not literally the network
[11:53:25] <andrewbogott>	 but of course if a different OSD is having trouble that might look like timeouts on any other OSD
[11:57:06] <dhinus>	 thanks topranks btw for looking into this! I think there's nothing pointing to the network being the root cause so far
[11:57:12] * andrewbogott agrees
[11:58:26] <topranks>	 ok np guys 
[11:58:56] <dhinus>	 I think downgrading 1035-1037 is probably a reasonable thing to try. wdyt?
[11:59:12] <topranks>	 I posted an update on the task with some details of what I could see 
[12:00:38] <andrewbogott>	 I definitely think downgrading is the next step. Then potentially we can go to the next version of ceph on bullseye before doing OS upgrades so we avoid this exact combination.
[12:00:46] <andrewbogott>	 But it would be nice to actually know what's wrong :/
[12:02:40] <andrewbogott>	 The sw raid rebuild corresponds to some but not all of the lockups...
[12:02:47] <andrewbogott>	 it's hard for me to blame the raid thing on ceph
[12:03:39] <andrewbogott>	 but why does 1036 keep flapping then? It should already be done with the raid rebuild and it doesn't seem to be repeating that
[12:14:35] <andrewbogott>	 btw the reimage is going badly because the partman phase fails. Typically I'd get around that by reformatting the OS drives so it has to rebuild the raid but I'm not yet sure how to do that with the controller on this generation of hosts...
[12:31:32] <andrewbogott>	 bah, is there /any/ partition or raid management tool on the debian installer CLI?  Do I really have to write a custom partman script?
[12:36:33] <dhinus>	 no idea :( I'm on my phone atm, I can have a look in 30 mins, but I have limited knowledge of the partitioning and raid setup
[12:39:07] <dhinus>	 I see that toolsdb is also alerting now, don't worry about it I'll fix it later
[12:39:23] <andrewbogott>	 ok, thanks
[12:39:26] <andrewbogott>	 are you off today, officially?
[13:04:57] <dhinus>	 andrewbogott: no I'm working, and I'm back at my laptop now :)
[13:05:47] <andrewbogott>	 ok! I'm enlisting Ben to help with the partman issue above
[13:05:56] <dhinus>	 +1
[13:06:04] <dhinus>	 I'll look at toolsdb
[13:06:08] <andrewbogott>	 And should probably step away for a few and eat something so my brain doesn't seize up
[13:11:11] <dhinus>	 please do! things seem relatively stable right now
[13:11:20] <dhinus>	 I've just restarted toolsdb replication and I think it's working
[13:14:50] <dhinus>	 toolsdb replication is back in sync
[14:20:17] <andrewbogott>	 dhinus: can you log in to /any/ cloud-vps hosts right now?
[14:20:23] <dhinus>	 andrewbogott: let me try
[14:20:28] <dhinus>	 there's a report in Slack about "I was trying to create an object storage container and am getting upstream request timeout errors."
[14:20:52] <dhinus>	 hmm I can't ssh to the tools-db host I used one hour ago
[14:20:58] <dhinus>	 so I guess it's a "no"
[14:21:43] <andrewbogott>	 Maybe the bastion is busted? I'll restart it
[14:21:49] <andrewbogott>	 https://openstack-browser.toolforge.org/ works, so this isn't a total network outage
[14:22:24] <dhinus>	 I'm reopening the incident
[14:22:35] <andrewbogott>	 no idea if this is related
[14:24:44] <andrewbogott>	 I rebooted bastion-r and
[14:24:46] <andrewbogott>	 [/sbin/fsck.ext4 (1) -- /dev/sda1] fsck.ext4 -a -C0 /dev/sda1 
[14:24:46] <andrewbogott>	 /dev/sda1: recovering journal
[14:24:57] <andrewbogott>	 so something bad has happened filesystemwise.
[14:25:10] <andrewbogott>	 things seem able to recover, but... this is not consistent with how I would expect ceph to fail.
[14:25:22] <andrewbogott>	 Unless it just caused a hard crash on vms that messed up journaling
[14:25:30] <andrewbogott>	 there wasn't ever a ceph message implying data loss was there?
[14:27:36] <andrewbogott>	 dhinus: my thought is... get bastionr working so that cumin works so that I can do some cloud-wide tests to see what VMs are locked up by surprise
[14:29:13] <dhinus>	 didn't see anything implying data loss so far
[14:29:23] <dhinus>	 I'm updating the incident doc timeline at https://docs.google.com/document/d/1CLY_iZyXDTyJEl4fKYeU1aRSNsheO9-TZcjyW9wFyEk/edit?tab=t.0
[14:29:37] <dhinus>	 things seem to get worse after 14:12 UTC
[14:30:19] <andrewbogott>	 1007 is having a thing now.
[14:30:31] <andrewbogott>	 So we have two nodes down at once, which is probably what causes actual user-facing effects.
[14:42:34] <andrewbogott>	 dhinus: cumin works on VMs again, can you do some searching and rebooting of frozen toolforge nodes?
[14:43:20] <dhinus>	 you mean the usual "D state" frozen?
[14:43:29] <dhinus>	 or other types of frozen?
[14:43:57] <andrewbogott>	 probably they'll just appear as down in alertmanager
[14:44:08] <andrewbogott>	 or you can see which ones time out for cumin. They're, like, all the way down. 
[14:44:18] <andrewbogott>	 at least, there are lots on deployment-prep in that state
[14:44:54] <andrewbogott>	 maybe they'll recover on their own but that's not obvious to me if they are
[14:47:17] <dhinus>	 ok
[14:47:53] <andrewbogott>	 thx
[14:52:55] <andrewbogott>	 are you seeing many, many unreachable hosts?
[14:53:24] <dhinus>	 30% in project tools
[14:53:43] <dhinus>	 there's also this nice graph https://grafana.wmcloud.org/d/bfcWngjVk/taavi-cloud-vps-issue-detection-tests
[14:54:57] <andrewbogott>	 I wish I understood better what's happening with those down nodes. Ceph should've just been a little slow during the flaps, right?
[14:55:03] <andrewbogott>	 It should make a VM lock up
[14:55:11] <andrewbogott>	 *shouldn't*
[14:55:32] <dhinus>	 this morning they always self-recovered, but now they're not
[14:55:39] <dhinus>	 probably because we had 2 ceph nodes down together
[14:55:46] <andrewbogott>	 yeah
[14:56:03] <andrewbogott>	 once we get 1036 back healthy I'm pretty sure this particular case won't happen again
[14:56:46] <andrewbogott>	 dhinus: a useful thing would be a way to enumerate VM IDs from the list of unresponsive hosts.
[14:56:50] <dhinus>	 cumin is not very usable or maybe I'm not using the right flags
[14:57:07] <andrewbogott>	 I can't think of a /good/ way to do that, but it's possible with a lot of searches
[14:58:44] <andrewbogott>	 are you having any luck getting those stuck hosts unstuck?
[14:58:59] <andrewbogott>	 dhinus: I'm doing things like "sudo cumin O{project:deployment-prep} 'true' --timeout 10 --force "
[14:59:13] <andrewbogott>	 and figuring out that the timeout VMs are stuck
[15:01:30] <dhinus>	 my cumin command is completely stuck so I don't even have a list of hosts
[15:01:52] <dhinus>	 I'm also doing IC and trying to update the timeline so I have limited bandwidth :)
[15:02:51] <andrewbogott>	 I'm running on cloudcumin1001 and it's working reliably
[15:03:03] <dhinus>	 what command are you running?
[15:03:19] <andrewbogott>	 sudo cumin O{project:deployment-prep} 'true' --timeout 10 --force 
[15:03:20] <andrewbogott>	 just now
[15:03:25] <andrewbogott>	 seems to be working
[15:03:41] <dhinus>	 cloudcumin itself work, but my attempt of "cumin 'O{project:tools}' id" was never returning
[15:03:50] <dhinus>	 of course I was missing "--timeout 10" :)
[15:04:41] <andrewbogott>	 yeah, otherwise the stuck VMs make you wait forever
[15:05:23] <dhinus>	 then what should we do with the ones that fail?
[15:06:30] <andrewbogott>	 I'm not sure. The first few I tried came back after a hard reboot...
[15:44:09] <andrewbogott>	 dhinus: do you have a google doc or just the phab task?
[15:44:18] <dhinus>	 incident doc linked from the task
[15:44:40] <dhinus>	 found it?
[15:50:36] <andrewbogott>	 yep
[15:51:32] <dhinus>	 I'm officially handing off IC to you then :)
[15:52:43] <dhinus>	 if you have a handy list of bookworm-upgraded hosts, I would add it to the top of the doc
[15:54:11] <andrewbogott>	 ok
[15:54:28] <andrewbogott>	 If I'm both repairing and IC then I will lean on the repairing bit
[15:54:43] <dhinus>	 makes sense
[16:23:40] * dhinus offline
[17:12:55] <andrewbogott>	 I am still reimaging things but I believe the incident to be over and the current Ceph state should prevent remaining issues from being visible to users.
[17:14:37] <andrewbogott>	 perfect timing, stashbot
[18:47:05] <andrewbogott>	 I am now going to try to leave ceph alone. Everything is back to Bullseye and ceph is doing a bit of rebalancing but seems otherwise happy. The volume that vanished from cloudcephosd1013 is concerning but seems unrelated to the rest of what's happening so it can be investigated later.