[07:01:27] <arturo>	 I just got paged by ceph
[07:01:56] <arturo>	 I'm on a doctor appointment and dont have the laptop with me
[07:03:51] <arturo>	 dhinus: are you able to take a look?
[07:10:45] * dcaro paged
[07:10:51] <dcaro>	 it just went away
[07:10:53] <dcaro>	 looking
[07:10:56] <dhinus>	 I was also paged
[07:13:34] <dcaro>	 I don't see anything hapenning right now
[07:13:47] <dcaro>	 there was an alert for 1 slow ops, and ceph unknown state
[07:13:56] <arturo>	 a ticket about slow ops was just created
[07:15:58] <dcaro>	 the metric for ceph unknown status `(ceph_health_status{job="ceph_eqiad"} or on () vector(-1)) == -1` does not show anything to me :/, feels like the one we had for the openstack apis
[07:16:19] <dcaro>	 as if we were getting different data different times
[07:16:49] <dcaro>	 and that one is also triggering right now xd
[07:16:50] <dcaro>	 Got no data from the Openstack API to check the response times. It may mean that the control plane is unreliable.
[07:17:06] <dcaro>	 may be worth it to ping o11y about it
[07:17:13] <dcaro>	 see if they know what's what
[07:17:42] <arturo>	 do you think they are related?
[07:18:05] <arturo>	 are cloudcontrol servers involved in the ceph metric scrapes?
[07:18:09] <dcaro>	 both are related to no data being found, but both show some data when queried (and show gaps sometimes when queried too)
[07:18:21] <arturo>	 like, is the ceph metric exporter running on cloudcontrol?
[07:18:37] <arturo>	 ok
[07:19:02] <dcaro>	 no, they get scraped by prometheus directly, but I'm thinking that it might be that we are hitting different thanos/prometheus or similar, and they have different data for some reason
[07:19:12] <dcaro>	 or something like that
[07:20:43] <arturo>	 ah, I see
[07:22:31] <dcaro>	 that happened in the past on toolforge iirc
[07:26:15] * dcaro off
[07:26:28] <dcaro>	 I'll let you handle this during regular work hours :)
[07:27:29] <arturo>	 sure
[07:28:03] <arturo>	 I'm hoping to be back in the laptop in max 2h from now
[07:29:00] <arturo>	 thanks for showing up in your PTO day!
[08:58:21] * arturo in the laptop now
[09:36:05] <arturo>	 I have created this ticket: T374599
[09:36:05] <stashbot>	 T374599: cloud: prometheus: investigate weirdness with metrics and alertmanager - https://phabricator.wikimedia.org/T374599
[10:07:52] * dhinus paged ToolsToolsDBWritable
[10:08:44] * dcaro paged too
[10:08:46] <dhinus>	 tools-db-1 is running, but read-only
[10:08:55] <dcaro>	 ceph is having issues
[10:09:34] <dhinus>	 mariadb crashed and recovered
[10:09:44] <dcaro>	 hmm... the slow ops are not showing on ceph status
[10:09:45] <dcaro>	 :/
[10:09:47] <dhinus>	 it's normal that it restarts as read-only after a crash
[10:09:53] <dcaro>	 it seems it recovered
[10:10:00] <arturo>	 yeah, slow ops are flapping 
[10:10:08] <arturo>	 it was present in ceph status just a few moments ago
[10:10:11] <dhinus>	 I will SET GLOBAL read_only=OFF;
[10:10:51] <dcaro>	 I don't see anything in `ceph crash ls`
[10:11:37] <dcaro>	 would be nice to have a `ceph health detail` when the slow ops happened
[10:11:49] <arturo>	 dcaro: I have some info in my backscroll
[10:11:49] <dcaro>	 (that would show the osds/nodes that were having issues)
[10:11:51] <arturo>	 https://www.irccloud.com/pastebin/R35wgI8G/
[10:11:57] <dhinus>	 the toolsdb alert is not firing anymore
[10:12:37] <dcaro>	 those are 1023 (D5) and 1030 (F4)
[10:13:17] <dcaro>	 root@cloudcephmon1006:~# ceph device ls  | grep 175
[10:13:17] <dcaro>	 HFS1T9G32FEH-BA10A_KN0CNA143I040120F  cloudcephosd1023:sdc  osd.175                 
[10:13:17] <dcaro>	 root@cloudcephmon1006:~# ceph device ls  | grep 242
[10:13:17] <dcaro>	 HFS1T9G32FEH-BA10A_KJA8N5701I0308I4P  cloudcephosd1030:sdg  osd.242  
[10:14:41] <arturo>	 https://www.irccloud.com/pastebin/GX7hOfP3/
[10:15:08] <arturo>	 is this the known hard drive error we are negotiating with dell?
[10:15:13] <dcaro>	 yep
[10:15:39] <dcaro>	 did it increase?
[10:16:20] <arturo>	 let me check
[10:16:56] <dcaro>	 for 1023 it did not increase
[10:17:14] <arturo>	 also, not for 1030
[10:18:15] <dcaro>	 there's two services reporting about the counter, one is smartd, and the other smart_failure (the latter does not like it)
[10:18:18] <dcaro>	 https://www.irccloud.com/pastebin/RG9BPIgu/
[10:18:37] <dcaro>	 but, no movement on that drive for that counter yep
[10:18:53] <arturo>	 ack
[10:19:57] <dcaro>	 can you attach the new task to T348643 and note the lack of increase in the counter?
[10:19:57] <stashbot>	 T348643: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643
[10:20:08] <dcaro>	 (and any other notes you want to add)
[10:20:19] <arturo>	 ok
[10:20:23] <arturo>	 which new task?
[10:20:35] <arturo>	 do you think we should create a ticket about today's slow ops?
[10:21:29] <dcaro>	 I thought it had created an automated task
[10:21:35] <dcaro>	 then just a note there is enough
[10:21:46] <dcaro>	 I think the cluster is struggling a bit extra due to the drainage
[10:22:07] <arturo>	 maybe T373632
[10:22:08] <stashbot>	 T373632: CephSlowOps  Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T373632
[10:22:39] <arturo>	 but I thought T373632 was about the prometheus weirdness rather than ceph having actual issues
[10:23:00] <arturo>	 (the ticket was created earlier today, then updated with this new alert)
[10:23:12] <dcaro>	 that's the ticket yep
[10:23:22] <dcaro>	 I think that the one about ceph not getting data did not create a ticket
[10:23:39] <arturo>	 ok, then I'll remove the parent task
[10:23:48] <dcaro>	 ack, thanks
[10:24:23] * dcaro disappears into the shadows
[10:24:23] <arturo>	 the ceph no data did create a task BTW, it is here: T374593
[10:24:24] <stashbot>	 T374593: CephClusterInUnknown - https://phabricator.wikimedia.org/T374593
[10:24:40] <dcaro>	 👍
[10:24:47] <arturo>	 thanks for showing up
[11:16:30] <arturo>	 new alert: tools k8s workers with many D procs
[11:17:46] <arturo>	 which seems to be a legit failure in this case
[11:24:03] <arturo>	 the worker with more D procs is tools-k8s-worker-nfs-16
[11:24:23] <arturo>	 but I just checked and it has a bunch of containers getting oomkilled
[11:24:44] <arturo>	 and the D procs are those perl containers from the checkwiki tool
[11:33:21] <arturo>	 the nfs servers seems happy
[11:33:33] <arturo>	 tools-k8s-worker-nfs-28 is stuck, wont let me in via ssh
[11:33:35] <arturo>	 will reboot it
[11:36:15] <arturo>	 opened T374612 to track this work
[11:36:16] <stashbot>	 T374612: toolforge: workers with many procs (2024-09-12 edition) - https://phabricator.wikimedia.org/T374612
[16:36:33] * dhinus offline