[07:01:06] <godog>	 greetings
[08:31:34] <volans>	 morning, yesterday I uploaded an older version of alloy to the registry (I mistakenly pasted the version of the chart instead of the software itself, :facepalm:). Is there an easy way to delete it as it's not needed by any chance?
[09:50:11] <dhinus>	 hello!
[09:51:46] <dhinus>	 volans: which registry?
[09:52:07] <volans>	 docker-registry.toolforge.org
[09:53:25] <dhinus>	 https://wikitech.wikimedia.org/wiki/Docker-registry#Deleting_images
[09:53:31] <dhinus>	 never tried it myself though
[09:53:51] <volans>	 isn't that for the prod registry?
[09:53:58] <dhinus>	 ah sorry you're right
[09:54:36] <dhinus>	 https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Docker-registry#Delete_old_images
[09:55:44] <volans>	 thx
[09:56:58] <volans>	 possibly not worth for a ~100MB image
[10:09:44] <dhinus>	 yeah agreed
[10:14:42] <godog>	 we didn't get paged for yesterday cloudweb outage did we?
[10:15:32] * dhinus checks
[10:16:00] <dhinus>	 we did not
[10:16:07] <volans>	 sigh
[10:16:09] <godog>	 :(
[10:16:14] <godog>	 it is task time
[10:16:28] <volans>	 :D
[10:16:48] <dhinus>	 in my mailbox I find a "page" cloudweb alert from october, but it was a nagios one
[10:17:00] <dhinus>	 this time I only received an alertmanager one, with severity=warning
[10:17:47] <dhinus>	 it's correct because it's about the single node being down
[10:18:07] <dhinus>	 I think we miss an alert that checks the availability of the services running on cloudweb*
[10:19:18] <godog>	 indeed, that's now T411470
[10:19:18] <stashbot>	 T411470: Page on cloudweb/horizon down - https://phabricator.wikimedia.org/T411470
[10:19:25] <volans>	 also review the pybal config
[10:20:49] <godog>	 indeed, my understanding is that pybal did the right thing in this case in the sense that both realservers were failing healthchecks
[12:13:55] <taavi>	 dhinus: https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/92
[12:15:33] <dhinus>	 taavi: thanks! +1d
[13:28:19] <volans>	 from a shell in a tool, where I can find the labstore-secondary-tools-home directory? is it mounted somewhere?
[13:28:23] <volans>	 to test the nfs tracing
[13:28:50] <godog>	 does /proc/mounts list it?
[13:29:36] <volans>	 no, but it might be hidden elsewhere :D
[13:30:16] <volans>	 I only see the 2 dumps, the scratch and the tools-project one
[13:30:26] <godog>	 lol indeed
[13:31:50] <taavi>	 would be at /home if it was mounted
[13:32:15] <volans>	 ack, do you know anything in toolsbeta that mounts it?
[13:32:28] <volans>	 that's where I looked first and was empty indeed
[13:35:57] <taavi>	 if it's not mounted in tools then I don't think it is mounted in toolsbeta either
[13:36:44] <volans>	 I checked on one tool only, or is it mounted either for all or for none?
[13:47:02] <taavi>	 the mount config would be the same on all tools
[13:50:47] <volans>	 ok thx
[16:00:41] <dhinus>	 tools-db is alerting, looking
[16:00:58] <dhinus>	 perfect timing given I just closed two toolsdb tasks as "resolved" :D
[16:02:23] <dhinus>	 andrewbogott: did you reboot tools-db-7 by any chance?
[16:02:51] <dhinus>	 I thought mariadb would auto-restart after a reboot but it didn't
[16:03:21] <dhinus>	 restarted now
[16:03:25] <andrewbogott>	 I did, can you revive it?
[16:03:51] <dhinus>	 yes already revived
[16:04:19] <dhinus>	 in the past, I'm pretty sure mariadb was starting automatically after a reboot
[16:04:39] <dhinus>	 that still meant you had to enable read-write on the primary, but the replica would resume without manual intervention
[16:04:51] <dhinus>	 this time even if it was the replica being restarted, it needed a manual "systemctl start mariadb"
[16:04:52] <taavi>	 iirc it's explicitely configured to not automatically start?
[16:05:15] <volans>	 in prod mariadb is not restarted on reboot
[16:05:21] <volans>	 not sure if you use a specific setting to do that
[16:05:34] <dhinus>	 taavi: in production dbs, yes. in toolsdb I'm pretty sure it's configured to start as read-only, but to start anyway
[16:05:49] <dhinus>	 I will check if i can track down when/if this config changed
[16:21:32] <dhinus>	 unrelated: is there a way to email all the members of a cloudvps project? (context: https://phabricator.wikimedia.org/T409668#11424478)
[16:21:40] <taavi>	 no
[16:21:59] <taavi>	 (well, there is a script in puppet to do that once, but that's not very helpful for that situation)
[16:22:05] <dhinus>	 ack thanks!
[16:30:27] <andrewbogott>	 dduvall: there are a few gitlab-runner things on https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/2RV6C3TKSILUX6BGZZY4MFLLIJ6IEVDE/ -- is it ok if I reboot them now (one at a time) or do you need to depool someplace first?
[18:20:41] * dhinus off