[07:43:34] Morning! [10:08:12] for T432325 I'm looking at roll-restarting all tools that mount nfs, do we have a ready-made solution that's simpler than 'cookbook wmcs.toolforge.k8s.reboot --cluster-name toolsbeta --all-nfs-workers' ? [10:08:13] T432325: Make sure dumps-nfs mount/umount is propagated inside Toolforge containers - https://phabricator.wikimedia.org/T432325 [10:08:22] blunt tool but it'll work [10:09:01] also good morning [10:09:20] \o, not really I think [10:09:52] ok yeah I'll just do that, we know it'll work [10:10:14] technically not needed but oh well [10:10:22] it might take a while on tools though (many workers) [10:11:00] true, I don't particularly mind as maybe roll-restarting all nfs tools might equally take a long time ? not sure [10:11:17] 👍 [10:11:33] it should take less if you don't care of them coming up xd [10:11:40] lol [10:31:37] * godog lunch [11:15:49] I have a two easy and one not so easy mrs for the utils repo https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests [11:31:27] I'll take a look [11:34:07] ok https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/6 is a little more than I have time for, rest LGTM [11:34:25] thanks! [11:35:31] np, unrelated but toolsbeta now has on-demand dumps-nfs mounts and new volume-admission, everything seems to work as expected [11:35:55] \o/ [11:35:57] I'll push volume-admission to tools (no on-demand dumps-nfs yet) and merge the bump branch [11:37:14] hmpf... gitlab was telling me that I needed to run the pipeline before merging, so I hit 'merge' and it shows a pop up with "run pipeline", I hit it, it says it failed and to try again, so I hit merge again and now it merged without pipeline :/ [11:37:28] :( [12:02:31] unless there are objections I'll start with roll-restart all tools nfs workers [12:04:42] ack [13:40:55] dhinus: can you give me a quick catch-up about clouddb1026? I got as far as 'services monitoring s3 are failing because there's no mysqld running for s3' but didn't actually answer the question 'should there be an s3 agent?' [13:41:40] andrewbogott: see https://phabricator.wikimedia.org/T432745#12158573 [13:42:48] let me know if that answers your question or I can give you more details :) [13:42:55] ah, thanks! So, we can ignore those alerts until manual tells us enough to clean up the alerts. [13:43:22] yep, I cleaned all the other clouddb-related ones but I'm waiting for manuel's input on that one [13:43:32] 'remove the alerts' was my first instinct and then I was like um... best not to make a db design decision over the weekend [13:49:11] bd808: do you have good tools to use to test the new toolforge opensearch cluster (which'll replace the old elasticsearch one)? [15:18:12] We don't have nested virt enabled anywhere right? Can we test one hypervisor setup for with it? (that would unblock building and running lima-kilo inside a VM there) [15:32:57] here is the ceph vdev_enable_discard patch: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318163 [15:33:24] dcaro: It was briefly enabled and worked OK but wound up being kind of a drag to keep supported. Users do ask for it occasionally... [15:34:47] dcaro: here was the initial tracking task: https://phabricator.wikimedia.org/T267433 and here was the deathknell https://phabricator.wikimedia.org/T276208 [15:35:11] I guess it's possible that that latter issue is no longer present. [15:39:32] according to this it should work on kernels >5.3 (we have >6 everywhere) https://www.kernel.org/doc/html/v5.7/virt/kvm/running-nested-guests.html#live-migration-with-nested-kvm [15:42:17] nice! I wouldn't object to someone running a trial but it's likely to end up a bit low on my list for this quarter. [15:48:41] ack, yep I know, also on mine, but it would be sooooo nice [15:52:28] FYI all, I'm about to start a ceph osd rolling reboot. In theory the new cookbook won't cause as many spurious alerts, but in practice... [15:53:07] dcaro: it would be nice [15:53:25] and would patch an abstraction leak that bothers me :) [15:55:10] andrewbogott: did you see my comment in the ceph patch? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318163 [15:55:38] oh, it's not merged, so it's not deploying that right? [15:55:41] (the rolling reboot) [15:56:36] correct, not related to the reboot [15:56:49] but I'm sure you're right that it's a global setting. Seems most everything is :/ [16:01:18] I'm not sure what's the criteria they use to put it in one section or another, sometimes it feels kinda random yep [17:02:47] taavi: I think I could point some test deploy of stashbot and bash at it. CRUD operations on an index is I think mostly the service contract today. That should be a relatively small surface to test the basics of. If both clusters are up in parallel we can probably work out a data and app migration recommendation. [17:04:18] bd808: yeah, I have both clusters running now, just s/elastic/open/ in the hostname [17:05:15] my plan was to make a tiny wrapper around https://docs.opensearch.org/latest/api-reference/document-apis/reindex/#cross-cluster-reindexing that you could just tell the index name and it'd copy it over (and probably re-name the source index to make it clear it's been moved). but I want to try that with a single tool manually first before announcing [17:05:15] it to the world. [17:12:56] dcaro: does https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1318200 look right to you? I think we must not have traversed that code before, seems like it wouldn't have ever worked. [17:13:42] consider_maintenance_healthy should take care of the noout/norebalance no? [17:13:47] (maybe we changed that xd) [17:17:26] I do not see that in the code, but will read again... [17:18:05] https://www.irccloud.com/pastebin/gMyQHWoG/ [17:18:31] pretty sure that doesn't help since it would == 2 in this case or possibly ==3 [17:20:00] https://www.irccloud.com/pastebin/xsom0lwP/ [17:20:30] I think that it expects the health checks to be just 1, and osdflags to be noout, norebalance and/or noin [17:21:49] there is some sturcture to the json, looking [17:22:56] it does not take into account the muted slow ops [17:23:00] https://www.irccloud.com/pastebin/cyxjcue5/ [17:23:08] oh, so /i/ broke it :) [17:23:19] it never took them into account I think xd [17:23:37] (as in, never checked if they were muted or not) [17:24:28] so the proper fix will be in get_osdmap_set_flags [17:26:00] I think it would be in `is_cluster_in_maintenance`, it should change ` len(temp_status["health"]["checks"]) == 1` to filter out the muted health checks [17:26:22] really? Shouldn't get_osdmap_set just not report muted flags? [17:26:31] that one is osd flags, not status checks [17:27:26] you can check, but I think that `get_osdmap_set_flags` is returning the correct info [17:28:30] it just parses the string under the `OSDMAP_FLAGS` check, and splits the flags in it `noout,norebalance flag(s) set` -> `{"noout", "norebalance"}` kinda things [17:29:02] oh, I see what you're saying, the muted thing doesn't appear in the flags, we hit it earlier on [17:29:28] yep [17:31:32] * dcaro off [17:32:01] andrewbogott: gtg, but I can check later if you want, or tomorrow morning (in case you want review) [17:32:12] ok! later [17:37:23] taavi: the smokiest of smoke tests works -- https://bd808-test.toolforge.org/opensearch.php [17:37:48] The same style of dump for the legacy cluster is https://bd808-test.toolforge.org/elastic7.php [17:41:42] The bash, irc-*, and sal indexes are all Stashbot related. bash and sal have the viewing frontends that folks generally know about. The csp-* indexes are maintained and used by https://csp-report.toolforge.org/ . Those are all the ones I know things about personally. [18:09:44] bd808: thanks. I wrote https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318228 and used that to try to copy the 'bash' index over which went fine. I also just tried to point bash.toolforge.org against it (in read-only mode), which also seems to have worked just fine. [18:10:07] i assume stashbot only has one 'server' setting, instead of allowing it to move features one by one? [19:38:18] Hi, I remember there was something that could redirect tools. It would be great if all urls to https://scheherazade-temp.toolforge.org/ would get redirected to https://scheherazade.toolforge.org/. I had to do that due to a bug in the name validation of tools but that's fixed now [19:42:55] Amir1: https://wikitech.wikimedia.org/wiki/Tool:Containers#Redirect_container [19:43:40] ah thanks [20:37:01] taavi: stashbot does indeed currently only have one shared server setup to all three index collections it maintains. [20:37:51] I'll need to stare at my calendar for a bit to figure out when I can make time to migrate its multiple parts. [20:44:20] I thought I was a PAWS admin but I don't see the 'admin' button up top when I log in. Assuming other admins can see it... can someone please make 'AndrewBogott (WMF)' an admin so I can help a blocked-but-legit user? [21:46:02] andrewbogott: In think you should be an admin again [21:46:29] It looks like you were using your `Andrewbogott` account for that before. [21:46:41] both have the admin hat now [21:47:26] I think I've used both at different times, but couldn't find the admin button for either. [21:47:31] Let's see if I can find it now... [21:48:08] After you login you probably need to go to File>Hub Control Panel [21:48:15] and then Admin will be in the top nave there [21:48:22] *nav [21:48:49] Or manually go to https://hub-paws.wmcloud.org/hub/admin#/ [21:49:14] oooh you can't see it if you're viewing a notebook... I see it now. [21:49:16] ty! [21:49:58] The UI changed at some point, probably quite a while ago now. [21:50:09] I had to hunt for the menu item