[04:23:41] Reminder: Our next Toolforge Monthly meeting is this Tuesday, 28th. [04:23:41] The meeting agenda and meeting link are on the wiki page below: [04:23:41] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Monthly_meeting [06:41:19] greetings [07:30:43] Morning! [10:15:52] * godog lunch [12:44:01] es->opensearch migration script ready for review: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318228 [12:49:38] bd808: the other option is that we just migrate all of the indexes and tools stashbot touches at once :P [12:56:14] I'm seeking reviews of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318139 [12:56:23] looking [12:56:54] thank you [12:58:12] godog: I'm having a bit of trouble understanding the cleanup logic.. in which scenario will dumps_use_nfs_lb be false but the legacy hostname paths will be symlinks that need to be removed? [12:58:24] is that something that we'd need in case of a rollback? [12:58:58] taavi: yes correct, during rollback so then labstore::nfs_mount can do its thing [12:59:26] rollback and cleanup after transition phase ends actually [13:00:15] ah [13:00:55] +1'd [13:01:00] thank you [13:01:37] re: migration script, LGTM from a quick look, and if dataset is not huge actually might not be a bad idea to stop the world and migrate (?) [13:05:33] dcaro: are the sal repeated entries for wmcs.openstack.get_project_for_proxy expected ? [13:06:18] oh, nope, I'm crating it anew, it does not need to log anything [13:06:40] ack [13:06:46] hmm, that did not use to work from my laptop xd [13:07:13] success (?) [13:09:27] I'll have it, yay \o/ [13:11:20] godog: for the ES->OS migration I want to make it initially something that the tool maintainer explicitely chooses to start so that they're around to see if there are compatibility issues with the new version or something [13:11:54] fair yeah [13:11:55] fwiw for sizes, the biggest index is ~18M records, there's one more that's in the millions and the rest are in the hundreds of thousands or smaller [13:12:45] nice thank you, for some reason I was imagining a much larger dataset [13:13:01] not complaining tho [13:16:29] godog: the cookbook making the mess ready for review now https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1318716 xd, no more sallogs from it [13:19:47] ack, yeah any reason why we can't query the proxy itself via http for that info? [13:19:49] hrm the haproxy migration work might kill that redis instance entirely so i'd rather not add any more reads to it :P [13:20:02] it's in the mariadb database already or you can just ^F https://openstack-browser.toolforge.org/proxy/ [13:21:28] "proxy itself via http for that info" I don't know of an api to do it, happy to use one if it's there [13:22:01] I'd prefer not rely on toolforge for cloudvps admin stuff (re using https://openstack-browser.toolforge.org/proxy/), where does it get it from? [13:22:30] those ceph emails are me doing kernel reboots. Just one left [13:23:20] the proxy API [13:24:04] there is yes! [13:24:07] 👀 [13:24:28] do we have any cookbooks using the openstack api already? (to reuse auth and such) [13:26:00] nope, our cookbooks still don't know how to talk to and authenticate to the openstack api directly :P [13:26:07] there might be something using the `wmcs-webproxy` CLI though [13:26:25] ohhh, nice, I can do the cli for now xd [13:26:59] hmm, it requires the project xd [13:27:18] godog: fwiw migrating an index with ~400k records takes about a minute, that can probably be extrapolated up for the couple bigger indexes [13:28:14] (that is with 1+1 replication) [13:28:41] not too bad, I'd imagine/expect iops to be the bottleneck [13:42:18] it's going to take some extra effort to adapt the cookbook to use any of the web proxy API and/or web proxy cli :/, little by little [14:38:24] is cloudvirt1048 getting worked on and I'm just bad at phab search? Or should I blanket-ignore cloudvirt alerts due to rebalancing work? [14:43:02] the silence expired for NeutronAgentDown though yes T431682 [14:43:03] T431682: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682 [14:43:06] I'll ack [14:43:34] thanks. I see no VMs running there so it was pretty clear something intentional was happening [14:44:50] also sorry about the annual o11y nag... maybe the right thing is for me to just always ignore monitoring for that project. Or figure out how to opt it out altogether. [14:51:19] very last-minute agenda reminder for the Toolforge monthly meeting [14:53:46] thanks, added a note [14:53:51] taavi: one big move was what I was thinking about for the stashbot/sal/bash group. I'd like to work out the config changes before I try to guess when I would do that. It is probably not a big deal. [15:02:03] dcaro: meeting time [15:30:31] I have the draft of the bullseye deprecation follow up message that will go out later this week here: [15:30:31] https://docs.google.com/document/d/1Qp1x23-kiHTVnpN-uWR092FcwSMROCmSyL8N_gWuQKE/edit?tab=t.u9wkyqvqz0c5 [15:49:45] that looks good to me but I added one change suggestion [15:57:36] andrewbogott: thanks! [16:20:08] taavi: I sent this DM on the updates to the clinic duties approval process: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Project_membership_request_approval [16:20:08] It's basically what we discussed on account approval process. [16:42:12] * dcaro off [16:42:15] cya tomorrow! [17:45:48] d????????? ? ? ? ? ? labstore-secondary-project [17:45:48] d????????? ? ? ? ? ? labstore-secondary-home [17:45:51] you love to see it [17:46:31] (trying to see if I can replace the nfs scratch server without rebooting everything everywhere) [17:47:55] komla: yeah, that seems good to me. thanks [17:49:08] OMG it recovered without a reboot! [17:49:26] And didn't seem to lock up the system in the meantime, so... not bad [18:21:57] andrewbogott: woot! [18:25:17] komla: in the message deprecation - if you send it later this week, it will already be too late (since you start shutting things down August 1 according to the message). Also i may suggestion to mention this is the second notice ? (as a notice was sent out some time ago) [19:13:48] here's another proofreading request: https://etherpad.wikimedia.org/p/scratchmaintenance [23:25:39] bliviero: noted! we mentioned that it's a follow up message. I will make it more explicit. I intend to send it out tomorrow the 29th. will that still work with the July 31st date since it's a follow up? [23:25:53] taavi: thanks!