[09:10:34] <arturo>	 good morning
[09:11:12] <arturo>	 dcaro: do you plan to follow up with robh regarding hardware for ceph?
[09:11:30] <arturo>	 (email)
[09:12:49] <dcaro>	 done already
[09:13:03] <dcaro>	 added a note in the network meeting too to discuss the switch space and such
[09:13:42] <dcaro>	 there's not much, so if we want to spread the nodes we'll have to either shift other hosts around, or if QoS is ready, then we can try the single port setup
[09:22:16] <arturo>	 ok, thanks
[09:22:51] <arturo>	 dcaro: there was this plan to create a document with the different ceph HA strategies we had been exploring. Do you know if it was finally created?
[09:23:18] <dcaro>	 The document was created, it has not yet been filled up (PTO + other more urgent work)
[09:25:12] <dcaro>	 https://docs.google.com/document/d/1UtMK8ZLLfn1CFbcgBccBvzTlIuab244XAs98_gTuBlg/edit?tab=t.0#heading=h.qqx9jzis1t6 feel free to add anything if you have time
[09:26:54] <arturo>	 ok, thanks
[10:14:03] <dcaro>	 arturo: thanks, added a bit more structure if you want to keep adding stuff
[10:14:13] <arturo>	 ack
[11:38:21] * dcaro lunch
[12:38:49] <dhinus>	 heads up I'm testing the NodeDown alerts by shutting down cloudvirt1063 (which is already in maintenance)
[12:45:44] <arturo>	 ack
[13:06:21] <dcaro>	 ack
[13:11:37] <dhinus>	 we're starting the toolsbeta upgrade with Raymond_Ndibe 
[13:16:40] <dcaro>	 dhinus: ack, I'm in a public place without headphones xd, let me know if you need anything from me
[13:28:43] <dhinus>	 ok!
[13:41:01] <dcaro>	 dhinus: Raymond_Ndibe there's one worker node (nfs-7) in toolsbeta that's scheduling disabled, is that you?
[13:41:02] <dhinus>	 hmm not sure, unless it's the cookbook doing it?
[13:41:02] <dhinus>	 what can cause the worker node to be "scheduling disabled"?
[13:41:02] <dhinus>	 we're also seeing one error in functional tests (haven't started the upgrade yet)
[13:41:02] <dcaro>	 the instructions usually start from the control nodes
[13:41:03] <dcaro>	 I think it might be manually set (or with cookbook)
[13:41:03] <dhinus>	 we are at line 22 of https://etherpad.wikimedia.org/p/k8s-1.27-to-1.28-upgrade
[13:41:03] <Raymond_Ndibe>	 ummm no, that shouldn't be happening. We are about to begin toolsbeta upgrade but so far the only command we've executed is `sudo cookbook wmcs.toolforge.k8s.prepare_upgrade --cluster-name toolsbeta --src-version 1.27.16 --dst-version 1.28.14 --cluster-name toolsbeta --task-id <id_of_task>`
[13:41:04] <dcaro>	 we should make the tests pass before upgrading though
[13:41:04] <dcaro>	 what's the faiure?
[13:41:04] <dcaro>	 *failure
[13:41:04] <Raymond_Ndibe>	 I don 't expect that command have that effect
[13:41:04] <dcaro>	 btw. I'll uncordon the node, see if it works ok
[13:41:04] <dhinus>	 dcaro: ok
[13:41:04] <dcaro>	 Raymond_Ndibe: I don't think the command did it, should have been someone manually or an old run of the cookbook or something
[13:41:04] <Raymond_Ndibe>	 the failure is coming from `get components health without auth works`
[13:41:05] <dcaro>	 looking, the new node seems to be unable to start pods :/
[13:41:05] <Raymond_Ndibe>	 I wasn't expecting that to be an issue until we get to tools but yea, I'll fix it before continuing
[13:41:05] <dcaro>	 not really, the node is running a build ok
[13:41:05] <dcaro>	 aaaahhh, the pod was a test....
[13:41:05] <dcaro>	 nm, the node seems to be working ok
[13:41:05] <dcaro>	 looking into components, I think it might be the old image issue
[13:41:05] <dcaro>	 hmm, it's failing to pull the new image, but the old one is still running, what's the output of that test?
[13:41:05] <Raymond_Ndibe>	 yeaa it is
[13:41:06] <Raymond_Ndibe>	 https://www.irccloud.com/pastebin/VV23og7o/
[13:41:06] <dcaro>	 hmm, sometimes it works, sometimes it does not :/
[13:41:06] <dcaro>	 https://www.irccloud.com/pastebin/CTeRUCeV/
[13:42:07] <dcaro>	 I'll update the components-api deployment by hand to an image that exists :/
[13:42:41] <Raymond_Ndibe>	 I don't expect that to solve this particular problem
[13:43:38] <dcaro>	 using tools-harbor.wmcloud.org/toolforge/components-api:image-0.0.27-20241030094558-9279575a
[13:43:47] <dcaro>	 it should be able to create all the pods correctly now
[13:44:22] <dcaro>	 Raymond_Ndibe: why do you think so?
[13:45:06] <dcaro>	 this seems to be working now
[13:45:10] <dcaro>	 https://www.irccloud.com/pastebin/WyMsRUUc/
[13:45:19] <dhinus>	 dcaro: is that a newer or older version?
[13:45:23] <dcaro>	 newer
[13:45:36] <dcaro>	 latest one I found in tools harbor
[13:45:46] <dhinus>	 what was the previous version?
[13:45:52] <dcaro>	 0.0.19
[13:45:56] <Raymond_Ndibe>	 the previous version was 0.0.19
[13:45:57] <Raymond_Ndibe>	 yes
[13:46:53] <dcaro>	 are the tests passing now?
[13:47:36] <Raymond_Ndibe>	 seems to be working. {"status": "OK"} vs {data: {"status": "OK"}...}
[13:48:05] <dcaro>	 yep, it should not care as long as it's a 200 
[13:48:10] <dcaro>	 and json parseable
[13:48:12] <dcaro>	 iirc
[13:48:17] <dhinus>	 I thought we were running 0.0.42 after this MR? https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/564
[13:48:58] <dcaro>	 looks like it should yep
[13:49:02] <dcaro>	 was that deployed?
[13:50:02] <dcaro>	 that's what's is the code, so yep, maybe it was never deployed?
[13:50:23] <dcaro>	 (we should use the same for tools btw)
[13:50:24] <Raymond_Ndibe>	 I didn't run the deploy 🤦🏽
[13:50:45] <dcaro>	 ack, that makes sense then :)
[13:50:57] <dcaro>	 we have pending adding some kind of check for things not having been deployed xd
[13:51:20] <Raymond_Ndibe>	 yes we didn't add it for tools because we wanted to discuss it with you dcaro. But if you think we should use the same version for tools that should be great
[13:51:46] <dcaro>	 yep, we can bump to the latest, though we might want to have the 'disable' flag before that just in case
[13:52:30] <dhinus>	 do you think we should deploy 0.42 to toolsbeta before or after the 1.28 upgrade?
[13:52:51] <Raymond_Ndibe>	 my suggestion is to continue toolsbeta upgrade, then I add the disable flag, then we deploy that for components-api before we do tools upgrade next week
[13:54:07] <dcaro>	 we can go with this upgrade, and deploy later
[13:56:18] <dhinus>	 the upgrade cookbook is running :rocket:
[13:56:51] <dcaro>	 \o/
[14:09:06] * dcaro have to relocate again, be back in a bit
[14:48:50] <dcaro>	 control plane look all upgraded and well, are tests passing?
[15:01:36] <dcaro>	 the whole cluster is upgraded now, Raymond_Ndibe how are tests going?
[15:02:21] <Raymond_Ndibe>	 all tests are passing
[15:02:46] <Raymond_Ndibe>	 going to let it run a bit more but everything looks good
[15:04:16] <dcaro>	 \o/
[16:18:06] <dcaro>	 Raymond_Ndibe: it's everything done for toolsbeta upgrade then? Anything missing I can help with?
[16:18:55] <dhinus>	 arturo: do you know if cloudcontrol2006-dev can be powered down? see last comment in T370401
[16:18:55] <stashbot>	 T370401: cloudcontrol2006-dev struggling with memory - https://phabricator.wikimedia.org/T370401
[16:19:08] <arturo>	 dhinus: I'll reply soon
[16:19:20] <arturo>	 but the answer is yes
[16:19:20] <dhinus>	 ack
[16:19:32] <arturo>	 thanks!
[16:58:02] * arturo offline
[17:50:16] <Raymond_Ndibe>	 dcaro: yes everything is done for toolsbeta. test ran for approx 3hrs without errors so yea
[17:51:27] <Raymond_Ndibe>	 obviously there are still some minor things left like removing the flag thing and some version upgrade for components but those can be handled
[17:56:17] <dcaro>	 Raymond_Ndibe: ack, kudos! :)
[18:02:46] <taavi>	 dhinus: re T375101#10278327, we have a cookbook for this, which also includes the meta_p update run which you missed
[18:02:47] <stashbot>	 T375101: Prepare and check storage layer for nrwiki - https://phabricator.wikimedia.org/T375101
[18:05:27] <dhinus>	 taavi: thanks, I followed https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas/DNS which does not mention the cookbook, I will update the wiki
[18:05:55] <taavi>	 i recommend following https://wikitech.wikimedia.org/wiki/Add_a_wiki#Maintain_views instead
[18:06:04] <taavi>	 that wiki page is about the dns mechanism specifically, not the general process
[18:06:46] <dhinus>	 this section seemed to match what I needed... https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas/DNS#Adding_a_new_wikidb_alias
[18:07:00] <dhinus>	 "When a new wikidb is added"
[18:07:38] <taavi>	 let me add a colored box pointing to the other docs
[18:07:44] <dhinus>	 thanks :)
[18:08:59] <dhinus>	 I noticed that the script run updated a few other aliases, I wonder if some other wikis were added in the past and the cookbook was not run
[18:09:19] <dhinus>	 how can we make sure all the DNS records are in sync?
[18:09:36] <taavi>	 by running the script and seeing if it notices any not in sync :-)
[18:10:53] <dhinus>	 I'm running the cookbook now, let's see what's the output
[18:11:12] <dhinus>	 though the cookbook takes the single wiki as an argument
[18:11:23] <dhinus>	 and I don't want to run it once per each wiki :)
[18:11:45] <dhinus>	 I think the DNS script will check all the aliases
[18:11:50] <dhinus>	 but what about meta_p?
[18:12:06] <taavi>	 if you mean checking if the DNS is in sync, you do that with the wikireplicas-dns script
[18:12:32] <taavi>	 i don't think the meta-p script has an option to detect missyncs, it just reloads the data
[18:12:37] <dhinus>	 yes sorry, my question was incomplete
[18:12:50] <dhinus>	 I will do a full manual run of the DNS script, and that should sort DNS
[18:13:11] <taavi>	 didn't you say you did that already?
[18:13:25] <taavi>	 anyhow, none of this would be a problem if everyone used the cookbook :-)
[18:13:25] <dhinus>	 only with --shard s5
[18:15:42] <dhinus>	 the cookbook runs the dns script without the "--shard" option so that will sort out things
[18:15:49] <dhinus>	 even if it takes longer....
[18:16:22] <dhinus>	 I though I was smarted and saved some time by adding "--shard s5" but karma is getting me
[18:16:28] <dhinus>	 *thought
[18:16:32] <dhinus>	 *smarter
[18:16:40] * dhinus cannot type
[18:17:55] <taavi>	 that script could probably be made much faster by listing all of the existing records at once instead of calling the designate API once per wanted record to get the current state :-)
[18:25:39] <dhinus>	 last question (I hope): by a quick look at the maintain-meta_p script, it looks like it's only adding the wiki you specify in the args... I worry that if the same mistake of not using the cookbook was made in the past, we might have some wikis missing from the meta_p.wiki table?
[18:26:50] <dhinus>	 or maybe I'm reading it wrong and it's looping? I see there's also --purge
[18:27:44] <taavi>	 i think it also has a flag to re-create the entire table, but yes, if you only ran it with a wiki specified now, it's entirely possible that it's missing something
[18:28:27] <dhinus>	 so I should probably run that script manually with --all-databases --purge
[18:28:42] <dhinus>	 what's relying on that table?
[18:29:11] <taavi>	 all the bots that need to iterate each wiki
[18:29:17] <taavi>	 and things like https://db-names.toolforge.org/
[18:30:00] <dhinus>	 gotcha
[18:34:00] <dhinus>	 any concerns before I run "maintain-meta_p --purge --all-databases" on clouddb1014 and clouddb1018?
[18:36:34] <dhinus>	 the cookbook completed successfully in the meantime
[18:37:42] <dhinus>	 I'm testing maintain-meta_p with --dry-run
[18:38:52] <dhinus>	 it fails with "WARNING failed request for https://labtestwikitech.wikimedia.org"
[18:39:06] <dhinus>	 I'll postpone this to tomorrow :)
[18:39:35] <dhinus>	 thanks taavi for the help, and for spotting I had missed the meta_p update!
[18:40:08] * dhinus logs off
[18:51:39] * dcaro off