[00:02:17] Southparkfan: Do you remember from your work on deployment-prep bullseye stuff what changes are needed to pool a new MediaWiki server? Context is T378752 and my next step desire to route traffic to that PHP 8.1 node. [00:02:18] T378752: Figure out how to install PHP 8.1 on bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752 [00:05:29] ah ha, I bet T361387 will tell me a lot [00:05:30] T361387: Replace or delete deployment-mediawiki[11-12].deployement-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361387 [00:06:03] bd808: I'm about to log off to get some sleep, but what I can tell for sure is that I had to provision the new appservers with puppet, then add them to their appropriate dsh groups so that they get the scap syncs [00:07:04] However, I don't think all appservers are pooled in ATS for cache_text traffic, iirc only one of the two appservers is pooled right now. Technically, the other one can also serve traffic just fine, though [00:08:12] Southparkfan: good to know. I found some of what you did. I'll ask you more if I end up really stuck. :) [00:09:27] I also recall having issues installing new VMs with certain puppet roles right away, at some point I just decided to bootstrap them with the 'base puppet role' (i.e. the default role for Cloud VPS instances, nothing app-specific applied), then changing the puppet role to the appropriate role [00:10:49] I had "fun" with puppetmaster certs on the new instance I built. It felt like there has been some regression in switching to a self-hosted puppetmaster [00:10:58] (otherwise the host could get stuck in some unknown state, where it boot# fine, but wouldn't allow me to log in) [00:11:34] I luckily have all the s3cr3t cloud root juice if that happens [00:12:03] Oh heh, I didn't have issues with the puppetmaster certificates. I did have issues with incorrect permissions on the git clone of the puppet repo on the puppetmaster in deployment-prep, but I think it has been fixed since [00:12:59] Ahaha :D magic sauce for those who shall not depend on working LDAP clients [00:13:59] yeah, I can even enter through the vm's root console if things are very messed up. [00:14:18] Feel free to brain dump the installation process somewhere, I can surely take a look later today (or tomorrow I think, with regards to your TZ - it's already past midnight here) [00:15:39] Southparkfan: thanks for the offer. Get some good sleep. :) [00:16:19] Yes, will surely do :-p good luck, we'll catch up later [16:06:33] the tools-proxy seems to be having problems? : https://tools-static.wmflabs.org/bridgebot/57ede3a7/file_66698.jpg [16:06:46] ah, now it’s working again [16:07:36] there was a small outage on one switch, making the ceph cluster hiccup (and toolforge/cloudvps with it) [16:38:06] I'm getting a "Something went wrong!" page when visiting https://horizon.wikimedia.org/project/proxy/ [16:38:27] (deployment-prep project) [16:42:14] dancy: discussion in -releng suggests that might have been caused by the outage mentioned above and should recover by itself [16:42:16] (https://wm-bot.wmcloud.org/browser/index.php?start=11%2F21%2F2024&end=11%2F21%2F2024&display=%23wikimedia-releng) [16:42:23] oh. wait. I’m stupid [16:42:33] that would make sense for deployment-prep in general but not horizon ^^ [16:42:36] nevermind me [16:44:07] …and I’m just noticed that’s the same person in both conversations 🤦‍♂️ [16:59:10] dancy: I get the same :/, looking, andrewbogott are you doing anything with proxy/dns? [17:03:37] it's broken also for other projects (ex. tools) [17:09:57] found it, it's on horizon side [17:11:32] T380511 [17:11:33] T380511: [horizon] failing to show the proxies tab for some projects - https://phabricator.wikimedia.org/T380511 [17:12:01] that's news to me but I can try to look soon [17:12:40] dancy: can you subscribe to that task so I can enlist you for later testing? [17:28:42] Absolutely [18:15:21] dancy: try it now? [18:15:32] * dancy presses F5 [18:15:45] Success! [18:33:19] great! [18:58:03] And now for the next problem: I get the Something went wrong! page when accessing https://horizon.wikimedia.org/project/instances/60461772-4536-4055-ab8e-9180c719cf25/?tab=instance_details__interfaces [18:59:26] dancy: what project/VM is that? horizon urls are very session-dependent. [18:59:58] ah, I see. That's deployment-deploy04 in deployment-prep [19:00:13] I get similar weirdness when trying to view the "Interfaces" tab of that VM. [19:00:24] (in that case it just never populates the page w/ info) [19:01:54] ok, that's probably worth making a bug about although I don't think there are any self-serve options available on that tab anyway [19:03:10] Will do. [19:06:52] https://phabricator.wikimedia.org/T380531 filed [19:22:47] bd808: hi! How's the deployment-prep work going? (: [20:40:12] Southparkfan: it really has not progressed far since our last conversation, but I have been looking at some very long lived Phabricator tasks and thinking about what my plan for rollout (and rollback if things go poorly) might look like. [20:41:13] As you are well aware there are plenty of things about deployment-prep that could use some more thought and attention so I have to also try and stay out of the deep rabbit holes and infinitely sticky tar pits. [20:43:15] !log tools.bullseye add toolforge-standards-committee as maintainer for processing of T380537 [20:43:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bullseye/SAL [20:43:18] T380537: Adoption request for bullseye - https://phabricator.wikimedia.org/T380537 [22:44:24] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 3486d60faf (l10n updates: tcy) [22:44:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL