[08:19:29] cloudinfra-idp-1.cloudinfra.eqiad1.wikimedia.cloud has been spamming me puppet failure emails for a while [08:56:16] taavi: yeah same here, cc slyngs [08:58:13] I see it. Missing rredis password [09:05:32] morning [09:24:18] arturo / taavi cloudinfra-idp-1 has been unbroken. [10:18:46] thanks slyngs [10:21:00] Not that it wouldn't have broken in any case, we also need to upgrade cloudinfra-idp-1 in the near future. [10:38:11] ok! [10:44:23] I need a +1 on a quota request: T379514 [10:44:24] T379514: Increase CPUs for webperformancetest - https://phabricator.wikimedia.org/T379514 [10:52:17] +1d [10:52:21] thanks [11:11:27] does anybody remember what's the mechanism that adds users to the "bastion" project? T379478 [11:11:28] T379478: Members of project-a4b03ac235d443159feaeafec24a92dd not seemingly added to project-bastion - https://phabricator.wikimedia.org/T379478 [11:16:03] dhinus: it should happen automatically when adding an user to any project [11:16:25] maybe it was the wmfkeystone hook, but I don't remember exactly [11:17:04] I wonder if the issue related to the new random project ids [11:18:44] or to the fact the project was created with opentofu. did you delete any hooks? [11:21:53] I think the hook is still there, I have not deleted it [11:22:09] however, I think I deleted the logic to create security groups [11:22:17] so definitely there have been changes [11:22:28] this is not the first project that has been created using tofu-infra [11:23:53] but it's possible other projects did not show the issue, because members were already part of "bastion" [11:24:20] that is a good theory indeed [11:25:29] it's also possible it started to break when random project ids were introduced some months ago, we haven't created that many projects since [12:03:04] dhinus: for now, I will add the missing members to the bastion project by hand [12:09:50] arturo: ok! [12:10:07] I have also created T379550 to track the root problem [12:10:08] T379550: openstack: keystone may be failing to add users to the bastion project - https://phabricator.wikimedia.org/T379550 [12:10:13] and will close the user-created ticket [13:35:23] FYI I will move the 1K python LOC for the nova_fullstack daemon from puppet.git into a dedicated gitlab repository, so I can add gitlab CI, tox, etc [13:35:32] modules/openstack/files/nova/fullstack/nova_fullstack_test.py <-- file [13:38:37] tox/ci is available in the puppet repo too, but +1 for the move [13:40:32] mmm right. I thought writing pytest was harder on ops/puppet.git [13:46:38] Raymond_Ndibe: are you running functional tests in toolsbeta? [13:48:38] Raymond_Ndibe: I stopped your loop as it was blocking me from deploying anything (can't run tests in parallel) [13:49:37] if you want to keep the tests running, you might wan to consider using a different tool user so it does not block the deployments (--test-tool option to run_functional_tests.sh) [13:50:01] (as in running in the background on a loop) [13:55:21] arturo: +1 for moving that python script to its own repo [13:55:55] ack [14:42:36] dhinus: when did we upgrade spicerack? do you know if anything changed in the way ttys are allocated (the vm console cookbook is failing with PTY allocation failed now) [14:43:45] I did upgrade it a couple weeks ago [14:44:12] I don't remember anything about ttys, let me check the changelog [14:46:08] this was the upgrade: https://sal.toolforge.org/log/-Ofv05IBFk7ipym_I4bW [14:50:59] I can reproduce, and I have another cookbook that's failing on spicerack/remote.py so maybe it's a similar issue? [15:00:46] scratch that, the other failure is a puppet failure, so not the same issue [15:01:20] ack, let me try using the older version from my laptop [15:06:22] this works [15:06:24] https://www.irccloud.com/pastebin/wGJWoEdR/ [15:10:21] I would open a task and ping volans [15:10:37] unrelated quick review: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1089806 [15:59:39] dcaro: I ran again "puppetserver ca clean" from the right host (cloudinfra-cloudvps-puppetserver-1) and it worked! [16:00:42] 💃 yay! [16:07:53] dhinus: I have not been able to reproduce the console cookbook issue locally... I have re-upgraded to latest spicerack and now it works :/, also using the same version of clustershell than cloudcumin, and it's still working for me... [16:23:57] hmmm [16:24:30] maybe we have some dependency that is out of date on cloudcumin? [16:24:54] I have to log off shortly, but I can do some tests tomorrow [16:30:42] dhinus: created https://phabricator.wikimedia.org/T379570 [16:31:18] thanks! [16:40:43] * arturo offline [18:10:09] * dcaro off