[08:25:53] <wm-bb>	 <IVeertje> Glamorous seems to be down https://glamtools.toolforge.org/glamorous.php
[08:26:07] <wm-bb>	 <IVeertje> Webservice request timed out
[08:52:04] <arturo>	 !log admin T387828 depooled galera on cloudcontrol1005
[08:52:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[08:52:11] <stashbot>	 T387828: openstack galera no recent writes 2025-03-04 - https://phabricator.wikimedia.org/T387828
[09:04:46] <dcaro>	 IVeertje: I've restarted the webservice and it's back up
[10:50:20] <wm-bb>	 <mar_zh9> Hi! I am setting up my first Toolforge tool, and it seems my database credentials are missing:
[10:50:22] <wm-bb>	 <mar_zh9> tools.monax-ukwiki-queries@tools-bastion-13:~$ ls -a
[10:50:23] <wm-bb>	 <mar_zh9> .  ..  .bash_logout  .bashrc  .kube  .local  .profile  .python_history  .toolskube  logs
[10:50:25] <wm-bb>	 <mar_zh9> The docs advised me to ask for help, so there's that (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database). Or am I missing something here? Thanks in advance
[11:11:43] <dcaro>	 mar_zh9: looking
[11:15:06] <dcaro>	 oh, the process that creates the credentials was running in cloudcontrol1005 xd
[11:15:10] <dcaro>	 arturo: ^
[11:15:32] <dcaro>	 are we any closer to getting that node up? should I relocate the maintain-dbusers system?
[11:15:35] <arturo>	 ha!
[11:15:48] <arturo>	 dcaro: we are blocked on DCops operations, so better to relocate
[11:16:31] <dcaro>	 ack, I'll start on that
[11:18:36] <dcaro>	 got two new api slow alerts popping up
[11:21:04] <arturo>	 maybe expected
[11:23:07] <dcaro>	 okok
[11:23:44] <dcaro>	 patch to change the control node https://gerrit.wikimedia.org/r/c/operations/puppet/+/1124391
[11:24:08] <arturo>	 +1'd
[11:25:18] <dcaro>	 thanks!
[11:26:08] <dcaro>	 we might hit some db credentials issues...
[11:26:11] <dcaro>	 (hopefully not)
[11:26:25] <dcaro>	 grants to create users and such in the replicas
[11:26:36] <arturo>	 right
[11:30:44] <dcaro>	 it worked :)
[11:33:30] <dcaro>	 mar_zh9: your credentials are created \o/ (note that it's preferred to use the envvars to the file for your tools, `toolforge envvars list`, https://wikitech.wikimedia.org/wiki/Help:Toolforge/Envvars)
[11:35:45] <wm-bb>	 <mar_zh9> >oh, the process that creates the credentials was running in cloudcontrol1005 xd
[11:35:46] <wm-bb>	 <mar_zh9> I guess I am very lucky :)
[11:35:58] <wm-bb>	 <mar_zh9> thanks, will take a look
[13:40:11] <dhinus>	 !log tools reboot tools-legacy-redirector-2 (http probes failing more than usual)
[13:40:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:30:16] <arturo>	 !log admin-monitoring manually cleaning up list of fullstackd VMs
[15:30:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL