[06:53:52] <wm-bb>	 <sohom_datta> Quick question, Toolforge should have any issues if I connect to a OpenAI endpoint right ?
[06:54:04] <wm-bb>	 <sohom_datta> (Using my own keys)
[07:15:09] <wm-bb>	 <sohom_datta> -- Pod 'generate-new-data-daily-28696730-f8pcv'. Phase: 'pending'. Container state: 'waiting'. With reason 'ErrImagePull'. With message: 'rpc error: code = Unknown desc = failed to pull and unpack image "tools-harbor.wmcloud.org/tool (http://tools-harbor.wmcloud.org/tool-wsstats/tool-wsstats:latest)-wsstats/tool-wsstats:latest (http
[07:15:09] <wm-bb>	 //tools-harbor.wmcloud.org/tool-wsstats/tool-ws <clipped message>
[07:15:10] <wm-bb>	 <sohom_datta> stats:latest)": failed to resolve reference "tools-harbor.wmcloud.org/tool (http://tools-harbor.wmcloud.org/tool-wsstats/tool-wsstats:latest)-wsstats/tool-wsstats:latest (http://tools-harbor.wmcloud.org/tool-wsstats/tool-wsstats:latest)": failed to authorize: failed to fetch anonymous token: unexpected status: 504 Gateway Time-out'. Is this intended ?
[07:32:41] <RhinosF1>	 dhinus: ^ expected due to outage?
[07:33:14] <dhinus>	 RhinosF1: expected yes, tools-harbor is unresponsive
[07:33:30] <dhinus>	 !status tools-harbor is unresponsive
[07:58:15] <dcaro>	 !status toolforge ongoing outage, builds not working, new pods might fail to start
[08:06:06] <dhinus>	 !status ok
[11:51:33] <Guest61>	 !help It is mentioned on https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database
[11:51:33] <wm-bot>	 If you don't get a response in 15-30 minutes, please email the cloud@ mailing list -- https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication
[11:51:33] <Guest61>	 that "Database credentials are generated on account creation and placed in a file called replica.my.cnf in the home directory of both a Tool and a Tools user account. This file cannot be modified or removed by users."
[11:51:34] <Guest61>	 I have done ssh into toolforge and also done "become mytool"
[11:51:34] <Guest61>	 But where can i see my replica.my.cnf file.
[11:51:35] <Guest61>	 I am really struggling with https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases.
[11:52:29] <arturo>	 Guest61: what is your tool name?
[11:53:01] <Guest61>	 I have just created a tool wikievolution.
[11:53:08] <Guest61>	 Not added any code so far.
[11:53:23] <Guest61>	 I want to create a database for my tool on tools db
[11:57:03] <arturo>	 let me check a few things
[11:59:14] <arturo>	 !log admin restarted maintain-dbusers.service in cloudcontrol1005, it was stuck doing nothing
[11:59:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:59:22] <arturo>	 Guest61: you should be all set now
[11:59:43] <arturo>	 Guest61: you have a replica.my.cnf file now in your tool home dir
[12:00:23] <Guest61>	 Yeah thankyou :)
[12:01:02] <arturo>	 np
[12:04:56] <Guest61>	 when trying to connect to tools.db.svc.wikimedia.cloud with the replica.my.cnf credentials,
[12:04:56] <Guest61>	 it says:
[12:04:57] <Guest61>	 ERROR 1045 (28000): Access denied for user 's56037'@'172.16.1.16' (using password: YES)
[12:21:02] <Guest61>	 I mean after doing "beome mytool" I am not able to connect to tools.db, but by directly connecting to tools.db through my toolforge account, i am able to connect. Why is it so?
[12:21:56] <dcaro>	 yep, it did not work
[12:21:58] <dcaro>	 Jul 24 12:20:57 cloudcontrol1005 maintain-dbusers[1353413]: ERROR [root.populate_accountsdb:756] problem populating new account: Request to create replica.my.cnf file for account_type tool and account_id tools.wikievolution failed without response.
[12:22:00] <dcaro>	 looking
[12:27:08] <dcaro>	 Guest61: done :), can you try now?
[12:27:35] <dcaro>	 the fix is https://gerrit.wikimedia.org/r/c/operations/puppet/+/1056489, though I want to see why it was not caught by the functional tests before merging
[12:28:26] <dcaro>	 hmm... the tests don't pass :/
[12:28:36] <Guest61>	 Yeah i am able to do it now
[12:29:03] <dcaro>	 will investigate, Guest61 thanks for your patience! let us know if there's anything else that does not work :)
[12:29:27] <Guest61>	 Yeah Thankyou so much for your quick response.:)
[16:02:45] <Southparkfan>	 !log deployment-prep moved sessionstorage/kask from sessionstorage04 to sessionstorage06 T370461
[16:02:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[16:02:49] <stashbot>	 T370461: Remove or replace deployment-sessionstore04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370461
[19:12:53] <don-vip>	 I don't understand why the bookworm instances I created on 2024-07-02 are already being displayed as "deprecated 2024-07-03" in horizon?
[19:20:50] <stw>	 don-vip: it just means that a new version of the bookworm image has been published since you created your instances
[19:21:59] <stw>	 occasionally new image versions are created to apply updates etc so newly-created instances have those updates preinstalled - there shouldn't be any need to worry about it
[19:32:17] <don-vip>	 ok thank you, it worried me :)
[19:56:45] <don-vip>	 !log video Shutting down all celery workers to migrate the Redis database, then encoder instances
[19:56:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[20:09:17] <mutante>	 I have a cloud VPS project with a local mariadb install. The setup is also puppetized, don't have a particular issue with it. Except I should have offsite backups and not having to maintain mariadb wouldn't hurt.  Would you say I should still look at using trove instead?
[20:28:54] <bd808>	 mutante: trove will not give you backups as far as I know
[20:37:09] <mutante>	 bd808: I just created a trove instance out of curiosity and saw a "Create Backup" button next to it.. just now wondering
[20:37:46] <mutante>	 I am debating if it's making things simpler or more complex to use the extra instance vs just keeping local mariadb ...
[20:38:15] <bd808>	 https://wikitech.wikimedia.org/wiki/Help:Trove_database_user_guide#Backup_or_snapshot_your_databases
[20:38:25] <mutante>	 ah, thanks
[20:39:15] <bd808>	 the task that the snapshot line mentions is resolved so ¯\_(ツ)_/¯ 
[20:39:21] <mutante>	 I do get a "Backup Database" pop-up window when hitting that button
[20:39:39] <mutante>	 asks for swift container name
[20:40:35] <don-vip>	 !log video drop gfg and encoding07 instances (unused) as per T360711 + T365154
[20:40:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[20:40:39] <stashbot>	 T360711: Replace or remove Debian Buster VMs in 'video' cloud-vps project - https://phabricator.wikimedia.org/T360711
[20:40:40] <stashbot>	 T365154: video2commons general failure - https://phabricator.wikimedia.org/T365154
[20:40:58] <mutante>	 *nod* seems like that task was to enable this but we would still need swift
[20:41:24] <don-vip>	 !log video migrated redis database from video-redis-buster to video-redis-bookworm as per T360711 + T365154
[20:41:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[20:41:33] <mutante>	 I should probably just get that cinder volume and store my mysqldump there
[21:42:45] <mutante>	 I am trying to let puppet mount a cinder volume. I copied the code from profile::labs::cindermount::srv and just adjusted the mountpoint. The command it runs fails with "Refusing to format volume in non-interactive mode without --allow-unattended-format"
[21:44:18] <mutante>	 sounds like that option has to be added somewhere. though .. when I run that myself it feels like that is "interactive"
[21:45:02] <mutante>	 puppet just runs the  /usr/local/sbin/wmcs-prepare-cinder-volume   here 
[21:45:23] <don-vip>	 I need a disk quota incrase to complte the video bookworm migration ^^' https://phabricator.wikimedia.org/T370964
[22:37:14] <andrewbogott>	 don-vip, just the storage quota change or do you also need more VM things?
[22:42:35] <mutante>	 !log devtools gitlab-prod-1002 out of disk again when attempting package upgrade - apt-get clean; rm /var/log/syslog.2*.gz rm /var/log/messages.2*.gz to get space
[22:42:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[23:38:59] <don-vip>	 andrew sorry I didn't see your question here. For now only disk, thank you :)
[23:39:45] <don-vip>	 !log video updated and restarted video2commons frontend to use the bookworm redis instance. Setup encoding01 as the first new bookworm instance as per T360711 + T365154
[23:39:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[23:39:49] <stashbot>	 T360711: Replace or remove Debian Buster VMs in 'video' cloud-vps project - https://phabricator.wikimedia.org/T360711
[23:39:50] <stashbot>	 T365154: video2commons general failure - https://phabricator.wikimedia.org/T365154
[23:40:15] <don-vip>	 I hope video2commons is still working...
[23:40:33] <don-vip>	 I'lee see the outcome of my actions tomorrow, it's late here. Good night :)