[06:53:52] <wm-bb> <sohom_datta> Quick question, Toolforge should have any issues if I connect to a OpenAI endpoint right ? [06:54:04] <wm-bb> <sohom_datta> (Using my own keys) [07:15:09] <wm-bb> <sohom_datta> -- Pod 'generate-new-data-daily-28696730-f8pcv'. Phase: 'pending'. Container state: 'waiting'. With reason 'ErrImagePull'. With message: 'rpc error: code = Unknown desc = failed to pull and unpack image "tools-harbor.wmcloud.org/tool (http://tools-harbor.wmcloud.org/tool-wsstats/tool-wsstats:latest)-wsstats/tool-wsstats:latest (http [07:15:09] <wm-bb> //tools-harbor.wmcloud.org/tool-wsstats/tool-ws <clipped message> [07:15:10] <wm-bb> <sohom_datta> stats:latest)": failed to resolve reference "tools-harbor.wmcloud.org/tool (http://tools-harbor.wmcloud.org/tool-wsstats/tool-wsstats:latest)-wsstats/tool-wsstats:latest (http://tools-harbor.wmcloud.org/tool-wsstats/tool-wsstats:latest)": failed to authorize: failed to fetch anonymous token: unexpected status: 504 Gateway Time-out'. Is this intended ? [07:32:41] <RhinosF1> dhinus: ^ expected due to outage? [07:33:14] <dhinus> RhinosF1: expected yes, tools-harbor is unresponsive [07:33:30] <dhinus> !status tools-harbor is unresponsive [07:58:15] <dcaro> !status toolforge ongoing outage, builds not working, new pods might fail to start [08:06:06] <dhinus> !status ok [11:51:33] <Guest61> !help It is mentioned on https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database [11:51:33] <wm-bot> If you don't get a response in 15-30 minutes, please email the cloud@ mailing list -- https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication [11:51:33] <Guest61> that "Database credentials are generated on account creation and placed in a file called replica.my.cnf in the home directory of both a Tool and a Tools user account. This file cannot be modified or removed by users." [11:51:34] <Guest61> I have done ssh into toolforge and also done "become mytool" [11:51:34] <Guest61> But where can i see my replica.my.cnf file. [11:51:35] <Guest61> I am really struggling with https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases. [11:52:29] <arturo> Guest61: what is your tool name? [11:53:01] <Guest61> I have just created a tool wikievolution. [11:53:08] <Guest61> Not added any code so far. [11:53:23] <Guest61> I want to create a database for my tool on tools db [11:57:03] <arturo> let me check a few things [11:59:14] <arturo> !log admin restarted maintain-dbusers.service in cloudcontrol1005, it was stuck doing nothing [11:59:18] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:59:22] <arturo> Guest61: you should be all set now [11:59:43] <arturo> Guest61: you have a replica.my.cnf file now in your tool home dir [12:00:23] <Guest61> Yeah thankyou :) [12:01:02] <arturo> np [12:04:56] <Guest61> when trying to connect to tools.db.svc.wikimedia.cloud with the replica.my.cnf credentials, [12:04:56] <Guest61> it says: [12:04:57] <Guest61> ERROR 1045 (28000): Access denied for user 's56037'@'172.16.1.16' (using password: YES) [12:21:02] <Guest61> I mean after doing "beome mytool" I am not able to connect to tools.db, but by directly connecting to tools.db through my toolforge account, i am able to connect. Why is it so? [12:21:56] <dcaro> yep, it did not work [12:21:58] <dcaro> Jul 24 12:20:57 cloudcontrol1005 maintain-dbusers[1353413]: ERROR [root.populate_accountsdb:756] problem populating new account: Request to create replica.my.cnf file for account_type tool and account_id tools.wikievolution failed without response. [12:22:00] <dcaro> looking [12:27:08] <dcaro> Guest61: done :), can you try now? [12:27:35] <dcaro> the fix is https://gerrit.wikimedia.org/r/c/operations/puppet/+/1056489, though I want to see why it was not caught by the functional tests before merging [12:28:26] <dcaro> hmm... the tests don't pass :/ [12:28:36] <Guest61> Yeah i am able to do it now [12:29:03] <dcaro> will investigate, Guest61 thanks for your patience! let us know if there's anything else that does not work :) [12:29:27] <Guest61> Yeah Thankyou so much for your quick response.:) [16:02:45] <Southparkfan> !log deployment-prep moved sessionstorage/kask from sessionstorage04 to sessionstorage06 T370461 [16:02:48] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [16:02:49] <stashbot> T370461: Remove or replace deployment-sessionstore04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370461 [19:12:53] <don-vip> I don't understand why the bookworm instances I created on 2024-07-02 are already being displayed as "deprecated 2024-07-03" in horizon? [19:20:50] <stw> don-vip: it just means that a new version of the bookworm image has been published since you created your instances [19:21:59] <stw> occasionally new image versions are created to apply updates etc so newly-created instances have those updates preinstalled - there shouldn't be any need to worry about it [19:32:17] <don-vip> ok thank you, it worried me :) [19:56:45] <don-vip> !log video Shutting down all celery workers to migrate the Redis database, then encoder instances [19:56:48] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [20:09:17] <mutante> I have a cloud VPS project with a local mariadb install. The setup is also puppetized, don't have a particular issue with it. Except I should have offsite backups and not having to maintain mariadb wouldn't hurt. Would you say I should still look at using trove instead? [20:28:54] <bd808> mutante: trove will not give you backups as far as I know [20:37:09] <mutante> bd808: I just created a trove instance out of curiosity and saw a "Create Backup" button next to it.. just now wondering [20:37:46] <mutante> I am debating if it's making things simpler or more complex to use the extra instance vs just keeping local mariadb ... [20:38:15] <bd808> https://wikitech.wikimedia.org/wiki/Help:Trove_database_user_guide#Backup_or_snapshot_your_databases [20:38:25] <mutante> ah, thanks [20:39:15] <bd808> the task that the snapshot line mentions is resolved so ¯\_(ツ)_/¯ [20:39:21] <mutante> I do get a "Backup Database" pop-up window when hitting that button [20:39:39] <mutante> asks for swift container name [20:40:35] <don-vip> !log video drop gfg and encoding07 instances (unused) as per T360711 + T365154 [20:40:39] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [20:40:39] <stashbot> T360711: Replace or remove Debian Buster VMs in 'video' cloud-vps project - https://phabricator.wikimedia.org/T360711 [20:40:40] <stashbot> T365154: video2commons general failure - https://phabricator.wikimedia.org/T365154 [20:40:58] <mutante> *nod* seems like that task was to enable this but we would still need swift [20:41:24] <don-vip> !log video migrated redis database from video-redis-buster to video-redis-bookworm as per T360711 + T365154 [20:41:27] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [20:41:33] <mutante> I should probably just get that cinder volume and store my mysqldump there [21:42:45] <mutante> I am trying to let puppet mount a cinder volume. I copied the code from profile::labs::cindermount::srv and just adjusted the mountpoint. The command it runs fails with "Refusing to format volume in non-interactive mode without --allow-unattended-format" [21:44:18] <mutante> sounds like that option has to be added somewhere. though .. when I run that myself it feels like that is "interactive" [21:45:02] <mutante> puppet just runs the /usr/local/sbin/wmcs-prepare-cinder-volume here [21:45:23] <don-vip> I need a disk quota incrase to complte the video bookworm migration ^^' https://phabricator.wikimedia.org/T370964 [22:37:14] <andrewbogott> don-vip, just the storage quota change or do you also need more VM things? [22:42:35] <mutante> !log devtools gitlab-prod-1002 out of disk again when attempting package upgrade - apt-get clean; rm /var/log/syslog.2*.gz rm /var/log/messages.2*.gz to get space [22:42:37] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [23:38:59] <don-vip> andrew sorry I didn't see your question here. For now only disk, thank you :) [23:39:45] <don-vip> !log video updated and restarted video2commons frontend to use the bookworm redis instance. Setup encoding01 as the first new bookworm instance as per T360711 + T365154 [23:39:49] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [23:39:49] <stashbot> T360711: Replace or remove Debian Buster VMs in 'video' cloud-vps project - https://phabricator.wikimedia.org/T360711 [23:39:50] <stashbot> T365154: video2commons general failure - https://phabricator.wikimedia.org/T365154 [23:40:15] <don-vip> I hope video2commons is still working... [23:40:33] <don-vip> I'lee see the outcome of my actions tomorrow, it's late here. Good night :)