[06:46:11] Hi, I'm running into a strange issue with the Toolforge database. My Toolforge app is unable to connect, returning a permissions denied error (https://meta.wikimedia.org/wiki/Global_reminder_bot/log). This started about two weeks before the PHP issue, so it's not related to that. I noticed that the credentials were rotated (not by me) around the [06:46:11] time of the failure. I tried to rotate it again (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Rotate_your_credentials), confirmed that the credentials were rotated, but it still seems to be failing. I haven't changed the code at all so that isn't the reason. Not sure what I could be missing. [08:02:08] !log jeanfred@tools-bastion-15 tools.integraality Deploy 44f14cc (Fix crash when creating a missing grouping page) for T438301 [08:02:10] !log jeanfred@tools-bastion-15 tools.integraality Deploy be8878d (Match SSE section keys on the last underscore) for T425477 [08:02:12] !log jeanfred@tools-bastion-15 tools.integraality Deploy 9ecbeff (Stream grouping page creation progress over SSE) for T425477 [08:02:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [08:02:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [08:02:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [08:41:40] !help [08:41:40] If you don't get a response in 15-30 minutes, please email the cloud@ mailing list -- https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication [08:42:13] Leaderboard: which tool account is that? [08:42:57] taavi https://iw.toolforge.org/permissions-expiry-reminder [08:46:20] Leaderboard: the credentials in the tool home directory seem to be working: https://phabricator.wikimedia.org/P97021. it seems like you have copied the file to userrights-reminder-bot/replica.my.cnf which would be outdated after rotating the credentials, I would start from fixing that [08:49:44] Thanks, I didn't realise that there was a duplicate. [10:03:36] !log jeanfred@tools-bastion-15 tools.integraality Deploy 3cfcf64 (Deduplicate Listeria columns differing only by reference check) for T368001 [10:03:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [10:32:41] !log jeanfred@tools-bastion-15 tools.integraality Deploy 415f8be (Mark grouping-page creation complete and fix count wording) for T425477 [10:32:43] !log jeanfred@tools-bastion-15 tools.integraality Deploy 0bd0104 (Hyperlink created grouping pages in live update progress) for T425477 [10:32:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [10:32:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [11:20:46] !log striker delete striker-docker-02 (bookworm) which has a too old docker version to run modern striker, and would need an upgrade to get rid of old PHP versions T329687 [11:20:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL [11:20:50] T329687: Rebuild Striker demo server - https://phabricator.wikimedia.org/T329687 [12:05:00] #Node2shell [12:05:00] Hello all, [12:05:01] I'm aware of the `Node2Shell` vulnerability pointed out on #cloud-announce (https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/TCUZGJX6H4OYZKOIPNVNKB63LIPGQFG6/) : [12:05:03] ``` [12:05:04] If you host a Node project on Toolforge or Cloud VPS, you should [12:05:06] immediately confirm whether your project is running a vulnerable version of [12:05:07] Node, which would be any version previous to Node 22.23.4, 24.22.1 or [12:05:09] 26.12.1. You should also review these steps [2] to confirm whether your [12:05:10] project is currently vulnerable. If you determine that your [12:05:12] Toolforge-hosted tool or Cloud VPS project is affected by this issue, you [12:05:13] should disable it until WMCS can provide new, secure NodeJS images for your [12:05:15] service.``` [12:05:16] But as of today, as I am initing a new `category-downloader.toolforge.org` server, I ran into this : [12:05:18] ``` [12:05:19] $ toolforge webservice node24 shell [12:05:21] type must be one of: [12:05:22] bookworm [12:05:24] * buildservice [12:05:25] * jdk21 [12:05:27] * node18 [12:05:28] * node20 [12:05:30] * perl5.36 [12:05:31] * perl5.40 [12:05:33] * php8.2 [12:05:34] * php8.4 [12:05:36] * python3.11 [12:05:37] * python3.13 [12:05:39] * ruby3.1 [12:05:40] * ruby3.3 [12:05:45] * tcl8.6 [12:05:47] * trixie``` [12:39:07] On which exact URL? (re @wmtelegram_bot: I am getting an 'error 500' when trying to create a Wikipedia developer account.) [13:47:44] !log jeanfred@tools-bastion-15 tools.integraality Deploy a86b16b (Centralize the default Redis host in one place) [13:47:46] !log jeanfred@tools-bastion-15 tools.integraality Deploy 745f4da (Strip template param values when parsing config) for T440134 [13:47:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [13:47:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [15:37:31] !log jeanfred@tools-bastion-15 tools.integraality Deploy 54312eb (Extract GroupingConfiguration.with_grouping_type seam) [15:37:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [15:37:33] !log jeanfred@tools-bastion-15 tools.integraality Deploy 2e015ff (Add two databases tables for the dashboard configurations) for T440134 [15:37:35] !log jeanfred@tools-bastion-15 tools.integraality Deploy 665ad02 (Add configuration_id column to the dashboard_runs table) for T440134 [15:37:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [15:37:37] !log jeanfred@tools-bastion-15 tools.integraality Deploy db57dcf (Add canonical config serialization + config_hash) for T440134 [15:37:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [15:37:39] !log jeanfred@tools-bastion-15 tools.integraality Deploy fe2243a (Add live config writer at successful assembly) for T440134 [15:37:41] !log jeanfred@tools-bastion-15 tools.integraality Deploy 747acaa (Add current-config snapshot one-shot) for T440134 [15:37:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [15:37:43] !log jeanfred@tools-bastion-15 tools.integraality Deploy e700e45 (Link runs to configs via dashboard_runs.configuration_id) for T440134 [15:37:45] !log jeanfred@tools-bastion-15 tools.integraality Deploy 6c78812 (Add full-history config backfiller) for T440134 [15:37:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [15:37:47] !log jeanfred@tools-bastion-15 tools.integraality Deploy 5de28a0 (Cache grouping-type resolution in the backfiller) [15:37:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [15:37:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [15:37:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [15:37:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [16:31:00] mutante: any chance we can just delete deploy-1006.devtools.eqiad1.wikimedia.cloud? It's running on bullseye with oudated php, seems to be the last blocker to restoring post-php2shell normalcy to that project, as well as T429494 [16:31:05] T429494: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494 [16:33:14] andrewbogott: I can't make that call (alone). please ask Arnold who is currently assigned to that or my team as a whole and we have to give you a response later. [16:34:19] ok! [16:36:00] !log lucaswerkmeister-wmde@tools-bastion-15 tools.wmde-access tried to bring the tool back up under php8.4 but it now crashes trying to parse the puppet data.yaml file; webservice stopped again [16:36:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmde-access/SAL [16:39:53] mutante: is IRC the right place to talk to your team or should I ask on slack? [16:46:05] andrewbogott: since you are asking this way. async is the right way to go. sre-collaboration-services@wikimedia.org [16:46:57] ok [16:50:06] !log integration running puppet-git-sync-upstream manually on integration-puppetserver-01 to pick up 1356455 (containerd root beneath docker data-root) - T440670 [16:50:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/SAL [16:50:10] T440670: Upgrade integration-agent-docker instances to trixie - https://phabricator.wikimedia.org/T440670 [16:50:16] o/ had a question about node use on VMs. [16:50:53] debian trixie comes with node v20 and so I have two VMs with node v20 and i have stopped services on that given the node2shell incident. [16:50:59] and https://lists.wikimedia.org/hyperkitty/list/cloud@lists.wikimedia.org/thread/TCUZGJX6H4OYZKOIPNVNKB63LIPGQFG6/ says no node v20. [16:51:13] so, how are folks proceeding with this? [16:51:58] I was pondering these two options (a) install nvm and then use nvm to get node versions (b) install docker, fresh-node, and see if you can make the services work with fresh-node [16:52:27] But, if new secure node images might show up for WMCS, I can wait for a little bit to use that instead. [16:52:48] curious if there are any recommendations from WMCS folks / others who use node on your VMs. [16:52:50] thanks! :) [16:57:01] subbu: my current understanding is that if you have the debian-provided node version trixie, with the latest debian security updates for libbrotli1, you should be fine [16:58:34] okay .. so, I'll run 'sudo apt-get update && sudo apt-get upgrade' once more for good measure and move on. :) Thanks! [17:02:35] Do we need to send a clarifying followup to aputhin's email saying that? Or have we already? [17:18:28] We haven't I think, but we can [17:41:21] woohoo managed to get mediawiki running on toolforge by hacking out the version check for mariadb to allow the older version ;) [17:41:26] i'm sure it's fine