[07:56:07] greetings [07:58:30] taking a look at quarry [08:46:11] no joy so far in understanding what's wrong, other than "workers are not responding" [09:02:54] morning! [09:03:00] do you want another pair of eyes? [09:03:14] yes please, and thank you dcaro [09:03:17] (I have not yet gone around my email/backscroll) [09:03:42] I'm also not sure what I am supposed to do on the bastion with quarry.git cloned to get 'kubectl' to work [09:04:12] there should be a kube config file somewhere around added by tofu (or ansible, not sure) [09:05:09] usually under $REPO/tofu/kube.config [09:05:44] (not sure if that's what you are asking xd) [09:06:12] what's the problem though? [09:06:47] heh I'm not sure yet, anyways I'm prod oncall and need to look into a thing rn [09:06:59] ack [09:07:03] I'll give it a look [09:07:09] ack cheers [09:07:34] │ sqlite3.OperationalError: unable to open database file [09:07:46] is there a task somewhere? (I'll create a new one if not) [09:32:53] morning [09:46:24] so it seems quarry is having issues trying to get some? result databases (sqlite on NFS), I tried a random existing one and I was able to get it [09:46:29] https://www.irccloud.com/pastebin/29zhBBHF/ [09:46:50] so it's not an "all or nothing" type of issue [09:49:38] to help debugging: [09:49:38] https://github.com/toolforge/quarry/pull/110 [09:52:06] +1d but tox is complaining [09:53:17] ack, let me see [09:55:04] hmmm... failed deploying :/ [09:56:18] it tries to create an extra worker it seems :/ [09:59:23] it says 'UPDATE_FAILED' in horizon [09:59:25] https://usercontent.irccloud-cdn.com/file/ipYkPbU0/image.png [09:59:31] and does not allow resizing [10:00:14] anyhow, I was able to get one of the failing result files [10:00:18] /results/942/8965/78988.sqlite [10:00:21] it does not exist [10:03:06] everything is broken :D [10:06:26] xd [10:07:39] this should make it not crash, though does not fix why they are not there [10:07:40] https://github.com/toolforge/quarry/pull/111 [10:10:09] now it just logs it, but does not crash [10:10:14] https://www.irccloud.com/pastebin/dW43T5UY/ [10:11:27] it's quite annoying that tox is not running ok locally xd [10:14:16] created T443411 to follow up [10:14:17] T443411: [quarry] started failing when results directory does not exist - https://phabricator.wikimedia.org/T443411 [10:19:55] it seems more stable, but it's still failing some liveness checks for some reason [10:36:36] I think that there's also a bit of missalignment between the resources available, and the ones used for the web, and the liveness probes could be tweaked, I'll send another MR to updated those a bit [10:43:00] changed this manually in the cluster https://github.com/toolforge/quarry/pull/112 [10:43:05] so far seems way more stable, and faster [10:43:19] (there's no grafana for worker mem usage and such though :/) [10:43:58] oops, talked too fast xd [10:44:57] wait no, I had a 503 cached or something? :/ [10:46:06] oh, https://quarry.wmcloud.org/run/250706/output/12/csv gives error [10:46:14] (not sure if it should) [10:46:47] it probably should, resultset 1 does exist for that query run [10:46:54] should be nicer though xd [10:51:52] dhinus: thanks for the reviews :), there's still the "making the deploy work, and figure out why it started failing now", but that is less urgent [10:54:14] thank you for the fixes! I think a.ndrew redeployed quarry recently, so might be worth checking with him later today [10:54:31] ack, I'll ping him later. /me reading backlog, let me know if there's anything else I should look into first, otherwise might take me some time to catch up [11:31:12] quarry sent an alert again [14:02:39] quick review for ssh-yubikey change? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1361937 [14:03:47] {done} [14:04:58] volans: thanks! what do you mean add my nickname? [14:05:32] (I followed the guide that said to name it like that) [14:07:13] I haven't checked this specific file but usually the keys endup in a file one per line and without any name it's hard to know to who they belong [14:07:32] it's under my name `dcaro:` [14:08:30] that's in hiera, but ok the file is created with a separator [14:08:30] # volans [14:08:37] /etc/ssh/userkeys/root [14:08:41] so ok [14:09:38] ohh, in the destination file, ack [14:09:52] yep, but there is a separator with the name, so all good [15:14:10] apparently confd wants you to explicitely turn on authentication to the backend, otherwise it just silently ignores any credentials given to it [15:14:13] don't ask why i know [15:15:06] :D [15:30:01] /o\ [15:37:26] a user asked if we can update the list of tools running deprecated php versions https://wikitech.wikimedia.org/wiki/Talk:Incidents/2026-10-21_PHP_security_update/Community/Tools_running_deprecated_PHP_versions [15:37:43] dcaro: can you re-run the script you used to generate that list? [15:39:15] 👀 [15:40:42] fyi it's [15:40:43] kubectl get pods -A -o json | jq -r '.items[] | select(any(.spec.containers[]; .image | test("php8"; "i") == false and test("php"; "i") )) | [.metadata.namespace, .metadata.name, ([.spec.containers[].image] | join(", "))] | @tsv' [15:41:02] just looks for pods with images with php but not php8 [15:41:36] nice ^ [15:42:19] that shows the image too though (and matches sample-php-* xd) [15:44:55] looks like we're down to only 23 tools, did you get the same result? [15:46:05] yep [15:46:16] that seems suspicious [15:46:23] why would those Pod objects even exist at this point? [15:46:32] like, shouldn't the admission controller block them? [15:46:46] I would suspect you need to look at Deployment or CronJob objects instead? [15:53:55] taavi: not sure about the admission controller, but I think checking CronJob makes sense, I'll amend the command [15:56:09] if the deployment/cronjob existed already, then the pods will start getting scheduled, and fail due to the registry-admission not letting them get through [15:56:12] but they get created [15:56:28] registry-admission should stop the pod objects from being created entirely [15:56:45] it does not, it stops deployments/cronjobs [15:57:13] has it been changed since i last touched it? [15:57:17] let me double check, but that's the behavior I was seeing during the last tays [16:02:07] the pods show in Error for ~12d [16:52:46] since when cookbook dry-runs are not really dry-runs? [16:52:49] gitlab.exceptions.GitlabHttpError: 400: A branch called 'add_project_lingua-data-s3' already exists. Switch to that branch in order to make changes [16:53:02] I've only run a dry-run before [16:54:55] the gitlab branches and merge requests are created also in dry-run for the create_project cookbook, FYI [16:55:27] I've closed the MRs and deleted the branches [17:28:14] I have updated the list at https://wikitech.wikimedia.org/wiki/Incidents/2026-10-21_PHP_security_update/Community/Tools_running_deprecated_PHP_versions [17:28:39] and pasted the wrong command :( the one I used did also check cronjobs! [17:33:39] dhinus: the list is missing a bunch of tools, for example `ace2018` has a php5 Deployment but is missing from there [17:33:46] yes sorry I'm fixing it [17:35:09] I thought the new command gave me the same list, but actually I messed up the commands :/ [17:38:40] the command in the shell history of control-9 should have deployments and cronjobs [17:38:50] (the actual on I used xd, not the one for the pods) [17:40:05] https://www.irccloud.com/pastebin/fwMhIim8/ [17:40:50] I used a similar one and updated the page, let me compare it with your command [17:41:28] now the page has extra `tool-` prefixes in the links [17:44:07] taavi: sorry I'm tired, bad replacemente in VIM :) fixing [17:48:29] now should be better [17:48:39] ty! [17:48:48] now the diff is more useful https://wikitech.wikimedia.org/w/index.php?title=Incidents/2026-10-21_PHP_security_update/Community/Tools_running_deprecated_PHP_versions&curid=460894&diff=2476884&oldid=2475883 [17:51:19] nice! [17:53:24] I compared the output of my command (the one in the wiki page) and the one that dcaro pasted above (https://www.irccloud.com/pastebin/fwMhIim8/). my one returns exactly the same tools, plus 13 extra ones, that's because I also include jobs and pods [17:54:46] some jobs are old jobs that are no longer running so they should not be included actually [17:58:59] I think that the pods stuck in Error are there because they got stuck whet k8s was loading and kyverno webhook was not yet up and running somehow, and now can't be deleted correctly (there was a task for it somewhere, changing the image allows the pod to get updated and cleaned up) [18:16:41] updated the page again, the new script should exclude the old jobs and pods. 10 tools were removed from the list. [18:19:12] * dhinus off, see you tomorrow! [18:30:37] * dcaro off [18:30:39] cya! [18:41:41] echoing here too, I've started the first cleanup from https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull [21:19:03] FWIW last thursday I asked gemini to count the tools that were on that (deprecated php) page (oct 27 capture) and there were 554. It now see 557. So it seems the needle didn't really move [22:35:52] Hi all, not sure where this belongs; I have uploaded https://gerrit.wikimedia.org/r/c/operations/puppet/+/1362216 to switch intermediate CAs for services relying on the cloud PKI (well, Cloud VPS clients running Envoy and getting their leaf certificates from the "pki" Cloud VPS project). The current intermediate is going to expire in January 2027 and I only happened to discover this by "accident". [22:36:31] Not sure if there are Cloud VPS projects other than devtools and deployment-prep using the cloud PKI [22:38:16] Southparkfan: it was already tracked in https://phabricator.wikimedia.org/T442012 [22:39:02] volans: that seems to be a different intermediate CA used in the production PKI stack [22:40:35] ah ok, I thought that task was tracking also that [22:40:52] This `discovery` intermediate CA is part of the cfssl stack hosted in the `pki` Cloud VPS, and its certificate is going to expire on Jan 27 12:02:00 2027 GMT, so we have a bit more time. T442012 seems to be more urgent [22:40:52] T442012: The Cloud intermediate CA is going to expire on Dec 12th - https://phabricator.wikimedia.org/T442012 [22:41:04] err, `pki` Cloud VPS project [22:41:22] But yes, T442012 is very similar, just a different intermediate CA [22:42:37] thanks for reporting it [22:42:50] * volans gotta go, really late here [22:43:25] Cya