[00:00:25] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107#10750539 (10bd808) 05Resolved→03Open [00:00:38] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107#10750541 (10bd808) [00:00:40] 10Quarry: [bug] Quarry queries don't run - https://phabricator.wikimedia.org/T392169#10750544 (10bd808) →14Duplicate dup:03T392107 [00:01:15] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107#10750549 (10bd808) Down again per {T392169}. The magical restart fix did not hold unfortunately. [00:12:14] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107#10750571 (10bd808) `lang=shell-session bd808@laptop$ ssh root@quarry-bastion.quarry.eqiad1.wikimedia.cloud root@quarry-bastion:~# export KUBECONFIG=/home/rook/quarry/tofu/kube.c... [00:23:06] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107#10750590 (10bd808) `kubectl delete pod -n quarry --all` seems to be the magic temporary fix. I'm going to assume that there is something out of the normal that keeps getting sta... [00:28:44] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107#10750678 (10bd808) @Liz Things should be working again. I just ran a query myself on the system. I am going to leave this task open for now since the last "victory" only lasted... [00:36:08] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" - https://phabricator.wikimedia.org/T392107#10750695 (10bd808) The redis pod's storage looks good following the pod restart: `lang=shell-session root@quarry-bastion:~# kubectl -n quarry exec -it pod/redis-676b955f95-gn4d9... [00:36:45] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" due to redis pod running out of disk space - https://phabricator.wikimedia.org/T392107#10750696 (10bd808) [00:37:31] 06cloud-services-team, 10Quarry: No alerting for quarry - https://phabricator.wikimedia.org/T392138#10750697 (10bd808) [00:37:32] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" due to redis pod running out of disk space - https://phabricator.wikimedia.org/T392107#10750698 (10bd808) [00:37:37] 06cloud-services-team, 10Quarry: Update quarry redis deployment - https://phabricator.wikimedia.org/T392141#10750699 (10bd808) [00:37:43] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" due to redis pod running out of disk space - https://phabricator.wikimedia.org/T392107#10750700 (10bd808) [00:37:50] 06cloud-services-team, 10Quarry: Quarry: Why so many web pods? - https://phabricator.wikimedia.org/T392143#10750701 (10bd808) [00:37:57] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" due to redis pod running out of disk space - https://phabricator.wikimedia.org/T392107#10750702 (10bd808) [00:45:13] 06cloud-services-team, 10Quarry, 07Documentation: [[wikitech:Portal:Data Services/Admin/Quarry]] documents legacy Quarry setup - https://phabricator.wikimedia.org/T392181 (10bd808) 03NEW [01:05:07] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" due to redis pod running out of disk space - https://phabricator.wikimedia.org/T392107#10750725 (10Liz) I think this may be resolved. [01:32:47] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" due to redis pod running out of disk space - https://phabricator.wikimedia.org/T392107#10750743 (10Liz) Thank you, much appreciated. This situation seems to happen every 6-9 months. But, luckily, not too often. Thanks again. [03:00:53] (03update) 10raymond-ndibe: Draft: [envvars-api] fix envvars-api EnvvarName regex bug [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/54 (https://phabricator.wikimedia.org/T391966) [03:01:28] (03update) 10raymond-ndibe: harbor: start on VM reboot [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/236 (owner: 10dcaro) [03:01:30] (03approved) 10raymond-ndibe: harbor: start on VM reboot [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/236 (owner: 10dcaro) [05:10:58] 06cloud-services-team, 10Quarry: No alerting for quarry - https://phabricator.wikimedia.org/T392138#10750857 (10taavi) a:03taavi [05:32:19] 06cloud-services-team, 10Quarry: No alerting for quarry - https://phabricator.wikimedia.org/T392138#10750862 (10github-toolforge-bot) supertassu opened https://github.com/toolforge/quarry/pull/74 [05:45:54] 06cloud-services-team, 10Toolforge: Bot is restarting continuously - https://phabricator.wikimedia.org/T392188 (101997kB) 03NEW [05:54:03] 06cloud-services-team, 10Toolforge: Bot is restarting continuously - https://phabricator.wikimedia.org/T392188#10750893 (10taavi) 05Open→03Invalid This is https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Health_checks working as intended. If your tool does not expose a web thing on $PORT, [[ http... [06:06:44] 06cloud-services-team, 10Quarry: Update quarry redis deployment - https://phabricator.wikimedia.org/T392141#10750937 (10github-toolforge-bot) supertassu opened https://github.com/toolforge/quarry/pull/75 [06:07:06] 06cloud-services-team, 10Quarry: Update quarry redis deployment - https://phabricator.wikimedia.org/T392141#10750938 (10taavi) a:03taavi [07:11:05] 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" due to redis pod running out of disk space - https://phabricator.wikimedia.org/T392107#10750989 (10dcaro) Pods are failing to run on node1 again, it seems to have gotten out of inodes: ` dcaro@quarry-bastion:~$ kubectl ge... [07:13:40] !log dcaro@acme quarry START - Cookbook wmcs.openstack.cloudvirt.vm_console [07:13:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [07:13:48] !log dcaro@acme quarry END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [07:13:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [07:13:58] !log dcaro@acme quarry START - Cookbook wmcs.openstack.cloudvirt.vm_console [07:13:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [07:14:03] !log dcaro@acme quarry END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [07:14:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [07:14:10] !log dcaro@acme quarry START - Cookbook wmcs.openstack.cloudvirt.vm_console [07:14:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [07:16:04] !log dcaro@acme quarry END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [07:16:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [07:24:46] !log dcaro@acme quarry START - Cookbook wmcs.openstack.cloudvirt.vm_console [07:24:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [07:47:00] (03merge) 10dcaro: harbor: start on VM reboot [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/236 [08:54:03] (03update) 10fnegri: start-devenv: don't ask if you want to edit config [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/237 [08:55:30] (03merge) 10fnegri: start-devenv: don't ask if you want to edit config [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/237 [09:07:34] (03close) 10fnegri: toolforge: introduce /etc/openstack/clouds.yaml [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/228 (https://phabricator.wikimedia.org/T379030) (owner: 10aborrero) [09:11:19] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [envvars-api] bug in envvars-api EnvvarName validation Regex - https://phabricator.wikimedia.org/T391966#10751285 (10Raymond_Ndibe) a:03Raymond_Ndibe [09:16:43] 06cloud-services-team, 10Cloud-VPS: tofuinfratest creates many pages in wikitech - https://phabricator.wikimedia.org/T376888#10751289 (10taavi) 05Open→03Resolved a:03taavi I'm declaring this fixed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1114111. [09:20:05] 06cloud-services-team, 10Cloud-VPS: Error parsing "/var/lib/prometheus/node.d/node_cloudvirt_libvirt_stats.prom" - https://phabricator.wikimedia.org/T289563#10751295 (10taavi) 05Open→03Resolved No longer seems to be a problem, closing. [09:20:59] 06cloud-services-team, 10Cloud-VPS: Error parsing "/var/lib/prometheus/node.d/node_cloudvirt_libvirt_stats.prom" - https://phabricator.wikimedia.org/T289563#10751297 (10taavi) 05Resolved→03Open a:03taavi Actually, no, this is still there on cloudvirtlocals. [09:24:33] (03update) 10raymond-ndibe: [jobs-api] delete completed one-off job of same name in create_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/157 (https://phabricator.wikimedia.org/T352989) [09:26:05] (03update) 10raymond-ndibe: [jobs-cli] health_check and quota refactor [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T389118) [09:48:24] 10wikitech.wikimedia.org, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 10Wikimedia-Interwiki-links, 13Patch-For-Review: Enable interwiki links to/from Wikitech - https://phabricator.wikimedia.org/T290147#10751338 (10JoelyRooke-WMDE) @taavi As the [[ https://phabricator.wikimedia.org/project/... [09:56:07] (03update) 10raymond-ndibe: Draft: [envvars-api] fix envvars-api EnvvarName regex bug [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/54 (https://phabricator.wikimedia.org/T391966) [10:21:02] 06cloud-services-team, 10Quarry: No alerting for quarry - https://phabricator.wikimedia.org/T392138#10751390 (10github-toolforge-bot) supertassu closed https://github.com/toolforge/quarry/pull/74 [10:24:16] (03update) 10raymond-ndibe: Draft: [envvars-api] fix envvars-api EnvvarName regex bug [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/54 (https://phabricator.wikimedia.org/T391966) [10:25:08] 06cloud-services-team, 10Quarry: Update quarry redis deployment - https://phabricator.wikimedia.org/T392141#10751415 (10github-toolforge-bot) supertassu closed https://github.com/toolforge/quarry/pull/75 [10:26:35] 06cloud-services-team, 10Quarry: Quarry: Why so many web pods? - https://phabricator.wikimedia.org/T392143#10751416 (10github-toolforge-bot) supertassu opened https://github.com/toolforge/quarry/pull/76 [10:28:35] 06cloud-services-team, 10Quarry: Quarry: Why so many web pods? - https://phabricator.wikimedia.org/T392143#10751417 (10taavi) a:03taavi [10:53:02] 06cloud-services-team, 10Quarry: Quarry: Why so many web pods? - https://phabricator.wikimedia.org/T392143#10751491 (10github-toolforge-bot) supertassu closed https://github.com/toolforge/quarry/pull/76 [11:03:13] !log dcaro@acme quarry END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [11:03:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [11:03:36] 06cloud-services-team, 10Quarry: Quarry: Why so many web pods? - https://phabricator.wikimedia.org/T392143#10751517 (10taavi) 05Open→03Resolved [11:03:53] 06cloud-services-team, 10Toolforge, 03Wikimedia-Hackathon-2025: [Session] Introducing and exploring Toolforge UI with prospective users - https://phabricator.wikimedia.org/T383149#10751519 (10Sarai-WMF) Hey @debt! After discussing this further, we'd like to rather use an Unconference slot for the "Introducti... [11:07:02] 06cloud-services-team, 10Quarry: No alerting for quarry - https://phabricator.wikimedia.org/T392138#10751526 (10taavi) 05Open→03Resolved ` MariaDB [prometheusconfig]> insert into alerts (project_id, name, expr, duration, severity, annotations) values (16, 'Down', 'up{project="quarry",job="app"} == 0',... [11:08:21] 06cloud-services-team, 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10751542 (10Ladsgroup) I'd appericiate it you do it. My plate has been overflowing :( I can make the patch. The exact change is that `ipblocks` and `ipblocks_compat... [12:25:43] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/17 [12:25:43] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/34 [12:44:36] 06cloud-services-team, 10Quarry: Update quarry redis deployment - https://phabricator.wikimedia.org/T392141#10751742 (10taavi) The merged patch migrated the deployment to a replicaset, and added an emptyDir volume for the data directory to use. If we care that data enough (job queue + sessions) I guess we coul... [13:45:29] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 13Patch-For-Review: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10752035 (10fnegri) 05Open→03In progress a:03fnegri Thanks @Ladsgroup I'll send an email to cloud-announce and think... [13:47:01] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 13Patch-For-Review: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10752040 (10fnegri) Realized June 1st is a Sunday, so maybe June 2nd is a better date :) [13:48:17] (03update) 10dcaro: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] (move_most_custom_validations_out_of_api_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T389118) (owner: 10raymond-ndibe) [13:50:12] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 13Patch-For-Review: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10752055 (10Ladsgroup) Sounds good to me. Some documentation on this: https://www.mediawiki.org/wiki/Manual:Block_table an... [14:04:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-meta-monitor-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:26:16] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10752207 (10fnegri) > We should comp air inlet air temps from any neighboring device before assuming that is bad though. The next two servers in t... [14:33:59] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad, 06SRE: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10752236 (10fnegri) p:05High→03Medium [14:34:07] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad, 06SRE: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10752237 (10fnegri) 05Stalled→03In progress [15:12:11] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] Remove apt pinning and upgrade to latest version - https://phabricator.wikimedia.org/T385885#10752410 (10fnegri) 05In progress→03Resolved a:05fnegri→03taavi [15:54:15] (03open) 10jeanfred: Use 'mul' default value for label when creating tracks and compositions [toolforge-repos/multitrack-drafting] - 10https://gitlab.wikimedia.org/toolforge-repos/multitrack-drafting/-/merge_requests/1 (https://phabricator.wikimedia.org/T392139) [16:02:54] 06cloud-services-team, 10Toolforge: [lima-kilo] when using "--ha", some containers are not restarting after restarting the VM - https://phabricator.wikimedia.org/T385082#10752704 (10fnegri) a:05fnegri→03None [16:08:20] (03approved) 10dcaro: [jobs-api] delete completed one-off job of same name in create_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/157 (https://phabricator.wikimedia.org/T352989) (owner: 10raymond-ndibe) [16:21:13] (03merge) 10dcaro: T389694 - rework envvar parsing to keep text after an additional '=' character [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/70 (owner: 10don-vip) [16:22:52] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.130-20250417162126-9d5e4c89 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/749 [16:23:25] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [16:25:06] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder [16:25:06] (03update) 10dcaro: [jobs-api] delete completed one-off job of same name in create_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/157 (https://phabricator.wikimedia.org/T352989) (owner: 10raymond-ndibe) [16:25:42] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [envvars-api] bug in envvars-api EnvvarName validation Regex - https://phabricator.wikimedia.org/T391966#10752805 (10dcaro) p:05Triage→03Medium [16:28:14] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [16:38:49] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [16:45:12] (03update) 10dcaro: [envvars-cli] hide envvar if truncate true [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/78 (https://phabricator.wikimedia.org/T363544) (owner: 10raymond-ndibe) [16:45:26] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [16:46:23] 06cloud-services-team, 10Toolforge: [builds-api,builds-service,builds-cli] toolforge build --envvar does not accept values containing equals character - https://phabricator.wikimedia.org/T389694#10752923 (10dcaro) a:03dcaro [16:52:15] 06cloud-services-team, 10Toolforge: [functional-tests,deploy,cookbook] Run only selected tests when deploying a component - https://phabricator.wikimedia.org/T381011#10752960 (10dcaro) >>! In T381011#10736390, @Raymond_Ndibe wrote: > @dcaro Is the priority here really `medium`? I know that we can use `--filter... [16:54:06] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): Environment variables are not being passed - https://phabricator.wikimedia.org/T390845#10752963 (10dcaro) [16:57:12] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [17:01:48] (03approved) 10dcaro: builds-builder: bump to 0.0.130-20250417162126-9d5e4c89 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/749 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:01:51] (03update) 10dcaro: builds-builder: bump to 0.0.130-20250417162126-9d5e4c89 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/749 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:01:52] (03merge) 10dcaro: builds-builder: bump to 0.0.130-20250417162126-9d5e4c89 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/749 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:02:30] 06cloud-services-team, 10Toolforge: [builds-api,builds-service,builds-cli] toolforge build --envvar does not accept values containing equals character - https://phabricator.wikimedia.org/T389694#10753014 (10dcaro) This has been deployed :~) thanks for the fixes!! I'll close, but if you still find issues pleas... [17:02:42] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [builds-api,builds-service,builds-cli] toolforge build --envvar does not accept values containing equals character - https://phabricator.wikimedia.org/T389694#10753015 (10dcaro) [17:02:55] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [builds-api,builds-service,builds-cli] toolforge build --envvar does not accept values containing equals character - https://phabricator.wikimedia.org/T389694#10753017 (10dcaro) 05Open→03Resolved [17:19:15] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): Environment variables are not being passed - https://phabricator.wikimedia.org/T390845#10753071 (10dcaro) Some notes from a quick check: The envvar is there when executing inside the container, and it's kept even after running the `launcher` wrapper... [17:22:45] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): Environment variables are not being passed - https://phabricator.wikimedia.org/T390845#10753107 (10dcaro) I think I found something, the command that the webservice runs is actually set by the golang buildpack, not the procfile one it seems, might be... [19:02:45] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/17 (owner: 10l10n-bot) [19:02:48] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/17 (owner: 10l10n-bot) [19:03:42] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/34 (owner: 10l10n-bot) [19:03:44] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/34 (owner: 10l10n-bot) [20:41:08] 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T388928#10753811 (10LibUp-bot) A new upstream version of OpenRefine is now available: 3.9.3. * https://github.com/OpenRefine/OpenRefine/releases/tag/3.9.3 [21:59:28] 06cloud-services-team, 10Toolforge, 03Wikimedia-Hackathon-2025: [Session] Introducing and exploring Toolforge UI with prospective users - https://phabricator.wikimedia.org/T383149#10754012 (10debt) Got it - it's been moved to 10am on Saturday. :)