[01:02:35] (03close) 10raymond-ndibe: use __ENTRYPOINT__ keyword to support empty k8s command buildservice images [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/340 (https://phabricator.wikimedia.org/T431273) [01:04:01] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: jobs-api: allow default command for buildservice images - https://phabricator.wikimedia.org/T431273#12092786 (10Raymond_Ndibe) >>! In T431273#12090854, @dcaro wrote: > Another option: > * Use the `web` command for buildservice webservic... [01:04:23] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: jobs-api: allow default command for buildservice images - https://phabricator.wikimedia.org/T431273#12092788 (10Raymond_Ndibe) 05In progress→03Resolved [01:05:17] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: resolve some env variable differences between jobs-api and webservice-cli - https://phabricator.wikimedia.org/T431274#12092792 (10Raymond_Ndibe) 05Open→03Resolved a:03Raymond_Ndibe [01:06:17] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: HTTP health-check probes missing User-Agent headers - https://phabricator.wikimedia.org/T431275#12092797 (10Raymond_Ndibe) 05Open→03Resolved a:03Raymond_Ndibe [01:10:06] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: significant resources request/limit differences vs. webservice-cli - https://phabricator.wikimedia.org/T431281#12092801 (10Raymond_Ndibe) **Decision:** * this is a bit concerning, but we can ignore it until it becomes a bottleneck [01:10:36] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: significant resources request/limit differences vs. webservice-cli - https://phabricator.wikimedia.org/T431281#12092803 (10Raymond_Ndibe) 05Open→03Resolved a:03Raymond_Ndibe [01:10:48] 10Toolforge, 06tools-platform-team: [code,ci,general] 1st of the month updates - https://phabricator.wikimedia.org/T430995#12092808 (10Raymond_Ndibe) 05In progress→03Resolved [01:16:32] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api: resolve some env variable differences between jobs-api and webservice-cli - https://phabricator.wikimedia.org/T431274#12092822 (10Raymond_Ndibe) 05Resolved→03Open [02:04:19] (03open) 10raymond-ndibe: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) [02:05:11] (03update) 10raymond-ndibe: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) [02:23:28] (03update) 10raymond-ndibe: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) [03:18:55] (03update) 10raymond-ndibe: k8s/jobs.py: set PORT env whenever a job exposes a port [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/341 (https://phabricator.wikimedia.org/T431274) [03:45:43] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092) [04:06:59] (03merge) 10countcount: Update pnpm to v11.10.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/1 (owner: 10renovatebot) [04:07:23] (03merge) 10countcount: Tag releases and gate deploys on green CI [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/21 [04:08:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1071 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [04:10:13] (03merge) 10countcount: Tag releases and gate deploys on green CI [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/36 [04:11:32] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-32 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:11:36] FIRING: InstanceDown: Project toolsbeta instance toolsbeta-test-k8s-worker-nfs-8 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:12:10] FIRING: CloudVirtDown: Cloudvirt node cloudvirt1071 is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CloudVirtDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1071 - https://alerts.wikimedia.org/?q=alertname%3DCloudVirtDown [04:14:49] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 [04:15:17] (03update) 10raymond-ndibe: cleanup by image name [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 (owner: 10dcaro) [04:16:20] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/20 [04:17:23] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-32 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [04:17:24] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-worker-nfs-8 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [04:22:00] 06cloud-services-team, 10Cloud-VPS: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374 (10Andrew) 03NEW [04:22:10] RESOLVED: CloudVirtDown: Cloudvirt node cloudvirt1071 is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CloudVirtDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1071 - https://alerts.wikimedia.org/?q=alertname%3DCloudVirtDown [04:22:23] FIRING: [2x] ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-32 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [04:23:18] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1071.eqiad.wmnet' (T431374) [04:23:24] T431374: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374 [04:23:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1071 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [04:26:32] RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-32 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:26:36] RESOLVED: InstanceDown: Project toolsbeta instance toolsbeta-test-k8s-worker-nfs-8 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:27:23] RESOLVED: [2x] ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-32 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [04:27:24] RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-worker-nfs-8 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [04:27:39] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1071.eqiad.wmnet' (T431374) [04:33:19] (03update) 10raymond-ndibe: Replace black+isort with ruff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/339 (https://phabricator.wikimedia.org/T431305) (owner: 10tlepage) [04:33:20] (03approved) 10raymond-ndibe: Replace black+isort with ruff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/339 (https://phabricator.wikimedia.org/T431305) (owner: 10tlepage) [04:43:46] 06cloud-services-team, 10Cloud-VPS: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12092983 (10Andrew) And, just like T410470, a totally useless syslog during the downtime: ` 2026-07-07T04:05:10.149266+00:00 cloudvirt1071 neutron-openvswitch-agent: 2026-07-07 04:05:10.147 2094 INFO ne... [04:50:31] 06cloud-services-team, 10Cloud-VPS: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12092988 (10Andrew) For now: this host is in the maintenance aggregate and drained of all user VMs. [08:06:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [08:11:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:06:57] (03open) 10taavi: eqiad1: project-proxy: Add HAProxy exporter ports to security groups [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/319 (https://phabricator.wikimedia.org/T429960) [09:06:57] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/319 [09:07:19] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/319 [09:12:40] (03merge) 10taavi: eqiad1: project-proxy: Add HAProxy exporter ports to security groups [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/319 (https://phabricator.wikimedia.org/T429960) [09:13:06] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [09:14:20] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [09:23:36] FIRING: TargetDown: Job main-haproxy is unreachable in project project-proxy instance proxy-6 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:28:36] FIRING: [2x] TargetDown: Job main-haproxy is unreachable in project project-proxy instance proxy-5 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:48:36] RESOLVED: [2x] TargetDown: Job main-haproxy is unreachable in project project-proxy instance proxy-5 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [10:28:00] (03open) 10dcaro: refactor: move utils k8s specific methods to runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/342 [11:26:54] (03update) 10raymond-ndibe: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] (black-to-ruff) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) (owner: 10tlepage) [11:30:04] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T388092 https://phabricator.wikimedia.org/T423410) [11:31:46] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [11:32:28] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092) [11:33:30] (03open) 10samwilson: Improve CSS complexity test [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/14 (https://phabricator.wikimedia.org/T271595) [11:34:11] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [11:36:00] (03update) 10raymond-ndibe: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T431403) [11:36:39] (03update) 10raymond-ndibe: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T431403) [11:39:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [11:44:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [11:53:12] (03update) 10raymond-ndibe: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T431403) [12:01:50] (03update) 10raymond-ndibe: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T431403) [12:03:41] (03open) 10seanleong-wmde: commit message [toolforge-repos/wmde-github-phab-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-github-phab-bot/-/merge_requests/3 [12:03:47] (03merge) 10seanleong-wmde: commit message [toolforge-repos/wmde-github-phab-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-github-phab-bot/-/merge_requests/3 [12:11:51] (03open) 10seanleong-wmde: user link [toolforge-repos/wmde-github-phab-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-github-phab-bot/-/merge_requests/4 [12:12:03] (03merge) 10seanleong-wmde: user link [toolforge-repos/wmde-github-phab-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-github-phab-bot/-/merge_requests/4 [12:36:33] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T431410) [12:36:43] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T431410) [12:39:46] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Data-Services, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12094337 (10fgiunchedi) >>! In T411248#12084087, @fgiunchedi wrote: >>>! In T411248#12080727,... [12:44:24] (03open) 10dcaro: runtime: return unknown status on exception [repos/cloud/toolforge/jobs-api] (move_k8s_utils) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/343 [12:49:54] (03update) 10dcaro: refactor: move utils k8s specific methods to runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/342 [13:04:19] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T431410) [13:08:09] (03merge) 10ranjithsiji: Finalmodification [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/9 (owner: 10arpithhhaaa) [13:08:36] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T431410) [13:29:16] (03update) 10dcaro: runtime: return unknown status on exception [repos/cloud/toolforge/jobs-api] (move_k8s_utils) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/343 [13:39:11] (03merge) 10tlepage: Replace black+isort with ruff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/339 (https://phabricator.wikimedia.org/T431305) [13:39:12] (03update) 10tlepage: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) [13:42:38] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.527-20260707133922-840e7aad [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1321 (https://phabricator.wikimedia.org/T431305) [13:42:46] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.527-20260707133922-840e7aad [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1321 (https://phabricator.wikimedia.org/T431305) [13:47:11] 06cloud-services-team, 10Toolforge: [builds-builder] Pull upstream heroku images weekly - https://phabricator.wikimedia.org/T431125#12094551 (10DamianZaremba) This would be a significant improvement, as a data point comparing a (Python based) build (yesterday) that uses the Toolforge image vs the current upstr... [13:56:39] (03update) 10tlepage: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) [14:01:50] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad: New thermal paste for cloudvirt1071.eqiad.wmnet - https://phabricator.wikimedia.org/T431429 (10Andrew) 03NEW [14:01:57] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad: New thermal paste for cloudvirt1071.eqiad.wmnet - https://phabricator.wikimedia.org/T431429#12094665 (10Andrew) Oh -- it's out of service now, so y'all can switch it off whenever. [14:11:36] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 95.22%) at host paws-nfs-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [14:28:00] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge, 06tools-platform-team: DRAFT Decision request - Focus for improving lima-kilo developer experience - https://phabricator.wikimedia.org/T403051#12094853 (10dcaro) [14:28:06] 10Cloud Services Proposals, 06cloud-services-team, 06tools-platform-team: Decision request - Update and expand on coding standards for toolforge - https://phabricator.wikimedia.org/T408782#12094854 (10dcaro) [14:28:18] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge, 06tools-platform-team: [DRAFT] Decision Request - Initial product approach to integrating Toolforge UI functionality with Toolsadmin - https://phabricator.wikimedia.org/T393010#12094855 (10dcaro) [14:33:17] 06cloud-services-team, 10Cloud-VPS: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12094914 (10BLiviero-WMF) i would recommend bouncing it over to DC Ops and having them run a memtest to ensure all memory is ok and any other diagnostics / BIOS checks possible [14:33:42] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12094916 (10BLiviero-WMF) [14:47:42] (03update) 10raymond-ndibe: refactor: move utils k8s specific methods to runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/342 (owner: 10dcaro) [14:52:13] (03update) 10dcaro: runtime: return unknown status on exception [repos/cloud/toolforge/jobs-api] (move_k8s_utils) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/343 [15:03:59] (03update) 10dcaro: refactor: move utils k8s specific methods to runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/342 [15:09:09] (03update) 10dcaro: runtime: return unknown status on exception [repos/cloud/toolforge/jobs-api] (move_k8s_utils) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/343 [15:09:20] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work, 13Patch-For-Review: Fix description and example issues for transform paths in the MediaWiki REST API OAD (part 1) - https://phabricator.wikimedia.org/T428151#12095176 (10KBach) [15:11:04] (03update) 10raymond-ndibe: refactor: move utils k8s specific methods to runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/342 (owner: 10dcaro) [15:11:05] (03approved) 10raymond-ndibe: refactor: move utils k8s specific methods to runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/342 (owner: 10dcaro) [15:24:59] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:26:17] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: New thermal paste for cloudvirt1071.eqiad.wmnet - https://phabricator.wikimedia.org/T431429#12095333 (10Andrew) Let's do a memory scan on this server too, while it's out of service. Thanks! [15:37:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:37:24] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:51:20] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:52:05] (03merge) 10dcaro: jobs-api: bump to 0.0.527-20260707133922-840e7aad [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1321 (https://phabricator.wikimedia.org/T431305) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:00:23] (03merge) 10dcaro: refactor: move utils k8s specific methods to runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/342 [16:00:30] (03update) 10dcaro: runtime: return unknown status on exception [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/343 [16:03:15] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.528-20260707160044-7e31c643 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1322 (https://phabricator.wikimedia.org/T431385) [16:03:23] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.528-20260707160044-7e31c643 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1322 (https://phabricator.wikimedia.org/T431385) [16:13:55] (03update) 10raymond-ndibe: runtime: return unknown status on exception [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/343 (owner: 10dcaro) [16:14:13] (03update) 10raymond-ndibe: runtime: return unknown status on exception [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/343 (owner: 10dcaro) [16:14:16] (03approved) 10raymond-ndibe: runtime: return unknown status on exception [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/343 (owner: 10dcaro) [16:18:15] (03merge) 10dcaro: runtime: return unknown status on exception [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/343 [16:21:03] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.529-20260707161829-6741a688 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1322 (https://phabricator.wikimedia.org/T431385) [16:21:07] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.529-20260707161829-6741a688 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1322 (https://phabricator.wikimedia.org/T431385) [16:23:23] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:29:13] (03open) 10dcaro: fix date stuff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/344 [16:35:15] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [16:35:22] (03update) 10dcaro: fix date stuff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/344 [16:35:35] (03update) 10dcaro: status: parse restarted_at with different format [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/344 [16:35:36] (03update) 10dcaro: status: parse restarted_at with different format [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/344 [16:36:11] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:51:19] (03update) 10dcaro: status: parse restarted_at with different format [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/344 [16:51:33] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [16:54:22] (03approved) 10dcaro: jobs-api: bump to 0.0.529-20260707161829-6741a688 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1322 (https://phabricator.wikimedia.org/T431385) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:54:28] (03merge) 10dcaro: jobs-api: bump to 0.0.529-20260707161829-6741a688 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1322 (https://phabricator.wikimedia.org/T431385) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:55:33] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [16:55:37] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api [16:55:47] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [16:56:23] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [16:56:39] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:01:52] (03update) 10raymond-ndibe: status: parse restarted_at with different format [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/344 (owner: 10dcaro) [17:01:54] (03approved) 10raymond-ndibe: status: parse restarted_at with different format [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/344 (owner: 10dcaro) [17:05:09] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [17:05:53] (03open) 10dcaro: bump_version: improve the message when bumping the version [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/142 [17:07:52] (03update) 10dcaro: bump_version: improve the message when bumping the version [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/142 [17:08:16] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:08:23] (03update) 10dcaro: bump_version: improve the message when bumping the version [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/142 [17:08:30] (03update) 10dcaro: bump_version: improve the message when bumping the version [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/142 [17:17:22] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [17:33:29] (03PS1) 10Majavah: views: Use singular backend property [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1308167 (https://phabricator.wikimedia.org/T429960) [17:35:50] !log andrew@cloudcumin1001 puppet-diffs START - Cookbook wmcs.nfs.add_server [17:38:11] !log andrew@cloudcumin1001 puppet-diffs END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97) [17:38:54] !log andrew@cloudcumin1001 dumps START - Cookbook wmcs.nfs.add_server [17:47:55] !log andrew@cloudcumin1001 dumps END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) [17:50:39] 10Tool-humaniki-2: Fix link to contrib guide in readme and update version workflow - https://phabricator.wikimedia.org/T428258#12096257 (10Danya) 05Open→03In progress [17:51:21] 10Tool-humaniki-2: Fix link to contribution guide - https://phabricator.wikimedia.org/T427394#12096264 (10Danya) 05Open→03In progress [17:52:20] !log andrew@cloudcumin1001 dumps START - Cookbook wmcs.nfs.migrate_service [17:52:29] !log andrew@cloudcumin1001 dumps END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) [17:54:37] !log andrew@cloudcumin1001 dumps START - Cookbook wmcs.nfs.migrate_service [17:57:07] !log andrew@cloudcumin1001 dumps END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0) [17:57:24] 10Tool-humaniki-2: Fix link to contrib guide in readme and update version workflow - https://phabricator.wikimedia.org/T428258#12096312 (10Danya) Done directly on `develop` branch. [17:57:41] 10Tool-humaniki-2: Fix link to contrib guide in readme and update version workflow - https://phabricator.wikimedia.org/T428258#12096314 (10Danya) 05In progress→03Resolved [18:00:26] 10Tool-humaniki-2: Fix link to contribution guide - https://phabricator.wikimedia.org/T427394#12096349 (10Danya) Done directly on `develop` branch. [18:00:35] 10Tool-humaniki-2: Fix link to contribution guide - https://phabricator.wikimedia.org/T427394#12096361 (10Danya) 05In progress→03Resolved [18:00:41] 10Tool-humaniki-2: Missing vertical axis units in bar charts - https://phabricator.wikimedia.org/T428051#12096364 (10Danya) 05Open→03In progress [18:04:24] 06cloud-services-team: Dumps: replace Bullseye nfs server with Trixie - https://phabricator.wikimedia.org/T430590#12096374 (10Andrew) this is done. The old server dumps-nfs-1 is shutdown and can be deleted after a few days of good health. [18:13:39] 10Tool-humaniki-2: Missing vertical axis units in bar charts - https://phabricator.wikimedia.org/T428051#12096419 (10Danya) Done directly on `develop` branch. [18:13:51] 10Tool-humaniki-2: Missing vertical axis units in bar charts - https://phabricator.wikimedia.org/T428051#12096422 (10Danya) 05In progress→03Resolved [18:28:32] (03update) 10tlepage: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) [18:29:43] 10Tool-humaniki-2: Rename project - https://phabricator.wikimedia.org/T431186#12096488 (10Danya) 05Open→03In progress [18:33:06] (03update) 10tlepage: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) [18:33:59] (03update) 10tlepage: runtimes/k8s: fix flaky test_get_one_off_job_status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/338 (https://phabricator.wikimedia.org/T430550) [18:38:49] (03PS1) 10Btullis: Add dummy keytabs for new an-test nodes. [labs/private] - 10https://gerrit.wikimedia.org/r/1308182 (https://phabricator.wikimedia.org/T406746) [18:39:08] (03CR) 10Btullis: [V:03+2 C:03+2] Add dummy keytabs for new an-test nodes. [labs/private] - 10https://gerrit.wikimedia.org/r/1308182 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [19:38:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [20:08:49] 10Tool-wiki-gender-stats: Show current page name in tab title - https://phabricator.wikimedia.org/T431490 (10Danya) 03NEW [20:08:51] 10Tool-wiki-gender-stats: Show current page name in tab title - https://phabricator.wikimedia.org/T431490#12096947 (10Danya) p:05Triage→03Medium [20:21:24] 10Tool-wiki-gender-stats: Rename project - https://phabricator.wikimedia.org/T431186#12097051 (10Danya) Done: * Phabricator project has been renamed. * Tool has been migrated to its new location on Toolforge: `/data/project/wiki-gender-stats/` * GitLab repository has been renamed: https://gitlab.wikimedia.org/t... [20:26:16] (03open) 10danyya: Draft: Resolve T431186 "Rename project" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/68 [20:38:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [20:48:47] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: New thermal paste for cloudvirt1071.eqiad.wmnet - https://phabricator.wikimedia.org/T431429#12097198 (10VRiley-WMF) a:03VRiley-WMF [21:04:04] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12097230 (10VRiley-WMF) a:03VRiley-WMF [21:09:54] (03update) 10danyya: Draft: Resolve T431186 "Rename project" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/68 [21:13:08] (03update) 10danyya: Resolve T431186 "Rename project" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/68 [21:13:26] (03merge) 10danyya: Resolve T431186 "Rename project" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/68 [21:15:11] 10Tool-wiki-gender-stats, 13Patch-For-Review: Rename project - https://phabricator.wikimedia.org/T431186#12097256 (10Danya) 05In progress→03Resolved [21:16:08] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 [21:16:09] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 [21:17:06] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [21:17:12] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [21:18:32] 10Cloud-VPS, 06tools-infrastructure-team: Rebalance WMCS instance backups between the two hosts - https://phabricator.wikimedia.org/T430018#12097259 (10BLiviero-WMF) What is now the time for backups completion between the two servers? [21:33:53] 06cloud-services-team: Update all cloud-vps Bullseye NFS servers - https://phabricator.wikimedia.org/T429793#12097287 (10Andrew) [21:48:32] FIRING: InstanceDown: Project gitlab-runners instance runner-1035 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [22:02:59] (03PS1) 10Alien4444: UserRights: fix removal of redundant auto-removals [labs/xtools] - 10https://gerrit.wikimedia.org/r/1308215 [22:05:47] (03open) 10aghirelli: chore: update ruleset version by pointing to the NPM package [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/26 [22:06:03] (03merge) 10aghirelli: chore: update ruleset version by pointing to the NPM package [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/26 [22:12:14] (03CR) 10Alien4444: [C:03+2] AppExtension: cache git hash shown in footer of every page [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307265 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [22:13:18] (03Merged) 10jenkins-bot: AppExtension: cache git hash shown in footer of every page [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307265 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [22:15:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 [22:15:50] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [22:23:55] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work: Deploy the OpenAPI linter in CI - https://phabricator.wikimedia.org/T422920#12097375 (10AGhirelli-WMF) 05Open→03In progress [22:24:02] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work, 13Patch-For-Review: Publish the Wikimedia Spectral ruleset to NPM - https://phabricator.wikimedia.org/T425627#12097380 (10AGhirelli-WMF) 05In progress→03Resolved [22:35:08] 06cloud-services-team, 10Horizon: Rebuild Horizon with new blubber version - https://phabricator.wikimedia.org/T430164#12097422 (10Andrew) 05Open→03Resolved this needs more testing before I ship it to eqiad1, but the specific blubber update is done. [22:37:41] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Make webservice-cli use jobs-api - https://phabricator.wikimedia.org/T431421#12097427 (10bd808) > `webservice shell` stays as it is, unless we add support for shelling into containers in jobs-api too. {T431276} [23:15:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 [23:16:27] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [23:48:32] RESOLVED: InstanceDown: Project gitlab-runners instance runner-1035 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [23:49:07] 06cloud-services-team, 10Openstack-Magnum: debian packaging for magnum-cluster-api - https://phabricator.wikimedia.org/T426431#12097589 (10Andrew) 05Open→03Resolved [23:50:54] 06cloud-services-team, 10Openstack-Magnum: debian packaging for magnum-cluster-api - https://phabricator.wikimedia.org/T426431#12097592 (10Andrew)