[00:16:29] FIRING: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:21:29] RESOLVED: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:06:39] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:11:39] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:39:36] 10Toolforge (Toolforge iteration 14): [jobs-api] move jobs load feature to the backend - https://phabricator.wikimedia.org/T366209#10041185 (10dcaro) [07:39:48] 10Toolforge (Toolforge iteration 14): [jobs-cli] enforce proper validation for load jobs before calculate_changes - https://phabricator.wikimedia.org/T366211#10041187 (10dcaro) [07:40:01] 06cloud-services-team, 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [infra,k8s] prepare deb packages for k8s 1.26 - https://phabricator.wikimedia.org/T370246#10041183 (10dcaro) [07:40:14] 10Toolforge (Toolforge iteration 14): [toolforge] simplify calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377#10041193 (10dcaro) [07:40:23] 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: `webservice` requires effective user to be the tool user and listed in NSS passwd data - https://phabricator.wikimedia.org/T369569#10041189 (10dcaro) [07:41:01] 10Toolforge (Toolforge iteration 14), 07Upstream: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#10041195 (10dcaro) [07:41:04] 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [jobs-api] Remove authentication and use the api-gateway provided headers - https://phabricator.wikimedia.org/T367180#10041181 (10dcaro) [07:41:11] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 14), 05Goal: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#10041201 (10dcaro) [07:42:07] 10Toolforge (Toolforge iteration 14), 07Upstream: [builds-builder,jobs-api,upstream] Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016#10041197 (10dcaro) [07:42:08] 06cloud-services-team, 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#10041191 (10dcaro) [07:42:17] 10Toolforge (Toolforge iteration 14), 13Patch-For-Review, 07Upstream: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417#10041207 (10dcaro) [07:43:11] 10Toolforge (Toolforge iteration 14), 07Documentation: [harbor,docs] Improve Harbor quota handling and docs - https://phabricator.wikimedia.org/T351092#10041199 (10dcaro) [07:43:41] 06cloud-services-team, 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: Toolforge: Replace all bastion with grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665#10041203 (10dcaro) [07:43:42] 10Toolforge (Toolforge iteration 14): [builds-cli,builds-api] `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701#10041209 (10dcaro) [07:43:55] 06cloud-services-team, 10Toolforge (Toolforge iteration 14): toolforge: Refresh certs that are not controlled by kubeadm (mid 2024 edition) - https://phabricator.wikimedia.org/T309782#10041205 (10dcaro) [07:44:51] 06cloud-services-team, 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [lima-kilo, k8s] Upgrade Kubernetes in lima-kilo to version 1.26 - https://phabricator.wikimedia.org/T370244#10041215 (10dcaro) [07:45:12] 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [builds-api,jobs-api,envvars-api,api-gateway] Figure out and document how to do non-backwards compatible changes - https://phabricator.wikimedia.org/T356974#10041217 (10dcaro) [07:45:20] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14): [components-api] Get a skeleton of API webservice and implement `/tool//deploy` with build-only features - https://phabricator.wikimedia.org/T362069#10041213 (10dcaro) [07:46:32] 10Toolforge (Toolforge iteration 14): envvars-api 0.0.50 depends on unreleased envvars-cli changes - https://phabricator.wikimedia.org/T367961#10041223 (10dcaro) [07:46:48] 10Toolforge (Toolforge iteration 14): Toolforge Aptfile not producing working copy of `ffmpeg` - https://phabricator.wikimedia.org/T365633#10041221 (10dcaro) [07:46:52] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [sct.frontend] Show the backend status - https://phabricator.wikimedia.org/T370324#10041211 (10dcaro) [07:47:13] 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [jobs-api] Split the API, business, and k8s models - https://phabricator.wikimedia.org/T359808#10041229 (10dcaro) [07:48:00] 10Toolforge (Toolforge iteration 14): [toolforge-weld] support back python 3.7 - https://phabricator.wikimedia.org/T370932#10041225 (10dcaro) [07:48:03] 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [jobs-api] Save business models in a DB - https://phabricator.wikimedia.org/T359650#10041219 (10dcaro) [07:48:10] 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#10041231 (10dcaro) [07:48:15] 06cloud-services-team, 10Toolforge (Toolforge iteration 14): [infra,k8s,monitoring] Add an alert to warn when the prometheus k8s cert is about to expire - https://phabricator.wikimedia.org/T366579#10041235 (10dcaro) [07:48:53] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14), 07Epic: [Hypothesis] WE6.3.1 Consulting Toolforge roots/maintainers - https://phabricator.wikimedia.org/T368601#10041237 (10dcaro) [07:48:57] 10Toolforge (Toolforge iteration 14): [k8s,infra,cookbook] change the hiera under the -k8s-control prefix whet adding/removing an etcd node - https://phabricator.wikimedia.org/T371370#10041241 (10dcaro) [07:49:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 14): [infra,k8s] remove deprecated kubelet flags before 1.27 upgrade - https://phabricator.wikimedia.org/T370245#10041242 (10dcaro) [07:49:09] 10Toolforge (Toolforge iteration 14): [gateway-api] something is caching the openapi docs - https://phabricator.wikimedia.org/T371033#10041243 (10dcaro) [07:49:11] 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [k8s] Add node anti-affinity topologySpreadConstraints to infrastructure components where relevant - https://phabricator.wikimedia.org/T358203#10041227 (10dcaro) [07:49:13] 10Toolforge (Toolforge iteration 14): [harbor] Investigate how to deactivate wal from trove for postrges databases - https://phabricator.wikimedia.org/T370845#10041244 (10dcaro) [07:49:15] 10Toolforge (Toolforge iteration 14): [toolforge-cli,jobs-cli,builds-cli,envvars-cli] Explore OpenAPI SDK tooling for client consolidation - https://phabricator.wikimedia.org/T356261#10041245 (10dcaro) [07:49:17] 10Toolforge (Toolforge iteration 14): [harbor] 2024-07-24 Tools harbor db out of space - https://phabricator.wikimedia.org/T370843#10041246 (10dcaro) [07:49:21] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14): [sct.backend] Create worker and connect to redis - https://phabricator.wikimedia.org/T370321#10041248 (10dcaro) [07:49:23] 06cloud-services-team, 10Toolforge (Toolforge iteration 14): toolforge: puppetserver got OOMkilled - https://phabricator.wikimedia.org/T369797#10041247 (10dcaro) [07:49:25] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14), 07Epic: [Hypothesis] WE6.3.2 Create "standard" tool (Sample Complex Tool, SCT) to measure the number of steps for a deployment - https://phabricator.wikimedia.org/T368602#10041239 (10dcaro) [07:49:26] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14): [sct.backend] Create trove database - https://phabricator.wikimedia.org/T370317#10041249 (10dcaro) [07:49:27] 10Toolforge (Toolforge iteration 14): [toolforge deploy] direct-api tests fail intermittently on toolsbeta - https://phabricator.wikimedia.org/T369891#10041250 (10dcaro) [07:49:31] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 14): Intermittent redis connection timeouts in Toolforge - https://phabricator.wikimedia.org/T318479#10041233 (10dcaro) [07:49:35] 10Toolforge (Toolforge iteration 14): [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images - https://phabricator.wikimedia.org/T369693#10041251 (10dcaro) [07:49:40] 10Toolforge (Toolforge iteration 14), 07Epic: [jobs-cli,builds-cli,toolforge-cli,webservice] Consolidate the Toolforge CLIs - https://phabricator.wikimedia.org/T356262#10041253 (10dcaro) [07:49:43] 06cloud-services-team, 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#10041252 (10dcaro) [07:49:47] 06cloud-services-team, 10Toolforge (Toolforge iteration 14): [api-gateway] add alert for uptime - https://phabricator.wikimedia.org/T348633#10041254 (10dcaro) [07:49:52] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge (Toolforge iteration 14), 07Epic, 05Goal: Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#10041256 (10dcaro) [08:08:14] (03CR) 10David Caro: [C:03+2] ceph.wait_for_rebalance: new handy cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054510 (owner: 10David Caro) [08:11:30] (03update) 10dcaro: run_functional_tests: don't tabulate the versions [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/460 [08:12:13] (03Merged) 10jenkins-bot: ceph.wait_for_rebalance: new handy cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1054510 (owner: 10David Caro) [08:19:24] (03approved) 10dcaro: bump openapi version [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/30 (https://phabricator.wikimedia.org/T356974) [08:19:26] (03update) 10dcaro: bump openapi version [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/30 (https://phabricator.wikimedia.org/T356974) [08:19:30] (03merge) 10dcaro: bump openapi version [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/30 (https://phabricator.wikimedia.org/T356974) [08:21:56] FIRING: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:22:40] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.34-20240805081943-a45c7fa3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/467 (https://phabricator.wikimedia.org/T356974) [08:30:50] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [08:31:01] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [08:32:46] (03approved) 10dcaro: run_functional_tests: don't tabulate the versions [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/460 [08:32:49] (03merge) 10dcaro: run_functional_tests: don't tabulate the versions [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/460 [08:32:51] (03update) 10dcaro: run_functional_tests: don't tabulate the versions [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/460 [08:33:12] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/25 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:33:16] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/25 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:34:04] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/180 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:34:07] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/180 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:34:15] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/56 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:34:18] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/56 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:34:22] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/50 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:34:24] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/50 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:34:30] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/84 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:34:34] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/84 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:34:41] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/61 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:34:44] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/61 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:34:49] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/2 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:34:52] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/2 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:37:17] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.7-20240805083504-73f7e174 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/468 [08:38:29] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [08:38:40] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [08:42:25] 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [builds-api,jobs-api,envvars-api,api-gateway] Figure out and document how to do non-backwards compatible changes - https://phabricator.wikimedia.org/T356974#10041347 (10dcaro) [08:44:56] (03approved) 10dcaro: api-gateway: bump to 0.0.34-20240805081943-a45c7fa3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/467 (https://phabricator.wikimedia.org/T356974) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:45:01] (03update) 10dcaro: api-gateway: bump to 0.0.34-20240805081943-a45c7fa3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/467 (https://phabricator.wikimedia.org/T356974) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:45:26] (03merge) 10dcaro: api-gateway: bump to 0.0.34-20240805081943-a45c7fa3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/467 (https://phabricator.wikimedia.org/T356974) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:02:43] (03open) 10dcaro: docs: fix links [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/33 [09:03:33] (03approved) 10dcaro: docs: fix links [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/33 [09:03:35] (03update) 10dcaro: docs: fix links [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/33 [09:04:05] (03merge) 10dcaro: docs: fix links [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/33 [09:06:04] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.35-20240805090419-f68a21c8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/469 [09:06:08] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.35-20240805090419-f68a21c8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/469 [09:07:33] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [09:07:43] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [09:09:26] RESOLVED: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:18:15] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [09:18:26] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [09:19:31] PROBLEM - toolschecker: Make sure enwiki dumps are not empty on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/nfs/dumps - 324 bytes in 60.004 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [09:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:20:49] (03approved) 10dcaro: api-gateway: bump to 0.0.35-20240805090419-f68a21c8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/469 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:20:51] (03update) 10dcaro: api-gateway: bump to 0.0.35-20240805090419-f68a21c8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/469 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:20:52] (03merge) 10dcaro: api-gateway: bump to 0.0.35-20240805090419-f68a21c8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/469 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:23:15] RECOVERY - toolschecker: Make sure enwiki dumps are not empty on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 36.075 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [09:26:44] 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [builds-api,jobs-api,envvars-api,api-gateway] Figure out and document how to do non-backwards compatible changes - https://phabricator.wikimedia.org/T356974#10041418 (10dcaro) 05In progress→03Resolved [09:42:11] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:05:48] (03open) 10dcaro: docs: update component cookbook help [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/470 [10:11:33] 10Striker: Add one-time-code autocomplete type to TOTP field on toolsadmin login page - https://phabricator.wikimedia.org/T371794 (10XtexChooser) 03NEW [10:12:13] (03PS4) 10XtexChooser: labsauth: Set OATH input autocomplete to one-time-code [labs/striker] - 10https://gerrit.wikimedia.org/r/1058616 (https://phabricator.wikimedia.org/T371794) [10:15:56] FIRING: SystemdUnitDown: The service unit systemd-timedated.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:23:37] (03open) 10dcaro: merge: merge the components/responses structure too [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/34 [10:30:56] RESOLVED: SystemdUnitDown: The service unit systemd-timedated.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:31:04] (03approved) 10fnegri: merge: merge the components/responses structure too [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/34 (owner: 10dcaro) [10:48:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:51:07] (03approved) 10dcaro: merge: merge the components/responses structure too [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/34 [10:51:10] (03update) 10dcaro: merge: merge the components/responses structure too [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/34 [10:51:12] (03merge) 10dcaro: merge: merge the components/responses structure too [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/34 [10:53:25] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.36-20240805105124-23271f37 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/471 [11:03:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:36:37] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [11:36:48] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [11:39:58] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#10041781 (10fnegri) `clouddb1021` was decommed today by @BTullis before reading this comment, so it can no longer be used as a test host. :/ @Maroste... [11:42:04] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [11:42:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [11:43:39] (03approved) 10dcaro: api-gateway: bump to 0.0.36-20240805105124-23271f37 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/471 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:43:42] (03update) 10dcaro: api-gateway: bump to 0.0.36-20240805105124-23271f37 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/471 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:43:42] (03merge) 10dcaro: api-gateway: bump to 0.0.36-20240805105124-23271f37 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/471 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:44:11] (03update) 10dcaro: docs: update component cookbook help [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/470 [11:44:31] (03approved) 10dcaro: docs: update component cookbook help [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/470 [11:44:33] (03update) 10dcaro: docs: update component cookbook help [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/470 [11:44:37] (03merge) 10dcaro: docs: update component cookbook help [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/470 [11:44:48] (03update) 10dcaro: components-api: bump to 0.0.7-20240805083504-73f7e174 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/468 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:44:52] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#10041786 (10fnegri) [12:03:03] 10Toolforge (Toolforge iteration 14): [toolforge] simplify calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377#10041836 (10dcaro) The current API is stable now (https://wikitech.wikimedia.org/wiki/Help:Toolforge/API), we are still working on making the clis... [12:06:13] (03approved) 10dcaro: toolforge_deploy_mr: use the correct name when registering an mr [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/179 [12:06:14] (03update) 10dcaro: toolforge_deploy_mr: use the correct name when registering an mr [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/179 [12:06:18] (03update) 10dcaro: toolforge_deploy_mr: use the correct name when registering an mr [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/179 [12:07:43] (03merge) 10dcaro: toolforge_deploy_mr: use the correct name when registering an mr [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/179 [12:29:01] 06cloud-services-team, 10Cloud-VPS, 06collaboration-services, 13Patch-For-Review: puppet problems mounting cinder volumes (and suggested fixes) - https://phabricator.wikimedia.org/T371573#10041908 (10LSobanski) [13:14:29] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance gitlab-runners-puppetserver-01 in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:21:52] (03update) 10dcaro: Draft: show backend status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/1 (https://phabricator.wikimedia.org/T370324) [13:33:20] (03update) 10dcaro: Draft: show backend status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/1 (https://phabricator.wikimedia.org/T370324) [13:34:16] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api [13:34:21] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal, 13Patch-For-Review: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#10042079 (10fnegri) I'm going forward and reimaging clouddb1019 pairing with @BTullis [13:34:26] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api [13:34:51] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api [13:35:01] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api [13:35:59] (03approved) 10dcaro: components-api: bump to 0.0.7-20240805083504-73f7e174 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/468 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [13:36:01] (03merge) 10dcaro: components-api: bump to 0.0.7-20240805083504-73f7e174 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/468 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [13:36:30] (03approved) 10dcaro: use kubeconfig for toolname [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569) [13:36:34] (03update) 10dcaro: use kubeconfig for toolname [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569) [13:38:01] (03merge) 10dcaro: use kubeconfig for toolname [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569) [13:38:07] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/24 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:38:11] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/24 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:38:28] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/55 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:38:32] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/55 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:38:54] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/3 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:38:58] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/3 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:39:55] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/43 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:39:59] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/43 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:40:00] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/107 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:40:04] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/32 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:40:07] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/31 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:40:08] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/107 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:40:17] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/31 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:40:18] (03update) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/32 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:40:27] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/24 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:40:32] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/55 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:40:34] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/112 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:40:41] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/112 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:41:13] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/31 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:41:32] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/111 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:42:08] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.9-20240805133909-34a6a2c5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/472 [13:42:31] PROBLEM - mysqld processes on clouddb1019 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [13:44:03] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/31 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:44:34] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.327-20240805134054-27dfc026 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/473 [13:44:56] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal, 13Patch-For-Review: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#10042092 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by fnegri@cumin1002 for host clouddb1019.eqiad.wmnet with O... [13:46:20] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.37-20240805134412-40551e32 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/474 [13:46:34] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-api: bump to 0.0.59-20240805134018-35ffcb05 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/475 [13:49:58] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.167-20240805134031-250f2aef [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/476 [13:50:40] (03update) 10dcaro: show backend status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/1 (https://phabricator.wikimedia.org/T370324) [13:51:27] (03update) 10dcaro: show backend status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/1 (https://phabricator.wikimedia.org/T370324) [13:51:46] (03update) 10dcaro: show backend status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/1 (https://phabricator.wikimedia.org/T370324) [13:57:03] (03PS1) 10David Caro: toolforge.component.deploy: remove the k8s prefix [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059890 [14:00:37] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: remove the k8s prefix [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059890 (owner: 10David Caro) [14:17:20] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:17:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:17:50] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [14:17:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:17:52] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:17:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:27:25] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [14:27:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:27:30] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:27:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:27:48] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [14:27:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:28:40] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:28:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:29:12] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [14:29:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:29:14] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:29:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:29:31] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [14:29:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:29:38] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10042190 (10SLyngshede-WMF) p:05Triage→03Medium [14:29:44] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:29:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:30:13] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [14:30:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:30:14] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:30:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:31:13] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [14:31:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:31:15] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:31:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:31:36] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 10Data-Platform-SRE (2024.07.29 - 2024.08.16), 13Patch-For-Review: Remove or replace deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud (Buster depre... - https://phabricator.wikimedia.org/T370465#10042208 [14:32:47] 10Striker, 10Bitu, 06Infrastructure-Foundations: Remove feature to connect SUL account to Striker (read from Bitu instead) - https://phabricator.wikimedia.org/T371595#10042201 (10SLyngshede-WMF) p:05Triage→03Medium [14:33:32] 10Tools, 06Infrastructure-Foundations: Requested offboarding-to-volunteer of HTriedman // Transfer ownership of SpinachBot from HTriedman (WMF) to HTriedman - https://phabricator.wikimedia.org/T371644#10042209 (10SLyngshede-WMF) 05Open→03In progress p:05Triage→03Medium a:03SLyngshede-WMF [14:50:14] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [14:50:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:51:01] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10042298 (10jijiki) [14:54:19] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:54:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:54:43] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [14:54:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:57:54] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:57:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:58:10] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [14:58:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:58:54] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:58:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:59:10] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [14:59:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:00:43] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#10042329 (10fnegri) 05Open→03In progress [15:00:53] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:00:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:01:12] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [15:01:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:02:49] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:02:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:03:04] !log dcaro@urcuchillay toolsbeta END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=1) [15:03:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:03:36] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:03:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:03:52] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [15:03:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:04:11] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:04:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:04:26] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [15:04:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:09:13] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:09:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:09:31] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [15:09:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:11:06] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:11:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:11:21] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [15:11:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:11:51] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:11:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:12:07] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [15:12:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:12:26] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:12:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:12:42] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [15:12:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:13:27] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:13:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:13:43] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [15:13:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:14:02] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:14:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:14:18] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [15:14:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:16:26] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#10042384 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by fnegri@cumin1002 for host clouddb1019.eqiad.wmnet with OS bookworm completed: - cl... [15:18:20] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services: [wikireplicas] frequent replag spikes in clouddb hosts - https://phabricator.wikimedia.org/T367778#10042399 (10fnegri) I skipped the reimage of clouddb1021 (see comments in T365424) and proceeded with the reimage of clouddb1019, which is now running B... [15:23:45] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:23:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:24:08] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [15:24:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:27:49] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikidumpparse" project Buster deprecation - https://phabricator.wikimedia.org/T367561#10042457 (10Andrew) @notconfusing can you confirm that you still plan to work on this? [15:28:14] 10Cloud-VPS (Debian Buster Deprecation), 10Wikispore: Rebuild Wikispore Vagrant boxes on Bullseye or Bookworm - https://phabricator.wikimedia.org/T365934#10042459 (10Andrew) It's a new week and a new nag! @tgr if you would like to prevent these VMs from being shut down please upgrade them soon. Thanks! [15:29:33] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:29:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:29:56] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [15:29:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:31:59] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#10042501 (10fnegri) [15:36:43] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:36:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:41:52] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [15:41:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:51:31] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:51:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:51:44] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [15:51:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:51:51] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:51:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:52:00] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [15:52:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:52:16] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:52:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:52:34] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [15:52:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:52:50] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:52:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:53:14] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [15:53:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:53:36] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:53:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:54:06] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [15:54:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:55:04] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:55:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:55:15] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [15:55:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:55:31] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:55:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:56:02] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [15:56:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:58:48] (03PS1) 10David Caro: toolforge.component.deploy: use bump_ as default branch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059905 [15:58:48] (03PS1) 10David Caro: run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 [15:58:48] (03PS1) 10David Caro: toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 [16:02:08] (03CR) 10CI reject: [V:04-1] run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 (owner: 10David Caro) [16:02:11] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: use bump_ as default branch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059905 (owner: 10David Caro) [16:02:42] (03CR) 10CI reject: [V:04-1] toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 (owner: 10David Caro) [16:04:08] (03PS2) 10David Caro: toolforge.component.deploy: remove the k8s prefix [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059890 [16:04:08] (03PS2) 10David Caro: toolforge.component.deploy: use bump_ as default branch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059905 [16:04:08] (03PS2) 10David Caro: run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 [16:04:09] (03PS2) 10David Caro: toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 [16:07:46] (03CR) 10CI reject: [V:04-1] run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 (owner: 10David Caro) [16:08:09] (03CR) 10CI reject: [V:04-1] toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 (owner: 10David Caro) [16:13:37] 10Cloud-Services, 10Catalyst (PatchDemo GoLive): Moving proxies across wmcs projects for patchdemo.wmflabs.org - https://phabricator.wikimedia.org/T370080#10042794 (10thcipriani) [16:14:46] (03PS3) 10David Caro: run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 [16:14:46] (03PS3) 10David Caro: toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 [16:15:24] 10Cloud-VPS (Debian Buster Deprecation), 10Wikispore: Rebuild Wikispore Vagrant boxes on Bullseye or Bookworm - https://phabricator.wikimedia.org/T365934#10042803 (10Tgr) I'll try to get it done during the hackathon. [16:17:56] (03CR) 10CI reject: [V:04-1] run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 (owner: 10David Caro) [16:18:00] (03CR) 10CI reject: [V:04-1] toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 (owner: 10David Caro) [16:32:43] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [16:32:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:33:16] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [16:33:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:35:56] 10Cloud-Services, 10Catalyst (PatchDemo GoLive): Moving proxies across wmcs projects for patchdemo.wmflabs.org - https://phabricator.wikimedia.org/T370080#10042871 (10thcipriani) Current status: - patchdemo.wmflabs.org -- redirecting to patchdemo.wmcloud.org currently - patchdemo.wmcloud.org -- pointing to th... [16:43:31] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Quarry: Allow Quarry to query its own database - https://phabricator.wikimedia.org/T367415#10042952 (10bd808) [16:51:41] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.run_tests [16:51:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:52:19] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [16:52:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:58:55] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Quarry: Allow Quarry to query its own database - https://phabricator.wikimedia.org/T367415#10043015 (10bd808) >>! In T367415#10038537, @fnegri wrote: > @bd808 I'm interested in your opinion on this one. I created a pull request, but I'm also wondering if anybody sti... [17:14:39] (03PS4) 10David Caro: run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 [17:14:39] (03PS4) 10David Caro: toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 [17:14:40] (03PS1) 10David Caro: openstack.tofu: use run_script instead of reimplementing it [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059919 [17:14:40] (03PS1) 10David Caro: WMCSCookbookRunnerBase: load the wmcs config if it's there [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059920 [17:14:41] (03PS1) 10David Caro: DRAFT: toolforge.deploy: run tests and add note to MR [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921 [17:18:12] (03CR) 10CI reject: [V:04-1] run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 (owner: 10David Caro) [17:18:18] (03CR) 10CI reject: [V:04-1] toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 (owner: 10David Caro) [17:18:28] (03CR) 10CI reject: [V:04-1] openstack.tofu: use run_script instead of reimplementing it [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059919 (owner: 10David Caro) [17:18:39] (03CR) 10CI reject: [V:04-1] WMCSCookbookRunnerBase: load the wmcs config if it's there [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059920 (owner: 10David Caro) [17:18:48] (03CR) 10CI reject: [V:04-1] DRAFT: toolforge.deploy: run tests and add note to MR [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921 (owner: 10David Caro) [17:22:07] (03PS5) 10David Caro: run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 [17:22:08] (03PS5) 10David Caro: toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 [17:22:08] (03PS2) 10David Caro: openstack.tofu: use run_script instead of reimplementing it [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059919 [17:22:08] (03PS2) 10David Caro: WMCSCookbookRunnerBase: load the wmcs config if it's there [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059920 [17:22:09] (03PS2) 10David Caro: DRAFT: toolforge.deploy: run tests and add note to MR [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921 [17:25:29] (03CR) 10CI reject: [V:04-1] run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 (owner: 10David Caro) [17:25:38] (03CR) 10CI reject: [V:04-1] openstack.tofu: use run_script instead of reimplementing it [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059919 (owner: 10David Caro) [17:25:39] (03CR) 10CI reject: [V:04-1] WMCSCookbookRunnerBase: load the wmcs config if it's there [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059920 (owner: 10David Caro) [17:25:39] (03CR) 10CI reject: [V:04-1] DRAFT: toolforge.deploy: run tests and add note to MR [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921 (owner: 10David Caro) [17:25:39] (03CR) 10CI reject: [V:04-1] toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 (owner: 10David Caro) [17:29:45] (03PS3) 10David Caro: WMCSCookbookRunnerBase: load the wmcs config if it's there [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059920 [17:29:45] (03PS3) 10David Caro: toolforge.component.deploy: remove the k8s prefix [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059890 [17:29:45] (03PS3) 10David Caro: toolforge.component.deploy: use bump_ as default branch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059905 [17:29:46] (03PS6) 10David Caro: run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 [17:29:47] (03PS6) 10David Caro: toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 [17:29:48] (03PS3) 10David Caro: openstack.tofu: use run_script instead of reimplementing it [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059919 [17:29:52] (03PS3) 10David Caro: DRAFT: toolforge.deploy: run tests and add note to MR [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921 [17:29:56] (03PS1) 10David Caro: openstack.tofu: use gitlab token from wmcs config [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059925 [17:33:01] (03CR) 10CI reject: [V:04-1] WMCSCookbookRunnerBase: load the wmcs config if it's there [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059920 (owner: 10David Caro) [17:33:23] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: remove the k8s prefix [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059890 (owner: 10David Caro) [17:33:30] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: use bump_ as default branch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059905 (owner: 10David Caro) [17:33:34] (03CR) 10CI reject: [V:04-1] run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 (owner: 10David Caro) [17:33:37] (03CR) 10CI reject: [V:04-1] openstack.tofu: use run_script instead of reimplementing it [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059919 (owner: 10David Caro) [17:33:37] (03CR) 10CI reject: [V:04-1] DRAFT: toolforge.deploy: run tests and add note to MR [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921 (owner: 10David Caro) [17:33:39] (03CR) 10CI reject: [V:04-1] toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 (owner: 10David Caro) [17:33:52] (03CR) 10CI reject: [V:04-1] openstack.tofu: use gitlab token from wmcs config [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059925 (owner: 10David Caro) [17:40:14] (03PS4) 10David Caro: WMCSCookbookRunnerBase: load the wmcs config if it's there [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059920 [17:40:14] (03PS2) 10David Caro: openstack.tofu: use gitlab token from wmcs config [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059925 [17:40:14] (03PS4) 10David Caro: toolforge.component.deploy: remove the k8s prefix [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059890 [17:40:15] (03PS4) 10David Caro: toolforge.component.deploy: use bump_ as default branch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059905 [17:40:16] (03PS7) 10David Caro: run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 [17:40:17] (03PS7) 10David Caro: toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 [17:40:21] (03PS4) 10David Caro: openstack.tofu: use run_script instead of reimplementing it [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059919 [17:40:25] (03PS4) 10David Caro: DRAFT: toolforge.deploy: run tests and add note to MR [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921 [17:43:25] (03CR) 10CI reject: [V:04-1] WMCSCookbookRunnerBase: load the wmcs config if it's there [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059920 (owner: 10David Caro) [17:43:55] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: remove the k8s prefix [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059890 (owner: 10David Caro) [17:43:59] (03CR) 10CI reject: [V:04-1] openstack.tofu: use gitlab token from wmcs config [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059925 (owner: 10David Caro) [17:44:03] (03CR) 10CI reject: [V:04-1] openstack.tofu: use run_script instead of reimplementing it [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059919 (owner: 10David Caro) [17:44:04] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: use bump_ as default branch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059905 (owner: 10David Caro) [17:44:05] (03CR) 10CI reject: [V:04-1] toolforge.run_tests: use the functional tests [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059907 (owner: 10David Caro) [17:44:18] (03CR) 10CI reject: [V:04-1] run_one_raw,run_script: add run_script and user parameter [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059906 (owner: 10David Caro) [17:44:22] (03CR) 10CI reject: [V:04-1] DRAFT: toolforge.deploy: run tests and add note to MR [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921 (owner: 10David Caro) [17:47:26] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [17:47:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:48:01] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [17:48:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:48:43] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [17:48:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:49:15] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [17:49:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:50:16] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [17:50:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:55:59] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [17:56:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:57:51] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [17:57:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:57:57] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [17:57:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:57:59] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [17:58:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:58:52] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [17:58:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:04:41] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [18:04:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:05:33] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [18:05:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:09:48] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [18:09:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:10:46] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [18:10:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:21:52] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [18:21:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:22:25] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [18:22:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:24:29] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance gitlab-runners-puppetserver-01 in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:25:49] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [18:25:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:26:18] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [18:26:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:27:09] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [18:27:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:28:08] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [18:28:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:44:11] (03PS1) 10GauriGuptaa: Fix the typo error from one to on [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1059930 [19:00:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [19:03:18] (03CR) 10Dzahn: [C:03+2] frontend: Enable php-opcache, Debian 11 Bullseye to 12 Bookworm, PHP 8.1 to 8.3 [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1058203 (owner: 10Krinkle) [19:04:09] (03Merged) 10jenkins-bot: frontend: Enable php-opcache, Debian 11 Bullseye to 12 Bookworm, PHP 8.1 to 8.3 [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1058203 (owner: 10Krinkle) [20:13:51] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [20:24:28] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [20:35:33] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [20:47:19] 06Toolforge-standards-committee: Refresh membership of Toolforge standards committee - https://phabricator.wikimedia.org/T370474#10043711 (10bd808) * On-wiki nominations open: https://wikitech.wikimedia.org/w/index.php?title=Help_talk:Toolforge/Toolforge_standards_committee&diff=prev&oldid=2212409 * Announced to... [20:47:35] 06Toolforge-standards-committee: Refresh membership of Toolforge standards committee - https://phabricator.wikimedia.org/T370474#10043714 (10bd808) [21:04:44] 06Toolforge-standards-committee: Refresh membership of Toolforge standards committee - https://phabricator.wikimedia.org/T370474#10043731 (10bd808) [21:08:28] 06Toolforge-standards-committee: Refresh membership of Toolforge standards committee - https://phabricator.wikimedia.org/T370474#10043735 (10bd808) Added note to Tech News for its 2024/33 edition: https://meta.wikimedia.org/w/index.php?title=Tech/News/2024/33&diff=prev&oldid=27237246 [21:13:34] 06Toolforge-standards-committee, 07User-notice: Refresh membership of Toolforge standards committee - https://phabricator.wikimedia.org/T370474#10043739 (10JJMC89) [21:28:03] 06Toolforge-standards-committee, 07User-notice: Refresh membership of Toolforge standards committee - https://phabricator.wikimedia.org/T370474#10043751 (10bd808) I have thanked all 6 past committee members for their service with a wikilove post to their Wikitech talk page awarding each of them the [[https://c... [22:04:25] 06Toolforge-standards-committee: Adoption request for image-metadata-viewer - https://phabricator.wikimedia.org/T338558#10043793 (10bd808) @mdaniels5757 are you still interested in adopting this tool? There is no source code or other data available in the tool's $HOME on Toolforge, so the only thing you would be... [22:38:10] 06Toolforge-standards-committee: Adoption request for vrb - https://phabricator.wikimedia.org/T338556#10043819 (10bd808) Jobs for the bot were disabled by inactivity on {T320129}. Secrets removed: * $HOME/LicenseReviewerBot/user-password.py (bot password for https://commons.wikimedia.org/wiki/User:LicenseReview... [22:43:26] 06Toolforge-standards-committee: Adoption request for vrb - https://phabricator.wikimedia.org/T338556#10043822 (10bd808) 05Open→03Resolved a:03bd808 Sorry this got stuck for so long @mdaniels5757. You now have access to the tool and its source on Toolforge. [23:47:33] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [23:48:19] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [23:55:57] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650)