[01:20:41] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:38:49] 10Tool-nfp: NFP is down - https://phabricator.wikimedia.org/T376683 (10JayCubby) 03NEW [02:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:12:49] 10Tool-nfp: NFP is down - https://phabricator.wikimedia.org/T376683#10209203 (10JayCubby) Back up again. Resolving... [02:12:53] 10Tool-nfp: NFP is down - https://phabricator.wikimedia.org/T376683#10209204 (10JayCubby) 05Open→03Resolved [03:28:05] (03merge) 10legoktm: Tweaks for Wikitech-on-SUL era [toolforge-repos/ldap] - 10https://gitlab.wikimedia.org/toolforge-repos/ldap/-/merge_requests/4 (https://phabricator.wikimedia.org/T376560) (owner: 10taavi) [03:34:49] 10Tool-ldap: Display linked SUL account - https://phabricator.wikimedia.org/T376560#10209230 (10Legoktm) 05Open→03Resolved a:03taavi Deployed. Also I gave you access to the tool but also happy to review patches in the future. [03:47:23] 10Tool-ldap: Indicate if account is locked - https://phabricator.wikimedia.org/T362046#10209244 (10Legoktm) Hmm, I can't find where in LDAP that is...do you have an example account that's disabled? [05:20:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:48:14] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10209425 (10SunAfterRain) Will the Two-factor Authentication previously set up in Wikitech still be effective for LDAP, or will it become invalid alon... [06:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:27:15] 10Striker, 13Patch-For-Review: Striker should use ID instead of username to identify SUL accounts - https://phabricator.wikimedia.org/T359428#10209449 (10Bugreporter) Striker ideally should not store any SUL connection information at all. {T371595} [06:31:37] 06cloud-services-team, 10Toolforge: Support jdk21 on toolforge - https://phabricator.wikimedia.org/T346477#10209451 (10Benjavalero) Thanks for the profuse clarification! [07:19:30] 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T376505#10209496 (10dcaro) a:03aborrero +1 approved [07:23:39] 10VPS-project-Codesearch, 10VPS-project-Extdist, 06collaboration-services, 10Gerrit, 13Patch-For-Review: Move clients off of gerrit-replica.wikimedia.org back to gerrit.wikimedia.org - https://phabricator.wikimedia.org/T336710#10209504 (10hashar) >>! In T336710#10189664, @Dzahn wrote: > A thing that chan... [08:00:38] (03open) 10aborrero: maintain-kubeusers: bump quota for video-answer-tool [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/550 (https://phabricator.wikimedia.org/T376505) [08:02:12] (03approved) 10dcaro: maintain-kubeusers: bump quota for video-answer-tool [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/550 (https://phabricator.wikimedia.org/T376505) (owner: 10aborrero) [08:02:17] (03update) 10dcaro: maintain-kubeusers: bump quota for video-answer-tool [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/550 (https://phabricator.wikimedia.org/T376505) (owner: 10aborrero) [08:03:33] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain_kubeusers [08:03:40] !log aborrero@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain_kubeusers [08:03:58] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [08:07:56] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [08:08:10] !log aborrero@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers [08:08:35] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [08:08:58] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [08:10:24] (03merge) 10aborrero: maintain-kubeusers: bump quota for video-answer-tool [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/550 (https://phabricator.wikimedia.org/T376505) [08:12:51] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [08:13:39] 10Toolforge (Quota-requests), 13Patch-For-Review: Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T376505#10209593 (10aborrero) 05Open→03Resolved This is done. Note, we can give you additional resources if required. Please consult with us before moving... [08:14:54] 06cloud-services-team, 10Toolforge: Support jdk21 on toolforge - https://phabricator.wikimedia.org/T346477#10209600 (10dcaro) >>! In T346477#10209451, @Benjavalero wrote: > Thanks for the profuse clarification! you are welcome :), sorry if I overexplain things a bit, I find it's usually better than the ot... [08:24:42] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [08:24:46] !log aborrero@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers [08:24:54] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [08:26:12] (03merge) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [08:26:33] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [08:28:28] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: DONOTMERGE components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 [08:34:37] (03PS1) 10Arturo Borrero Gonzalez: toolforge.component.deploy: don't print to stdout the raw MR note [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078611 [08:36:00] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 (https://phabricator.wikimedia.org/T362070) [08:36:23] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment): Implement Global Contributions as a central page on Meta and implement redirects from other projec... - https://phabricator.wikimedia.org/T376612#10209652 [08:36:58] (03update) 10dcaro: slavina/clean up code [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/11 (https://phabricator.wikimedia.org/T362070) [08:37:10] (03update) 10dcaro: global: add mypy checks and fixes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/12 (https://phabricator.wikimedia.org/T362070) [08:37:18] (03update) 10dcaro: api: wrap responses inside routes [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/13 (https://phabricator.wikimedia.org/T362070) (owner: 10sstefanova) [08:37:26] (03update) 10dcaro: api: wrap errors in ApiResponse [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/15 (https://phabricator.wikimedia.org/T362070) (owner: 10sstefanova) [08:37:33] (03update) 10dcaro: api: add auth [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/16 (https://phabricator.wikimedia.org/T362070) (owner: 10sstefanova) [08:39:26] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15), 07Epic: [components-api] First iteration of the component API - https://phabricator.wikimedia.org/T362051#10209665 (10dcaro) [08:39:56] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15): [components-api] Get a minimal version of the config with build-only data - https://phabricator.wikimedia.org/T362070#10209667 (10dcaro) [08:40:01] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15), 07Epic: [components-api] First iteration of the component API - https://phabricator.wikimedia.org/T362051#10209673 (10dcaro) 05Open→03In progress [08:40:08] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15): [components-api] Get a minimal version of the config with build-only data - https://phabricator.wikimedia.org/T362070#10209670 (10dcaro) 05Open→03Resolved [08:41:11] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15), 07Epic: [Hypothesis] WE6.3.4 By building an "orchestrator" toolforge component (components-api) we will be able to automate most manually-triggered deployments - https://phabricator.wikimedia.org/T375199#10209677 (10dcaro) 05Op... [08:48:16] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15): [components-api] Get a skeleton of API webservice and implement `/tool//deploy` with build-only features - https://phabricator.wikimedia.org/T362069#10209691 (10dcaro) [08:58:30] 06cloud-services-team, 10ContentTranslation, 10LPL Essential: Setup Wiki Family on CX / SX staging - https://phabricator.wikimedia.org/T345340#10209716 (10aborrero) >>! In T345340#9599781, @santhosh wrote: > > There is a problem to expose this wiki instances in a public wmcloud instance. ideally, we would l... [09:20:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:24:12] (03PS11) 10David Caro: toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 [09:24:24] (03PS12) 10David Caro: toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 [09:24:33] (03CR) 10David Caro: [C:03+2] toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 (owner: 10David Caro) [09:27:37] (03approved) 10dcaro: Adapt home archiving configuration for NFS-on-VMs usage [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/20 (https://phabricator.wikimedia.org/T372701) (owner: 10taavi) [09:28:06] (03Merged) 10jenkins-bot: toolforge.component.deploy: show the MR comment link [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 (owner: 10David Caro) [09:28:30] (03PS3) 10David Caro: ceph.undrain_rack: undrain from different hosts in parallel [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 [09:28:37] (03CR) 10CI reject: [V:04-1] ceph.undrain_rack: undrain from different hosts in parallel [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 (owner: 10David Caro) [09:28:49] (03CR) 10David Caro: [C:03+2] ceph.undrain_rack: undrain from different hosts in parallel (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 (owner: 10David Caro) [09:28:56] (03CR) 10CI reject: [V:04-1] ceph.undrain_rack: undrain from different hosts in parallel [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 (owner: 10David Caro) [09:29:12] (03CR) 10David Caro: [C:03+2] "Merge conflict, looking" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 (owner: 10David Caro) [09:29:18] (03CR) 10CI reject: [V:04-1] ceph.undrain_rack: undrain from different hosts in parallel [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 (owner: 10David Caro) [09:33:55] (03PS4) 10David Caro: ceph.undrain_rack: undrain from different hosts in parallel [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 [09:34:24] (03PS5) 10David Caro: ceph.undrain_rack: undrain from different hosts in parallel [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 [09:38:36] (03CR) 10CI reject: [V:04-1] ceph.undrain_rack: undrain from different hosts in parallel [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 (owner: 10David Caro) [09:41:57] (03PS6) 10David Caro: ceph.undrain_rack: undrain from different hosts in parallel [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 [09:46:22] (03CR) 10CI reject: [V:04-1] ceph.undrain_rack: undrain from different hosts in parallel [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 (owner: 10David Caro) [09:49:19] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment), 10Trust and Safety Product Sprint (Sprint Cello (Oct 7 - 18)): Implement Global Contributions a... - https://phabricator.wikimedia.org/T376612#10209831 [09:49:43] (03PS7) 10David Caro: ceph.undrain_rack: undrain from different hosts in parallel [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 [09:54:55] 06cloud-services-team, 10Cloud-VPS: tofu-infra: update to opentofu 1.8.x - https://phabricator.wikimedia.org/T376704 (10aborrero) 03NEW [09:57:22] (03PS4) 10David Caro: wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075892 (https://phabricator.wikimedia.org/T371393) [09:57:29] (03CR) 10David Caro: wmcs.vps.create_project: replace logic with message about deprecation (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075892 (https://phabricator.wikimedia.org/T371393) (owner: 10David Caro) [10:00:22] 06cloud-services-team, 10Cloud-VPS: tofu-infra: update to openstack provider - https://phabricator.wikimedia.org/T376705 (10aborrero) 03NEW [10:00:35] 06cloud-services-team, 10Cloud-VPS: tofu-infra: update to openstack provider to 3.x - https://phabricator.wikimedia.org/T376705#10209882 (10aborrero) [10:01:26] 06cloud-services-team, 10Cloud-VPS: tofu-infra: update to opentofu 1.8.x - https://phabricator.wikimedia.org/T376704#10209896 (10aborrero) p:05Triage→03Medium [10:01:56] 06cloud-services-team, 10Cloud-VPS: tofu-infra: update to openstack provider 3.x - https://phabricator.wikimedia.org/T376705#10209901 (10aborrero) [10:02:00] 06cloud-services-team, 10Cloud-VPS: tofu-infra: update to openstack provider 3.x - https://phabricator.wikimedia.org/T376705#10209897 (10aborrero) p:05Triage→03Medium a:05aborrero→03None [10:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:06:54] (03CR) 10FNegri: [C:03+1] wmcs.vps.create_project: replace logic with message about deprecation (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075892 (https://phabricator.wikimedia.org/T371393) (owner: 10David Caro) [10:07:45] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10209932 (10aborrero) See also: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 by @fnegri [10:08:37] (03CR) 10FNegri: [C:03+1] ceph.undrain_rack: undrain from different hosts in parallel [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 (owner: 10David Caro) [10:41:05] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 07Epic, and 2 others: [Epic] Implement global contributions feature - https://phabricator.wikimedia.org/T337089#10210128 (10Niharika) [10:48:07] (03open) 10aborrero: tofu-infra: bump openstack provider to 3.0.0 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/94 (https://phabricator.wikimedia.org/T376705) [10:51:34] (03approved) 10fnegri: tofu-infra: bump openstack provider to 3.0.0 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/94 (https://phabricator.wikimedia.org/T376705) (owner: 10aborrero) [10:55:38] (03open) 10dsavuljesku: Draft: Add rich text support [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/3 [10:56:13] (03update) 10dsavuljesku: Draft: Add rich text support [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/3 [11:15:57] (03merge) 10aborrero: tofu-infra: bump openstack provider to 3.0.0 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/94 (https://phabricator.wikimedia.org/T376705) [11:16:05] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:16:31] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:55:40] (03open) 10sstefanova: api: add auth to delete route [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/17 (https://phabricator.wikimedia.org/T362069) [11:58:20] (03open) 10dcaro: deployment: Add the deployment endpoints and mock storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/18 (https://phabricator.wikimedia.org/T362069) [12:05:45] (03update) 10dcaro: builds-builder: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 (https://phabricator.wikimedia.org/T374908) [12:16:19] (03open) 10sstefanova: tests: add basic tests for delete config route [repos/cloud/toolforge/components-api] (slavina/add-auth-to-delete-route) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T362069) [12:23:37] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15): [builds-builder,builds-api] After the upgrade build actions take >10s - https://phabricator.wikimedia.org/T376710 (10dcaro) 03NEW [12:29:47] (03open) 10dcaro: api_client: increase timeout to 20s [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/59 (https://phabricator.wikimedia.org/T376710) [12:36:48] (03approved) 10sstefanova: api_client: increase timeout to 20s [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/59 (https://phabricator.wikimedia.org/T376710) (owner: 10dcaro) [12:37:15] (03merge) 10dcaro: api_client: increase timeout to 20s [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/59 (https://phabricator.wikimedia.org/T376710) [12:37:15] (03update) 10dcaro: api_client: increase timeout to 20s [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/59 (https://phabricator.wikimedia.org/T376710) [12:38:01] (03update) 10fnegri: cloudvps_flavors: Add missing g3 flavors [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/4 (owner: 10taavi) [12:43:51] (03open) 10dcaro: d/changelog: bump to 1.6.4 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/60 (https://phabricator.wikimedia.org/T376710) [12:44:14] (03update) 10sstefanova: api: add auth to delete route [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/17 (https://phabricator.wikimedia.org/T362069) [12:44:19] (03close) 10sstefanova: api: add auth to delete route [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/17 (https://phabricator.wikimedia.org/T362069) [12:44:40] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15), 13Patch-For-Review: [builds-builder,builds-api] After the upgrade build actions take >10s - https://phabricator.wikimedia.org/T376710#10210407 (10dcaro) p:05Triage→03High [12:44:56] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10210400 (10Tacsipacsi) Who will receive the notification emails? The //Author// column contains the first (why only the first?) author of the tool – which may not be the same as the maintain... [12:45:42] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15), 13Patch-For-Review: [builds-builder,builds-api] Upgrade tekton - https://phabricator.wikimedia.org/T374908#10210415 (10dcaro) 05In progress→03Resolved [12:46:45] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15), 13Patch-For-Review: [builds-builder,builds-api] Upgrade tekton - https://phabricator.wikimedia.org/T374908#10210411 (10dcaro) Upgrade done, there's the subtask to investigate the increased slowness on tools (did not notice a... [12:47:39] (03update) 10sstefanova: tests: add basic tests for delete config route [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T362069) [12:48:08] (03open) 10sstefanova: auth: use APIKeyHeader auth scheme [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/20 [12:52:46] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld (T376710) [12:52:48] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld (T376710) [12:52:51] T376710: [builds-builder,builds-api] After the upgrade build actions take >10s - https://phabricator.wikimedia.org/T376710 [12:53:14] 06cloud-services-team, 10Toolforge: [infra, k8s, webservice] remove deprecated kubectl --wait flag before k8s 1.29 upgrade - https://phabricator.wikimedia.org/T373866#10210434 (10Slst2020) Heh, I think I'm that someone. What happened is that Toolsbeta was in disarray for a while during the last k8s upgrade... [12:54:15] (03close) 10dcaro: d/changelog: bump to 1.6.4 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/60 (https://phabricator.wikimedia.org/T376710) [12:54:32] (03open) 10dcaro: d/changelog: bump to 1.6.4 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/61 (https://phabricator.wikimedia.org/T376710) [12:55:30] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld (T376710) [12:58:01] (03CR) 10David Caro: [C:03+2] ceph.undrain_rack: undrain from different hosts in parallel [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 (owner: 10David Caro) [12:59:26] 06cloud-services-team, 10Toolforge: [k8s, infra] update pause image to 3.6 - https://phabricator.wikimedia.org/T374193#10210463 (10dcaro) p:05Triage→03Low [12:59:39] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld (T376710) [12:59:43] T376710: [builds-builder,builds-api] After the upgrade build actions take >10s - https://phabricator.wikimedia.org/T376710 [13:02:35] (03Merged) 10jenkins-bot: ceph.undrain_rack: undrain from different hosts in parallel [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 (owner: 10David Caro) [13:20:42] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:26:45] 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T376505#10210580 (10derenrich) thanks so much! i'm actually still having an issue with this though. starting the service with the command ` toolforge webservice buildservice resta... [13:27:16] 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T376505#10210582 (10derenrich) 05Resolved→03Open [13:27:23] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld (T376710) [13:27:29] T376710: [builds-builder,builds-api] After the upgrade build actions take >10s - https://phabricator.wikimedia.org/T376710 [13:28:32] (03approved) 10dcaro: builds-builder: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 (https://phabricator.wikimedia.org/T374908) [13:29:15] (03merge) 10dcaro: builds-builder: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 (https://phabricator.wikimedia.org/T374908) [13:29:28] (03update) 10dcaro: builds-builder: upgrade tekton [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531 (https://phabricator.wikimedia.org/T374908) [13:34:22] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld (T376710) [13:34:26] T376710: [builds-builder,builds-api] After the upgrade build actions take >10s - https://phabricator.wikimedia.org/T376710 [13:37:13] (03update) 10sstefanova: auth: use APIKeyHeader auth scheme [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/20 [13:47:00] (03approved) 10dcaro: d/changelog: bump to 1.6.4 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/61 (https://phabricator.wikimedia.org/T376710) [13:47:04] (03update) 10dcaro: d/changelog: bump to 1.6.4 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/61 (https://phabricator.wikimedia.org/T376710) [13:47:06] (03merge) 10dcaro: d/changelog: bump to 1.6.4 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/61 (https://phabricator.wikimedia.org/T376710) [13:50:12] (03PS1) 10David Caro: toolforge.component.deploy: handle non-mr package deploys [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 [13:52:49] 06cloud-services-team: alerting: detect if a kernel had a panic - https://phabricator.wikimedia.org/T376719 (10aborrero) 03NEW [13:53:07] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: handle non-mr package deploys [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 (owner: 10David Caro) [13:54:04] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: cloudgw1002: network interface problem - https://phabricator.wikimedia.org/T376589#10210732 (10aborrero) checked the server today. No kernel panic. [13:58:46] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/20 [14:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:28:55] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: cloudgw1002: network interface problem - https://phabricator.wikimedia.org/T376589#10211007 (10Jclark-ctr) @aborrero i did just update idrac from 4.4 to 7.0. unrelated. but since i was logged in and causes no reboot. let us know if you w... [14:30:14] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10211008 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm [14:32:04] (03update) 10sstefanova: auth: use APIKeyHeader auth scheme [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/20 [14:34:50] (03update) 10sstefanova: auth: use APIKeyHeader auth scheme [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/20 [14:36:13] (03update) 10sstefanova: auth: use APIKeyHeader auth scheme [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/20 [14:39:05] (03update) 10sstefanova: auth: use APIKeyHeader auth scheme [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/20 [14:40:22] (03open) 10dcaro: toolforge_components: when waiting, use a higher timeout [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/194 [14:40:28] (03update) 10dcaro: toolforge_components: when waiting, use a higher timeout [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/194 [14:44:49] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: cloudgw1002: network interface problem - https://phabricator.wikimedia.org/T376589#10211051 (10aborrero) >>! In T376589#10211007, @Jclark-ctr wrote: > @aborrero i did just update idrac from 4.4 to 7.0. unrelated. but since i was logged in a... [14:47:22] (03update) 10dcaro: toolforge_components: when waiting, use a higher timeout [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/194 [14:48:19] 06cloud-services-team, 13Patch-For-Review: alerting: detect if a kernel had a panic - https://phabricator.wikimedia.org/T376719#10211053 (10aborrero) 05Open→03In progress p:05Triage→03Low patch should be ready for review. [14:48:27] (03update) 10dcaro: toolforge_components: when waiting, use a higher timeout [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/194 [14:51:31] (03open) 10sstefanova: api: fix exceptions handler responses [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/21 [14:53:40] 06cloud-services-team, 10Cloud-VPS: tofu-infra: update to openstack provider 3.x - https://phabricator.wikimedia.org/T376705#10211063 (10aborrero) 05Open→03Resolved a:03aborrero [15:23:26] (03open) 10dcaro: ansible: add timestamps to the logs [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/195 [15:25:04] (03approved) 10fnegri: ansible: add timestamps to the logs [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/195 (owner: 10dcaro) [15:25:19] (03approved) 10fnegri: toolforge_components: when waiting, use a higher timeout [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/194 (owner: 10dcaro) [15:25:31] (03merge) 10dcaro: ansible: add timestamps to the logs [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/195 [15:29:07] (03update) 10dcaro: toolforge_components: when waiting, use a higher timeout [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/194 [15:30:06] (03merge) 10dcaro: toolforge_components: when waiting, use a higher timeout [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/194 [15:33:15] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10211167 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm [15:53:38] (03approved) 10dcaro: tests: add basic tests for delete config route [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T362069) (owner: 10sstefanova) [15:53:43] (03update) 10dcaro: tests: add basic tests for delete config route [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T362069) (owner: 10sstefanova) [15:56:02] (03update) 10dcaro: deployment: Add the deployment endpoints and mock storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/18 (https://phabricator.wikimedia.org/T362069) [15:57:45] (03approved) 10dcaro: auth: use APIKeyHeader auth scheme [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/20 (owner: 10sstefanova) [15:58:06] (03update) 10dcaro: auth: use APIKeyHeader auth scheme [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/20 (owner: 10sstefanova) [16:29:55] 10Tools, 06Infrastructure-Foundations: Requested offboarding-to-volunteer of HTriedman // Transfer ownership of SpinachBot from HTriedman (WMF) to HTriedman - https://phabricator.wikimedia.org/T371644#10211460 (10Htriedman) @SLyngshede-WMF Hello! Today is my first day back as a contractor. As of right now,... [16:37:37] (03PS5) 10David Caro: wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075892 (https://phabricator.wikimedia.org/T371393) [16:37:43] (03CR) 10David Caro: wmcs.vps.create_project: replace logic with message about deprecation (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075892 (https://phabricator.wikimedia.org/T371393) (owner: 10David Caro) [16:37:54] (03CR) 10David Caro: [C:03+2] wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075892 (https://phabricator.wikimedia.org/T371393) (owner: 10David Caro) [16:41:46] (03Merged) 10jenkins-bot: wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075892 (https://phabricator.wikimedia.org/T371393) (owner: 10David Caro) [16:44:01] 10Toolforge (Quota-requests): Request increased quota for sqid Toolforge tool - https://phabricator.wikimedia.org/T376338#10211498 (10dcaro) a:03aborrero +1 approved [16:44:19] 06cloud-services-team, 10Toolforge (Toolforge iteration 15), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10211504 (10dcaro) a:05dcaro→03Raymond_Ndibe [16:47:51] 10Tools, 06Infrastructure-Foundations: Requested offboarding-to-volunteer of HTriedman // Transfer ownership of SpinachBot from HTriedman (WMF) to HTriedman - https://phabricator.wikimedia.org/T371644#10211509 (10JLam-WMF) Hi Hal, your Okta and Office Wiki access will be restored momentarily. [16:48:12] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10211511 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm [16:48:30] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10211512 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm executed w... [16:49:10] 06cloud-services-team, 10Toolforge (Toolforge iteration 15), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10211503 (10dcaro) [16:49:25] (03merge) 10sstefanova: auth: use APIKeyHeader auth scheme [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/20 [16:51:51] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: DONOTMERGE components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 [16:52:55] (03update) 10sstefanova: tests: add basic tests for delete config route [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T362069) [16:53:37] (03CR) 10FNegri: "A different solution for the same problem was implemented in I58f8e1b82d73a743ac0d6169299075eba3e36ed6" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078611 (owner: 10Arturo Borrero Gonzalez) [16:56:23] (03update) 10sstefanova: api: fix exceptions handler responses [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/21 [16:57:11] 10VPS-project-Codesearch: Retain input when switching betwen tabs - https://phabricator.wikimedia.org/T376732 (10Ammarpad) 03NEW [16:58:57] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [16:59:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:59:03] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld [16:59:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:02:35] 10Tools, 06Infrastructure-Foundations: Requested offboarding-to-volunteer of HTriedman // Transfer ownership of SpinachBot from HTriedman (WMF) to HTriedman - https://phabricator.wikimedia.org/T371644#10211571 (10JLam-WMF) @Htriedman Please check your personal GMail inbox. I sent a follow up message. There... [17:04:39] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [17:04:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:05:11] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06Data-Platform-SRE: Automate maintain-views replica depooling - https://phabricator.wikimedia.org/T300427#10211582 (10fnegri) > Killing existing sessions on depooled servers still doesn't work as expected. This problem surfaced when @BTullis was t... [17:09:44] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld [17:09:46] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [17:09:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:09:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:09:51] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld [17:09:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:10:09] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [17:10:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:17:22] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld [17:17:25] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [17:17:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:17:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:20:42] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:24:28] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld [17:24:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:24:49] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [17:24:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:34:18] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld [17:34:21] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [17:34:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:34:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:35:02] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld [17:35:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:35:16] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [17:35:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:41:20] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [17:41:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:42:41] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [17:42:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:43:42] !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld [17:43:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:45:54] (03PS2) 10David Caro: toolforge.component.deploy: handle non-mr package deploys [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 [17:47:09] (03PS3) 10David Caro: toolforge.component.deploy: handle non-mr package deploys [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 [17:51:11] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: handle non-mr package deploys [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 (owner: 10David Caro) [18:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [18:32:18] FIRING: NodeDown: The node cloudgw1002 is unreachable. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudgw1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:37:18] RESOLVED: NodeDown: The node cloudgw1002 is unreachable. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudgw1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [19:23:38] 10Tool-ldap: Indicate if account is locked - https://phabricator.wikimedia.org/T362046#10212211 (10taavi) >>! In T362046#10209244, @Legoktm wrote: > Hmm, I can't find where in LDAP that is... `pwdPolicySubentry` is set to `cn=disabled,ou=ppolicies,dc=wikimedia,dc=org`. https://ldap.toolforge.org/user/evuvave is... [19:28:43] 10Tool-video-answer-tool, 06Future-Audiences: Add ability to edit attribution and swap image in video tool - https://phabricator.wikimedia.org/T376659#10212234 (10etz) 05Open→03Resolved [19:54:47] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10212323 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm [21:00:05] 10VPS-project-Codesearch: Retain input when switching between tabs - https://phabricator.wikimedia.org/T376732#10212574 (10Aklapper) [21:20:42] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:31:56] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10212646 (10NealMcB) I was very confused to see this pop up, especially because it doesn't define "SUL" anywhere or link to background. So this is my naive attempt to do so. Please correct an... [22:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [22:20:45] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10212749 (10bd808) >>! In T368816#10201603, @Gopavasanth wrote: > @bd808/@Slst2020 , I would love to hear your thoughts and suggestions on this idea. My main feedback is that the system sho... [22:47:32] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 16d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [23:11:47] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10212839 (10bd808) >>! In T161859#10209425, @SunAfterRain wrote: > Will the Two-factor Authentication previously set up in Wikitech still be effective... [23:24:25] 10Tool-video-answer-tool, 06Future-Audiences: Create video for arbitrary topic - https://phabricator.wikimedia.org/T376660#10212856 (10derenrich) implemented in https://gitlab.wikimedia.org/repos/future-audiences/video-answer-tool/-/merge_requests/47 [23:26:25] 10Tool-video-answer-tool, 06Future-Audiences: Create video for arbitrary topic - https://phabricator.wikimedia.org/T376660#10212858 (10derenrich) {F57599450}barebones ui [23:42:52] 10wikitech.wikimedia.org: DeploymentCalendarTool needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376221#10212883 (10thcipriani) [23:44:49] 10wikitech.wikimedia.org: DeploymentCalendarTool needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376221#10212886 (10thcipriani) 05Open→03Resolved a:03thcipriani [23:50:53] 10Tool-ldap, 10Phabricator: https://ldap.toolforge.org/ integration assumes that `cn` and `uid` are equivalent - https://phabricator.wikimedia.org/T376769#10212894 (10Peachey88)