[00:36:28] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment), 10Trust and Safety Product Sprint (Sprint Cello (Oct 7 - 18)): Implement Global Contributions a... - https://phabricator.wikimedia.org/T376612#10212903 [00:36:50] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment), 10Trust and Safety Product Sprint (Sprint Cello (Oct 7 - 18)): Implement Global Contributions a... - https://phabricator.wikimedia.org/T376612#10212904 [00:43:37] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2024 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T374993#10212906 (10JJMC89) There seems to be a process gap for getting Legal to look at #WMF-NDA requests. @KFrancis, would you start the process f... [00:51:45] (03PS1) 10Krinkle: frontend: Change `apt autoremove` to apt-get to address build warning [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1078770 [00:51:46] (03PS1) 10Krinkle: frontend: Update php-composer from 2.3 to 2.8 to fix PHP 8 warnings [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1078771 [00:52:08] (03CR) 10Krinkle: [C:03+2] frontend: Change `apt autoremove` to apt-get to address build warning [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1078770 (owner: 10Krinkle) [00:52:12] (03CR) 10Krinkle: [C:03+2] frontend: Update php-composer from 2.3 to 2.8 to fix PHP 8 warnings [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1078771 (owner: 10Krinkle) [00:52:59] (03Merged) 10jenkins-bot: frontend: Change `apt autoremove` to apt-get to address build warning [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1078770 (owner: 10Krinkle) [00:53:05] (03Merged) 10jenkins-bot: frontend: Update php-composer from 2.3 to 2.8 to fix PHP 8 warnings [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1078771 (owner: 10Krinkle) [01:20:42] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:16:21] 10Tool-ldap: Indicate if account is locked - https://phabricator.wikimedia.org/T362046#10212988 (10Legoktm) a:03Legoktm [04:21:49] 10Tool-ldap: Indicate if account is locked - https://phabricator.wikimedia.org/T362046#10212991 (10Legoktm) 05Open→03Resolved https://gitlab.wikimedia.org/toolforge-repos/ldap/-/commit/336f2308cc0d6bd3c2131d3b32653643d6f77c53 {F57599955} [04:27:51] 10Tool-ldap, 10Phabricator: https://ldap.toolforge.org/ integration assumes that `cn` and `uid` are equivalent - https://phabricator.wikimedia.org/T376769#10212996 (10Legoktm) This is a silly question - is it possible to register "BryanDavis" as a uid? (probably not, because it has to be lower case?) The mai... [05:20:42] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:31:03] 10Tool-techcontribs: Sort numbers by value, not alphabetically - https://phabricator.wikimedia.org/T376567#10213064 (10Chlod) 05Open→03Resolved p:05Triage→03Medium a:03Chlod Sorry for getting to this late! My email filters weren't working well and I completely lost the Phabricator email in the nois... [06:04:41] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:14:12] (03update) 10sstefanova: tests: add basic tests for delete config route [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T362069) [06:20:04] (03update) 10sstefanova: tests: add basic tests for delete config route [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T362069) [06:21:56] (03merge) 10sstefanova: tests: add basic tests for delete config route [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/19 (https://phabricator.wikimedia.org/T362069) [06:23:13] (03update) 10sstefanova: api: fix exceptions handler responses [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/21 [06:24:08] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: DONOTMERGE components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 (https://phabricator.wikimedia.org/T362069) [06:38:38] (03update) 10sstefanova: api: fix exceptions handler responses [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/21 [06:38:54] (03update) 10sstefanova: api: fix exceptions handler responses [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/21 [08:08:57] 06cloud-services-team, 10wikitech.wikimedia.org: Labslogbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376220#10213180 (10aborrero) p:05Triage→03High it is not immediately clear to me what is the use of this bot, but I can try to get this done sometime... [08:14:48] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment), 10Trust and Safety Product Sprint (Sprint Cello (Oct 7 - 18)): Implement Global Contributions a... - https://phabricator.wikimedia.org/T376612#10213208 [08:19:59] (03approved) 10dcaro: api: fix exceptions handler responses [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/21 (owner: 10sstefanova) [08:20:02] (03update) 10dcaro: api: fix exceptions handler responses [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/21 (owner: 10sstefanova) [08:20:50] (03merge) 10sstefanova: api: fix exceptions handler responses [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/21 [08:22:43] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: DONOTMERGE components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 (https://phabricator.wikimedia.org/T362069) [08:22:46] (03CR) 10Btullis: [C:03+1] Provision dummy secrets for ceph-csi users [labs/private] - 10https://gerrit.wikimedia.org/r/1078401 (https://phabricator.wikimedia.org/T376407) (owner: 10Brouberol) [08:24:18] (03CR) 10Brouberol: [C:03+2] Provision dummy secrets for ceph-csi users [labs/private] - 10https://gerrit.wikimedia.org/r/1078401 (https://phabricator.wikimedia.org/T376407) (owner: 10Brouberol) [08:24:20] (03CR) 10Brouberol: [V:03+2 C:03+2] Provision dummy secrets for ceph-csi users [labs/private] - 10https://gerrit.wikimedia.org/r/1078401 (https://phabricator.wikimedia.org/T376407) (owner: 10Brouberol) [08:35:56] (03update) 10dcaro: deployment: Add the deployment endpoints and mock storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/18 (https://phabricator.wikimedia.org/T362069) [09:04:45] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15), 13Patch-For-Review: [builds-builder,builds-api] After the upgrade build actions take >10s - https://phabricator.wikimedia.org/T376710#10213363 (10dcaro) Done a bit of investigating, the issue seems to be the way the conversion h... [09:08:00] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15), 13Patch-For-Review: [builds-builder,builds-api] After the upgrade build actions take >10s - https://phabricator.wikimedia.org/T376710#10213386 (10dcaro) This would be fixed over time, as the old builds get cleaned (so the stored... [09:09:21] (03open) 10aborrero: maintain-kubeusers: bump quota for squid tool [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/551 (https://phabricator.wikimedia.org/T376338) [09:11:33] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [09:11:58] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [09:16:28] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [09:17:42] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [09:18:29] (03merge) 10aborrero: maintain-kubeusers: bump quota for squid tool [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/551 (https://phabricator.wikimedia.org/T376338) [09:20:24] (03open) 10aborrero: maintain-kubeusers: bump quota again for video-answer-tool tool [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/552 (https://phabricator.wikimedia.org/T376505) [09:20:33] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [09:20:37] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [09:20:42] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:24:36] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [09:25:55] 06cloud-services-team, 10Toolforge, 07Epic: [Hypothesis] WE6.3.6: Toolforge Modernization Discovery - https://phabricator.wikimedia.org/T375914#10213420 (10Sarai-WMF) [09:27:25] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [09:27:51] (03merge) 10aborrero: maintain-kubeusers: bump quota again for video-answer-tool tool [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/552 (https://phabricator.wikimedia.org/T376505) [09:29:05] 10Toolforge (Quota-requests), 13Patch-For-Review: Request increased quota for sqid Toolforge tool - https://phabricator.wikimedia.org/T376338#10213429 (10aborrero) 05Open→03Resolved [09:29:34] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10213430 (10hashar) There wa... [09:31:01] 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T376505#10213427 (10aborrero) 05Open→03Resolved please, try again! [10:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:09:05] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10213527 (10Ladsgroup) SUL refers to the account in Wikimedia projects, most notably Wikimedia Wikis. The question is that do you have an account in mediawiki.org or en.wikipedia.org or other... [10:11:51] 10wikitech.wikimedia.org: Wikitech double redirect bot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376224#10213532 (10Ladsgroup) Might sound stupid but with SUL working, a bot can simply use the pywikibot's standard redirect.py script. I don't know who runs t... [10:15:25] (03open) 10raymond-ndibe: Draft: [lima-kilo] cache container images [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/196 [10:25:02] (03update) 10raymond-ndibe: Draft: [lima-kilo] cache container images [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/196 [10:30:19] (03open) 10fnegri: Reformat all files with "tofu fmt -recursive" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95 [10:32:34] (03approved) 10aborrero: Reformat all files with "tofu fmt -recursive" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95 (owner: 10fnegri) [10:33:44] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [10:34:06] !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [10:34:17] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95 [10:34:48] !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95 [10:35:13] (03merge) 10fnegri: Reformat all files with "tofu fmt -recursive" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95 [10:38:34] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [10:42:10] (03update) 10raymond-ndibe: Draft: [lima-kilo] cache container images [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/196 [10:46:33] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for Antonin Delpeuch (Pintoch) - https://phabricator.wikimedia.org/T374995#10213609 (10Aklapper) Per previous comment I was about to add Pintoch to #wmf-nda but then realized that instructions on https://wikitech.wikimedia.org/wiki/Volunteer_ND... [10:52:44] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [10:52:54] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [10:53:00] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [10:53:39] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [10:53:40] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [10:54:06] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [10:55:38] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [10:55:38] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [10:55:54] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [10:58:28] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [10:58:28] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [10:58:49] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [11:03:50] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [11:20:30] PROBLEM - Host wikitech-static.wikimedia.org is DOWN: PING CRITICAL - Packet loss = 100% [11:22:54] RECOVERY - Host wikitech-static.wikimedia.org is UP: PING OK - Packet loss = 0%, RTA = 22.33 ms [11:44:32] (03update) 10raymond-ndibe: Draft: [lima-kilo] cache container images [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/196 [13:00:09] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10213969 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm [13:20:42] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:30:05] 06cloud-services-team: tf-infra-test fails creating dbs and k8s cluster - https://phabricator.wikimedia.org/T376802 (10fnegri) 03NEW [13:30:15] 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T376505#10214054 (10derenrich) Thanks so much! When trying to start the service with `toolforge webservice buildservice restart --mount=all -c=6 -m=8Gi` I'm getting errors like... [13:30:23] 06cloud-services-team, 10Cloud-VPS: tf-infra-test fails creating dbs and k8s cluster - https://phabricator.wikimedia.org/T376802#10214057 (10fnegri) [13:43:04] 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T376505#10214137 (10aborrero) This is a known problem with higher quotas, specially memory, as documented in here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Conta... [13:47:26] 10Toolforge (Quota-requests): Request increased quota for video-answer-tool Toolforge tool - https://phabricator.wikimedia.org/T376505#10214162 (10derenrich) Yeah I figured that was the issue. I tried walking down the requested resources and it actually seemed like the CPU was more of the limiting factor tha... [14:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:05:53] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: tf-infra-test fails creating dbs and k8s cluster - https://phabricator.wikimedia.org/T376802#10214396 (10fnegri) p:05Triage→03Medium a:03fnegri [14:16:09] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [14:16:09] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [14:16:27] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [14:19:04] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [14:19:06] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [14:19:20] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [14:20:13] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [14:20:15] (03update) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) [14:20:25] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10214504 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm executed w... [14:20:31] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [14:39:35] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10214600 (10CDanis) |**Wikitech account/LDAP:**| CDanis| |**SUL account**| CDanis (WMF)| |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]** |IDK! IDM serves me a 500 Internal Ser... [14:42:55] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10214610 (10Ladsgroup) @CDanis, I renamed your account, can you try doing password reset with "CDanis (WMF)" again? [14:44:37] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10214619 (10CDanis) >>! In T376267#10214610, @Ladsgroup wrote: > @CDanis, I renamed your account, can you try doing password reset with "CDanis (WMF)" again? Works now, thanks very much! [15:13:20] (03PS4) 10David Caro: toolforge.component.deploy: handle non-mr package deploys [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 [15:15:24] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: tf-infra-test fails creating dbs and k8s cluster - https://phabricator.wikimedia.org/T376802#10214720 (10fnegri) I fixed the db errors by resetting the trove quotas for the project as described in https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin... [15:17:04] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: handle non-mr package deploys [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 (owner: 10David Caro) [15:33:59] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: tf-infra-test fails creating dbs and k8s cluster - https://phabricator.wikimedia.org/T376802#10214815 (10fnegri) p:05Medium→03High I can replicate the issue if I try manually creating a new Trove instance from Horizon, in a different project. [15:52:12] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: tf-infra-test fails creating dbs and k8s cluster - https://phabricator.wikimedia.org/T376802#10214871 (10fnegri) 05Open→03In progress [15:56:29] 06cloud-services-team, 10wikitech.wikimedia.org, 06Trust-and-Safety: Account recovery help needed for MediaWiki account JMando (WMF) - https://phabricator.wikimedia.org/T376819 (10JLam-WMF) 03NEW [16:02:06] 06cloud-services-team, 10wikitech.wikimedia.org, 06Trust-and-Safety: Account recovery help needed for MediaWiki account JMando (WMF) - https://phabricator.wikimedia.org/T376819#10214915 (10bd808) #wikitech.wikimedia.org no longer has local 2FA protections following the 2024-10-01 detachment from Developer ac... [16:07:08] 10wikitech.wikimedia.org: Wikitech double redirect bot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376224#10214920 (10bd808) The bot has always run the standard redirect.py script. @taavi has used the account to test the production of a pywikibot container fo... [16:26:15] 10Tool-ldap, 10Phabricator: https://ldap.toolforge.org/ integration assumes that `cn` and `uid` are equivalent - https://phabricator.wikimedia.org/T376769#10214990 (10bd808) >>! In T376769#10212996, @Legoktm wrote: > This is a silly question - is it possible to register "BryanDavis" as a uid? (probably not, be... [16:28:12] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for Antonin Delpeuch (Pintoch) - https://phabricator.wikimedia.org/T374995#10214992 (10Pppery) While we're cleaning up the process we should get someone to remove the bullet point "* Get sign off by a C-level staff of the Wikimedia Foundation".... [16:30:30] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, and 2 others: dns: integrate PTR support for 2a02:ec80:a100::/48 - https://phabricator.wikimedia.org/T376462#10214998 (10cmooney) The delegations for the 4 subnets used so far on the infra-side are working also: ` cmooney@cumin1002... [16:52:52] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15): [components-api] Develop the webhook mechanism to trigger a deployment - https://phabricator.wikimedia.org/T362066#10215068 (10dcaro) [16:55:17] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 15), 13Patch-For-Review: [components-api] Develop the webhook mechanism to trigger a deployment - https://phabricator.wikimedia.org/T362066#10215070 (10dcaro) 05Open→03In progress [16:58:01] 10Tool-ldap, 10Phabricator: https://ldap.toolforge.org/ integration assumes that `cn` and `uid` are equivalent - https://phabricator.wikimedia.org/T376769#10215091 (10Legoktm) a:03Legoktm Ack, thanks for the explanation. I'll create a new `/cn/` route then and will send a MR to update Phabricator's links too. [17:01:25] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10215097 (10Jc3s5h) Wikitech account/LDAP: none SUL account: Jc3s5h Account linked on IDM: na I have visited MediaWiki:Loginprompt: Y I have tried to reset my password using Special:PasswordR... [17:20:42] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:07:18] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [18:12:30] 06cloud-services-team, 10wikitech.wikimedia.org: Labslogbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376220#10215289 (10bd808) >>! In T376220#10213180, @aborrero wrote: > it is not immediately clear to me what is the use of this bot, but I can try to get... [18:59:58] 10Tool-paulina: Add Wikisource links to works (feature request) - https://phabricator.wikimedia.org/T376570#10215450 (10Pepe_piton) p:05Triage→03Medium [19:00:41] 10Tool-paulina: Add Wikisource links to works (feature request) - https://phabricator.wikimedia.org/T376570#10215453 (10Pepe_piton) a:03Pepe_piton [19:00:43] 10Tool-paulina: Add Wikisource links to works (feature request) - https://phabricator.wikimedia.org/T376570#10215445 (10Pepe_piton) This feature is already implemented, but in a way that it seems is not intuitive. The link to the Wikisource item is only displayed when the person browses Paulina in the same lang... [19:21:25] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for Antonin Delpeuch (Pintoch) - https://phabricator.wikimedia.org/T374995#10215499 (10KFrancis) Because there isn't a specific inbox for NDA requests, in the past, we've used contracts@wikimedia.org. You may direct them there. You can also d... [19:45:39] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2024 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T374993#10215543 (10KFrancis) Hello @JJMC89, @Pintoch, @LucasWerkmeister, @waldyrious, @SD0001, @theprotonade, please email your full name, mailing a... [19:54:14] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2024 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T374993#10215584 (10LucasWerkmeister) In my case, it turned out that I already signed the NDA but wasn’t added to the #wmf-nda project (see T375001);... [19:58:23] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2024 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T374993#10215596 (10KFrancis) Hi @LucasWerkmeister if you've already signed an NDA, then no need. Thanks for checking! [21:21:27] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:07:18] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [22:47:33] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 15d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [23:02:58] 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure: Quota increase for Integration project (Jenkins CI runners) - https://phabricator.wikimedia.org/T376847 (10bd808) 03NEW [23:17:00] 06cloud-services-team, 10Toolforge: Allow to remap/rename secrets from envvars when running specific toolforge job - https://phabricator.wikimedia.org/T376849 (10Edgars2007) 03NEW [23:31:51] (03CR) 10Dzahn: "Hello Antoine, so the reason I hadn't just merged that was that I wanted to discuss it in meeting but we didn't get to that yet, partially" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/920243 (https://phabricator.wikimedia.org/T336710) (owner: 10Hashar) [23:33:56] (03CR) 10Dzahn: [C:03+2] Revert "switch to main gerrit server instead of using the replica" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/920243 (https://phabricator.wikimedia.org/T336710) (owner: 10Hashar) [23:34:00] (03PS4) 10Hashar: Revert "switch to main gerrit server instead of using the replica" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/920243 (https://phabricator.wikimedia.org/T336710) [23:35:39] 10VPS-project-Codesearch, 10VPS-project-Extdist, 06collaboration-services, 10Gerrit, 13Patch-For-Review: Move clients off of gerrit-replica.wikimedia.org back to gerrit.wikimedia.org - https://phabricator.wikimedia.org/T336710#10215968 (10Dzahn) Hello Antoine, so the reason I hadn't just merged that was... [23:37:49] 10VPS-project-Codesearch, 10VPS-project-Extdist, 06collaboration-services, 10Gerrit, 13Patch-For-Review: Move clients off of gerrit-replica.wikimedia.org back to gerrit.wikimedia.org - https://phabricator.wikimedia.org/T336710#10215974 (10Ladsgroup) BTW, last time we did the change, the hound storage fil... [23:41:49] 06cloud-services-team, 10Toolforge: Allow to remap/rename secrets from envvars when running specific toolforge job - https://phabricator.wikimedia.org/T376849#10215977 (10bd808) {T365048} is a thing that folks who are doing a lot of fiddling with envvars should be aware of. A remapping feature like this one co... [23:42:41] FIRING: PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted