[03:28:42] PROBLEM - MD RAID on cloudcephosd1031 is CRITICAL: CRITICAL: State: degraded, Active: 1, Working: 1, Failed: 0, Spare: 0 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering [03:28:43] ACKNOWLEDGEMENT - MD RAID on cloudcephosd1031 is CRITICAL: CRITICAL: State: degraded, Active: 1, Working: 1, Failed: 0, Spare: 0 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T364060 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering [03:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:52:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:57:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:52:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:57:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:02:23] 10Cloud-VPS (Project-requests), 10Fiwiki-Wikidata-Commons: Request creation of imagehash VPS project - https://phabricator.wikimedia.org/T364063 (10TuukkaH) 03NEW [06:06:29] 10Cloud-VPS (Project-requests), 10Fiwiki-Wikidata-Commons: Request creation of imagehash VPS project - https://phabricator.wikimedia.org/T364063#9765902 (10TuukkaH) [06:07:56] 10Cloud-VPS (Project-requests), 10Fiwiki-Wikidata-Commons, 03Wikimedia-Hackathon-2024: Request creation of imagehash VPS project - https://phabricator.wikimedia.org/T364063#9765903 (10TuukkaH) [08:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:22:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:27:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:28:36] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [labs/private] - 10https://gerrit.wikimedia.org/r/1026693 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [08:42:20] 10Tool-bub2, 10Internet-Archive: Documentation for BUB2 Tool - https://phabricator.wikimedia.org/T364082 (10wassan.anmol117) 03NEW [08:42:50] 10Tool-bub2, 10Internet-Archive: Documentation for BUB2 Tool - https://phabricator.wikimedia.org/T364082#9766371 (10wassan.anmol117) [08:42:51] 10Tools, 06Tech-Docs-Team, 07Documentation, 03Wikimedia-Hackathon-2024: [Hackathon 2024] Improve technical documentation of tools - https://phabricator.wikimedia.org/T358040#9766372 (10wassan.anmol117) [09:07:08] (03PS1) 10Muehlenhoff: Remove obsolete stub certs [labs/private] - 10https://gerrit.wikimedia.org/r/1026806 [09:13:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:28:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:32:27] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent - https://phabricator.wikimedia.org/T362233#9766560 (10aborrero) [09:34:15] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent - https://phabricator.wikimedia.org/T362233#9766565 (10aborrero) 05In progress→03Resolved a:03aborrero [09:58:24] 06cloud-services-team, 10Toolforge (Toolforge iteration 09): toolforge lima-kilo: refresh maintain-kubeusers test data - https://phabricator.wikimedia.org/T363482#9766659 (10CodeReviewBot) aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/19 maintain-kubeus... [10:07:57] 10Cloud-VPS (Project-requests), 10Internet-Archive: Request creation of BUB2 VPS project - https://phabricator.wikimedia.org/T364094 (10wassan.anmol117) 03NEW [10:08:51] 10Cloud-VPS (Project-requests), 10Internet-Archive: Request creation of BUB2 VPS project - https://phabricator.wikimedia.org/T364094#9766688 (10wassan.anmol117) [10:14:49] 06cloud-services-team, 10Cloud-VPS (Project-requests), 10Fiwiki-Wikidata-Commons, 03Wikimedia-Hackathon-2024: Request creation of imagehash VPS project - https://phabricator.wikimedia.org/T364063#9766699 (10bd808) +1 [10:15:44] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge lima-kilo: refresh maintain-kubeusers test data - https://phabricator.wikimedia.org/T363482#9766708 (10CodeReviewBot) aborrero closed https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests... [10:16:01] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [10:16:12] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [10:16:48] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [10:17:00] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [10:18:20] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge lima-kilo: refresh maintain-kubeusers test data - https://phabricator.wikimedia.org/T363482#9766723 (10CodeReviewBot) aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests... [10:18:58] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge lima-kilo: refresh maintain-kubeusers test data - https://phabricator.wikimedia.org/T363482#9766704 (10CodeReviewBot) aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests... [10:19:22] 06cloud-services-team, 10Cloud-VPS (Project-requests), 10Internet-Archive: Request creation of BUB2 VPS project - https://phabricator.wikimedia.org/T364094#9766730 (10bd808) +1 [10:21:02] 06cloud-services-team, 10Cloud-VPS (Project-requests), 10Fiwiki-Wikidata-Commons, 03Wikimedia-Hackathon-2024: Request creation of imagehash VPS project - https://phabricator.wikimedia.org/T364063#9766735 (10taavi) a:03taavi [10:21:09] 06cloud-services-team, 10Cloud-VPS (Project-requests), 10Internet-Archive: Request creation of BUB2 VPS project - https://phabricator.wikimedia.org/T364094#9766736 (10taavi) a:03taavi [10:21:40] !log taavi@cloudcumin1001 imagehash START - Cookbook wmcs.vps.create_project for project imagehash in eqiad1 (T364063) [10:21:41] taavi@cloudcumin1001: Unknown project "imagehash" [10:21:41] taavi@cloudcumin1001: Did you mean to say "tools.imagehash" instead? [10:21:41] T364063: Request creation of imagehash VPS project - https://phabricator.wikimedia.org/T364063 [10:22:15] !log taavi@cloudcumin1001 imagehash END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project imagehash in eqiad1 (T364063) [10:22:15] taavi@cloudcumin1001: Unknown project "imagehash" [10:22:15] taavi@cloudcumin1001: Did you mean to say "tools.imagehash" instead? [10:23:13] !log taavi@cloudcumin1001 imagehash START - Cookbook wmcs.vps.add_user_to_project for user 'tuukka' in role 'member' (T364063) [10:23:13] taavi@cloudcumin1001: Unknown project "imagehash" [10:23:13] taavi@cloudcumin1001: Did you mean to say "tools.imagehash" instead? [10:23:19] !log taavi@cloudcumin1001 imagehash END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tuukka' in role 'member' (T364063) [10:23:20] taavi@cloudcumin1001: Unknown project "imagehash" [10:23:20] taavi@cloudcumin1001: Did you mean to say "tools.imagehash" instead? [10:23:24] !log taavi@cloudcumin1001 imagehash START - Cookbook wmcs.vps.add_user_to_project for user 'zache-tool' in role 'member' (T364063) [10:23:24] taavi@cloudcumin1001: Unknown project "imagehash" [10:23:24] taavi@cloudcumin1001: Did you mean to say "tools.imagehash" instead? [10:23:30] !log taavi@cloudcumin1001 imagehash END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'zache-tool' in role 'member' (T364063) [10:23:30] taavi@cloudcumin1001: Unknown project "imagehash" [10:23:30] taavi@cloudcumin1001: Did you mean to say "tools.imagehash" instead? [10:23:52] 06cloud-services-team, 10Cloud-VPS (Project-requests), 10Fiwiki-Wikidata-Commons, 03Wikimedia-Hackathon-2024: Request creation of imagehash VPS project - https://phabricator.wikimedia.org/T364063#9766752 (10taavi) 05Open→03Resolved [10:24:20] !log taavi@cloudcumin1001 bub2 START - Cookbook wmcs.vps.create_project for project bub2 in eqiad1 (T364094) [10:24:21] taavi@cloudcumin1001: Unknown project "bub2" [10:24:21] taavi@cloudcumin1001: Did you mean to say "tools.bub2" instead? [10:24:21] T364094: Request creation of BUB2 VPS project - https://phabricator.wikimedia.org/T364094 [10:24:51] taavi: ^^^ the cookbook SAL integration needs some work [10:24:56] !log taavi@cloudcumin1001 bub2 END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project bub2 in eqiad1 (T364094) [10:24:56] taavi@cloudcumin1001: Unknown project "bub2" [10:24:56] taavi@cloudcumin1001: Did you mean to say "tools.bub2" instead? [10:25:03] i say that is a stashbot issue [10:25:48] !log taavi@cloudcumin1001 bub2 START - Cookbook wmcs.vps.add_user_to_project for user 'theprotonade' in role 'member' (T364094) [10:25:48] taavi@cloudcumin1001: Unknown project "bub2" [10:25:48] taavi@cloudcumin1001: Did you mean to say "tools.bub2" instead? [10:26:29] 06cloud-services-team, 10Cloud-VPS (Project-requests), 10Internet-Archive: Request creation of BUB2 VPS project - https://phabricator.wikimedia.org/T364094#9766768 (10taavi) > Developer account usernames of requestors: @wassan.anmol117 @theprotonade These are Phabricator usernames, not developer account use... [10:26:35] !log taavi@cloudcumin1001 bub2 END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'theprotonade' in role 'member' (T364094) [10:26:35] taavi@cloudcumin1001: Unknown project "bub2" [10:26:35] taavi@cloudcumin1001: Did you mean to say "tools.bub2" instead? [10:31:57] 06cloud-services-team, 10Cloud-VPS (Project-requests), 10Internet-Archive: Request creation of BUB2 VPS project - https://phabricator.wikimedia.org/T364094#9766799 (10wassan.anmol117) >>! In T364094#9766768, @taavi wrote: >> Developer account usernames of requestors: @wassan.anmol117 @theprotonade > > These... [10:36:59] !log taavi@cloudcumin1001 bub2 START - Cookbook wmcs.vps.add_user_to_project for user 'wassananmol' in role 'member' (T364094) [10:37:02] T364094: Request creation of BUB2 VPS project - https://phabricator.wikimedia.org/T364094 [10:37:43] !log taavi@cloudcumin1001 bub2 END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'wassananmol' in role 'member' (T364094) [10:39:13] 06cloud-services-team, 10Cloud-VPS (Project-requests), 10Internet-Archive: Request creation of BUB2 VPS project - https://phabricator.wikimedia.org/T364094#9766835 (10taavi) 05Open→03Resolved Thanks, added. [10:53:49] 14Grid-Engine-to-K8s-Migration, 03Wikimedia-Hackathon-2024: Migrate noclaims from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319927#9766866 (10Multichill) 05Declined→03In progress [10:55:17] 14Grid-Engine-to-K8s-Migration, 03Wikimedia-Hackathon-2024: Migrate noclaims from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319927#9766869 (10Multichill) 05In progress→03Resolved Works again, see https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.noclaims/SA... [11:18:56] 06cloud-services-team, 10Cloud-VPS (Project-requests), 10Fiwiki-Wikidata-Commons, 03Wikimedia-Hackathon-2024: Request creation of imagehash VPS project - https://phabricator.wikimedia.org/T364063#9766990 (10TuukkaH) Thank you @bd808 and @taavi! [11:27:38] 06cloud-services-team, 10Data-Platform-SRE (2024.04.15 - 2024.05.05): Streamline the creation of ceph storage cluster user accounts - https://phabricator.wikimedia.org/T364105 (10BTullis) 03NEW [11:39:30] 06cloud-services-team, 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: toolforge lima-kilo: refresh maintain-kubeusers test data - https://phabricator.wikimedia.org/T363482#9767080 (10aborrero) 05In progress→03Resolved [11:40:50] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Project WP1.0/mwoffliner requests Trove instance with 75 GB - https://phabricator.wikimedia.org/T363901#9767086 (10bd808) +1 [11:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:43:28] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Project WP1.0/mwoffliner requests Trove instance with 75 GB - https://phabricator.wikimedia.org/T363901#9767093 (10aborrero) a:03aborrero [11:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:50:02] (03Abandoned) 10Lewis Cawte: Set defaultbranch for git review [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1003081 (owner: 10Lewis Cawte) [11:52:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:56:31] 14Grid-Engine-to-K8s-Migration, 03Wikimedia-Hackathon-2024: Migrate geograph from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319765#9767150 (10Multichill) 05Declined→03Open [11:57:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:57:49] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudidm2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [11:58:10] 14Grid-Engine-to-K8s-Migration, 03Wikimedia-Hackathon-2024: Migrate geograph from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319765#9767155 (10Multichill) 05Open→03Resolved Working again. Just have to check in a couple of hours if I don't have a ton of queued jobs... [12:12:18] 06cloud-services-team, 10Data-Platform-SRE (2024.04.15 - 2024.05.05), 13Patch-For-Review: Streamline the creation of ceph storage cluster user accounts - https://phabricator.wikimedia.org/T364105#9767230 (10BTullis) p:05Triage→03Low [12:15:40] 06cloud-services-team, 10Data-Platform-SRE (2024.04.15 - 2024.05.05), 13Patch-For-Review: Streamline the creation of ceph storage cluster user accounts - https://phabricator.wikimedia.org/T364105#9767241 (10BTullis) @dcaro @aborrero - I have submitted a patch for this, but it's not blocking me. I have just n... [12:20:59] 06cloud-services-team, 10Toolforge: toolforge: identify and cache in our container registry all kyverno images - https://phabricator.wikimedia.org/T364113 (10aborrero) 03NEW [12:23:12] 06cloud-services-team, 10Toolforge: toolforge: identify and cache in our container registry all kyverno images - https://phabricator.wikimedia.org/T364113#9767278 (10aborrero) p:05Triage→03Medium [12:30:10] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Project WP1.0/mwoffliner requests Trove instance with 75 GB - https://phabricator.wikimedia.org/T363901#9767314 (10aborrero) 05Open→03Resolved I updated the quotas, but the [[ https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Trove#Adjusting... [12:45:00] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [12:45:11] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [12:46:02] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [12:46:13] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [13:04:33] 10Tool-itwiki: Creazione utenza Commissione arbitrale - https://phabricator.wikimedia.org/T364120 (10valcio) 03NEW [13:04:49] 10Tool-itwiki: Creazione utenza Commissione arbitrale - https://phabricator.wikimedia.org/T364120#9767486 (10valcio) 05Open→03In progress [13:11:05] 10Tool-itwiki: Creazione utenza Commissione arbitrale - https://phabricator.wikimedia.org/T364120#9767492 (10valcio) p:05Triage→03Medium [13:20:52] 10Tool-yearinreview, 10New-Engagement-Experiments, 03Wikimedia-Hackathon-2024: Improve the year in review tool for next year - https://phabricator.wikimedia.org/T362898#9767525 (10Gopavasanth) [13:20:56] 10Tool-yearinreview, 06Editing-team, 06Growth-Team, 10New-Engagement-Experiments, 06Web-Team-Backlog: "Year in Review" for Wikipedia Contributions - https://phabricator.wikimedia.org/T328385#9767526 (10Gopavasanth) [13:23:21] 10Tool-itwiki: Creazione utenza Commissione arbitrale - https://phabricator.wikimedia.org/T364120#9767540 (10Civvi) Utenza creata: https://it.wikipedia.org/w/index.php?title=Speciale:Registri&logid=56182630 [13:25:23] 10Tool-itwiki: Creazione utenza Commissione arbitrale - https://phabricator.wikimedia.org/T364120#9767556 (10valcio) 05In progress→03Resolved L'utenza è stata creata (cfr. [[ https://it.wikipedia.org/w/index.php?title=Speciale:Registri&logid=56182630 | log 56182630 ]]). Per ora eventuali richieste che ne... [13:32:15] 10Tool-wikiloves: Armenia in WLE statistic - https://phabricator.wikimedia.org/T364035#9767616 (10Aklapper) Hi @Beko, thanks for taking the time to report this! Where to see this problem? Please provide sufficient information that allows someone else to reproduce this problem. Is this about #Tool-wikiloves? [13:45:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:50:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:08:10] 10Tools, 06translatewiki.net, 10Language-Team (Language-2024-April-June), 03Localization Infrastructure FY2023-24, and 2 others: Make Wikidata Image Positions tool translatable on translatewiki.net - https://phabricator.wikimedia.org/T363626#9767779 (10siebrand) I'm interested in helping this move forward. [14:19:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-meta-monitor-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:19:29] 06cloud-services-team, 10Data-Platform-SRE (2024.04.15 - 2024.05.05), 13Patch-For-Review: Streamline the creation of ceph storage cluster user accounts - https://phabricator.wikimedia.org/T364105#9767860 (10aborrero) the proposed patch LGTM. [14:28:25] 10Tool-wikiloves: Armenia in WLE statistic - https://phabricator.wikimedia.org/T364035#9767901 (10Beko) Armenia has already uploaded pictures that are in the appropriate category, which is in the same category as all the other countries, but Armenia does not appear in the statistics. / https://wikiloves.toolfo... [14:29:28] (03PS1) 10Jforrester: release: Bump grunt-stylelint to 0.20.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1026943 [14:29:40] (03CR) 10Jforrester: [C:03+2] release: Bump grunt-stylelint to 0.20.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1026943 (owner: 10Jforrester) [14:29:48] (03CR) 10Majavah: [C:03+2] release: Bump grunt-stylelint to 0.20.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1026943 (owner: 10Jforrester) [14:30:15] (03Merged) 10jenkins-bot: release: Bump grunt-stylelint to 0.20.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1026943 (owner: 10Jforrester) [14:42:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:52:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:15:00] 06cloud-services-team, 10Data-Platform-SRE (2024.04.15 - 2024.05.05), 13Patch-For-Review: Streamline the creation of ceph storage cluster user accounts - https://phabricator.wikimedia.org/T364105#9768109 (10BTullis) Thanks @aborrero. I will look at deploying it next week. [15:17:36] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10XTools, 07Epic, 10Temporary accounts (Create/update essential tools/anti-abuse management): [Epic] Implement global user contributions feature - https://phabricator.wikimedia.org/T337089#9768115 (10kostajh) [15:33:43] (03CR) 10Bking: [C:03+1] Remove obsolete dummy cert [labs/private] - 10https://gerrit.wikimedia.org/r/1026439 (https://phabricator.wikimedia.org/T360439) (owner: 10Muehlenhoff) [15:37:58] 06cloud-services-team, 10Data-Platform-SRE (2024.05.06 - 2024.05.26), 13Patch-For-Review: Streamline the creation of ceph storage cluster user accounts - https://phabricator.wikimedia.org/T364105#9768225 (10Gehel) [15:40:24] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [15:40:34] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [15:40:50] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [15:41:03] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [15:45:25] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Project WP1.0/mwoffliner requests Trove instance with 75 GB - https://phabricator.wikimedia.org/T363901#9768296 (10Audiodude) Yes it seems to have worked! Thanks so much. [15:57:52] 10Data-Services: logging views are incredibly slow for specific targets - https://phabricator.wikimedia.org/T364151 (10MusikAnimal) 03NEW p:05Triage→03High [15:57:55] 10Data-Services: logging views are incredibly slow for specific targets - https://phabricator.wikimedia.org/T364151#9768380 (10MusikAnimal) This effects the #xtools Edit Counter and hence is very widespread. Having spoken with @taavi it sounds like there isn't a quick fix, so I will disable the effected queries... [15:59:13] 10Data-Services: logging views are incredibly slow for specific targets - https://phabricator.wikimedia.org/T364151#9768384 (10MusikAnimal) [16:15:10] 06cloud-services-team, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 13Patch-For-Review: puppet servers run out of inodes in puppet code volume - https://phabricator.wikimedia.org/T364047#9768450 (10jhathaway) ok, I think I understand what is happening now, my apologies for my ignorance on how g10... [16:18:31] (03PS1) 10Valerio Bozzolan: become: add more documentation and follow shellcheck tips [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1026967 [16:22:10] (03PS1) 10Jforrester: Revert "release: Bump grunt-stylelint to 0.20.0" [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1026841 [16:22:14] (03CR) 10Jforrester: [C:03+2] Revert "release: Bump grunt-stylelint to 0.20.0" [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1026841 (owner: 10Jforrester) [16:23:22] (03Merged) 10jenkins-bot: Revert "release: Bump grunt-stylelint to 0.20.0" [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1026841 (owner: 10Jforrester) [17:14:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:24:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:36:04] 14Grid-Engine-to-K8s-Migration: Migrate multichill from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319912#9768804 (10Multichill) 05Declined→03Open [17:36:33] 14Grid-Engine-to-K8s-Migration, 03Wikimedia-Hackathon-2024: Migrate multichill from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319912#9768805 (10Multichill) [18:35:48] 10Wikibugs: wikibugs SSE client does not see events from gitlab-webhooks in real-time. - https://phabricator.wikimedia.org/T363877#9768913 (10bd808) The issue seems to be related to HTTP/1.1 vs HTTP/2. As previously noted, `curl -v --no-buffer -H 'Accept: text/event-stream' 'https://gitlab-webhooks.toolforge.org... [18:44:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:54:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:58:40] 10Tool-bub2, 03Wikimedia-Hackathon-2024: Move BUB2 to Cloud VPS - https://phabricator.wikimedia.org/T364175 (10wassan.anmol117) 03NEW [19:59:03] 10Tool-bub2, 03Wikimedia-Hackathon-2024: Move BUB2 to Cloud VPS - https://phabricator.wikimedia.org/T364175#9769130 (10wassan.anmol117) a:03wassan.anmol117 [20:05:41] 10Tool-bub2, 03Wikimedia-Hackathon-2024: Commons Metadata is breaking if it contains special characters - https://phabricator.wikimedia.org/T364176 (10wassan.anmol117) 03NEW [20:12:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:22:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:42:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:52:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:29:38] 10Tool-inteGraality: Queries endpoint support for Property columns with qualifier - https://phabricator.wikimedia.org/T251008#9769368 (10JeanFred) [22:47:45] 10Tool-inteGraality: Queries endpoint support for Property columns with qualifier - https://phabricator.wikimedia.org/T251008#9769388 (10JeanFred) Ok, the code is now in a shape where I can do this. I’m only missing the SPARQL queries :) Let’s continue to take https://www.wikidata.org/wiki/User:Jean-Fr%C3%A9d%C3... [23:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:22:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:27:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:42:32] 10Data-Services: logging views are incredibly slow for specific targets - https://phabricator.wikimedia.org/T364151#9769515 (10Nintendofan885) [23:46:09] 10Data-Services: logging views are incredibly slow for specific targets - https://phabricator.wikimedia.org/T364151#9769517 (10Nintendofan885)