[00:31:27] 10MediaWiki-extensions-OpenStackManager, 06Diffusion-Repository-Administrators, 10Projects-Cleanup, 06translatewiki.net, 10Wikimedia-GitHub: Archive the OpenStackManager extension - https://phabricator.wikimedia.org/T367220#9887181 (10Pppery) [00:40:11] 10MediaWiki-extensions-OpenStackManager, 06Diffusion-Repository-Administrators, 10Projects-Cleanup, 06translatewiki.net, 10Wikimedia-GitHub: Archive the OpenStackManager extension - https://phabricator.wikimedia.org/T367220#9887190 (10Pppery) [02:08:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:15:56] FIRING: SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:37:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:42:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:52:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:10:56] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:37:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:07:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:12:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:22:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:36:22] 06cloud-services-team, 06SRE-OnFire, 10Sustainability (Incident Followup): [grafana,ceph] Add both ends of switch links to the error/discard dashboards and include them also in the health section - https://phabricator.wikimedia.org/T367336#9887501 (10dcaro) [07:07:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:21:40] (03update) 10dcaro: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (owner: 10raymond-ndibe) [07:22:05] 10Quarry: Users get logged out from Quarry every day (or two) - https://phabricator.wikimedia.org/T362025#9887551 (10SD0001) This is because of redis pod getting restarted. It'll also happen during deployments. Ideally, we should configure RDB or AOF persistence for it. [07:42:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:44:27] (03approved) 10dcaro: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (owner: 10raymond-ndibe) [07:44:28] (03update) 10dcaro: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (owner: 10raymond-ndibe) [07:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:52:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:53:55] 06cloud-services-team, 10Toolforge (Toolforge iteration 11): [api-gateway] add alert for uptime - https://phabricator.wikimedia.org/T348633#9887602 (10Slst2020) I'd like to do this if someone could guide me or point me to some similar past task [08:24:18] 10Toolforge (Toolforge iteration 11): [builds-cli,builds-api] `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701#9887644 (10Slst2020) This is unlikely to be fixed even with new robot permissions. Only admin accounts can access system-wide configurations (via `api/v2.0/configura... [08:31:29] (03update) 10sstefanova: vm: set fs.inotify.max_user_instances=1024 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/141 (owner: 10dcaro) [08:31:42] (03approved) 10sstefanova: vm: set fs.inotify.max_user_instances=1024 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/141 (owner: 10dcaro) [08:37:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:42:20] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9887687 (10fnegri) 05In progress→03Resolved The specific error described in this bug report (`Access denied for user 'quarry'@'172.16... [08:47:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:54:57] 06cloud-services-team, 10Toolforge, 07Kubernetes: toolforge: evaluate DNS (coredns) autoscale options - https://phabricator.wikimedia.org/T239404#9887713 (10aborrero) [08:57:45] 06cloud-services-team, 10Toolforge: toolforge: track PSP migration plan - https://phabricator.wikimedia.org/T364297#9887720 (10aborrero) [08:57:47] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): Incident: 2024-06-12 toolforge k8s control plane - https://phabricator.wikimedia.org/T367348#9887721 (10aborrero) [08:57:59] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): Incident: 2024-06-12 toolforge k8s control plane - https://phabricator.wikimedia.org/T367348#9887714 (10aborrero) [08:59:06] 06cloud-services-team, 10Toolforge: toolforge: track PSP migration plan - https://phabricator.wikimedia.org/T364297#9887717 (10aborrero) [09:05:54] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): Verify that kyverno policies are evaluated only for namespaced resources - https://phabricator.wikimedia.org/T367350#9887727 (10aborrero) [09:08:26] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): Verify that kyverno policies are evaluated only for namespaced resources - https://phabricator.wikimedia.org/T367350#9887732 (10aborrero) 05Open→03In progress p:05Triage→03High In the upstream docs: > Policies can be defined a... [09:08:34] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): Incident: 2024-06-12 toolforge k8s control plane - https://phabricator.wikimedia.org/T367348#9887737 (10aborrero) p:05Triage→03High [09:13:29] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): toolforge: kyverno has a track record of overloading the cluster, maybe on new ways - https://phabricator.wikimedia.org/T367386 (10aborrero) 03NEW [09:16:15] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): toolforge: kyverno has a track record of overloading the cluster, maybe on new ways - https://phabricator.wikimedia.org/T367386#9887775 (10aborrero) 05Open→03In progress p:05Triage→03High [09:18:56] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): toolforge: consider scaling the k8s control plane - https://phabricator.wikimedia.org/T367388 (10aborrero) 03NEW [09:20:07] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): toolforge: consider scaling the k8s control plane - https://phabricator.wikimedia.org/T367388#9887810 (10aborrero) 05Open→03In progress p:05Triage→03High [09:23:08] 10MediaWiki-extensions-OpenStackManager, 06Diffusion-Repository-Administrators, 10Projects-Cleanup, 06translatewiki.net, 10Wikimedia-GitHub: Archive the OpenStackManager extension - https://phabricator.wikimedia.org/T367220#9887823 (10taavi) [09:24:15] 06cloud-services-team, 10Cloud-VPS, 06SRE-OnFire, 10Sustainability (Incident Followup): [grafana,ceph] Add both ends of switch links to the error/discard dashboards and include them also in the health section - https://phabricator.wikimedia.org/T367336#9887824 (10taavi) [09:33:49] 10Tool-schedule-deployment: Leave a comment on the Gerrit change when it is scheduled for a backport - https://phabricator.wikimedia.org/T366763#9887844 (10kostajh) >>! In T366763#9885001, @bd808 wrote: > The current code includes a tag of `autogenerated:{settings.GERRIT_ROBOT_ID}` on the reviews it leaves.... [09:34:45] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): toolforge: improve HAproxy and k8s apiserver interaction - https://phabricator.wikimedia.org/T367389 (10aborrero) 03NEW [09:38:15] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): toolforge: improve HAproxy and k8s apiserver interaction - https://phabricator.wikimedia.org/T367389#9887862 (10aborrero) p:05Triage→03High [09:42:22] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): toolforge: improve HAproxy and k8s apiserver interaction - https://phabricator.wikimedia.org/T367389#9887873 (10taavi) a:05taavi→03None [09:42:24] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): toolforge: improve HAproxy and k8s apiserver interaction - https://phabricator.wikimedia.org/T367389#9887875 (10taavi) [09:47:39] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): [k8s,infra,alerting] improve HAproxy and k8s apiserver interaction - https://phabricator.wikimedia.org/T367389#9887908 (10dcaro) [09:47:43] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): [k8s,infra] consider scaling the k8s control plane - https://phabricator.wikimedia.org/T367388#9887911 (10dcaro) [09:47:56] 06cloud-services-team, 10Toolforge: [k8s,infra] track PSP migration plan - https://phabricator.wikimedia.org/T364297#9887912 (10dcaro) [09:48:09] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): [k8s,infra] Verify that kyverno policies are evaluated only for namespaced resources - https://phabricator.wikimedia.org/T367350#9887914 (10dcaro) [09:49:03] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): [k8s,infra,alerting] improve HAproxy and k8s apiserver interaction - https://phabricator.wikimedia.org/T367389#9887918 (10dcaro) [09:49:45] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): [k8s,infra] scale up coredns replicas - https://phabricator.wikimedia.org/T333934#9887922 (10dcaro) [09:50:01] 10Toolforge: [k8s,infra] k8s control plane freezing and other stability issues - https://phabricator.wikimedia.org/T333922#9887923 (10dcaro) [09:50:10] 10Cloud-Services, 06cloud-services-team, 10Sustainability (Incident Followup): Fix HA proxy load-balancer health check monitor to not poll nodes where the API is not responding - https://phabricator.wikimedia.org/T367349#9887916 (10dcaro) →14Duplicate dup:03T367389 [09:50:34] 06cloud-services-team, 10Toolforge, 07Kubernetes: [k8s,infra] evaluate DNS (coredns) autoscale options - https://phabricator.wikimedia.org/T239404#9887926 (10dcaro) [09:59:29] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): [k8s,infra] kyverno has a track record of overloading the cluster, maybe on new ways - https://phabricator.wikimedia.org/T367386#9887936 (10dcaro) [10:00:05] (03approved) 10dcaro: vm: set fs.inotify.max_user_instances=1024 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/141 [10:00:11] (03merge) 10dcaro: vm: set fs.inotify.max_user_instances=1024 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/141 [10:01:39] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 10Quarry, 13Patch-For-Review: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9887935 (10fnegri) After the fixes in T365374 (thanks to @SD0001 for the fix and thanks to @rook f... [10:02:57] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Quarry: Support queries against Quarry's own database and ToolsDB - https://phabricator.wikimedia.org/T151158#9887943 (10fnegri) p:05Low→03Medium [10:03:51] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Quarry: Support queries against Quarry's own database and ToolsDB - https://phabricator.wikimedia.org/T151158#9887941 (10fnegri) a:03fnegri [10:04:23] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Quarry: Support queries against Quarry's own database and ToolsDB - https://phabricator.wikimedia.org/T151158#9887944 (10fnegri) 05Open→03In progress [10:07:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:17:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:21:27] 06cloud-services-team, 10Toolforge (Toolforge iteration 11): [api-gateway] add alert for uptime - https://phabricator.wikimedia.org/T348633#9887996 (10dcaro) On one side you have to register the url to be added to the pingthing/blackbox monitoring config, same as these but for the new url: https://gerrit.wikim... [10:26:13] 10cloud-services-team (FY2023/2024-Q3-Q4), 10superset.wmcloud.org: Allow Superset to query ToolsDB public databases - https://phabricator.wikimedia.org/T367393 (10fnegri) 03NEW [10:34:48] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Quarry: Support queries against Quarry's own database and ToolsDB - https://phabricator.wikimedia.org/T151158#9888093 (10fnegri) We are finally close to resolving this task! After the work in T348407 I can successfully query ToolsDB from Quarry. Access at the moment... [11:07:15] PROBLEM - ensure kvm processes are running on cloudvirt1032 is CRITICAL: PROCS CRITICAL: 0 processes with regex args qemu-system-x86_64 https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [11:07:35] ^me, sorry [11:08:15] RECOVERY - ensure kvm processes are running on cloudvirt1032 is OK: PROCS OK: 1 process with regex args qemu-system-x86_64 https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [11:37:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:47:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:47:56] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:48:11] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:13:16] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): [k8s,infra] kyverno has a track record of overloading the cluster, maybe on new ways - https://phabricator.wikimedia.org/T367386#9888347 (10aborrero) opened upstream ticket: https://github.com/kyverno/kyverno/issues/10458 [12:31:12] (03update) 10sstefanova: api: remove unprefixed endpoints [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/33 [12:37:11] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#9888392 (10taavi) [12:42:21] (03approved) 10aborrero: api: auth and proxy requests to the backend APIs [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/23 (https://phabricator.wikimedia.org/T363983) (owner: 10dcaro) [12:50:52] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 10Quarry: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9888500 (10fnegri) [13:07:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:23:39] (03approved) 10dcaro: api: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/jobs-api] (add_messages_to_all_responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/92 (owner: 10sstefanova) [13:24:24] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2003-dev.codfw.wmnet' [13:29:47] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2003-dev.codfw.wmnet' [13:30:54] (03update) 10sstefanova: api: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/jobs-api] (add_messages_to_all_responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/92 [13:31:04] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#9888687 (10taavi) [13:37:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:38:23] 10Tool-containers, 10Toolforge: Provide a Redis container for use within a tool's namespace - https://phabricator.wikimedia.org/T360378#9888704 (10Pintoch) I have tried the approach described at https://wikitech.wikimedia.org/wiki/Tool:Containers#Redis_container and it works! The editgroups tool is functional... [13:40:07] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' [13:42:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:52:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:54:20] (03update) 10sstefanova: api: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/jobs-api] (add_messages_to_all_responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/92 [13:59:48] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' [14:08:49] 06cloud-services-team, 10Cloud-VPS, 06SRE-OnFire, 10Sustainability (Incident Followup): [grafana,ceph] Add both ends of switch links to the error/discard dashboards and include them also in the health section - https://phabricator.wikimedia.org/T367336#9888788 (10dcaro) Added two pannels to the health... [14:08:53] 06cloud-services-team, 10Cloud-VPS, 06SRE-OnFire, 10Sustainability (Incident Followup): [grafana,ceph] Add both ends of switch links to the error/discard dashboards and include them also in the health section - https://phabricator.wikimedia.org/T367336#9888795 (10dcaro) Added the discards also to the c... [14:08:57] 06cloud-services-team, 10Cloud-VPS, 06SRE-OnFire, 10Sustainability (Incident Followup): [grafana,ceph] Add both ends of switch links to the error/discard dashboards and include them also in the health section - https://phabricator.wikimedia.org/T367336#9888796 (10dcaro) 05Open→03Resolved [14:18:34] vivian-rook opened https://github.com/toolforge/quarry/pull/50 [14:19:24] 10PAWS: Update images - https://phabricator.wikimedia.org/T366055#9888847 (10rook) 05Open→03Resolved [14:20:11] 10PAWS: jupyterlab to 4.2.2 - https://phabricator.wikimedia.org/T367337#9888853 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/432 [14:20:16] 10PAWS: jupyterlab to 4.2.2 - https://phabricator.wikimedia.org/T367337#9888854 (10rook) 05Open→03Resolved [14:20:21] vivian-rook closed https://github.com/toolforge/paws/pull/432 [14:36:24] 10Quarry: Allow Quarry to query its own database - https://phabricator.wikimedia.org/T367415 (10fnegri) 03NEW [14:37:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:39:37] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Quarry: Support queries against Quarry's own database and ToolsDB - https://phabricator.wikimedia.org/T151158#9888977 (10fnegri) [14:41:36] 10Cloud-VPS (Quota-requests): Add one floating ip to webperformancetest - https://phabricator.wikimedia.org/T367266#9888984 (10dcaro) Yep, unless it's speaking http it will not be able to get through the proxy [14:44:59] 10cloud-services-team (FY2023/2024-Q3-Q4), 10superset.wmcloud.org: Allow Superset to query ToolsDB public databases - https://phabricator.wikimedia.org/T367393#9889004 (10fnegri) [14:45:01] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Quarry: Support queries against Quarry's own database and ToolsDB - https://phabricator.wikimedia.org/T151158#9889006 (10fnegri) [14:46:17] 10Cloud-VPS (Quota-requests): Add one floating ip to webperformancetest - https://phabricator.wikimedia.org/T367266#9889020 (10Andrew) +1 [14:46:19] 10Cloud-VPS (Quota-requests): collection-alt-renderer disk space - https://phabricator.wikimedia.org/T367007#9889022 (10Andrew) [14:47:46] 10Cloud-VPS (Quota-requests): collection-alt-renderer disk space - https://phabricator.wikimedia.org/T367007#9889027 (10Andrew) +1 @dhunniger when migrating to a new debian version please replace your old VMs rather than upgrading in place -- in-place upgrades tend to cause weird long-term tracking issues. [14:49:47] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 10Quarry: Allow Quarry to query ToolsDB public databases - https://phabricator.wikimedia.org/T348407#9889028 (10fnegri) [14:58:50] 10Cloud-VPS (Quota-requests): collection-alt-renderer disk space - https://phabricator.wikimedia.org/T367007#9889126 (10dcaro) 05Open→03In progress a:03dcaro [14:59:21] !log dcaro@urcuchillay collection-alt-renderer START - Cookbook wmcs.openstack.quota_increase [14:59:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Collection-alt-renderer/SAL [14:59:28] !log dcaro@urcuchillay collection-alt-renderer END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) [14:59:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Collection-alt-renderer/SAL [15:00:48] 10Cloud-VPS (Quota-requests): collection-alt-renderer disk space - https://phabricator.wikimedia.org/T367007#9889136 (10dcaro) 05In progress→03Resolved Done :) ` dcaro@urcuchillay$ wmcs-cookbooks wmcs.openstack.quota_increase --gigabytes 160 --cluster-name eqiad1 --project collection-alt-renderer Got ma... [15:01:01] !log dcaro@urcuchillay webperformancetest START - Cookbook wmcs.openstack.quota_increase [15:01:01] wmbot~dcaro@urcuchillay: Unknown project "webperformancetest" [15:01:02] !log dcaro@urcuchillay webperformancetest END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) [15:01:05] wmbot~dcaro@urcuchillay: Unknown project "webperformancetest" [15:01:11] !log dcaro@urcuchillay webperformancetest START - Cookbook wmcs.openstack.quota_increase [15:01:11] wmbot~dcaro@urcuchillay: Unknown project "webperformancetest" [15:01:19] !log dcaro@urcuchillay webperformancetest END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) [15:01:19] wmbot~dcaro@urcuchillay: Unknown project "webperformancetest" [15:02:20] 10Cloud-VPS (Quota-requests): Add one floating ip to webperformancetest - https://phabricator.wikimedia.org/T367266#9889139 (10dcaro) 05Open→03Resolved a:03dcaro Done: ` dcaro@urcuchillay$ wmcs-cookbooks wmcs.openstack.quota_increase --cluster-name eqiad1 --project webperformancetest --floating-ips 1 G... [15:12:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:22:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:08:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:15:01] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [16:15:07] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) [16:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:22:09] 10Cloud-VPS (Debian Buster Deprecation), 06The-Wikipedia-Library, 10Moderator-Tools-Team (Kanban): Replace deprecated Buster VMs in Cloud VPS - https://phabricator.wikimedia.org/T364399#9889744 (10Kgraessle) 05In progress→03Resolved [16:27:15] (03PS1) 10Andrew Bogott: ensure_canary: use new g4 flavor for canary instances [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043149 [16:29:51] 10Tools: Quickstatements failing to OAuth - https://phabricator.wikimedia.org/T367430#9889803 (10Bugreporter) [16:31:13] (03CR) 10Majavah: ensure_canary: use new g4 flavor for canary instances (032 comments) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1043149 (owner: 10Andrew Bogott) [16:32:51] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 06Research: Replace or remove Debian Buster VMs in 'wmf-research-tools' cloud-vps project - https://phabricator.wikimedia.org/T367444 (10Andrew) 03NEW [16:37:48] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.05.27 - 2024.06.16): Upgrade clouddb1021 to bookworm - https://phabricator.wikimedia.org/T365450#9889867 (10BTullis) I have started work on {T365453} before this ticket. It's quite possible that we will be able to decom clouddb1021 instead of upg... [16:50:23] (03open) 10taavi: cloudvps_flavors: Fix setting the epherimental disk size [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/3 [16:50:25] (03merge) 10taavi: cloudvps_flavors: Fix setting the epherimental disk size [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/3 [17:07:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:12:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:13:49] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643#9890464 (10CDanis) Hi all. @joanna_borun asked me to do some looking into this. I promise I skimmed the above, but... [19:15:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [19:16:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [19:34:15] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643#9890544 (10dcaro) >>! In T348643#9890463, @CDanis wrote: > Hi all. @joanna_borun asked me to do some looking into t... [19:36:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [19:38:41] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643#9890547 (10CDanis) Very helpful, thanks @dcaro and enjoy the pto! I'll be gentle, and definitely won't do any write... [19:41:28] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [19:50:28] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [19:51:34] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Launching new bullseye deployment-prep instances fails, no sudo access - https://phabricator.wikimedia.org/T361536#9890577 (10Andrew) This looks like T366028 but when I encounter that issue puppet still completes and I'm able to log in and... [19:52:08] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Launching new bullseye deployment-prep instances fails, no sudo access - https://phabricator.wikimedia.org/T361536#9890583 (10Andrew) p:05Triage→03High a:03Andrew [19:53:39] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Migrate cloudinfra project off of debian buster - https://phabricator.wikimedia.org/T360696#9890585 (10Andrew) 05Open→03Resolved a:03Andrew [19:55:04] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Migrate project-proxy off of Debian Buster - https://phabricator.wikimedia.org/T360693#9890589 (10Andrew) 05Open→03Resolved [19:56:28] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [20:00:50] 10Cloud-VPS (Debian Buster Deprecation), 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Upgrade centralnotice-staging to get off debian buster - https://phabricator.wikimedia.org/T360949#9890617 (10Andrew) Hello! If the old buster VM can be deleted, please do so! If you're having trouble with a... [20:21:29] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Quarry: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) - https://phabricator.wikimedia.org/T365374#9890676 (10Liz) I'm not getting that message any more but some queries are just running endlessly and never finishing. I had one that ran... [20:26:16] FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:27:57] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464 (10Liz) 03NEW [20:31:16] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:31:31] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:36:16] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:57:16] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Create new g4 flavors to support hypervisor migration from Linuxbridge to OVS Neutron agents - https://phabricator.wikimedia.org/T364458#9890825 (10taavi) 05Open→03Resolved [21:12:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:15:27] 10Tool-containers, 10Toolforge: Provide a Redis container for use within a tool's namespace - https://phabricator.wikimedia.org/T360378#9891200 (10bd808) >>! In T360378#9888704, @Pintoch wrote: > I have tried the approach described at https://wikitech.wikimedia.org/wiki/Tool:Containers#Redis_container and it w... [23:37:57] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10VPS-project-Codesearch: Replace or remove Debian Buster VMs in 'codesearch' cloud-vps project - https://phabricator.wikimedia.org/T367479 (10Andrew) 03NEW