[00:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:59:26] 10Tool-Global-user-contributions: เพิ่มรายการบันทึกไปยัง GUC - https://phabricator.wikimedia.org/T257621#10402083 (10Gussdafii) [02:01:51] 10Tool-Global-user-contributions: Add log entries to GUC - https://phabricator.wikimedia.org/T257621#10402084 (10JJMC89) [05:42:18] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-73 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [05:43:48] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-73 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [06:17:32] 06Toolforge-standards-committee: Adoption request for jawi - https://phabricator.wikimedia.org/T379340#10402199 (10Hakimi97) Hi @bd808 and @JJMC89, it has been 34 days since the last response from @Aviator. @Aviator has updated the license information to the GNU General Public License 3.0 or later. Is it possibl... [07:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:22:29] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73 [08:27:58] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73 [08:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:38:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-73 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:56:46] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for tigwiki - https://phabricator.wikimedia.org/T381378#10402315 (10Marostegui) a:03Marostegui Redacting data [09:09:36] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 [09:14:58] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 [09:19:33] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 [09:24:52] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 [09:29:37] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for tigwiki - https://phabricator.wikimedia.org/T381378#10402345 (10Marostegui) Data has been redacted - still pending user creation and database `_p` creation [10:00:40] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for tigwiki - https://phabricator.wikimedia.org/T381378#10402446 (10Marostegui) a:05Marostegui→03fnegri User created, database created. Ready for views creation. [10:19:33] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06DBA: Prepare and check storage layer for tigwiki - https://phabricator.wikimedia.org/T381378#10402490 (10fnegri) [10:24:01] 10PAWS: Cannot export from PAWS, nor publish as a public page - https://phabricator.wikimedia.org/T382120 (10awight) 03NEW [11:06:43] 06cloud-services-team, 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T381453#10402715 (10Xqt) [11:19:22] FIRING: HAProxyBackendUnavailable: HAProxy service nova-metadata-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [11:27:49] 06cloud-services-team, 10Cloud-VPS, 10SRE Observability (FY2024/2025-Q2): Remove librenms -> graphite integration, replace with gnmi - https://phabricator.wikimedia.org/T372457#10402746 (10fgiunchedi) I took another look at the dashboards and it looks like to me we now have all interesting switch port metric... [12:10:47] 10Cloud-Services, 10Wikidocumentaries: Why have all images disappeared from https://wikidocumentaries.wmcloud.org/? - https://phabricator.wikimedia.org/T382138#10402849 (10Susannaanas) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/... [12:11:27] 10VPS-Projects, 10Wikidocumentaries: Why have all images disappeared from https://wikidocumentaries.wmcloud.org/? - https://phabricator.wikimedia.org/T382138#10402852 (10Susannaanas) [13:24:47] 10PAWS: Cannot export from PAWS, nor publish as a public page - https://phabricator.wikimedia.org/T382120#10403012 (10rook) I believe that file can be found here: https://public-paws.wmcloud.org/User:Adam_Wight_(WMDE)/survey-analytics/TWL24-Statistics.ipynb Thank you for the note about the documentation, I'll u... [13:27:23] (03open) 10raymond-ndibe: d/changelog: bump to 1.6.5 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/68 (https://phabricator.wikimedia.org/T361120 https://phabricator.wikimedia.org/T379893) [13:31:56] (03approved) 10raymond-ndibe: [toolforge-deploy] refetch default to main branch [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/645 (https://phabricator.wikimedia.org/T358225) [13:32:02] (03merge) 10raymond-ndibe: [toolforge-deploy] refetch default to main branch [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/645 (https://phabricator.wikimedia.org/T358225) [13:38:19] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [13:38:51] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld [13:39:36] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [13:39:59] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld [13:40:31] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [13:40:55] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld [13:48:35] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [13:54:15] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [13:54:25] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [13:59:55] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [14:00:07] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [14:07:52] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [14:07:56] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [14:16:15] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [14:16:45] (03approved) 10raymond-ndibe: d/changelog: bump to 1.6.5 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/68 (https://phabricator.wikimedia.org/T361120 https://phabricator.wikimedia.org/T379893) [14:16:52] (03merge) 10raymond-ndibe: d/changelog: bump to 1.6.5 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/68 (https://phabricator.wikimedia.org/T361120 https://phabricator.wikimedia.org/T379893) [14:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:23:15] (03update) 10raymond-ndibe: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/67 (owner: 10dcaro) [14:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:49:56] 10Tool-openstack-browser: Openstack Browser lists a user a user that horizon doesn't - https://phabricator.wikimedia.org/T382152 (10Reedy) 03NEW [14:49:59] 10Tool-openstack-browser: Openstack Browser lists a user a user that horizon doesn't - https://phabricator.wikimedia.org/T382152#10403225 (10Reedy) p:05Triage→03Low [15:19:22] FIRING: HAProxyBackendUnavailable: HAProxy service nova-metadata-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:47:35] 10VPS-Projects, 10Wikidocumentaries: Why have all images disappeared from https://wikidocumentaries.wmcloud.org/? - https://phabricator.wikimedia.org/T382138#10403331 (10TheDJ) The HTML contains: Error creating thumbnail: Unable to save thumbnail to destination That's either a permission issue, or the disk of... [16:49:22] RESOLVED: HAProxyBackendUnavailable: HAProxy service nova-metadata-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:39:29] (03CR) 10Jforrester: "check experimental" [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1101499 (owner: 10L10n-bot) [19:39:50] (03CR) 10Jforrester: "check experimental" [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1101516 (owner: 10L10n-bot) [19:40:30] (03CR) 10Jforrester: "check experimental" [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/961766 (owner: 10L10n-bot) [19:41:41] (03CR) 10Jforrester: "check experimental" [labs/tools/graphql] - 10https://gerrit.wikimedia.org/r/606696 (owner: 10Dbarratt) [22:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:28:51] 10Tools, 10Montage: Tool "montage" loads fonts from google - https://phabricator.wikimedia.org/T172741#10404048 (10Jayprakash12345) 05Open→03Resolved a:03Jayprakash12345 Done: https://github.com/hatnote/montage/pull/248 Montage: https://montage.toolforge.org/#/ Montage-beta: https://montage-beta.too... [22:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:57:44] 10VPS-project-Codesearch, 06collaboration-services: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#10404078 (10Dzahn) We have been evaluating software for a refreshed codesearch and https://www.sourcebot.dev/ seems like a viable candidate. [23:36:29] 10VPS-project-Wikistats: since all updates run with the "extinfo" parameter some tables did not get updated numbers - https://phabricator.wikimedia.org/T381623#10404199 (10Dzahn) 05Open→03Resolved