[06:22:45] 10Tool-wsindex, 10Wikisource Reader App: Add an option to sort list of books in a randomized manner - https://phabricator.wikimedia.org/T410595 (10Bodhisattwa) 03NEW [06:33:00] 10Tool-wsindex, 10Wikisource Reader App: Add an option to find new books added to the app - https://phabricator.wikimedia.org/T410597#11390930 (10Bodhisattwa) [06:36:19] 10Tool-wsindex, 10Wikisource Reader App: Add an option to find new books added to the app - https://phabricator.wikimedia.org/T410597#11390946 (10Bodhisattwa) [09:40:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:45:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:46:55] !log volans@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing [09:47:13] !log volans@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing [10:04:36] 10Toolforge, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11391233 (10fgiunchedi) >>! In T407140#11383672, @cmooney wrote: > Ok thanks @fgiunchedi for the info. > > I think that seems doab... [10:07:15] 10Toolforge, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11391235 (10taavi) >>! In T407140#11391233, @fgiunchedi wrote: > There are also NFS read only shares towards clouddump hosts, thoug... [11:26:04] (03update) 10volans: tracing: add tracing loki instance [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1040 (https://phabricator.wikimedia.org/T399313) [11:27:26] !log volans@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing [11:27:42] !log volans@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing [11:39:35] (03open) 10volans: logging: run only job-api tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1086 [11:51:16] (03CR) 10Majavah: [C:03+2] repos: add paws repository [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1165049 (owner: 10David Caro) [11:52:29] (03Merged) 10jenkins-bot: repos: add paws repository [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1165049 (owner: 10David Caro) [12:21:37] 10Toolforge, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11391608 (10cmooney) >>! In T407140#11391233, @fgiunchedi wrote: > The NFS shares host tools data and scratch space, specifically:... [13:02:03] 10VPS-project-Codesearch: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624 (10Aklapper) 03NEW [13:15:23] 10VPS-project-Codesearch: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11391921 (10Ladsgroup) How slow are we talking about? :D [13:15:30] (03CR) 10Divyaratann Srivastava: "recheck" [labs/striker] - 10https://gerrit.wikimedia.org/r/1195355 (https://phabricator.wikimedia.org/T405526) (owner: 10Divyaratann Srivastava) [13:15:42] (03CR) 10Divyaratann Srivastava: "recheck" [labs/striker] - 10https://gerrit.wikimedia.org/r/1195355 (https://phabricator.wikimedia.org/T405526) (owner: 10Divyaratann Srivastava) [13:18:45] 10Toolforge, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11391938 (10ayounsi) Not sure if it has been discussed but what do you think of using Calico's [[ https://docs.tigera.io/calico/lat... [13:20:42] 10VPS-project-Codesearch: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11391940 (10Ladsgroup) ` amir@amir:~$ time git clone https://netbox-exports.wikimedia.org/dns.git dns2 Klone nach 'dns2'... Anfordern... [13:21:19] 10VPS-project-Codesearch: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11391942 (10Ladsgroup) Sigh ` amir@amir:~$ time git clone --depth 1 https://netbox-exports.wikimedia.org/dns.git dns3 Klone nach 'dns... [13:21:50] 10VPS-project-Codesearch: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11391944 (10taavi) >>! In T410624#11391921, @Ladsgroup wrote: > How slow are we talking about? :D From Phab: `lang=shell-session ta... [13:30:13] 10VPS-project-Codesearch: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11391980 (10Ladsgroup) I think it's fine [13:33:08] 10Toolforge, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11392000 (10cmooney) >>! In T407140#11391938, @ayounsi wrote: > But instead of doing network separation using a complex VRF and `ip... [13:52:17] (03CR) 10Ssingh: [C:03+1] hcaptcha_proxy: remove unused parameters [labs/private] - 10https://gerrit.wikimedia.org/r/1207265 (owner: 10Kamila Součková) [13:54:26] (03CR) 10Kamila Součková: "check experimental" [labs/private] - 10https://gerrit.wikimedia.org/r/1207265 (owner: 10Kamila Součková) [14:03:46] (03approved) 10dcaro: logging: run only job-api tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1086 (owner: 10volans) [14:06:09] (03CR) 10Kamila Součková: [V:03+2 C:03+2] hcaptcha_proxy: remove unused parameters [labs/private] - 10https://gerrit.wikimedia.org/r/1207265 (owner: 10Kamila Součková) [14:06:21] !log volans@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logging [14:13:14] !log volans@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging [14:13:47] (03merge) 10volans: logging: run only job-api tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1086 [14:18:13] (03update) 10volans: tracing: add tracing loki instance [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1040 (https://phabricator.wikimedia.org/T399313) [14:40:51] !log volans@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [14:46:56] !log volans@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission [14:51:32] (03CR) 10Harsh_Kushwaha07: "recheck" [labs/striker] - 10https://gerrit.wikimedia.org/r/1195355 (https://phabricator.wikimedia.org/T405526) (owner: 10Divyaratann Srivastava) [14:54:34] !log volans@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [14:56:17] (03open) 10deerbee: Extend matching pattern [toolforge-repos/wmde-github-phab-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-github-phab-bot/-/merge_requests/2 (https://phabricator.wikimedia.org/T1 https://phabricator.wikimedia.org/T410608) [15:01:10] !log volans@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission [15:17:13] !log volans@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [15:23:40] !log volans@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission [15:28:17] !log volans@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [15:37:15] !log volans@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [15:37:46] !log volans@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [15:47:16] !log volans@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [15:50:10] 06cloud-services-team, 10Toolforge: Add paging alert if Toolforge HAProxy connection limit is reached - https://phabricator.wikimedia.org/T410421#11392721 (10dcaro) a:05dcaro→03None will not be able to get to this [15:51:07] !log volans@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logging [15:56:21] !log volans@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging [16:10:54] (03PS1) 10Ssingh: Revert "hcaptcha_proxy: remove unused parameters" [labs/private] - 10https://gerrit.wikimedia.org/r/1207897 [16:12:29] 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS: Standardize on opentofu management of all projects in the default keystone domain - https://phabricator.wikimedia.org/T410659 (10Andrew) 03NEW [16:13:57] 06cloud-services-team, 10Cloud-VPS: Standardize on opentofu management of all projects in the default keystone domain - https://phabricator.wikimedia.org/T410659#11392889 (10taavi) [16:19:21] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: MediaWiki REST API appropriately defines array types for parameters - https://phabricator.wikimedia.org/T409113#11392935 (10HCoplin-WMF) p:05Triage→03Medium [16:32:28] (03CR) 10Ssingh: [V:03+2 C:03+2] Revert "hcaptcha_proxy: remove unused parameters" [labs/private] - 10https://gerrit.wikimedia.org/r/1207897 (owner: 10Ssingh) [16:36:49] !log volans@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logging [16:45:33] !log volans@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging [16:55:35] !log volans@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing [17:13:43] !log volans@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing [17:14:48] (03merge) 10volans: tracing: add tracing loki instance [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1040 (https://phabricator.wikimedia.org/T399313) [18:07:33] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34 [18:14:28] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-34 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:19:28] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-34 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:20:23] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-34 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [18:26:23] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34 [18:37:53] RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-34 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [18:48:36] 06cloud-services-team, 10Tool-iw, 10Toolforge: Toolforge interwiki link handling no longer strips URL-encoding before redirecting when it previously did, breaking existing on-wiki links - https://phabricator.wikimedia.org/T409493#11393837 (10bd808) p:05High→03Low I am changing priority here from high ->... [18:57:24] 06cloud-services-team, 10Tool-iw, 10Toolforge: iw.toolforge.org does not support URL-encoded query parameters ([[toolforge:foo?bar]]) - https://phabricator.wikimedia.org/T345783#11393860 (10bd808) 05Open→03Declined I am declining this not because it is a bad idea for interwiki links to support query... [19:00:20] 10Tool-toolinfo-scraper: Port tool to build system and job framework - https://phabricator.wikimedia.org/T410472#11393866 (10bd808) The redirect container migration has now happened. [19:48:29] 10VPS-project-Codesearch, 06collaboration-services: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11393966 (10Dzahn) [21:48:31] (03open) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [21:48:39] (03update) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [22:07:16] (03update) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [22:15:34] (03update) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [22:20:50] (03update) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [22:26:18] (03update) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [23:07:19] 06cloud-services-team, 10Toolforge: Track global and per-tool concurrent requests and 503 rate limiting responses from Toolforge CDN edge - https://phabricator.wikimedia.org/T409794#11394510 (10bd808)