[00:24:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [00:51:14] 10Tool-etherpad-backup: Setup proof of concept storage and retrieval from WMCS object storage - https://phabricator.wikimedia.org/T422958#11810227 (10bd808) I followed the directions from https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide#S3_API to create credentials for the [[https://ldap.toolfo... [00:56:17] 10Tool-etherpad-backup: Setup proof of concept storage and retrieval from WMCS object storage - https://phabricator.wikimedia.org/T422958#11810229 (10bd808) I tried using `s3cmd ws-create s3://html` to make a bucket that would act as a webservice and automatically serve up an index.html file. Unfortunately the c... [04:44:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [04:46:25] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [05:41:25] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [06:10:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [07:14:26] (03CR) 10MusikAnimal: "Frontend changes are still unreviewed, but this enough for one round of feedback, I assume! Most of it mere nitpicks. Wikisource is still " [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444) [07:40:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [08:08:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [08:37:50] 10Tool-humaniki-2: Wrap API responses - https://phabricator.wikimedia.org/T422987 (10Danya) 03NEW [08:38:11] 10Tool-humaniki-2: Wrap API responses - https://phabricator.wikimedia.org/T422987#11810445 (10Danya) p:05Triage→03High [09:30:11] 10Tool-humaniki-2: Wrap API responses - https://phabricator.wikimedia.org/T422987#11810493 (10Danya) 05Open→03In progress [09:56:10] 10VPS-project-Codesearch: Codesearch stuck at Feb 12th? - https://phabricator.wikimedia.org/T421147#11810517 (10A_smart_kitten) Is this reoccuring (or e.g. did it never stop occurring, at least for `operations/mediawiki-config` in the 'Everything' group)? https://codesearch.wmcloud.org/search/?q=MetricsPlat... [10:20:33] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810535 (10Nokib_Sarkar) Hi, any updates? I cannot start the webservice event today for campwiz. is it possible to start by you on behalf of me from backend? [10:21:11] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280372486 (https://github.com/cluebotng/component-configs/commits/5b34645dff3f37bc9f974635e03cd6b8436f37d1) [10:21:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [10:26:09] (03open) 10danyya: Draft: Wrap api responses [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/5 [10:29:55] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280508671 (https://github.com/cluebotng/component-configs/commits/5953d3fb9c5e414df6995740382b7bd3be49ced2) [10:29:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [10:30:23] 10Tool-humaniki-2: Add explicit return types - https://phabricator.wikimedia.org/T422990 (10Danya) 03NEW [10:30:34] 10Tool-humaniki-2: Add explicit return types - https://phabricator.wikimedia.org/T422990#11810549 (10Danya) p:05Triage→03Low [10:30:38] 10Tool-humaniki-2: Add explicit return types - https://phabricator.wikimedia.org/T422990#11810551 (10Danya) 05Open→03In progress [10:38:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [11:18:52] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810576 (10HakanIST) @Nokib_Sarkar Just a note on the latest traceback: the 409 Conflict on the `/deployments` endpoint usually means the deployment already exists... [11:25:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [12:12:01] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810618 (10Nokib_Sarkar) @HakanIST ` tools.campwiz-backend@tools-bastion-15:~$ toolforge webservice buildservice start --mount all Traceback (most recent call las... [12:33:08] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810691 (10HakanIST) @Nokib_Sarkar try `toolforge webservice buildservice restart --mount all` (with the `buildservice` flag). The plain `toolforge webservice resta... [12:40:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [12:41:25] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [14:02:45] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810828 (10Nokib_Sarkar) So, the issue was I had a job named `campwiz-backend` using push-to-deploy. My tool name was also `campwiz-backend`. So, when I tried to st... [14:03:48] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810829 (10Nokib_Sarkar) a:03taavi [14:04:04] 06cloud-services-team, 10Toolforge: Connection with `k8s.tools.eqiad1.wikimedia.cloud` hits SSL error - https://phabricator.wikimedia.org/T422538#11810830 (10Nokib_Sarkar) 05Open→03Resolved [16:05:39] (03update) 10danyya: Draft: Wrap api responses [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/5 [16:11:46] (03update) 10danyya: Draft: Wrap api responses [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/5 [16:13:41] (03update) 10danyya: Wrap api responses [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/5 [16:13:48] (03merge) 10danyya: Wrap api responses [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/5 [16:14:06] 10Tool-humaniki-2: Wrap API responses - https://phabricator.wikimedia.org/T422987#11810930 (10Danya) 05In progress→03Resolved [16:14:07] 10Tool-humaniki-2: Add explicit return types - https://phabricator.wikimedia.org/T422990#11810932 (10Danya) 05In progress→03Resolved [16:16:08] 10Tool-humaniki-2: Retro-active statistics - https://phabricator.wikimedia.org/T422342#11810933 (10Danya) p:05Low→03Triage [16:16:14] 10Tool-humaniki-2: Setup push to deploy - https://phabricator.wikimedia.org/T422347#11810934 (10Danya) p:05Low→03Triage [16:21:37] 10Tool-humaniki-2: Favicon - https://phabricator.wikimedia.org/T423004 (10Danya) 03NEW [16:21:54] 10Tool-humaniki-2: Favicon - https://phabricator.wikimedia.org/T423004#11810962 (10Danya) p:05Triage→03Low