[00:03:55] FIRING: MaxConntrack: Max conntrack at 86.37% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:05:55] FIRING: MaxConntrack: Max conntrack at 90.02% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:06:06] 06cloud-services-team: MaxConntrack Netfilter: Maximum number of allowed connection tracking entries alert on cloudvirt1050:9100 - https://phabricator.wikimedia.org/T372693#10079852 (10phaultfinder) [00:06:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:10:55] RESOLVED: MaxConntrack: Max conntrack at 91.15% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:11:28] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:12:55] FIRING: MaxConntrack: Max conntrack at 90.04% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:13:06] 06cloud-services-team: MaxConntrack Netfilter: Maximum number of allowed connection tracking entries alert on cloudvirt1050:9100 - https://phabricator.wikimedia.org/T372693#10079855 (10phaultfinder) [00:32:55] RESOLVED: MaxConntrack: Max conntrack at 90.56% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:33:55] FIRING: MaxConntrack: Max conntrack at 90.75% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:34:49] 06cloud-services-team: MaxConntrack Netfilter: Maximum number of allowed connection tracking entries alert on cloudvirt1050:9100 - https://phabricator.wikimedia.org/T372693#10079873 (10phaultfinder) [00:37:25] RESOLVED: MaxConntrack: Max conntrack at 89.88% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:38:55] RESOLVED: MaxConntrack: Max conntrack at 90.38% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:41:23] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/6 [00:41:31] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/65 [00:41:34] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/5 [00:41:41] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/59 [00:41:47] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/65 [00:41:55] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/6 [00:42:09] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/12 [00:42:16] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/59 [00:42:33] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/9 [00:42:36] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/12 [00:42:37] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/85 [00:42:39] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/86 [00:42:41] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/9 [00:42:44] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/85 [00:42:51] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/52 [00:42:53] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/86 [00:43:03] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/120 [00:43:07] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/52 [00:43:09] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/58 [00:43:15] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/120 [00:43:19] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/121 [00:43:21] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/59 [00:43:25] FIRING: MaxConntrack: Max conntrack at 87.77% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:43:35] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/36 [00:43:40] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/58 [00:43:40] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/121 [00:43:42] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/11 [00:43:45] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/26 [00:43:47] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/16 [00:43:47] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/59 [00:43:51] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/36 [00:43:56] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/109 [00:43:58] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/11 [00:44:02] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/37 [00:44:06] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/184 [00:44:10] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/16 [00:44:14] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/59 [00:44:19] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/32 [00:44:23] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/26 [00:44:27] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/27 [00:44:31] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/109 [00:44:35] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/37 [00:44:39] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/59 [00:44:43] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/184 [00:44:47] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/32 [00:44:51] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/27 [00:48:25] RESOLVED: MaxConntrack: Max conntrack at 80.66% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:53:10] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/27 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:53:14] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/27 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:53:34] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/32 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:53:38] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/32 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:53:48] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/184 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:53:52] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/184 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:54:00] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/59 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:54:03] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/59 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:54:10] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/37 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:54:14] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/37 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:54:24] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/109 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:54:27] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/109 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:54:50] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/26 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:55:08] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:55:13] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/16 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:55:22] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/11 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:55:28] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/11 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:55:40] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-builder: bump to 0.0.115-20240821005414-c8b7a315 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/482 [00:55:41] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/36 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:55:48] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/59 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:55:53] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/59 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:56:00] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/121 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:56:05] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/121 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:56:20] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/120 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:56:38] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/26 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:56:41] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/26 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:56:56] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/58 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:57:08] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/52 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:57:13] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.39-20240821005426-da3b8a54 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/483 [00:57:15] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/52 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:57:29] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/12 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:57:34] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/12 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:58:16] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/9 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:58:20] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/9 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:58:36] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/5 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:58:41] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/5 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:58:53] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/65 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:58:54] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.330-20240821005619-4c3d6f04 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/484 [00:58:59] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.330-20240821005619-4c3d6f04 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/484 [00:59:03] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/65 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:59:04] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/59 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:59:13] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/59 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:59:19] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/86 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:59:27] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: volume-admission: bump to 0.0.53-20240821005523-419b3162 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/485 [00:59:34] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/86 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:59:34] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: volume-admission: bump to 0.0.53-20240821005523-419b3162 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/485 [00:59:38] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/36 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:59:44] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/36 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:59:52] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/58 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:59:55] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/58 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [01:00:04] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/120 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [01:00:13] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/120 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [01:00:23] (03update) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/85 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [01:00:46] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/6 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [01:00:50] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/6 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [01:02:00] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: ingress-admission: bump to 0.0.48-20240821005834-806c3cd8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/486 [01:02:01] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.168-20240821005444-14c18313 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/487 [01:02:05] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: ingress-admission: bump to 0.0.48-20240821005834-806c3cd8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/486 [01:02:08] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.168-20240821005444-14c18313 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/487 [01:02:14] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.39-20240821005426-da3b8a54 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/483 [01:02:50] (03approved) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/85 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [01:02:51] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.13-20240821010101-59ec75f4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/488 [01:02:55] (03merge) 10raymond-ndibe: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/85 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [01:02:56] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.13-20240821010101-59ec75f4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/488 [01:03:23] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.330-20240821005619-4c3d6f04 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/484 [01:05:14] (03update) 10raymond-ndibe: [builds-api] add topologySpreadConstraints to deployment [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/82 (https://phabricator.wikimedia.org/T358203) [01:13:27] (03update) 10raymond-ndibe: [ingress-admission] add topologySpreadConstraints to deployment [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/8 (https://phabricator.wikimedia.org/T358203) [01:13:48] (03update) 10raymond-ndibe: [registry-admission] add topologySpreadConstraints to deployment [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/11 (https://phabricator.wikimedia.org/T358203) [01:13:58] (03update) 10raymond-ndibe: [volume-admission] add topologySpreadConstraints to deployment [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/15 (https://phabricator.wikimedia.org/T358203) [01:14:10] (03update) 10raymond-ndibe: [envvars-admission] add topologySpreadConstraints to deployment [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/10 (https://phabricator.wikimedia.org/T358203) [01:38:51] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [01:38:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:39:57] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [01:39:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:42:30] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.run_tests [01:42:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:46:41] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [01:46:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:49:04] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [01:49:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:54:01] !log raymond@ubuntu toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [01:54:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [02:18:31] (03CR) 10Raymond Ndibe: "I think we forgot to refactor generate_wmcs_config.sh to auto-generate the new ./wmcs.yaml config and populate it with config parameters a" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921 (owner: 10David Caro) [02:48:37] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [02:48:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [02:53:40] !log raymond@ubuntu toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [02:53:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [02:55:06] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [02:55:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [02:59:58] !log raymond@ubuntu toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [03:00:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [03:06:20] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [03:06:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [03:12:19] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [03:12:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [03:13:04] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [03:13:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [03:17:48] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [03:17:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [03:19:07] !log raymond@ubuntu tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [03:19:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [03:28:08] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [03:28:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [03:33:46] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [03:33:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [03:34:32] (03approved) 10raymond-ndibe: builds-api: bump to 0.0.168-20240821005444-14c18313 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/487 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [03:34:36] (03merge) 10raymond-ndibe: builds-api: bump to 0.0.168-20240821005444-14c18313 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/487 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [03:35:13] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [03:35:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [03:41:37] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [03:41:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [03:41:55] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.330-20240821005619-4c3d6f04 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/484 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [03:42:35] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [03:42:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [03:45:33] (03CR) 10Raymond Ndibe: "Other than the above issue, everything seems to be working fine. currently using this PR to deploy on tools and toolsbeta with test result" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1059921 (owner: 10David Caro) [03:49:20] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [03:49:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [03:49:49] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.330-20240821005619-4c3d6f04 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/484 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [03:49:53] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.330-20240821005619-4c3d6f04 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/484 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [03:55:56] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.48-20240821005834-806c3cd8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/486 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [03:56:56] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [03:56:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [04:03:26] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [04:03:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [04:03:46] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [04:03:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:10:25] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [04:10:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:11:09] (03approved) 10raymond-ndibe: ingress-admission: bump to 0.0.48-20240821005834-806c3cd8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/486 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [04:11:12] (03merge) 10raymond-ndibe: ingress-admission: bump to 0.0.48-20240821005834-806c3cd8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/486 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [04:12:00] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [04:12:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [04:18:21] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [04:18:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [04:18:35] (03update) 10raymond-ndibe: volume-admission: bump to 0.0.53-20240821005523-419b3162 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/485 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [04:20:08] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [04:20:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:20:58] !log raymond@ubuntu tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission [04:20:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:21:52] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [04:21:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:22:40] !log raymond@ubuntu tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission [04:22:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:25:26] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [04:25:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:28:09] !log raymond@ubuntu tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission [04:28:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:36:48] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [04:36:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:43:32] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [04:43:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:44:15] (03approved) 10raymond-ndibe: volume-admission: bump to 0.0.53-20240821005523-419b3162 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/485 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [04:44:19] (03merge) 10raymond-ndibe: volume-admission: bump to 0.0.53-20240821005523-419b3162 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/485 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [04:44:40] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.39-20240821005426-da3b8a54 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/483 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [04:45:43] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [04:45:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [04:52:03] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [04:52:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [04:55:32] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [04:55:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [05:01:52] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [05:01:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [05:02:41] (03approved) 10raymond-ndibe: api-gateway: bump to 0.0.39-20240821005426-da3b8a54 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/483 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [05:02:45] (03merge) 10raymond-ndibe: api-gateway: bump to 0.0.39-20240821005426-da3b8a54 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/483 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [05:03:16] (03update) 10raymond-ndibe: builds-builder: bump to 0.0.115-20240821005414-c8b7a315 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/482 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [05:04:01] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [05:04:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [05:10:28] FIRING: InstanceDown: Project tools instance tools-prometheus-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:12:59] !log raymond@ubuntu toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder [05:13:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [05:14:40] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [05:14:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [05:19:39] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [05:19:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [05:20:46] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [05:20:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [05:27:16] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [05:27:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [05:29:09] (03approved) 10raymond-ndibe: builds-builder: bump to 0.0.115-20240821005414-c8b7a315 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/482 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [05:29:12] (03merge) 10raymond-ndibe: builds-builder: bump to 0.0.115-20240821005414-c8b7a315 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/482 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [05:29:41] (03update) 10raymond-ndibe: components-api: bump to 0.0.13-20240821010101-59ec75f4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/488 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [05:31:16] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [05:31:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [05:37:28] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [05:37:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [05:38:09] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [05:38:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [05:44:34] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [05:44:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [05:45:49] (03approved) 10raymond-ndibe: components-api: bump to 0.0.13-20240821010101-59ec75f4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/488 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [05:45:52] (03merge) 10raymond-ndibe: components-api: bump to 0.0.13-20240821010101-59ec75f4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/488 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [06:06:46] FIRING: Toolforge Kyverno no policy resources: Toolforge Kyverno has no policy resources - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/Toolforge_Kyverno_no_policy_resources - https://grafana-rw.wmcloud.org/d/kyverno/kyverno?orgId=1&var-DS_PROMETHEUS_KYVERNO=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforge+Kyverno+no+policy+resources [06:10:28] RESOLVED: InstanceDown: Project tools instance tools-prometheus-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:11:46] RESOLVED: Toolforge Kyverno no policy resources: Toolforge Kyverno has no policy resources - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/Toolforge_Kyverno_no_policy_resources - https://grafana-rw.wmcloud.org/d/kyverno/kyverno?orgId=1&var-DS_PROMETHEUS_KYVERNO=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforge+Kyverno+no+policy+resources [06:14:28] FIRING: InstanceDown: Project tools instance tools-prometheus-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:18:24] 10Data-Services: Optimize querying the page table by namespace - https://phabricator.wikimedia.org/T252122#10080069 (10komla) I took at look at this 20 seconds of execution time for this query is fine. Let us know if you think there are other bottlenecks. [07:14:28] RESOLVED: InstanceDown: Project tools instance tools-prometheus-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [08:15:48] 10Data-Services: Optimize querying the page table by namespace - https://phabricator.wikimedia.org/T252122#10080281 (10fnegri) > still ends up using filesort [...] Is it because the index is not just on page_namespace? Yes, that's the reason why it's using filesort. But I'm not sure if adding that index would m... [09:08:49] 10Data-Services: [toolsdb] Clean up users and manage as code - https://phabricator.wikimedia.org/T367772#10080348 (10fnegri) p:05Triage→03Low [09:16:52] 10Data-Services, 07Wikimedia-Slow-DB-Query: [wikireplicas] Log very slow queries - https://phabricator.wikimedia.org/T372859#10080366 (10joanna_borun) p:05Low→03Medium [09:20:54] 10Data-Services: Configure `report_host` on ToolsDB - https://phabricator.wikimedia.org/T355761#10080373 (10joanna_borun) p:05Triage→03Low [09:21:58] 06cloud-services-team, 10Cloud-VPS, 10Data-Services: [wmcs-cookbooks] Write cookbook for restarting ToolsDB - https://phabricator.wikimedia.org/T328282#10080377 (10joanna_borun) p:05Triage→03Medium [09:24:42] 06cloud-services-team, 10Data-Services: ToolsDB: discard obsolete GTID domains - https://phabricator.wikimedia.org/T334947#10080399 (10joanna_borun) p:05Triage→03Medium [09:29:17] 10Data-Services, 10Tools: EditGroups tool: Unstable connection to SQL database on Toolforge - https://phabricator.wikimedia.org/T241029#10080408 (10fnegri) 05Open→03Invalid This task is very old and the upstream issue has been closed. I'll resolve it as "invalid", please reopen if this is still an issue. [09:32:30] 06cloud-services-team, 10Data-Services, 10Tools: Determine if templatetiger is abandoned - https://phabricator.wikimedia.org/T253424#10080420 (10fnegri) p:05High→03Low [09:33:59] 06cloud-services-team, 10Data-Services, 10Tools: Determine if templatetiger is abandoned - https://phabricator.wikimedia.org/T253424#10080418 (10fnegri) @Kolossos I was wondering if you still need this data? 7GB is not huge, but if you don't need it anymore it would be good to remove it. [09:53:04] 06cloud-services-team, 10Data-Services: toolsdb: review alerting - https://phabricator.wikimedia.org/T306453#10080627 (10fnegri) p:05Medium→03High [10:12:15] 06cloud-services-team, 10Data-Services: Ensure all ToolsDB databases comply with current naming conventions - https://phabricator.wikimedia.org/T269609#10080721 (10fnegri) Some years later, there are still 22 dbs using the legacy naming with a single underscore. ` MariaDB [(none)]> USE information_schema; Rea... [10:21:40] 06cloud-services-team, 10Data-Services: Enable innodb_rollback_on_timeout for wmflabs databases - https://phabricator.wikimedia.org/T251030#10080765 (10fnegri) 05Open→03Declined I just found this old ticket going through the backlog. I will resolve it as "Declined", as ToolsDB is a shared database and... [10:27:20] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services: Ensure all ToolsDB databases comply with current naming conventions - https://phabricator.wikimedia.org/T269609#10080767 (10fnegri) p:05Medium→03High a:03fnegri [10:27:27] 06cloud-services-team, 10Observability-Metrics: Deploy 'cloud' Prometheus instance to codfw - https://phabricator.wikimedia.org/T350010#10080775 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi As far as I can tell this is done, boldly resolving [11:13:08] (03PS1) 10Slyngshede: P:idp add airflow_test_k8s dummy secret. [labs/private] - 10https://gerrit.wikimedia.org/r/1064357 (https://phabricator.wikimedia.org/T371209) [11:30:05] (03CR) 10Stevemunene: [C:03+1] "looks good!" [labs/private] - 10https://gerrit.wikimedia.org/r/1064357 (https://phabricator.wikimedia.org/T371209) (owner: 10Slyngshede) [11:30:30] (03CR) 10Slyngshede: [V:03+2 C:03+2] P:idp add airflow_test_k8s dummy secret. [labs/private] - 10https://gerrit.wikimedia.org/r/1064357 (https://phabricator.wikimedia.org/T371209) (owner: 10Slyngshede) [11:36:50] (03Abandoned) 10Stevemunene: Add airflow-analytics-test secret [labs/private] - 10https://gerrit.wikimedia.org/r/1057820 (https://phabricator.wikimedia.org/T371209) (owner: 10Stevemunene) [12:38:23] 10wikitech.wikimedia.org: Requesting Wikitech:Content administrators access for CDanis - https://phabricator.wikimedia.org/T373019 (10CDanis) 03NEW [12:39:38] 10PAWS: jupyterhub helm deploy show changed in ansible - https://phabricator.wikimedia.org/T366183#10081113 (10rook) 05Open→03Declined [14:04:56] 06cloud-services-team: CephClusterInUnknown - https://phabricator.wikimedia.org/T372573#10081489 (10Andrew) 05Open→03Resolved a:03Andrew [14:06:11] 06cloud-services-team: SystemdUnitDown Unit purge_vm_backup.service on node cloudbackup1004 has been down for long. - https://phabricator.wikimedia.org/T372143#10081496 (10Andrew) 05Open→03Resolved a:03Andrew [14:07:05] 06cloud-services-team: SystemdUnitDown Unit backup_vms.service on node cloudbackup1003 has been down for long. - https://phabricator.wikimedia.org/T372126#10081498 (10Andrew) 05Open→03Resolved a:03Andrew [14:08:38] 06cloud-services-team: SystemdUnitDown Unit rsync_enterprise_htmldumps.service on node clouddumps1001 has been down for long. - https://phabricator.wikimedia.org/T371690#10081514 (10Andrew) 05Open→03Resolved a:03Andrew [14:09:09] 06cloud-services-team, 10Wikidata, 10Wikidata Analytics: Delete the wmdeanalytics Cloud VPS machine - https://phabricator.wikimedia.org/T371696#10081500 (10Andrew) p:05Triage→03Medium a:03Andrew [14:11:14] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10081520 (10Andrew) Will this be moot after the move to k8s, or is this blocking the move to k8s? [14:12:16] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814#10081522 (10Andrew) p:05Triage→03Medium [14:12:32] 06cloud-services-team, 10Cloud-VPS: Cloud VPS: extend tofu-infra to cover quotas - https://phabricator.wikimedia.org/T371391#10081523 (10joanna_borun) p:05Triage→03Low [14:12:53] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Collect access metrics from cloud-vps web proxy - https://phabricator.wikimedia.org/T371382#10081524 (10Andrew) p:05Triage→03Medium [14:13:14] 06cloud-services-team: SystemdUnitDown Unit opentofu-infra-diff.service on node cloudcontrol1007 has been down for long. - https://phabricator.wikimedia.org/T370742#10081525 (10joanna_borun) 05Open→03Resolved [14:13:45] 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T370752#10081526 (10Andrew) 05Open→03Resolved a:03Andrew [14:13:51] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: investigate S3 spurious endpoint errors - https://phabricator.wikimedia.org/T370660#10081528 (10joanna_borun) p:05Triage→03Medium [14:14:37] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: introduce additional gitlab-ci automation - https://phabricator.wikimedia.org/T370652#10081529 (10joanna_borun) p:05Triage→03Low [14:15:39] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Remove or replace deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370460#10081530 (10joanna_borun) p:05Triage→03High [14:16:17] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace poolcounter06.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370458#10081531 (10joanna_borun) p:05Triage→03Medium [14:16:29] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T370412#10081532 (10joanna_borun) 05Open→03Resolved [14:16:51] 06cloud-services-team: PuppetFailure Puppet failure on cloudcontrol1006:9100 - https://phabricator.wikimedia.org/T370411#10081533 (10Andrew) 05Open→03Resolved a:03Andrew [14:17:19] 10cloud-services-team (Hardware): cloudcontrol2006-dev struggling with memory - https://phabricator.wikimedia.org/T370401#10081537 (10joanna_borun) p:05Triage→03Medium [14:17:37] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T370383#10081538 (10joanna_borun) 05Open→03Resolved [14:18:14] 06cloud-services-team: toolforge: prometheus server died - https://phabricator.wikimedia.org/T370143#10081539 (10joanna_borun) p:05Triage→03High [14:18:44] 06cloud-services-team: toolforge: prometheus server died - https://phabricator.wikimedia.org/T370143#10081541 (10Andrew) This happened again yesterday, multiple times. At least one of the times the oomkiller was firing. [14:18:51] 06cloud-services-team: SystemdUnitDown Unit purge_vm_rbd_images.service on node cloudcontrol1005 has been down for long. - https://phabricator.wikimedia.org/T369479#10081552 (10joanna_borun) 05Open→03Resolved [14:18:54] (03CR) 10JHathaway: [C:03+1] puppet8: add db_user [labs/private] - 10https://gerrit.wikimedia.org/r/1064113 (https://phabricator.wikimedia.org/T372664) (owner: 10JHathaway) [14:19:02] (03CR) 10JHathaway: [C:03+2] puppet8: add db_user [labs/private] - 10https://gerrit.wikimedia.org/r/1064113 (https://phabricator.wikimedia.org/T372664) (owner: 10JHathaway) [14:19:04] (03CR) 10JHathaway: [V:03+2 C:03+2] puppet8: add db_user [labs/private] - 10https://gerrit.wikimedia.org/r/1064113 (https://phabricator.wikimedia.org/T372664) (owner: 10JHathaway) [14:19:09] 06cloud-services-team: NovafullstackSustainedFailures The automated tests were unable to create, provision and decommission a VM in the last 5h - https://phabricator.wikimedia.org/T369316#10081554 (10joanna_borun) 05Open→03Resolved [14:19:39] 06cloud-services-team: SystemdUnitDown Unit backup_cinder_volumes.service on node cloudbackup1002-dev has been down for long. - https://phabricator.wikimedia.org/T368986#10081557 (10joanna_borun) 05Open→03Resolved [14:20:09] 06cloud-services-team, 10Data-Services, 10Infrastructure Security: [wikireplicas] Review grants and views - https://phabricator.wikimedia.org/T368748#10081558 (10joanna_borun) p:05Triage→03Low [14:20:47] 06cloud-services-team: NodeDown - https://phabricator.wikimedia.org/T368559#10081559 (10Andrew) 05Open→03Resolved a:03Andrew [14:21:17] 06cloud-services-team, 10Cloud-VPS: can the db server 'maps-test-2' be deleted? - https://phabricator.wikimedia.org/T368233#10081561 (10Andrew) p:05Triage→03Low [14:21:56] 06cloud-services-team: InterfaceSpeedError brq7425e328-56 on cloudvirt1063:9100 has the wrong speed: 1.25e+06. - https://phabricator.wikimedia.org/T368008#10081568 (10joanna_borun) 05Open→03Resolved [14:22:26] 06cloud-services-team: toolforge: prometheus server died - https://phabricator.wikimedia.org/T370143#10081563 (10Raymond_Ndibe) what could be the reason for this? any pointers? logs and servers to look at? etc It'd be nice to see what you @Andrew and @fnegri are already doing and looking at, and results so far [14:24:04] 06cloud-services-team: Replace 'download' cloud-vps project after we support per-tool object storage - https://phabricator.wikimedia.org/T367593#10081575 (10Andrew) 05Open→03Stalled p:05Triage→03Low [14:26:14] 06cloud-services-team, 10Toolforge: toolforge: make sure we cache in our repos/registries all helm charts and container images used in k8s - https://phabricator.wikimedia.org/T368630#10081578 (10Raymond_Ndibe) this looks like something we can handle with harbor @dcaro [14:26:16] 06cloud-services-team, 10Cloud-VPS: wmcs.ceph.osd.bootstrap_and_add cookbook should add fewer osds at once - https://phabricator.wikimedia.org/T372821#10081580 (10Andrew) p:05Triage→03High [14:26:18] 06cloud-services-team: PuppetFailure Puppet failure on cloudbackup2003:9100 - https://phabricator.wikimedia.org/T365638#10081581 (10joanna_borun) 05Open→03Resolved [14:26:22] 06cloud-services-team, 10Cloud-VPS: MaxConntrack Netfilter: Maximum number of allowed connection tracking entries alert on cloudvirt1042:9100 - https://phabricator.wikimedia.org/T365540#10081583 (10joanna_borun) 05Open→03Resolved [14:27:40] 06cloud-services-team: NeutronAgentDownForLong A Neutron agent has been down for more than 2h, VMs will have connectivity issues - https://phabricator.wikimedia.org/T365461#10081588 (10joanna_borun) 05Open→03Resolved [14:29:04] 06cloud-services-team, 10Cloud-VPS: Add a nicer interface in Spicerack/wmcs-cookbooks to downtime Cloud VPS instances - https://phabricator.wikimedia.org/T364733#10081595 (10joanna_borun) p:05Triage→03Low [14:29:33] 06cloud-services-team, 10Cloud-VPS: MaxConntrack Netfilter: Maximum number of allowed connection tracking entries alert on cloudvirt1042:9100 - https://phabricator.wikimedia.org/T365540#10081593 (10Andrew) p:05Triage→03Medium Did this get done? The alert isn't firing but it seems likely to recurr. [14:29:43] 06cloud-services-team: MaxConntrack Netfilter: Maximum number of allowed connection tracking entries alert on cloudvirt1040:9100 - https://phabricator.wikimedia.org/T371445#10081600 (10fnegri) 05Open→03Resolved a:03fnegri [14:29:52] 06cloud-services-team: SystemdUnitDown Unit kiwix-mirror-update.service on node clouddumps1002 has been down for long. - https://phabricator.wikimedia.org/T371576#10081598 (10fnegri) 05Open→03Resolved a:03fnegri [14:29:53] 06cloud-services-team: MaxConntrack Netfilter: Maximum number of allowed connection tracking entries alert on cloudvirt1050:9100 - https://phabricator.wikimedia.org/T372693#10081596 (10fnegri) 05Open→03Resolved a:03fnegri [14:29:55] 06cloud-services-team: SystemdUnitDown Unit rsync_nginxlogs.service on node clouddumps1001 has been down for long. - https://phabricator.wikimedia.org/T364819#10081604 (10fnegri) 05Open→03Resolved a:03fnegri [14:30:24] 06cloud-services-team: PuppetDisabled - https://phabricator.wikimedia.org/T364021#10081606 (10fnegri) 05Open→03Resolved a:03fnegri [14:30:29] 06cloud-services-team, 10Horizon, 10Striker: Striker/Horizon are running in Blubber built containers with a runtime UID that does not exist on the host machine - https://phabricator.wikimedia.org/T363452#10081611 (10joanna_borun) p:05Triage→03Medium [14:30:33] 06cloud-services-team: ProbeDown virt.cloudgw.eqiad1.wikimediacloud.org:0 failed when probed by icmp_virt_cloudgw_eqiad1_wikimediacloud_org_ip4 from codfw. Availability is 0%. - https://phabricator.wikimedia.org/T370477#10081602 (10fnegri) 05Open→03Resolved a:03fnegri [14:30:39] 06cloud-services-team: PuppetDisabled Puppet disabled on cloudrabbit1003:9100 - https://phabricator.wikimedia.org/T364020#10081608 (10fnegri) 05Open→03Resolved a:03fnegri [14:30:40] 06cloud-services-team, 10Toolforge: Large number of tools with servicegroup but no matching user - https://phabricator.wikimedia.org/T360379#10081613 (10Raymond_Ndibe) [14:30:45] 06cloud-services-team, 10Cloud-VPS: nova-api can get the listen queue of socket full - https://phabricator.wikimedia.org/T362956#10081616 (10Andrew) p:05Triage→03Medium [14:30:46] 06cloud-services-team, 10Data-Services, 10Toolforge, 07Performance Issue: NFS storage slow on Toolforge? - https://phabricator.wikimedia.org/T363621#10081615 (10Raymond_Ndibe) [14:30:48] 06cloud-services-team, 10wikitech.wikimedia.org, 10Observability-Alerting: Move wikitech-static monitoring off Icinga - https://phabricator.wikimedia.org/T362397#10081623 (10joanna_borun) p:05Triage→03Low [14:30:49] 06cloud-services-team, 10VPS-Projects, 10fundraising-tech-ops, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#10081625 (10Andrew) Hello! Can I get an update on this? [14:30:52] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Migrate cloudceph servers to nftables - https://phabricator.wikimedia.org/T361913#10081624 (10joanna_borun) p:05Triage→03High [14:31:15] 06cloud-services-team, 10Toolforge: Find a modern hostname for tools-static.wmflabs.org - https://phabricator.wikimedia.org/T361435#10081632 (10joanna_borun) p:05Triage→03Low [14:31:19] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace or remove deployment-echostore02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361383#10081637 (10joanna_borun) [14:31:26] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10081638 (10joanna_borun) [14:31:34] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Update monitoring project puppetmaster - https://phabricator.wikimedia.org/T361592#10081628 (10Andrew) 05Open→03Resolved a:03Andrew [14:31:46] 06cloud-services-team, 10Toolforge, 07Kubernetes: [infra] Upgrade Toolforge K8s etcd nodes to Bookworm - https://phabricator.wikimedia.org/T361237#10081639 (10joanna_borun) p:05Triage→03Medium [14:31:52] 06cloud-services-team, 10Cloud-VPS: Add check in nova-fullstack to ensure Puppet certificates are being revoked on instance deletion - https://phabricator.wikimedia.org/T361142#10081640 (10joanna_borun) p:05Triage→03Low [14:32:01] 06cloud-services-team, 10Data-Services: Move replica_cnf_api out of the Puppet repo - https://phabricator.wikimedia.org/T340754#10081642 (10Raymond_Ndibe) [14:32:09] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic: Set up a bitu instance for codfw1dev - https://phabricator.wikimedia.org/T360795#10081643 (10Andrew) @SLyngshede-WMF can this be closed now? [14:32:48] 06cloud-services-team, 10Toolforge: Large number of tools with servicegroup but no matching user - https://phabricator.wikimedia.org/T360379#10081644 (10joanna_borun) p:05Triage→03Medium [14:33:11] 06cloud-services-team, 10wikitech.wikimedia.org, 07LDAP: Replace wikitech as source of two-factor auth protection for developer accounts - https://phabricator.wikimedia.org/T359551#10081645 (10Andrew) a:03Andrew [14:33:12] 06cloud-services-team, 10wikitech.wikimedia.org, 07LDAP: Replace wikitech as source of two-factor auth protection for developer accounts - https://phabricator.wikimedia.org/T359551#10081646 (10joanna_borun) p:05Triage→03High a:05Andrew→03None [14:33:35] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: [trove] define process for updating docker images - https://phabricator.wikimedia.org/T359534#10081651 (10joanna_borun) p:05Triage→03Low [14:34:51] 06cloud-services-team: Replace or deprecate WMCS uses of report updater - https://phabricator.wikimedia.org/T357856#10081658 (10Andrew) @lbowmaker can we get a status update on this? [14:35:28] 06cloud-services-team, 10Toolforge: Toolforge: consider introducing some semantics for persistent storage - https://phabricator.wikimedia.org/T337192#10081662 (10Raymond_Ndibe) [14:35:49] 06cloud-services-team, 10Toolforge: Toolforge: consider introducing a command line for creating reverse proxies - https://phabricator.wikimedia.org/T337191#10081664 (10Raymond_Ndibe) [14:36:05] 06cloud-services-team, 10Openstack-Magnum: Hide fedora images from human Horizon users - https://phabricator.wikimedia.org/T356547#10081665 (10Andrew) Need to figure out if everyone can see these or just admins. [14:36:26] 06cloud-services-team, 10Openstack-Magnum: Hide fedora images from human Horizon users - https://phabricator.wikimedia.org/T356547#10081667 (10Andrew) p:05Triage→03Medium [14:36:48] 06cloud-services-team, 10Horizon: Horizon identity -> roles link logs user out when unauthorized - https://phabricator.wikimedia.org/T356162#10081668 (10joanna_borun) p:05Triage→03Low [14:36:58] 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): [k8s,infra] scale up coredns replicas - https://phabricator.wikimedia.org/T333934#10081669 (10Raymond_Ndibe) [14:37:36] 06cloud-services-team, 10Toolforge: toolforge k8s automation: introduce option to reboot a node if the uptime is higher than XYZ - https://phabricator.wikimedia.org/T356970#10081671 (10Raymond_Ndibe) [14:37:47] 06cloud-services-team, 10Cloud-VPS: wmcs.openstack.restart_openstack deals badly with down cloudvirts - https://phabricator.wikimedia.org/T355058#10081673 (10joanna_borun) p:05Triage→03Medium [14:39:12] 06cloud-services-team, 10Cloud-VPS, 07Upstream: [trove] does not expose amount of disk space used - https://phabricator.wikimedia.org/T354728#10081679 (10joanna_borun) p:05Triage→03Low [14:39:59] 06cloud-services-team, 10Toolforge, 07Kubernetes: [infra] Upgrade Toolforge K8s etcd nodes to Bookworm - https://phabricator.wikimedia.org/T361237#10081680 (10Raymond_Ndibe) [14:40:23] 10PAWS: update helm chart and jupyterhub - https://phabricator.wikimedia.org/T373029 (10rook) 03NEW [14:40:26] 06cloud-services-team: toolforge: prometheus server died - https://phabricator.wikimedia.org/T370143#10081703 (10Raymond_Ndibe) [14:41:55] 06cloud-services-team, 10Toolforge: toolforge k8s automation: introduce option to force reboot each node directly - https://phabricator.wikimedia.org/T356969#10081714 (10Raymond_Ndibe) [14:42:02] 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#10081715 (10Andrew) p:05Triage→03Medium [14:42:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance paws-puppetserver-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:43:13] 06cloud-services-team, 10Cloud-VPS: Neutron API backends are flapping - https://phabricator.wikimedia.org/T353796#10081710 (10Andrew) 05Open→03Resolved a:03Andrew I don't think this is happening anymore. Can reopen if we see it again. [14:44:37] 06cloud-services-team, 10Data-Services: Tools violating the connection handling policy - https://phabricator.wikimedia.org/T353551#10081722 (10joanna_borun) p:05Triage→03Medium [14:44:39] 10Tools: wdumps tool violates the Toolforge database connection handling policy - https://phabricator.wikimedia.org/T353553#10081724 (10joanna_borun) p:05Triage→03Medium [14:44:41] 06cloud-services-team, 10Cloud-VPS: Hide + disable 'key pair' tab when creating puppetized VMs - https://phabricator.wikimedia.org/T353331#10081723 (10Andrew) p:05Triage→03Low [14:44:43] 10Tools: montage tool violates the Toolforge database connection handling policy - https://phabricator.wikimedia.org/T353554#10081725 (10joanna_borun) p:05Triage→03Medium [14:44:50] 10Tools, 07Chinese-Sites: zhdeletionpedia tool violates the Toolforge database connection handling policy - https://phabricator.wikimedia.org/T353556#10081730 (10joanna_borun) p:05Triage→03Medium [14:45:02] 06cloud-services-team, 10Observability-Alerting, 10SRE Observability (FY2024/2025-Q1): Karma UI shows duplicate alerts - https://phabricator.wikimedia.org/T353457#10081731 (10joanna_borun) p:05Triage→03Low [14:45:33] 06cloud-services-team, 10Release-Engineering-Team (Priority Backlog 📥): Experiment with WMCS as a k8s provider for gitlab-cloud-runner cluster - https://phabricator.wikimedia.org/T353356#10081721 (10Andrew) releng people, still have interest in this? I still do :) [14:46:03] 06cloud-services-team, 10Cloud-VPS, 07PostgreSQL: Consider removing Postgres support from Trove - https://phabricator.wikimedia.org/T353018#10081727 (10Andrew) 05Open→03Declined postgres is now quite a bit better so we're going to keep it! [15:10:28] FIRING: InstanceDown: Project tools instance tools-prometheus-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:10:52] 06cloud-services-team: toolforge: prometheus server died - https://phabricator.wikimedia.org/T370143#10081819 (10Andrew) I have not done much beyond look at the syslog for 'oomkiller'. Presuming that's the problem we need to figure out what's gobbling RAM. [15:15:28] RESOLVED: InstanceDown: Project tools instance tools-prometheus-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:27:09] 06cloud-services-team, 10Release-Engineering-Team (Priority Backlog 📥): Experiment with WMCS as a k8s provider for gitlab-cloud-runner cluster - https://phabricator.wikimedia.org/T353356#10081899 (10bd808) {T372498} is semi-related work in that it is attempting to create a gitops system for working with Magnum... [15:29:58] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10081906 (10bd808) >>! In T371592#10081520, @Andrew wrote: > Will this be moot after the move to k8s, or is this block... [15:31:39] 10wikitech.wikimedia.org: Requesting Wikitech:Content administrators access for CDanis - https://phabricator.wikimedia.org/T373019#10081910 (10bd808) 05Open→03Resolved a:03bd808 https://wikitech.wikimedia.org/w/index.php?title=Special:Log&logid=973161 [15:57:12] 06cloud-services-team, 10Release-Engineering-Team (Priority Backlog 📥): Experiment with WMCS as a k8s provider for gitlab-cloud-runner cluster - https://phabricator.wikimedia.org/T353356#10082021 (10dduvall) >>! In T353356#10081721, @Andrew wrote: > releng people, still have interest in this? I still do :) In... [16:10:34] vivian-rook closed https://github.com/toolforge/paws/pull/448 [16:11:56] 10PAWS: Remove paws-127 cluster - https://phabricator.wikimedia.org/T373036 (10rook) 03NEW [16:11:58] 10PAWS: Remove paws-127 cluster - https://phabricator.wikimedia.org/T373036#10082079 (10rook) [16:11:58] 10PAWS: update helm chart and jupyterhub - https://phabricator.wikimedia.org/T373029#10082080 (10rook) [16:12:05] 10PAWS: update helm chart and jupyterhub - https://phabricator.wikimedia.org/T373029#10082081 (10rook) 05Open→03Resolved a:03rook [16:27:08] (03PS1) 10Krinkle: Skip edit if server responds with HTTP 50x [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1064414 [16:27:20] (03CR) 10Krinkle: [C:03+2] Skip edit if server responds with HTTP 50x [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1064414 (owner: 10Krinkle) [16:28:09] (03Merged) 10jenkins-bot: Skip edit if server responds with HTTP 50x [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1064414 (owner: 10Krinkle) [16:55:53] 06cloud-services-team, 10Web Team Visual Regression Framework, 10Quality-and-Test-Engineering-Team (Test Infrastructure): Move disk space and other Pixel metrics from Graphite to Prometheus - https://phabricator.wikimedia.org/T363969#10082323 (10Peter) Hi WMCS, I need your help/feedback! We run the [[ https:... [17:00:03] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10082373 (10Andrew) Right, but after the sul migration the ldap extension will be dead code, right? Or does it need to... [17:10:28] FIRING: InstanceDown: Project tools instance tools-prometheus-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:15:28] RESOLVED: InstanceDown: Project tools instance tools-prometheus-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:15:58] FIRING: [2x] InstanceDown: Project tools instance tools-prometheus-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:19:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:22:44] FIRING: Toolforge Kyverno unknown state: Toolforge Kyverno has unknown state. Kyverno might be down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/Toolforge_Kyverno_unknown_state - https://grafana-rw.wmcloud.org/d/kyverno/kyverno?orgId=1&var-DS_PROMETHEUS_KYVERNO=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforge+Kyverno+unknown+state [17:25:58] RESOLVED: [2x] InstanceDown: Project tools instance tools-prometheus-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:27:44] RESOLVED: Toolforge Kyverno unknown state: Toolforge Kyverno has unknown state. Kyverno might be down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/Toolforge_Kyverno_unknown_state - https://grafana-rw.wmcloud.org/d/kyverno/kyverno?orgId=1&var-DS_PROMETHEUS_KYVERNO=prometheus-tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforge+Kyverno+unknown+state [17:38:31] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "etytree" project Buster deprecation - https://phabricator.wikimedia.org/T367529#10082489 (10Andrew) I really need this dealt with by September 10th, when I am going on Sabbatical. The initial notice about this deprecation was sent in March , with a deadline of... [18:07:29] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Remove or replace deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370460#10082561 (10Eevans) Ok, status update: With th... [18:10:20] 06cloud-services-team: toolforge: prometheus server died - https://phabricator.wikimedia.org/T370143#10082574 (10fnegri) I also don't have many pointers, but this is happening on both tools-prometheus-6 and tools-prometheus-7. The server becomes unresponsive and it's not possible to ssh to it, or even to run `vi... [18:16:09] 06cloud-services-team: toolforge: prometheus server died - https://phabricator.wikimedia.org/T370143#10082612 (10fnegri) I can see multiple "Out of memory" errors, all related to the `prometheus` process. tools-prometheus-6: ` root@tools-prometheus-6:~# journalctl --since -7d -g "Killed process" --no-pager -- J... [18:17:57] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10082616 (10bd808) >>! In T371592#10082373, @Andrew wrote: > Right, but after the sul migration the ldap extension wil... [18:33:26] 06cloud-services-team, 10Data-Services, 07Epic: [toolsdb] convert myisam tables to innodb - https://phabricator.wikimedia.org/T306455#10082649 (10fnegri) Updated numbers: ` MariaDB [information_schema]> select ENGINE, count(*) from tables where ENGINE in ('InnoDB', 'MyISAM') group by ENGINE; +--------+-----... [18:33:31] FIRING: ToolsToolsDBWritableState: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [18:36:28] FIRING: InstanceDown: Project tools instance tools-db-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:46:28] RESOLVED: InstanceDown: Project tools instance tools-db-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:48:31] RESOLVED: ToolsToolsDBWritableState: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [18:53:31] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 06serviceops: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10082693 (10Andrew) Yep, that makes sense. I was confused because it popped up in our inbox as though it was an isolat... [19:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:48:20] 14Grid-Engine-to-K8s-Migration, 10Wiki-Loves-Monuments-Database: Migrate heritage from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319787#10082843 (10JeanFred) I got something working, will wait overnight to see if it went well, and will then send to Gerrit the necessary c... [19:50:57] (03CR) 10Lokal Profil: [C:03+2] "any particular reason for the port change (just out of curiosity)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1063853 (owner: 10Jean-Frédéric) [19:53:59] (03Merged) 10jenkins-bot: Upgrade docker-compose files from legacy syntax [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1063853 (owner: 10Jean-Frédéric) [20:40:37] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "wikicommunityhealth" project Buster deprecation - https://phabricator.wikimedia.org/T367560#10082978 (10CristianCantoro) Hi Andrew, sorry for the late reply. I have marked the project as "In use" in the [[ https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2024... [20:44:19] (03update) 10raymond-ndibe: [jobs-cli] remove unknown keys from dump [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/64 (https://phabricator.wikimedia.org/T341066) [20:46:14] (03update) 10raymond-ndibe: [jobs-cli] multi-replica support for continuous jobs [repos/cloud/toolforge/jobs-cli] (remove_unknown_keys_in_dump) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/63 (https://phabricator.wikimedia.org/T341066) [20:46:48] (03open) 10raymond-ndibe: [jobs-cli] upate autocomplete and man files [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/66 [20:47:15] (03update) 10raymond-ndibe: [jobs-cli] remove unknown keys from dump [repos/cloud/toolforge/jobs-cli] (update_autocomplete_and_man_files) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/64 (https://phabricator.wikimedia.org/T341066) [20:47:27] (03update) 10raymond-ndibe: [jobs-cli] remove unknown keys from dump [repos/cloud/toolforge/jobs-cli] (update_autocomplete_and_man_files) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/64 (https://phabricator.wikimedia.org/T341066) [20:49:41] (03update) 10raymond-ndibe: [jobs-cli] remove unknown keys from dump [repos/cloud/toolforge/jobs-cli] (update_autocomplete_and_man_files) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/64 (https://phabricator.wikimedia.org/T341066) [20:52:56] (03update) 10raymond-ndibe: [jobs-cli] multi-replica support for continuous jobs [repos/cloud/toolforge/jobs-cli] (remove_unknown_keys_in_dump) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/63 (https://phabricator.wikimedia.org/T341066) [20:53:52] (03update) 10raymond-ndibe: [jobs-cli] multi-replica support for continuous jobs [repos/cloud/toolforge/jobs-cli] (remove_unknown_keys_in_dump) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/63 (https://phabricator.wikimedia.org/T341066) [21:02:40] (03PS1) 10Lokal Profil: Add LICENSE to repo [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1064471 (https://phabricator.wikimedia.org/T174633) [21:18:12] (03PS1) 10Lokal Profil: Update documentation on localhost address [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1064474 [22:11:59] (03approved) 10raymond-ndibe: db: add database check and boileplate [toolforge-repos/sample-complex-app-backend] (add_worker) - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/2 (https://phabricator.wikimedia.org/T370317) (owner: 10dcaro) [22:12:35] (03approved) 10raymond-ndibe: worker: add simple task and worker process [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/1 (https://phabricator.wikimedia.org/T370321) (owner: 10dcaro) [22:15:49] 06cloud-services-team, 10Data-Services: Enable innodb_rollback_on_timeout for wmflabs databases - https://phabricator.wikimedia.org/T251030#10083230 (10mahmoud) Got it. It wasn't always the default, but it has been for at least a decade. I haven't been able to find why (and what would be the harms of enabl... [22:26:49] (03update) 10raymond-ndibe: show task status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/2 (https://phabricator.wikimedia.org/T370321) (owner: 10dcaro) [23:09:28] FIRING: InstanceDown: Project tools instance tools-prometheus-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown