[00:04:32] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications (Echo): Alerts are not being read correctly - https://phabricator.wikimedia.org/T387686#10594589 (10Reedy) I'm not quite sure if we have specific maintenance script that may help here... This may have helped? ` reedy@deploy2002:~$ mwscript extensions... [00:25:56] FIRING: MaxConntrack: Max conntrack at 80.12% on cloudvirt1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:30:56] RESOLVED: MaxConntrack: Max conntrack at 80.12% on cloudvirt1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:32:39] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:37:39] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:46:26] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications (Echo): Alerts are not being marked as read correctly - https://phabricator.wikimedia.org/T387686#10594621 (10Reedy) [00:48:56] FIRING: MaxConntrack: Max conntrack at 80.07% on cloudvirt1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:58:56] RESOLVED: MaxConntrack: Max conntrack at 80.03% on cloudvirt1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [04:22:39] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:27:39] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:02:39] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:21:06] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:26:06] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:34:50] 10VPS-project-Phabricator: phabricator.wmcloud.org says "Access denied for user 'app_user'@'localhost'" - https://phabricator.wikimedia.org/T387619#10594899 (10Anoop) [07:36:26] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org says "Access denied for user 'app_user'@'localhost'" - https://phabricator.wikimedia.org/T387619#10594901 (10LSobanski) [08:56:06] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:00:29] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org says "Access denied for user 'app_user'@'localhost'" - https://phabricator.wikimedia.org/T387619#10595064 (10ABran-WMF) Here is the full log: ` [Mon Mar 03 08:56:08.724529 2025] [proxy_fcgi:error] [pid 2950329:tid 2950521] [client 17... [09:01:06] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:06:06] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:13:33] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/40 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:13:36] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/40 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:13:50] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/232 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:13:53] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/232 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:14:31] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/76 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:14:34] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/76 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:15:49] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/61 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:15:52] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/61 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:16:29] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/72 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:16:33] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/72 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:18:49] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: api-gateway: bump to 0.0.63-20250303091606-9854d1cb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/682 [09:24:00] (03approved) 10aborrero: namespace: add comment about the tenancy label usage [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/66 (owner: 10dcaro) [09:24:04] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [09:24:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:28:44] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [09:28:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:31:02] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [09:31:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:38:00] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [09:38:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:40:10] (03approved) 10aborrero: volume-admission: add first test for disallowed path [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/675 (owner: 10dcaro) [09:43:49] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [09:43:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:50:36] (03approved) 10aborrero: admission: deny when unknown hostPath volumes passed [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 (owner: 10dcaro) [09:51:16] (03close) 10aborrero: Draft: kyverno: add cluster-wide-policy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/673 (https://phabricator.wikimedia.org/T386921) [09:52:14] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [09:52:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:53:02] (03approved) 10dcaro: api-gateway: bump to 0.0.63-20250303091606-9854d1cb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/682 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:53:07] (03merge) 10dcaro: api-gateway: bump to 0.0.63-20250303091606-9854d1cb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/682 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:07:01] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge-legacy-redirector: constant failed probes by prometheus - https://phabricator.wikimedia.org/T385908#10595374 (10dcaro) It seems to be mostly https traffic that get delayed (not sure yet why), @aborrero suggested skipping the http->https local... [10:08:43] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/101 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:08:47] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/101 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:09:11] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/19 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:09:14] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/19 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:09:16] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/85 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:09:20] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/85 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:12:31] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/67 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:12:36] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/67 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [10:14:15] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: builds-builder: bump to 0.0.126-20250303101244-a6b6f399 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/683 [10:21:53] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge-legacy-redirector: constant failed probes by prometheus - https://phabricator.wikimedia.org/T385908#10595415 (10dcaro) > It seems to be mostly https traffic that get delayed (not sure yet why), Some of the tests: ` # Quick'n'dirty stats on ho... [10:22:30] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [10:22:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:26:28] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [10:26:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:05:11] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [11:05:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:13:24] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [11:13:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:15:32] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [11:15:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:23:34] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [11:23:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:43:34] (03approved) 10dcaro: builds-builder: bump to 0.0.126-20250303101244-a6b6f399 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/683 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [11:43:37] (03merge) 10dcaro: builds-builder: bump to 0.0.126-20250303101244-a6b6f399 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/683 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [12:00:35] 06cloud-services-team, 10Cloud-VPS: openstack network problems (November 2024) - https://phabricator.wikimedia.org/T380882#10595640 (10aborrero) 05Open→03Resolved a:03aborrero I think the most important subtasks have been resolved, so I will close this ticket. We can keep working on the few subtasks... [12:01:08] (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804) [12:01:25] 06cloud-services-team, 10Toolforge: tools-nfs outage 2024-11-25 - https://phabricator.wikimedia.org/T380827#10595646 (10aborrero) [12:01:27] 06cloud-services-team, 10Toolforge: rsyslogd tls issues on tools-nfs-2 - https://phabricator.wikimedia.org/T380829#10595645 (10aborrero) [12:04:04] (03update) 10raymond-ndibe: [jobs-api] create seperate api.py and move flask things there [repos/cloud/toolforge/jobs-api] (diff_job_runtime_method) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [12:10:19] (03update) 10raymond-ndibe: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (split_logic_from_api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (https://phabricator.wikimedia.org/T359650) [12:10:51] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Migrate labweb-ssl LB VIPs to IPIP encapsulation - https://phabricator.wikimedia.org/T387305#10595661 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez [12:11:02] (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804) [12:13:08] 06cloud-services-team: Onboard Chuck Onwumelu - https://phabricator.wikimedia.org/T386715#10595666 (10aborrero) [12:22:52] 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS: Decision Request - How openstack projects relate to tofu-infra - https://phabricator.wikimedia.org/T385604#10595694 (10aborrero) >>! In T385604#10567062, @dcaro wrote: > I would like to see the 'TBD' parts filled up before expressing an opinion... [12:27:11] (03update) 10raymond-ndibe: [jobs-api] create seperate api.py and move flask things there [repos/cloud/toolforge/jobs-api] (diff_job_runtime_method) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [12:32:33] (03update) 10raymond-ndibe: [jobs-api] create seperate api.py and move flask things there [repos/cloud/toolforge/jobs-api] (diff_job_runtime_method) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [12:41:31] (03update) 10raymond-ndibe: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (split_logic_from_api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (https://phabricator.wikimedia.org/T359650) [12:42:33] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/45 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [12:42:35] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/45 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [12:45:12] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: maintain-harbor: bump to 0.0.20-20250303124254-00009de4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/684 [12:45:13] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: maintain-harbor: bump to 0.0.20-20250303124254-00009de4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/684 [12:45:24] (03update) 10raymond-ndibe: [jobs-api] create seperate api.py and move flask things there [repos/cloud/toolforge/jobs-api] (diff_job_runtime_method) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [12:46:39] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [12:46:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:51:07] (03update) 10raymond-ndibe: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (split_logic_from_api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (https://phabricator.wikimedia.org/T359650) [12:51:26] (03update) 10raymond-ndibe: [jobs-api] create seperate api.py and move flask things there [repos/cloud/toolforge/jobs-api] (diff_job_runtime_method) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [12:54:48] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [12:54:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [13:01:38] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [13:01:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:02:41] (03update) 10raymond-ndibe: [jobs-api] remove wait_for_job from runtime methods [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/138 (https://phabricator.wikimedia.org/T359804) [13:04:18] 06cloud-services-team, 10Cloud-VPS, 10Ceph, 06DC-Ops, and 2 others: evaluate new drives in cloudcephosd102[123] - https://phabricator.wikimedia.org/T386725#10595793 (10dcaro) I did a first check of the current values for the smartcl reported counters, all look good so far (no more Offline_Uncorrectable_Err... [13:07:04] (03update) 10raymond-ndibe: [jobs-api] remove wait_for_job from runtime methods [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/138 (https://phabricator.wikimedia.org/T359804) [13:10:31] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [13:10:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:11:10] 10Toolforge (Toolforge iteration 18): [jobs-api] rename variable/parameter type to job_type - https://phabricator.wikimedia.org/T387727 (10Raymond_Ndibe) 03NEW [13:12:14] (03update) 10raymond-ndibe: [jobs-api] remove wait_for_job from runtime methods [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/138 (https://phabricator.wikimedia.org/T359804) [13:12:22] (03update) 10raymond-ndibe: [jobs-api] remove wait_for_job from runtime methods [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/138 (https://phabricator.wikimedia.org/T359804) [13:13:29] (03merge) 10raymond-ndibe: [toolforge-weld] update jobs custom resources version in k8sclient [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/75 (https://phabricator.wikimedia.org/T359650) [13:27:39] (03approved) 10dcaro: maintain-harbor: bump to 0.0.20-20250303124254-00009de4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/684 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:27:41] (03merge) 10dcaro: maintain-harbor: bump to 0.0.20-20250303124254-00009de4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/684 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:27:57] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/120 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:28:03] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/120 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [13:29:48] 06cloud-services-team, 10Toolforge, 07Epic: [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools - https://phabricator.wikimedia.org/T127367#10595903 (10aborrero) >>! In T127367#10579141, @Andrew wrote: > We discussed loki storage a bit, it seems like we can get a... [13:30:09] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10595904 (10aborrero) read: https://opentofu.org/docs/language/modules/develop/refactoring/ [13:32:11] (03open) 10raymond-ndibe: d/changelog: bump to 1.6.8 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/76 (https://phabricator.wikimedia.org/T359650) [13:33:21] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [13:36:00] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: builds-api: bump to 0.0.179-20250303132834-fd3dba9e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/685 [13:40:12] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [13:40:53] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [13:49:48] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [13:50:58] (03update) 10raymond-ndibe: d/changelog: bump to 1.6.8 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/76 (https://phabricator.wikimedia.org/T359650) [13:51:00] (03approved) 10raymond-ndibe: d/changelog: bump to 1.6.8 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/76 (https://phabricator.wikimedia.org/T359650) [13:51:06] (03merge) 10raymond-ndibe: d/changelog: bump to 1.6.8 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/76 (https://phabricator.wikimedia.org/T359650) [13:55:23] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [13:56:22] 06cloud-services-team, 10Cloud-VPS, 10Ceph, 06DC-Ops, and 2 others: evaluate new drives in cloudcephosd102[123] - https://phabricator.wikimedia.org/T386725#10596009 (10dcaro) Just checked the number of operations/s (as a proxy for performance): * For cloudcephosd1021, comparing with 1018, there's a bit of... [13:56:47] 06cloud-services-team, 10Cloud-VPS, 10Ceph, 06DC-Ops, and 2 others: evaluate new drives in cloudcephosd102[123] - https://phabricator.wikimedia.org/T386725#10596011 (10dcaro) 05Open→03Resolved [13:58:05] (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804) [13:58:35] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [13:58:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [13:59:29] (03update) 10raymond-ndibe: [jobs-api] create seperate api.py and move flask things there [repos/cloud/toolforge/jobs-api] (diff_job_runtime_method) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [14:01:31] (03update) 10raymond-ndibe: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (split_logic_from_api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (https://phabricator.wikimedia.org/T359650) [14:02:48] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [14:03:39] FIRING: CloudVPSBastionlessUsers: Detected 1 cloud-vps users not in Bastion project - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Users_not_in_bastion_project - https://logstash.wikimedia.org/app/dashboards#/view/3ef008b0-c871-11eb-ad54-8bb5fcb640c0?_g=h@e78830b&_a=h@21715bf - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSBastionlessUsers [14:05:28] (03update) 10raymond-ndibe: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804) [14:07:48] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [14:07:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:08:23] 06cloud-services-team: Onboard Chuck Onwumelu - https://phabricator.wikimedia.org/T386715#10596050 (10aborrero) [14:08:35] 06cloud-services-team: Onboard Chuck Onwumelu - https://phabricator.wikimedia.org/T386715#10596051 (10aborrero) [14:10:38] (03update) 10raymond-ndibe: [jobs-api] create seperate api.py and move flask things there [repos/cloud/toolforge/jobs-api] (diff_job_runtime_method) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [14:24:09] 06cloud-services-team, 10Toolforge, 07Epic: [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools - https://phabricator.wikimedia.org/T127367#10596149 (10dcaro) > In my opinion, this design & architecture phase should be completed before any implementation work is s... [14:24:37] (03update) 10raymond-ndibe: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (split_logic_from_api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (https://phabricator.wikimedia.org/T359650) [14:29:43] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [14:31:36] (03open) 10matmarex: Allow scheduling changes for windows that are in progress [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/18 [14:32:29] (03update) 10raymond-ndibe: [builds-builder] create and use maintain-harbor robot account [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/66 (https://phabricator.wikimedia.org/T361698) [14:37:23] (03update) 10raymond-ndibe: [builds-builder] create and use maintain-harbor robot account [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/66 (https://phabricator.wikimedia.org/T361698) [14:37:24] (03update) 10raymond-ndibe: [builds-builder] create and use maintain-harbor robot account [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/66 (https://phabricator.wikimedia.org/T361698) [14:37:31] (03update) 10raymond-ndibe: [builds-builder] create and use maintain-harbor robot account [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/66 (https://phabricator.wikimedia.org/T361698) [14:39:12] (03update) 10raymond-ndibe: [do-not-merge][lima-kilo] test maintain-harbor robot account [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/231 (https://phabricator.wikimedia.org/T361698) [14:39:24] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [14:41:26] (03update) 10matmarex: Allow scheduling changes for windows that are in progress [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/18 [14:43:57] (03update) 10matmarex: Allow scheduling changes for windows that are in progress [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/18 [14:46:49] (03update) 10matmarex: Allow scheduling changes for windows that are in progress [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/18 [14:50:38] (03approved) 10dcaro: builds-api: bump to 0.0.179-20250303132834-fd3dba9e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/685 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [14:50:41] (03merge) 10dcaro: builds-api: bump to 0.0.179-20250303132834-fd3dba9e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/685 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [14:50:53] (03update) 10raymond-ndibe: [builds-builder] create and use maintain-harbor robot account [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/66 (https://phabricator.wikimedia.org/T361698) [14:53:20] (03update) 10raymond-ndibe: [builds-builder] create and use maintain-harbor robot account [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/66 (https://phabricator.wikimedia.org/T361698) [15:20:28] (03update) 10raymond-ndibe: [maintain-harbor] move image_retention_policy_template to config [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/44 (https://phabricator.wikimedia.org/T386953) [15:22:38] (03update) 10raymond-ndibe: [ maintain-harbor ] add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [15:25:33] (03update) 10raymond-ndibe: [maintain-harbor] move image_retention_policy_template to config [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/44 (https://phabricator.wikimedia.org/T386953) [15:28:17] 06cloud-services-team: Onboard Chuck Onwumelu - https://phabricator.wikimedia.org/T386715#10596453 (10Chuckonwumelu) [15:33:17] (03update) 10raymond-ndibe: [maintain-harbor] move image_retention_policy_template to config [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/44 (https://phabricator.wikimedia.org/T386953) [15:34:19] (03update) 10raymond-ndibe: [ maintain-harbor ] add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [15:35:38] (03update) 10raymond-ndibe: [maintain-harbor] move image_retention_policy_template to config [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/44 (https://phabricator.wikimedia.org/T386953) [15:39:17] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/51 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:39:23] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/51 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:45:24] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: envvars-api: bump to 0.0.65-20250303153943-68878700 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/686 [15:59:37] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [15:59:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:08:47] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [16:08:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:09:49] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [16:09:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:18:22] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [16:18:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:19:12] (03approved) 10dcaro: envvars-api: bump to 0.0.65-20250303153943-68878700 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/686 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:19:15] (03merge) 10dcaro: envvars-api: bump to 0.0.65-20250303153943-68878700 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/686 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:19:21] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/26 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:19:25] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/26 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [16:22:58] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: volume-admission: bump to 0.0.62-20250303161935-8fdaa3fc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/687 [16:34:32] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications (Echo): Alerts are not being marked as read correctly - https://phabricator.wikimedia.org/T387686#10596888 (10Szczot3k) >>! In T387686#10594589, @Reedy wrote: > I'm not quite sure if we have specific maintenance script that may help here... > > This m... [16:36:48] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications (Echo): Alerts are not being marked as read correctly - https://phabricator.wikimedia.org/T387686#10596905 (10Pppery) 05Open→03Resolved a:03Reedy [16:41:26] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [16:41:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:41:39] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org says "Access denied for user 'app_user'@'localhost'" - https://phabricator.wikimedia.org/T387619#10596945 (10LSobanski) @brennen for awareness. [16:46:40] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org says "Access denied for user 'app_user'@'localhost'" - https://phabricator.wikimedia.org/T387619#10596970 (10LSobanski) a:03Dzahn [16:48:42] (03update) 10aborrero: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 (https://phabricator.wikimedia.org/T375283) (owner: 10fnegri) [16:49:06] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [16:49:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:54:27] (03update) 10dcaro: deployment: update the deployment state during deploy [repos/cloud/toolforge/components-api] (deploy_source_build_too) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/54 [16:55:06] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [16:55:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:57:09] RESOLVED: CloudVPSBastionlessUsers: Detected 1 cloud-vps users not in Bastion project - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Users_not_in_bastion_project - https://logstash.wikimedia.org/app/dashboards#/view/3ef008b0-c871-11eb-ad54-8bb5fcb640c0?_g=h@e78830b&_a=h@21715bf - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSBastionlessUsers [16:59:57] (03update) 10dcaro: deployment: update the deployment state during deploy [repos/cloud/toolforge/components-api] (deploy_source_build_too) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/54 [17:02:47] (03CR) 10BryanDavis: "T208443 might make this less useful than it seems for new users." [labs/striker] - 10https://gerrit.wikimedia.org/r/1118564 (owner: 10Majavah) [17:03:09] 06cloud-services-team, 06DC-Ops, 10Ganeti, 06Infrastructure-Foundations, and 2 others: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10597032 (10Jclark-ctr) @fnegri @VRiley-WMF did this need to be reopened. idrac shows The system inlet temperature... [17:04:12] (03CR) 10BryanDavis: "Or am I completely misunderstanding that bug?" [labs/striker] - 10https://gerrit.wikimedia.org/r/1118564 (owner: 10Majavah) [17:04:30] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [17:04:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:14:49] (03approved) 10dcaro: volume-admission: bump to 0.0.62-20250303161935-8fdaa3fc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/687 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:14:50] (03merge) 10dcaro: volume-admission: bump to 0.0.62-20250303161935-8fdaa3fc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/687 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:15:05] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/60 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:15:11] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/60 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:16:37] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/60 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [17:18:25] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: api-gateway: bump to 0.0.64-20250303171648-bd834b88 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/688 [17:20:02] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [17:20:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:27:05] (03update) 10raymond-ndibe: [jobs-api] create seperate api.py and move flask things there [repos/cloud/toolforge/jobs-api] (diff_job_runtime_method) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [17:28:42] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [17:28:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:38:25] 10Tool-schedule-deployment: Allow scheduling for current backport window - https://phabricator.wikimedia.org/T381237#10597302 (10bd808) 05Open→03In progress a:03matmarex @matmarex started https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/18 which will address this issue. [17:38:38] 10Tool-schedule-deployment: Allow scheduling for current backport window - https://phabricator.wikimedia.org/T381237#10597306 (10bd808) p:05Triage→03Medium [17:42:26] (03update) 10raymond-ndibe: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (split_logic_from_api) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (https://phabricator.wikimedia.org/T359650) [18:10:48] (03update) 10matmarex: Allow scheduling changes for windows that are in progress [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/18 [18:10:49] (03update) 10matmarex: Allow scheduling changes for windows that are in progress [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/18 [18:35:00] 10Tool-video-answer-tool, 06Future-Audiences: User testing for short video - https://phabricator.wikimedia.org/T379309#10597735 (10DLin-WMF) 05Open→03Resolved [18:45:57] (03update) 10raymond-ndibe: [toolforge-deploy] maintain-harbor use robot account [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/672 (https://phabricator.wikimedia.org/T361698) [18:48:26] (03update) 10raymond-ndibe: [builds-api] use maintain-harbor robot account locally [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/119 (https://phabricator.wikimedia.org/T361698) [18:48:53] (03update) 10raymond-ndibe: [builds-api] use maintain-harbor robot account locally [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/119 (https://phabricator.wikimedia.org/T361698) [18:54:50] (03update) 10raymond-ndibe: [builds-api] use maintain-harbor robot account locally [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/119 (https://phabricator.wikimedia.org/T361698) [18:55:03] (03update) 10raymond-ndibe: [builds-api] use maintain-harbor robot account locally [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/119 (https://phabricator.wikimedia.org/T361698) [18:58:13] (03update) 10raymond-ndibe: [toolforge-deploy] maintain-harbor use robot account [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/672 (https://phabricator.wikimedia.org/T361698) [18:59:41] (03update) 10bd808: Allow scheduling changes for windows that are in progress [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/18 (https://phabricator.wikimedia.org/T381237) (owner: 10matmarex) [19:01:31] (03update) 10bd808: Allow scheduling changes for windows that are in progress [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/18 (https://phabricator.wikimedia.org/T381237) (owner: 10matmarex) [19:02:33] (03merge) 10bd808: Allow scheduling changes for windows that are in progress [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/18 (https://phabricator.wikimedia.org/T381237) (owner: 10matmarex) [19:05:56] (03open) 10raymond-ndibe: [toolforge-deploy] maintain-harbor use robot account [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/689 (https://phabricator.wikimedia.org/T361698 https://phabricator.wikimedia.org/T386953) [19:06:10] (03update) 10raymond-ndibe: [toolforge-deploy] maintain-harbor use robot account [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/689 (https://phabricator.wikimedia.org/T361698 https://phabricator.wikimedia.org/T386953) [19:06:24] (03update) 10raymond-ndibe: [toolforge-deploy] maintain-harbor use robot account [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/689 (https://phabricator.wikimedia.org/T361698 https://phabricator.wikimedia.org/T386953) [19:07:03] (03update) 10raymond-ndibe: [toolforge-deploy] refactor maintain-harbor config [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/689 (https://phabricator.wikimedia.org/T361698 https://phabricator.wikimedia.org/T386953) [19:07:21] (03update) 10raymond-ndibe: [toolforge-deploy] refactor maintain-harbor config [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/689 (https://phabricator.wikimedia.org/T386953) [19:09:20] (03update) 10raymond-ndibe: [maintain-harbor] move image_retention_policy_template to config [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/44 (https://phabricator.wikimedia.org/T386953) [19:09:33] (03update) 10raymond-ndibe: [maintain-harbor] move image_retention_policy_template to config [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/44 (https://phabricator.wikimedia.org/T386953) [19:09:37] 10Tool-schedule-deployment: Allow scheduling for current backport window - https://phabricator.wikimedia.org/T381237#10597898 (10bd808) 05In progress→03Resolved [19:11:30] 10Tool-schedule-deployment: ScheduleDeploymentBot should require confirmation to add more than 6 patches to a backport window - https://phabricator.wikimedia.org/T367229#10597899 (10bd808) [19:23:08] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [19:23:16] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (refactor_maintain_harbor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [19:25:53] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (refactor_maintain_harbor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [19:28:19] (03update) 10raymond-ndibe: [toolforge-deploy] maintain-harbor use robot account [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/672 (https://phabricator.wikimedia.org/T361698) [19:28:30] (03update) 10raymond-ndibe: [toolforge-deploy] maintain-harbor use robot account [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/672 (https://phabricator.wikimedia.org/T361698) [19:38:58] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (refactor_maintain_harbor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [19:41:55] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config [repos/cloud/toolforge/toolforge-deploy] (refactor_maintain_harbor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [19:43:01] (03open) 10raymond-ndibe: [toolforge-deploy] test maintain-harbor quota management [repos/cloud/toolforge/toolforge-deploy] (add_maintain_harbor_quota_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/690 (https://phabricator.wikimedia.org/T352417) [19:44:32] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config [repos/cloud/toolforge/toolforge-deploy] (refactor_maintain_harbor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [19:47:06] (03update) 10raymond-ndibe: [ maintain-harbor ] add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [19:47:36] (03update) 10raymond-ndibe: [ maintain-harbor ] add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [19:47:57] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org says "Access denied for user 'app_user'@'localhost'" - https://phabricator.wikimedia.org/T387619#10598049 (10Dzahn) What happened is mariadb got restarted on March 1st and the datadir was configured as the default /var/lib/mysql where... [19:48:23] (03update) 10raymond-ndibe: [toolforge-deploy] test maintain-harbor quota management [repos/cloud/toolforge/toolforge-deploy] (add_maintain_harbor_quota_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/690 (https://phabricator.wikimedia.org/T352417) [19:48:32] (03update) 10raymond-ndibe: [toolforge-deploy] test maintain-harbor quota management [repos/cloud/toolforge/toolforge-deploy] (add_maintain_harbor_quota_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/690 (https://phabricator.wikimedia.org/T352417) [19:49:31] (03update) 10raymond-ndibe: admission: deny when unknown hostPath volumes passed [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 (owner: 10dcaro) [19:49:36] (03update) 10raymond-ndibe: admission: deny when unknown hostPath volumes passed [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 (owner: 10dcaro) [19:49:40] (03update) 10raymond-ndibe: admission: deny when unknown hostPath volumes passed [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 (owner: 10dcaro) [19:49:42] (03approved) 10raymond-ndibe: admission: deny when unknown hostPath volumes passed [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 (owner: 10dcaro) [19:52:53] (03update) 10raymond-ndibe: [do-not-merge][lima-kilo] test maintain-harbor robot account [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/231 (https://phabricator.wikimedia.org/T361698) [19:53:16] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org says "Access denied for user 'app_user'@'localhost'" - https://phabricator.wikimedia.org/T387619#10598069 (10Dzahn) 05Open→03In progress [19:54:25] (03update) 10raymond-ndibe: [do-not-merge][lima-kilo] test maintain-harbor robot account [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/231 (https://phabricator.wikimedia.org/T361698) [20:03:08] 10VPS-project-Phabricator: Rename account on test instance of Phabricator - https://phabricator.wikimedia.org/T265360#10598095 (10Pppery) 05Open→03Invalid Looks like the Phabricator test database has been reset since. Neither https://phabricator.wmcloud.org/p/Zoranzoki21/ nor https://phabricator.wmcloud... [21:55:09] 10Tool-schedule-deployment: Allow scheduling for current backport window - https://phabricator.wikimedia.org/T381237#10598403 (10matmarex) Thanks @bd808! [23:05:05] 10Tool-schedule-deployment, 10WikimediaDebug: Allow schedule-deployment to determine if WikimediaDebug browser add-on is installed to simplify end user instructions - https://phabricator.wikimedia.org/T367213#10598753 (10bd808)