[00:13:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:13:59] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [00:14:00] FIRING: [6x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [00:18:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:27:23] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [00:35:06] Change on 12wikitech.wikimedia.org a page Help:Toolforge was modified, changed by Wargo link https://wikitech.wikimedia.org/w/index.php?diff=2384391 edit summary: [-20] Undid revision [[Special:Diff/2384357|2384357]] by [[Special:Contributions/Muusejaamac100|Muusejaamac100]] ([[User talk:Muusejaamac100|talk]]) [00:38:13] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [00:43:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:44:59] 06cloud-services-team, 10Toolforge: Node.js buildpack selects EOL Node.js version - https://phabricator.wikimedia.org/T418414#11652800 (10bd808) Probably the same core issue as {T408108}. The builder-24 stack added for {T380127} has not been tracking upstream changes yet so we have point in time snapshots from... [00:48:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:53:44] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [01:13:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:18:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:20:01] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [01:35:30] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [01:38:00] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [01:43:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:47:58] (03update) 10raymond-ndibe: logs-api,logs: test start, end params for logs-api and jobs-api log endpoint [repos/cloud/toolforge/toolforge-deploy] (add_logs_api_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [01:48:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:13:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:18:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:43:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:48:56] FIRING: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:13:18] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org account verification request: ajhalili2006 - https://phabricator.wikimedia.org/T418224#11652939 (10ajhalili2006) >>! In T418224#11652020, @Dzahn wrote: > It's fine if you want to test something around ACLs but I would like to poin... [03:13:56] RESOLVED: SystemdUnitDown: The service unit prometheus-node-textfile-wmcs-dnsleaks.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:16:56] FIRING: SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:19:46] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (Google Summer of Code (2026)): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11652942 (10Henryl746) Hi there! I'm Henry, a student full-stack developer with extensive experience in Next.js, React, TailwindCSS & MUI/ShadCN-UI. It's my f... [03:21:56] FIRING: [2x] SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:27:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster on deployment eqiad1 [03:29:17] FIRING: JobUnavailable: Reduced availability for job rabbitmq in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:30:32] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.rabbitmq.rebuild_rabbit_cluster (exit_code=0) on deployment eqiad1 [03:31:49] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services [03:34:17] RESOLVED: JobUnavailable: Reduced availability for job rabbitmq in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:35:22] FIRING: [9x] HAProxyBackendUnavailable: HAProxy service designate-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [03:39:56] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:40:22] RESOLVED: [9x] HAProxyBackendUnavailable: HAProxy service designate-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [03:44:56] RESOLVED: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:46:09] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services [03:46:56] FIRING: [2x] SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:53:00] RESOLVED: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [04:14:15] FIRING: [6x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:11:56] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [07:46:56] RESOLVED: SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [07:46:56] RESOLVED: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [07:52:25] 06cloud-services-team, 10Cloud-VPS: Increased openstack latency and rabbitmq cluster rebuild - https://phabricator.wikimedia.org/T418444 (10fgiunchedi) 03NEW [08:14:15] FIRING: [6x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:21:10] (03approved) 10eugene233: fix: remove responsive settings from campaign table DataTable initialization [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/66 (https://phabricator.wikimedia.org/T418228) (owner: 10gopavasanth) [08:21:19] (03merge) 10eugene233: fix: remove responsive settings from campaign table DataTable initialization [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/66 (https://phabricator.wikimedia.org/T418228) (owner: 10gopavasanth) [09:14:00] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE: Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11653365 (10fnegri) > Ah: This is covered by T416481: Adapt Sqoop for imagelinks schema changes Should we wait until... [09:59:54] (03update) 10taavi: Deploy istio and a Gateway [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1143 (https://phabricator.wikimedia.org/T418274) [10:04:30] 10Tool-redminbot: Handle cases of both frame and thumb file options being present - https://phabricator.wikimedia.org/T418060#11653502 (10Redmin) 05Open→03In progress [10:18:08] 06cloud-services-team, 10Cloud-VPS: Increased openstack latency and rabbitmq cluster rebuild - https://phabricator.wikimedia.org/T418444#11653538 (10dcaro) On the rabbitmq nodes, there's a spike in the amount of connections ata 20:30: ` root@cloudrabbit1001:~# for i in $(seq 10 59); do echo -n "20:$i:XX - ";... [10:33:53] 10Tool-redminbot: Handle cases of redundant format file options being present - https://phabricator.wikimedia.org/T418060#11653626 (10Redmin) [10:34:53] (03open) 10r4356th: Handle cases of redundant format file options being present [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/19 (https://phabricator.wikimedia.org/T418060) [10:35:34] (03merge) 10r4356th: Handle cases of redundant format file options being present [toolforge-repos/redminbot] - 10https://gitlab.wikimedia.org/toolforge-repos/redminbot/-/merge_requests/19 (https://phabricator.wikimedia.org/T418060) [10:38:41] 10Tool-redminbot, 13Patch-For-Review: Handle cases of redundant format file options being present - https://phabricator.wikimedia.org/T418060#11653640 (10Redmin) 05In progress→03Resolved a:03Redmin [10:39:11] 10Tool-redminbot: Handle cases of redundant format file options being present - https://phabricator.wikimedia.org/T418060#11653644 (10Redmin) [10:42:52] 06cloud-services-team, 10Cloud-VPS: Increased openstack latency and rabbitmq cluster rebuild - https://phabricator.wikimedia.org/T418444#11653651 (10dcaro) After the reboots, I see some of these errors on the rabbitmq logs: ` 2026-02-25 21:02:45.263581+00:00 [error] <0.426320.0> supervisor: {<0.426320.0>,r... [10:57:40] (03update) 10dcaro: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [10:58:20] (03update) 10dcaro: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [11:03:13] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE: Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11653706 (10Zabe) >>! In T417492#11653365, @fnegri wrote: >> Ah: This is covered by T416481: Adapt Sqoop for imagelin... [11:04:10] 06cloud-services-team, 10Toolforge, 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Create voter lists on VPS instead of production - https://phabricator.wikimedia.org/T404506#11653709 (10fgiunchedi) Given the requirements I am wondering: have you considered r... [11:08:02] 10Tool-redminbot: Remove duplicated file options - https://phabricator.wikimedia.org/T416766#11653738 (10Redmin) 05Open→03Resolved a:03Redmin Looks like I just handled this as part of T418060. [11:08:45] 10Tool-redminbot: Grab list of modules from wiki replicas instead of file - https://phabricator.wikimedia.org/T417783#11653746 (10Redmin) p:05Triage→03Medium [11:10:18] 10Tool-redminbot: Grab list of modules from wiki replicas instead of file - https://phabricator.wikimedia.org/T417783#11653752 (10Redmin) [11:15:07] 10Tool-redminbot: Mention what was fixed in the edit summary - https://phabricator.wikimedia.org/T416068#11653770 (10Redmin) [11:34:49] 06cloud-services-team, 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Create voter lists on VPS instead of production - https://phabricator.wikimedia.org/T404506#11653837 (10fgiunchedi) [11:42:37] 10Tool-redminbot: Fix pwrap-bug-workaround errors - https://phabricator.wikimedia.org/T418464 (10Redmin) 03NEW [11:56:21] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE: Drop support for cl_to, cl_collation and il_to from wikireplicas - https://phabricator.wikimedia.org/T417492#11653957 (10fnegri) @Zabe good catch, thanks. It looks like categorylinks was already adapted for Sqoop in {T397923}.... [12:21:39] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/34 [12:46:11] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org account verification request: ajhalili2006 - https://phabricator.wikimedia.org/T418224#11654122 (10Dzahn) Oh, yea, that makes more sense. thanks as well. [13:04:20] 06cloud-services-team, 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Create voter lists on VPS instead of production - https://phabricator.wikimedia.org/T404506#11654193 (10SD0001) >>! In T404506#11653709, @fgiunchedi wrote: > Given the requirements I am wonde... [13:18:44] 06cloud-services-team, 10Toolforge: Toolforge NFS tracing misses some dumps events - https://phabricator.wikimedia.org/T415199#11654262 (10taavi) 05Open→03Resolved [13:20:56] 06cloud-services-team, 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Create voter lists on VPS instead of production - https://phabricator.wikimedia.org/T404506#11654266 (10SD0001) I actually forgot to update this ticket with details of the full script run don... [13:33:31] (03update) 10dcaro: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [13:37:27] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component gateway-api [13:38:26] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Install Gateway API CRDs in Toolforge clusters - https://phabricator.wikimedia.org/T418275#11654323 (10taavi) 05Open→03Resolved [13:38:49] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Update access control logic for migration from Ingress to Gateway API objects - https://phabricator.wikimedia.org/T418276#11654326 (10taavi) [13:40:09] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component gateway-api [13:41:13] (03approved) 10taavi: docs: update the deployment docs using standard deploy [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/153 (owner: 10dcaro) [13:42:03] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Update access control logic for migration from Ingress to Gateway API objects - https://phabricator.wikimedia.org/T418276#11654335 (10taavi) p:05Triage→03High [13:42:08] 06cloud-services-team, 10Toolforge: Make webservice provision HTTPRoutes for each tool - https://phabricator.wikimedia.org/T418349#11654336 (10taavi) p:05Triage→03High [13:48:10] 06cloud-services-team, 10Toolforge: Port fourohfour tool to new Gateway system - https://phabricator.wikimedia.org/T418479 (10taavi) 03NEW [14:25:26] (03update) 10dcaro: Validate Gateway API HTTPRoute resources [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/37 (https://phabricator.wikimedia.org/T418276) (owner: 10taavi) [14:32:13] (03update) 10dcaro: Deploy istio and a Gateway [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1143 (https://phabricator.wikimedia.org/T418274) (owner: 10taavi) [14:38:34] (03update) 10dcaro: logs-api,logs: add logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) (owner: 10raymond-ndibe) [14:38:54] (03update) 10dcaro: jobs-api: test for proper handling of the diff variations of the --image argument [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1113 (https://phabricator.wikimedia.org/T414978 https://phabricator.wikimedia.org/T415322) (owner: 10raymond-ndibe) [14:39:11] (03update) 10dcaro: builds-builder: bump to 0.0.141-20260224092726-64d22acc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1139 (https://phabricator.wikimedia.org/T394902) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:39:35] (03update) 10dcaro: jobs-api: bump to 0.0.464-20260224094413-e10363a0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1138 (https://phabricator.wikimedia.org/T417503) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:39:45] (03merge) 10dcaro: builds-builder: bump to 0.0.141-20260224092726-64d22acc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1139 (https://phabricator.wikimedia.org/T394902) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:40:16] (03approved) 10pepepiton: login link for work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/181 (owner: 10nurahwakili) [14:40:26] (03merge) 10pepepiton: login link for work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/181 (owner: 10nurahwakili) [14:41:31] (03update) 10dcaro: [cli] support start and end params in logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/145 (https://phabricator.wikimedia.org/T400917) (owner: 10raymond-ndibe) [14:41:37] (03update) 10dcaro: [cli] support start and end params in logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/145 (https://phabricator.wikimedia.org/T400917) (owner: 10raymond-ndibe) [14:56:13] (03merge) 10dcaro: docs: update the deployment docs using standard deploy [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/153 [15:06:54] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.208-20260226145625-3d93cff0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1144 (https://phabricator.wikimedia.org/T407477) [15:07:02] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.208-20260226145625-3d93cff0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1144 (https://phabricator.wikimedia.org/T407477) [15:42:17] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:45:17] 06cloud-services-team, 10Toolforge: [logs-api] distinguish between container log entries and anything else - https://phabricator.wikimedia.org/T410055#11655132 (10dcaro) [15:47:32] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [15:49:14] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:55:06] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [15:56:39] (03approved) 10dcaro: builds-api: bump to 0.0.208-20260226145625-3d93cff0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1144 (https://phabricator.wikimedia.org/T407477) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:56:43] (03merge) 10dcaro: builds-api: bump to 0.0.208-20260226145625-3d93cff0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1144 (https://phabricator.wikimedia.org/T407477) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:14:44] 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408#11655350 (10Henryl746) @Nokib_Sarkar @Tiven2240 Just to confirm in regards to the CampaignEditForm component, this module is supposed to be migrated into /pages and not /components? In other words, it's supposed to be... [16:36:25] 06cloud-services-team, 10Cloud-VPS: Increased openstack latency and rabbitmq cluster rebuild - https://phabricator.wikimedia.org/T418444#11655467 (10fgiunchedi) [16:48:45] 06cloud-services-team, 10Cloud-VPS: Increased openstack latency and rabbitmq cluster rebuild - https://phabricator.wikimedia.org/T418444#11655551 (10fgiunchedi) [18:52:13] (03update) 10raymond-ndibe: images: return the hash if it was passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/272 (owner: 10dcaro) [18:52:15] (03update) 10raymond-ndibe: images: return the hash if it was passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/272 (owner: 10dcaro) [18:52:16] (03approved) 10raymond-ndibe: images: return the hash if it was passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/272 (owner: 10dcaro) [19:31:59] 10Toolforge (Toolforge iteration 25): [harbor,tools] Harbor object usage in S3 is steadily increasing - https://phabricator.wikimedia.org/T418528 (10dcaro) 03NEW [20:57:00] (03unapproved) 10raymond-ndibe: images: return the hash if it was passed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/272 (owner: 10dcaro) [21:32:17] (03open) 10aghirelli: feat(linter): add rules and tests for response [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/12 [22:18:03] 10Tool-wsindex, 10Wikisource Reader App (Android): Analytics table should have book title in original language - https://phabricator.wikimedia.org/T417148#11656724 (10Muguro) updated the app to save native language for downloaded book https://github.com/cis-india/Wikisource-Reader/pull/91 [23:07:29] FIRING: PuppetAgentNoResources: No Puppet resources found on instance extdist-06 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources