[00:14:08] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [00:14:58] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [02:49:30] (03PS1) 10Amanojaku: RCReader: Migrate from old IRC feed to standard EventStreams JSON [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1312290 [03:59:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:09:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:11:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:16:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:59:20] 10Tool-subtitler: Support Wikimedia API for machine translation for Subtitler - https://phabricator.wikimedia.org/T429916#12135583 (10psubhashish1) 05Open→03Resolved [08:04:44] 06tools-platform-team: [hypothesis] 5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564 (10dcaro) 03NEW [08:05:00] 06tools-platform-team, 07Epic: [hypothesis] 5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564#12135715 (10dcaro) p:05Triage→03High [08:06:10] (03approved) 10tlepage: rotate_clinic_duty: added some helper docstrings [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/4 (owner: 10dcaro) [08:06:20] (03update) 10tlepage: rotate_clinic_duty: added some helper docstrings [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/4 (owner: 10dcaro) [08:08:21] (03approved) 10tlepage: pypi: check for properly set auth [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/89 (owner: 10dcaro) [08:08:26] (03update) 10tlepage: pypi: check for properly set auth [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/89 (owner: 10dcaro) [08:08:41] 06tools-platform-team, 07Epic: [hypothesis] 5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564#12135723 (10dcaro) [08:10:21] 06tools-platform-team, 07Epic: [logs-api,loki] Implement a system-level logging endpoint - https://phabricator.wikimedia.org/T432565 (10dcaro) 03NEW [08:11:42] 06tools-platform-team: [logs-api] Implement an endpoint to retrieve system-level event logs - https://phabricator.wikimedia.org/T432566 (10dcaro) 03NEW [08:12:12] 06tools-platform-team: [logs-api,loki] Implement a system-level logging endpoint - https://phabricator.wikimedia.org/T432565#12135753 (10dcaro) [08:12:23] 10Toolforge, 06tools-platform-team, 07Epic: [hypothesis] 5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564#12135755 (10dcaro) [08:12:32] 10Toolforge, 06tools-platform-team: [logs-api,loki] Implement a system-level logging endpoint - https://phabricator.wikimedia.org/T432565#12135756 (10dcaro) [08:12:46] 10Toolforge, 06tools-platform-team: [logs-api] Implement an endpoint to retrieve system-level event logs - https://phabricator.wikimedia.org/T432566#12135767 (10dcaro) [08:15:27] 10Toolforge, 06tools-platform-team: [jobs-api] Send webservice start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570 (10dcaro) 03NEW [08:19:05] 10Toolforge, 06tools-platform-team, 07Epic: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571 (10dcaro) 03NEW [08:22:28] 10Toolforge, 06tools-platform-team: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572 (10dcaro) 03NEW [08:26:33] 10Tool-wikinewsie: WikiPortraits mode for Wikinewsie - https://phabricator.wikimedia.org/T432573 (10Pharos) 03NEW [08:30:42] 10Tools, 06Commons, 07Community-Wishlist: Image Editor for Commons - https://phabricator.wikimedia.org/T426106#12135879 (10mikez-WMF) →14Duplicate dup:03T428062 [08:30:45] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12135881 (10mikez-WMF) [08:31:53] 10Tools, 06Commons, 07Community-Wishlist: Image Editor for Commons - https://phabricator.wikimedia.org/T426106#12135886 (10mikez-WMF) We decided to merge this into the other related wish! I'll post a summary of all the work that's been done there. [08:35:32] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12135893 (10mikez-WMF) @Bugreporter2 yes, just closed it - thank you for the reminder! I'll post a summary soon. [08:37:00] 10Toolforge, 06tools-infrastructure-team: [toolsbeta,infra,admin] toolsbeta-test-k8s-haproxy7 is flaky on ip6 when replying to admin - https://phabricator.wikimedia.org/T432338#12135895 (10fnegri) →14Duplicate dup:03T426584 [08:37:01] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsbeta] probe flapping on ipv6 only - https://phabricator.wikimedia.org/T426584#12135897 (10fnegri) [08:38:05] 10Tool-wikinewsie: Live pageviews for Wikinewsie - https://phabricator.wikimedia.org/T432576 (10Pharos) 03NEW [08:38:07] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [toolsbeta] probe flapping on ipv6 only - https://phabricator.wikimedia.org/T426584#12135901 (10fnegri) [08:54:47] (03merge) 10dcaro: pypi: check for properly set auth [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/89 [08:55:34] (03merge) 10dcaro: rotate_clinic_duty: added some helper docstrings [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/4 [08:57:18] 10Cloud-VPS, 06tools-infrastructure-team: clean up nova IDs for newer cloudvirts - https://phabricator.wikimedia.org/T430933#12135951 (10fgiunchedi) >>! In T430933#12132469, @Andrew wrote: > I think I'm behind on something. When you say... > >> should not be hardcoded in puppet anymore > > Are you talking ab... [09:09:52] (03open) 10dcaro: tox: add coverage report [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/29 [09:14:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:19:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:20:28] (03update) 10dcaro: tox: add coverage report [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/29 [09:20:48] (03update) 10dcaro: tox: add coverage report [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/29 [09:21:32] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D{cloudcontrol1006.eqiad.wmnet}' [09:23:51] (03approved) 10fnegri: tox: add coverage report [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/29 (owner: 10dcaro) [09:23:59] (03update) 10fnegri: tox: add coverage report [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/29 (owner: 10dcaro) [09:24:22] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:24:24] PROBLEM - SSH on cloudcontrol1006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:25:10] FIRING: GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [09:26:34] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D{cloudcontrol1006.eqiad.wmnet}' [09:27:16] RECOVERY - SSH on cloudcontrol1006 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:29:11] (03approved) 10dcaro: tox: add coverage report [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/29 [09:29:16] (03merge) 10dcaro: tox: add coverage report [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/29 [09:29:22] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:30:10] RESOLVED: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [09:32:16] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.30-20260720092928-a41419a8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1343 [09:34:22] RESOLVED: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:49:18] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D{cloudcontrol1007.eqiad.wmnet}' [09:50:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:50:46] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [09:51:22] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:51:50] PROBLEM - Host cloudcontrol1007 is DOWN: PING CRITICAL - Packet loss = 100% [09:52:10] FIRING: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [09:53:14] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D{cloudcontrol1007.eqiad.wmnet}' [09:53:18] RECOVERY - Host cloudcontrol1007 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [09:55:17] FIRING: [2x] JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:56:22] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:57:01] (03update) 10dcaro: Remove mention of SRE meeting delegate [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/5 (owner: 10taavi) [09:57:03] (03approved) 10dcaro: Remove mention of SRE meeting delegate [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/5 (owner: 10taavi) [09:57:10] RESOLVED: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [09:58:22] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D{cloudcontrol1011.eqiad.wmnet}' [09:58:57] 10Striker, 06tools-platform-team, 10Toolhub: "GO TO TOOL" targeting toolsadmin.wikimedia.org pages fails because of remote cross-origin-opener-policy header value - https://phabricator.wikimedia.org/T432262#12136109 (10DamianZaremba) >>! In T432262#12125900, @bd808 wrote: >>>! In T432262#12125843, @DamianZar... [09:59:14] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [10:00:17] RESOLVED: [2x] JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:00:18] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [jobs-api] use `launcher` also for health-check script commands - https://phabricator.wikimedia.org/T403735#12136125 (10DamianZaremba) I noticed this again on T430578, which simplifies the behaviour for the main command. I'm going to propose a simpl... [10:00:40] FIRING: GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [10:00:50] PROBLEM - Host cloudcontrol1011 is DOWN: PING CRITICAL - Packet loss = 100% [10:01:08] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [10:01:22] FIRING: [30x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [10:02:18] RECOVERY - Host cloudcontrol1011 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [10:02:18] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D{cloudcontrol1011.eqiad.wmnet}' [10:02:25] FIRING: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [10:02:50] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12136138 (10Doc_James) We could add blurring to this possibly https://commons.wikimedia.org/wiki/Commons:ImageAnnotateTool We already have an SVG translation... [10:05:40] RESOLVED: [3x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [10:06:22] FIRING: [30x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [10:08:15] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudlb.safe_reboot on hosts matched by 'A:cloudlb AND A:eqiad' [10:08:47] FIRING: [3x] JobUnavailable: Reduced availability for job cloudlb-haproxy in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:11:13] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [10:11:22] RESOLVED: [30x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [10:12:05] FIRING: [2x] HostBGPDown: BGP session for cloudlb1001 (172.20.1.2) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [10:12:24] (03approved) 10dcaro: logs-api: bump to 0.0.30-20260720092928-a41419a8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1343 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:12:32] (03merge) 10dcaro: logs-api: bump to 0.0.30-20260720092928-a41419a8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1343 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:12:54] (03merge) 10taavi: Remove mention of SRE meeting delegate [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/5 [10:13:26] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudlb.safe_reboot (exit_code=0) on hosts matched by 'A:cloudlb AND A:eqiad' [10:13:47] RESOLVED: [3x] JobUnavailable: Reduced availability for job cloudlb-haproxy in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:17:05] RESOLVED: [3x] HostBGPDown: BGP session for cloudlb1001 (172.20.1.2) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [10:18:46] (03open) 10damian: health check - use `launcher` for build service [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/356 (https://phabricator.wikimedia.org/T403735) [10:18:50] 10Cloud-VPS, 06tools-infrastructure-team, 06tools-platform-team: Make sure dumps-nfs mount/umount is propagated inside containers - https://phabricator.wikimedia.org/T432325#12136158 (10fgiunchedi) Good point @dcaro, ok thank you! So as far as Toolforge is concerned we can move to bind-mounting `/mnt/nfs` wi... [10:19:08] 10Cloud-VPS, 06tools-infrastructure-team, 06tools-platform-team: Make sure dumps-nfs mount/umount is propagated inside Toolforge containers - https://phabricator.wikimedia.org/T432325#12136161 (10fgiunchedi) [10:21:43] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] use `launcher` also for health-check script commands - https://phabricator.wikimedia.org/T403735#12136169 (10DamianZaremba) @dcaro could you tag with needs review when you have chance please [10:23:04] 10PAWS, 06tools-infrastructure-team, 06tools-platform-team: Make sure dumps-nfs mounts are propagated inside PAWS containers - https://phabricator.wikimedia.org/T432583 (10fgiunchedi) 03NEW [10:25:39] 10Tool-techcontribs: Detect Wikimedia developer account from SUL username - https://phabricator.wikimedia.org/T432584 (10Chlod) 03NEW [10:26:18] 10Tool-techcontribs: Detect Wikimedia developer account from SUL username - https://phabricator.wikimedia.org/T432584#12136201 (10Chlod) p:05Triage→03Medium a:03Chlod [10:28:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [10:31:28] (03update) 10dcaro: health check - use `launcher` for build service [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/356 (https://phabricator.wikimedia.org/T403735) (owner: 10damian) [10:31:32] (03update) 10dcaro: health check - use `launcher` for build service [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/356 (https://phabricator.wikimedia.org/T403735) (owner: 10damian) [10:31:51] 10Toolforge, 06tools-platform-team: [jobs-cli] tab completion is broken - https://phabricator.wikimedia.org/T432586 (10DamianZaremba) 03NEW [10:32:38] 10Cloud-VPS, 06tools-infrastructure-team: Migrate to dumps-nfs.w.o in Cloud VPS - https://phabricator.wikimedia.org/T432587 (10fgiunchedi) 03NEW [10:34:31] 10Toolforge, 06tools-platform-team: [jobs-cli] tab completion is broken - https://phabricator.wikimedia.org/T432586#12136252 (10taavi) `lang=shell-session tools.cluebotng@tools-bastion-14:~$ toolforge jobs --debug list -o name [cut] [2026-07-20 10:33:10] [cli.py] ERROR: TjfCliError: Unknown error (list index... [10:35:44] 10Toolforge, 06tools-platform-team: [jobs-cli] tab completion is broken - https://phabricator.wikimedia.org/T432586#12136253 (10DamianZaremba) Actually jobs-cli just seems generally broken; ` tools.cluebotng@tools-bastion-15:~$ toolforge jobs show bot ERROR: TjfCliError: Unknown error (list index out of range)... [10:36:29] 10Toolforge, 06tools-platform-team: [jobs-cli] tab completion is broken - https://phabricator.wikimedia.org/T432586#12136255 (10DamianZaremba) ` tools.cluebotng@tools-bastion-15:~$ curl --cert .toolskube/client.crt --key .toolskube/client.key -k 'https://api.svc.tools.eqiad1.wikimedia.cloud:30003/jobs/v1/tool/... [10:36:51] 10Toolforge, 06tools-platform-team: [jobs-cli] tab completion is broken - https://phabricator.wikimedia.org/T432586#12136256 (10taavi) `lines=10 ERROR: Exception in ASGI application Traceback (most recent call last): File "/opt/lib/venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", lin... [10:39:18] 10Toolforge, 06tools-platform-team: [jobs-cli] tab completion is broken - https://phabricator.wikimedia.org/T432586#12136261 (10DamianZaremba) >>! In T432586#12136256, @taavi wrote: > `lines=10 > ERROR: Exception in ASGI application > Traceback (most recent call last): > File "/opt/lib/venv/lib/python3.13... [10:47:59] (03update) 10damian: health check - use `launcher` for build service [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/356 (https://phabricator.wikimedia.org/T403735) [10:53:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:02:13] 10Toolforge, 06tools-platform-team: [jobs-api] crashes if k8s object does not match expected format (un-guarded list access) - https://phabricator.wikimedia.org/T432586#12136301 (10DamianZaremba) [11:06:19] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work: Add parameter examples to TransformHandler.php - https://phabricator.wikimedia.org/T432228#12136310 (10AGhirelli-WMF) 05Open→03In progress [11:08:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:16:17] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29737812391 (https://github.com/cluebotng/component-configs/commits/10014b40131fb0e2f27f7de26102f6689287add4) [11:16:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [11:19:10] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudservices.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::services}' [11:20:17] FIRING: JobUnavailable: Reduced availability for job pdns_rec in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:20:47] PROBLEM - Host cloudservices1005 is DOWN: PING CRITICAL - Packet loss = 100% [11:22:25] RECOVERY - Host cloudservices1005 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [11:23:05] FIRING: [2x] HostBGPDown: BGP session for cloudservices1005 (172.20.2.4) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [11:23:21] PROBLEM - Host cloudservices1006 is DOWN: PING CRITICAL - Packet loss = 100% [11:25:17] FIRING: [2x] JobUnavailable: Reduced availability for job pdns in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:25:23] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudservices.safe_reboot (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::eqiad1::services}' [11:25:25] RECOVERY - Host cloudservices1006 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [11:28:05] RESOLVED: [4x] HostBGPDown: BGP session for cloudservices1005 (172.20.2.4) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [11:28:45] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [11:30:17] RESOLVED: [2x] JobUnavailable: Reduced availability for job pdns in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:30:20] 06tools-infrastructure-team, 06Data-Platform-SRE: Migrate to dumps-nfs.w.o in production - https://phabricator.wikimedia.org/T432212#12136362 (10fgiunchedi) [11:31:58] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29738720465 (https://github.com/cluebotng/component-configs/commits/661b4bae545b221090aa694fa30ff924415a5b22) [11:32:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [11:34:58] (03open) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in a 10-minute in-memory TTL cache [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59 [11:49:33] 10Cloud-VPS, 06tools-infrastructure-team: Migrate to dumps-nfs.w.o in Cloud VPS - https://phabricator.wikimedia.org/T432587#12136418 (10fgiunchedi) [11:51:41] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Upgrade Toolforge (Elastic|Open)Search cluster off of Bullseye - https://phabricator.wikimedia.org/T401818#12136431 (10taavi) a:03taavi [11:52:02] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.roll_reboot_cloudnets [11:54:38] PROBLEM - Host cloudnet1006 is DOWN: PING CRITICAL - Packet loss = 100% [11:54:52] (03update) 10taavi: Cleanup done imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/119 [11:54:54] (03update) 10taavi: tools: Provision Opensearch VMs [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/121 (https://phabricator.wikimedia.org/T401818) [11:54:55] (03update) 10taavi: tools: Add scaffolding for new OpenSearch service [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/120 (https://phabricator.wikimedia.org/T401818) [11:54:57] (03update) 10taavi: tools: Sort .svc. DNS entries [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/122 [11:54:59] (03update) 10taavi: tools: Add Opensearch svc DNS entries [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/123 (https://phabricator.wikimedia.org/T401818) [11:55:27] (03open) 10taavi: tools: Sort .svc. DNS entries [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/122 [11:55:28] (03open) 10taavi: Cleanup done imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/119 [11:55:28] (03open) 10taavi: tools: Provision Opensearch VMs [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/121 (https://phabricator.wikimedia.org/T401818) [11:55:29] (03open) 10taavi: tools: Add scaffolding for new OpenSearch service [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/120 (https://phabricator.wikimedia.org/T401818) [11:55:30] (03open) 10taavi: tools: Add Opensearch svc DNS entries [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/123 (https://phabricator.wikimedia.org/T401818) [11:55:31] (03update) 10countcount: Cache small wikis' /ipinfo phase-1 IP rows in a 10-minute in-memory TTL cache [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/59 [11:56:01] 10Toolforge, 06tools-platform-team: [jobs-api,jobs-cli] `toolforge jobs load` uses PATCH for one-off jobs too - https://phabricator.wikimedia.org/T432592 (10dcaro) 03NEW [11:57:25] RECOVERY - Host cloudnet1006 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [11:58:01] (03update) 10taavi: tools: Add Opensearch svc DNS entries [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/123 (https://phabricator.wikimedia.org/T401818) [11:58:04] (03update) 10taavi: tools: Sort .svc. DNS entries [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/122 [11:58:05] (03update) 10taavi: tools: Provision Opensearch VMs [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/121 (https://phabricator.wikimedia.org/T401818) [11:58:07] (03update) 10taavi: tools: Add scaffolding for new OpenSearch service [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/120 (https://phabricator.wikimedia.org/T401818) [12:17:28] FIRING: KeyholderUnarmed: 2 unarmed Keyholder key(s) on cloudcumin2001:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [12:18:49] (03merge) 10krinkle: channels: Remove wikimedia-infrastructure [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/75 [12:21:52] (03open) 10krinkle: mediawiki-parsoid: remove sbailey [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/36 [12:22:28] RESOLVED: KeyholderUnarmed: 2 unarmed Keyholder key(s) on cloudcumin2001:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [12:22:40] !log taavi@cloudcumin2001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-1.tools.eqiad1.wikimedia.cloud [12:22:55] 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12136526 (10klausman) [12:23:29] 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12136528 (10klausman) >>! In T431826#12131542, @elukey wrote: > Hey folks! IIUC this task is to track both Bo... [12:24:07] (03open) 10dcaro: core: guard against runtime issues [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/357 [12:24:26] !log taavi@cloudcumin2001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-1.tools.eqiad1.wikimedia.cloud [12:24:29] !log taavi@cloudcumin2001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-2.tools.eqiad1.wikimedia.cloud [12:26:11] !log taavi@cloudcumin2001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-2.tools.eqiad1.wikimedia.cloud [12:26:48] !log taavi@cloudcumin2001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-3.tools.eqiad1.wikimedia.cloud [12:28:35] !log taavi@cloudcumin2001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-3.tools.eqiad1.wikimedia.cloud [12:29:44] (03merge) 10krinkle: mediawiki-parsoid: remove sbailey [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/36 [12:30:34] (03update) 10dcaro: core: guard against runtime issues [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/357 [12:34:33] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] Refactor k8s object creation to have per-job-type functions - https://phabricator.wikimedia.org/T430071#12136558 (10dcaro) 05Open→03Resolved [12:35:17] (03open) 10krinkle: gerrit-channels: Remove wikimedia-services [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/76 [12:39:38] (03merge) 10krinkle: gerrit-channels: Remove wikimedia-services [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/76 [12:40:45] 10Toolforge, 06tools-platform-team: Request tool account: football-kit-sync - https://phabricator.wikimedia.org/T432450#12136572 (10dcaro) 05Invalid→03Resolved [12:41:13] (03open) 10dcaro: ci: add coverage [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/161 [12:49:46] 10Toolforge, 06tools-platform-team: Request tool account: football-kit-sync - https://phabricator.wikimedia.org/T432450#12136585 (10dcaro) 05Resolved→03Invalid [12:52:16] (03open) 10dcaro: cli: use create_job when updating a one-off [repos/cloud/toolforge/jobs-cli] (enable_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/162 [12:53:04] (03update) 10dcaro: ci: add coverage [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/161 [12:53:09] (03update) 10dcaro: ci: add coverage [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/161 [12:58:26] (03open) 10dcaro: tox: enable coverage in all the newer versions [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/91 [12:58:56] (03update) 10dcaro: tox: enable coverage in all the newer versions [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/91 [13:02:48] (03approved) 10taavi: tox: enable coverage in all the newer versions [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/91 (owner: 10dcaro) [13:04:09] (03merge) 10dcaro: tox: enable coverage in all the newer versions [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/91 [13:06:57] (03update) 10dcaro: ci: add coverage [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/161 [13:06:59] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29744681881 (https://github.com/cluebotng/component-configs/commits/8b5c46ef715a2e57f383d35d7acc37bfbfa4db2d) [13:07:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [13:07:55] 10Toolforge, 06tools-infrastructure-team, 13Patch-For-Review: Upgrade Toolforge (Elastic|Open)Search cluster off of Bullseye - https://phabricator.wikimedia.org/T401818#12136642 (10aputhin) [13:08:23] (03update) 10dcaro: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) (owner: 10raymond-ndibe) [13:10:18] (03update) 10fnegri: tox: enable coverage in all the newer versions [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/91 (owner: 10dcaro) [13:10:49] (03update) 10fnegri: tox: enable coverage in all the newer versions [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/91 (owner: 10dcaro) [13:16:55] (03update) 10dcaro: cli: use create_job when updating a one-off [repos/cloud/toolforge/jobs-cli] (enable_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/162 [13:17:23] 10Toolforge, 06tools-platform-team: [jobs-api,jobs-cli] `toolforge jobs load` uses PATCH for one-off jobs too - https://phabricator.wikimedia.org/T432592#12136671 (10dcaro) 05Open→03In progress p:05Triage→03High a:03dcaro [13:17:48] 10Toolforge, 06tools-platform-team: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12136675 (10dcaro) [13:21:38] (03update) 10fnegri: tox: enable coverage in all the newer versions [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/91 (owner: 10dcaro) [13:43:49] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Rebalance WMCS instance backups between the two hosts - https://phabricator.wikimedia.org/T430018#12136745 (10Andrew) I made another pass at rebalancing but 1004 is still taking a bit longer than 24 hours (24 hours and 6 minutes) to complete so th... [14:03:39] FIRING: CloudVPSBastionlessUsers: Detected 1 cloud-vps users not in Bastion project - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Users_not_in_bastion_project - https://logstash.wikimedia.org/app/dashboards#/view/3ef008b0-c871-11eb-ad54-8bb5fcb640c0?_g=h@e78830b&_a=h@21715bf - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSBastionlessUsers [14:11:16] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29749326972 (https://github.com/cluebotng/component-configs/commits/83d05d6d10640a88d80bdbc041955ca27603a0fe) [14:11:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [14:25:40] 10Cloud-VPS, 06tools-infrastructure-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#12137003 (10Andrew) > Our tool cannot run in Toolforge Can you tell us more about this? The default compute quotas for toolforge are a bit low but as long as your service scales ho... [14:26:18] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150#12137004 (10MusikAnimal) >>! In T432150#12132449, @taavi wrote: > What is the full double disk space needed for? Cinder volumes can be moved between VMs, one... [14:37:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:47:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:55:09] RESOLVED: CloudVPSBastionlessUsers: Detected 1 cloud-vps users not in Bastion project - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Users_not_in_bastion_project - https://logstash.wikimedia.org/app/dashboards#/view/3ef008b0-c871-11eb-ad54-8bb5fcb640c0?_g=h@e78830b&_a=h@21715bf - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSBastionlessUsers [15:04:07] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29753268267 (https://github.com/cluebotng/component-configs/commits/3cd242ed54e9146f6a3dfd9305f97d4580c2fc06) [15:04:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [15:22:20] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408#12137304 (10dcaro) 05Stalled→03In progress [15:23:54] (03update) 10dcaro: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) (owner: 10raymond-ndibe) [15:30:31] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12137389 (10LSobanski) a:03Arnoldokoth [15:32:59] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150#12137407 (10Andrew) > The stated quota increase is to make room for an identical VM to the main one we use now, which is of special flavour g4.cores24.ram122... [15:37:38] 10Cloud-VPS, 06tools-infrastructure-team: Improve purging of invalid backups in wmcs-backup - https://phabricator.wikimedia.org/T432610 (10Andrew) 03NEW [15:39:26] 10Cloud-Services, 06tools-platform-team, 10Diffusion: Create Conduit API method to lookup Phabricator accounts by MediaWiki user name - https://phabricator.wikimedia.org/T137001#12137464 (10valerio.bozzolan) (Thanks again) Surprisingly but understandably the API does not normalize underscores. So, "Valer... [15:41:42] 10Cloud-Services, 06tools-platform-team, 10Diffusion: Create Conduit API method to lookup Phabricator accounts by MediaWiki user name - https://phabricator.wikimedia.org/T137001#12137483 (10Aklapper) >>! In T137001#12137464, @valerio.bozzolan wrote: > Is taking care of this normalization a welcomed sub-t... [15:42:15] 10Cloud-Services, 10Diffusion: Create Conduit API method to lookup Phabricator accounts by MediaWiki user name - https://phabricator.wikimedia.org/T137001#12137502 (10taavi) [15:46:40] (03update) 10andrew: Add shared magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [15:52:56] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150#12137565 (10MusikAnimal) [15:53:07] (03approved) 10taavi: Add shared magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) (owner: 10andrew) [15:54:03] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150#12137566 (10MusikAnimal) [15:56:09] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150#12137582 (10MusikAnimal) >>! In T432150#12137407, @Andrew wrote: > Just to double-check... your initial request includes a request for another 14TB of volume... [15:57:04] 10Cloud-VPS, 06tools-infrastructure-team: clean up nova IDs for newer cloudvirts - https://phabricator.wikimedia.org/T430933#12137585 (10Andrew) ok, I'm caught up now! I hadn't realized that we had reproducible uuids for new hosts, I was still just taking nova's word for it :( [16:02:02] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150#12137611 (10MusikAnimal) I'll note we have really only have this week to work on the migration (assuming July 31 is a hard deadline), so I'm hoping this can... [16:03:41] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150#12137628 (10Andrew) +1 lgtm [16:04:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:09:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:16:51] 10Tool-documentation, 10Adiutor, 06Wiki-Mentor-Africa, 07good first task: [Documentation] Update and improve docs for Adiutor tool - https://phabricator.wikimedia.org/T390392#12137694 (10TBurmeister) I did notice there's a link to a Telegram channel on the page, you could try posting a message there? https... [16:24:37] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 07Incident Severity 3, and 2 others: 2026-07-11 Incident - Toolforge haproxies flapping - https://phabricator.wikimedia.org/T431946#12137717 (10LSobanski) [16:27:15] 10Cloud-VPS (Quota-requests): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617 (10jeena) 03NEW [16:34:45] 06cloud-services-team, 10Quarry, 06tools-platform-team, 06RoadToWiki, and 3 others: "1 rows" should be "1 row" - https://phabricator.wikimedia.org/T419564#12137803 (10Framawiki) Where did you push the commit that rolled-back \r changes? On https://github.com/toolforge/quarry/pull/98/commits I still see onl... [16:54:59] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [17:02:21] 10Toolforge, 06tools-platform-team, 07Epic: [hypothesis] ST5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564#12137983 (10fnegri) [17:05:45] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150#12138003 (10BLiviero-WMF) +1. Will you open a ticket to release the quota once migration is complete? [17:05:45] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [17:05:46] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [17:12:42] (03update) 10dcaro: kubernetes.py: detect conflict with httproutes created by jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/114 (https://phabricator.wikimedia.org/T431417) (owner: 10raymond-ndibe) [17:29:28] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [17:33:18] !log andrew@cloudcumin1001 wikiwho START - Cookbook wmcs.openstack.quota_increase by 23 cores, 122880 ram [17:33:26] !log andrew@cloudcumin1001 wikiwho END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 23 cores, 122880 ram [17:34:05] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150#12138096 (10Andrew) 05Open→03Resolved a:03Andrew I have granted the quota increase. Do please create a subticket about reverting, or follow up here... [17:42:41] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Quota increase request for project wikiwho - https://phabricator.wikimedia.org/T432150#12138136 (10MusikAnimal) Will do, thanks! [17:42:55] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [18:15:40] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Improve purging of invalid backups in wmcs-backup - https://phabricator.wikimedia.org/T432610#12138271 (10Andrew) 05Open→03Resolved [18:26:22] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: New thermal paste for cloudvirt1071.eqiad.wmnet - https://phabricator.wikimedia.org/T431429#12138313 (10VRiley-WMF) a:05Andrew→03VRiley-WMF [18:41:50] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [infra] Upgrade Toolforge K8s etcd nodes to Trixie - https://phabricator.wikimedia.org/T432634 (10Andrew) 03NEW [20:14:56] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [20:15:06] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/8 [20:16:53] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [20:17:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/43 [20:17:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [20:17:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [20:19:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [20:19:02] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [20:20:12] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/35 [20:20:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/35 [20:20:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/20 [20:20:47] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/20 [20:21:02] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/23 [20:21:04] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/23 [20:21:16] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/26 [20:21:18] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/26 [20:32:14] (03close) 10andrew: codfw1dev: Add two projects [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/284 (https://phabricator.wikimedia.org/T410659) [20:34:40] (03merge) 10andrew: Add shared magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [20:34:44] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [20:35:32] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [20:45:12] (03open) 10h2o: Update README [toolforge-repos/leximap] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/leximap/-/merge_requests/1 [20:46:32] (03merge) 10h2o: Update README [toolforge-repos/leximap] (dev) - 10https://gitlab.wikimedia.org/toolforge-repos/leximap/-/merge_requests/1 [20:51:04] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [20:51:36] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [21:02:02] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12138898 (10jsn.sherman) [21:12:25] (03open) 10andrew: codfw1dev magnum templates: use same ubuntu/k8s version as eqiad1 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/334 [21:12:52] (03merge) 10andrew: codfw1dev magnum templates: use same ubuntu/k8s version as eqiad1 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/334 [21:12:53] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [21:13:25] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [21:13:54] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12138931 (10jeena) [21:14:40] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [21:14:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/8 [21:15:33] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [21:15:33] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/43 [21:15:50] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [21:16:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [21:16:43] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/35 [21:16:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/20 [21:17:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/23 [21:17:07] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/26 [21:25:04] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [21:25:30] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [21:26:47] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [21:27:15] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [21:27:18] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [21:27:37] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [21:36:51] 10Cloud-VPS (Quota-requests), 10Catalyst (Luka Ijo Pimeja Jan): Quota increase request for project catalyst - https://phabricator.wikimedia.org/T432617#12139023 (10bd808) [22:34:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:39:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown