[01:42:56] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [01:46:14] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [02:02:00] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [02:06:55] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [02:07:06] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [02:45:22] (03update) 10raymond-ndibe: Draft: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [02:45:22] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [03:22:41] FIRING: [2x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:02:25] FIRING: [2x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:12:25] FIRING: [2x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:26:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:31:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:48:09] 10Toolforge (Quota-requests): Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12157514 (10Leaderboard) [06:26:27] (03update) 10countcount: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 (owner: 10renovatebot) [06:27:02] (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/20 (owner: 10renovatebot) [06:27:14] (03merge) 10countcount: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/23 (owner: 10renovatebot) [06:27:18] (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/26 (owner: 10renovatebot) [06:27:40] (03merge) 10countcount: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/43 (owner: 10renovatebot) [06:27:46] (03merge) 10countcount: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/8 (owner: 10renovatebot) [06:28:14] (03merge) 10countcount: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 (owner: 10renovatebot) [06:34:09] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12157532 (10Marostegui) clouddb1032 pooled: s4 and s6 [06:34:20] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12157533 (10Marostegui) [06:41:07] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12157536 (10Doc_James) We have added that ability to blur or pixilate part of the image with the image anotation tool. But yes we do need to organize these bett... [06:44:22] FIRING: [2x] HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-s5 backend clouddb1020.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [06:52:55] (03open) 10countcount: Update vulnerable transitive dependencies [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/13 [06:53:40] (03open) 10countcount: Update vulnerable transitive dependencies [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/45 [06:55:05] (03open) 10countcount: Resolve vulnerable Node.js transitive dependencies [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/26 [06:57:23] 06cloud-services-team, 10Horizon, 06tools-infrastructure-team: Horizon IDP/SSO logout doesn't really logout - https://phabricator.wikimedia.org/T374123#12157581 (10SLyngshede-WMF) This is also very implementation specific. We did look at the IDM flow a few weeks ago, to start fixing this issue. In that case... [06:59:00] (03merge) 10countcount: Update vulnerable transitive dependencies [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/13 [07:07:26] (03update) 10countcount: Update vulnerable transitive dependencies [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/45 [07:07:39] (03update) 10countcount: Update vulnerable transitive dependencies [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/45 [07:10:14] (03update) 10countcount: Resolve vulnerable Node.js transitive dependencies [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/26 [07:10:33] (03update) 10countcount: Resolve vulnerable Node.js transitive dependencies [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/26 [07:14:32] (03update) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [07:15:12] (03merge) 10countcount: Update vulnerable transitive dependencies [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/45 [07:15:31] (03merge) 10countcount: Resolve vulnerable Node.js transitive dependencies [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/26 [07:15:38] (03update) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [07:16:36] (03update) 10renovatebot: Update Rust crate serial_test to v4 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/61 [07:17:54] (03update) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [07:18:07] (03update) 10renovatebot: Update dependency jsdom to v30 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/27 [07:18:12] (03open) 10renovatebot: Update dependency jsdom to v30 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/27 [07:18:17] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [07:18:19] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [07:19:12] (03update) 10renovatebot: Update python Docker tag to v3.14.6 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [07:22:26] FIRING: [5x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:24:22] FIRING: [2x] HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-s5 backend clouddb1020.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [07:47:26] FIRING: [5x] SystemdUnitFailed: check-private-data.service on clouddb1026:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:02:22] (03open) 10dcaro: clinic_duty: add pinging script with dcaro only [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/6 [08:05:28] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api,loki] Implement a system-level logging endpoint (write) - https://phabricator.wikimedia.org/T432565#12157685 (10dcaro) [08:06:14] 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12157687 (10dcaro) [08:11:29] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [08:15:30] (03update) 10renovatebot: Update dependency jsdom to v30 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/27 [08:15:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [08:36:52] RESOLVED: HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-s8 backend clouddb1020.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [08:38:47] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12157747 (10Marostegui) [08:59:25] 10Toolforge, 06tools-platform-team, 10Data-Services (Quota-requests): Trove-only project request: Postgres database for Toolforge tool - omd - https://phabricator.wikimedia.org/T432818#12157799 (10fnegri) +1 [09:00:39] 10Toolforge, 06tools-platform-team, 10Data-Services (Quota-requests): Trove-only project request: Postgres database for Toolforge tool - omd - https://phabricator.wikimedia.org/T432818#12157801 (10dcaro) a:03dcaro [09:03:59] 10Cloud-VPS (Project-requests): Request creation of PanoCommons VPS project - https://phabricator.wikimedia.org/T433055#12157807 (10dcaro) Hi @Seb35, by the description of the project, it does not seem to be contributing anything back to commons or wikis in general, can you elaborate on how does this specific pr... [09:09:42] 10Toolforge (Quota-requests): Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12157828 (10dcaro) Hi @Leaderboard, currently 16G is the biggest of all the k8s workers we have, so given that no worker is idle, that would mean that your pods will probably never ha... [09:22:29] (03merge) 10filippo: server: default to HostToContainer propagated mounts [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/51 (https://phabricator.wikimedia.org/T432325) [09:28:08] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: volume-admission: bump to 0.0.87-20260727092249-b2d76db8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1348 (https://phabricator.wikimedia.org/T432325) [09:29:35] !log filippo@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission (T432325) [09:29:40] T432325: Make sure dumps-nfs mount/umount is propagated inside Toolforge containers - https://phabricator.wikimedia.org/T432325 [09:32:29] !log filippo@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission (T432325) [09:32:57] 10Cloud-VPS (Project-requests): Request creation of PanoCommons VPS project - https://phabricator.wikimedia.org/T433055#12157888 (10PanierAvide) Hi, thanks for your feedback. It doesn't help (yet !) contributing to Commons in terms of data. What we are looking for is making easier to reuse the Commons imagery in... [09:44:42] (03open) 10countcount: Add requested global lock history output [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/62 [09:47:29] (03update) 10dcaro: clinic_duty: add pinging script with dcaro only [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/6 [09:48:17] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:49:50] (03update) 10dcaro: clinic_duty: add pinging script with dcaro only [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/6 [09:50:31] (03open) 10countcount: Add API index page at service root [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/63 [09:52:14] (03merge) 10countcount: Add requested global lock history output [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/62 [09:53:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:53:17] (03update) 10dcaro: clinic_duty: add pinging script with dcaro only [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/6 [09:56:08] (03open) 10countcount: List global block exceptions [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64 [09:56:56] (03update) 10countcount: Add API index page at service root [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/63 [10:00:35] (03merge) 10countcount: Add API index page at service root [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/63 [10:01:20] (03open) 10countcount: Show global lock history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/11 [10:01:57] (03update) 10countcount: List global block exceptions [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64 [10:02:51] (03update) 10countcount: Show global lock history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/11 [10:02:54] (03update) 10countcount: Show global lock history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/11 [10:04:08] 10Toolforge (Quota-requests): Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12157991 (10Leaderboard) If we try 8 - 10 GB instead, (from your experience) is it still likely to be difficult to get the job scheduled? [10:05:16] (03update) 10countcount: Show global lock history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/11 [10:10:24] !log filippo@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers (T432325) [10:10:29] T432325: Make sure dumps-nfs mount/umount is propagated inside Toolforge containers - https://phabricator.wikimedia.org/T432325 [10:12:10] (03open) 10countcount: Replace serve to resolve security advisories [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/12 [10:12:35] (03update) 10countcount: Show global lock history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/11 [10:12:40] (03update) 10countcount: Show global lock history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/11 [10:14:46] (03merge) 10countcount: Replace serve to resolve security advisories [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/12 [10:16:05] (03update) 10renovatebot: Update Rust crate serial_test to v4 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/61 [10:17:41] (03update) 10countcount: Show global lock history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/11 [10:17:55] (03update) 10countcount: Show global lock history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/11 [10:20:39] (03open) 10countcount: Add AbchyZa22 block output integration test [toolforge-repos/multiuserinfo] (global-block-exceptions) - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/65 [10:22:04] (03merge) 10countcount: Show global lock history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/11 [10:30:18] !log filippo@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers (T432325) [10:30:24] T432325: Make sure dumps-nfs mount/umount is propagated inside Toolforge containers - https://phabricator.wikimedia.org/T432325 [10:36:14] (03merge) 10countcount: Add AbchyZa22 block output integration test [toolforge-repos/multiuserinfo] (global-block-exceptions) - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/65 [10:36:16] (03update) 10countcount: List global block exceptions [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/64 [10:58:43] (03update) 10hnowlan: Add API endpoint [toolforge-repos/template-radar] - 10https://gitlab.wikimedia.org/toolforge-repos/template-radar/-/merge_requests/1 (https://phabricator.wikimedia.org/T432270) [11:02:23] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [11:04:45] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12158228 (10Marostegui) clouddb1033 replicating s5 and s8 [11:06:27] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [11:06:50] (03open) 10dcaro: fix clinic duty carry [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/7 [11:08:14] (03update) 10dcaro: fix clinic duty carry [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/7 [11:11:21] (03open) 10dcaro: ci: add pre-commit and gitlab ci [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/8 [11:12:05] (03update) 10dcaro: fix clinic duty carry [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/7 [11:12:09] (03update) 10dcaro: fix clinic duty carry [repos/cloud/wmcs/utils] (add_precomitt) - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/7 [11:12:17] (03update) 10dcaro: fix clinic duty carry [repos/cloud/wmcs/utils] (add_precomitt) - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/7 [11:12:40] (03update) 10dcaro: rotate_clinic_duty: fix clinic duty section name [repos/cloud/wmcs/utils] (add_precomitt) - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/7 [11:13:11] (03update) 10dcaro: ci: add pre-commit and gitlab ci [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/8 [11:13:14] (03update) 10dcaro: rotate_clinic_duty: fix clinic duty section name [repos/cloud/wmcs/utils] (add_precomitt) - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/7 [11:14:00] (03update) 10dcaro: clinic_duty: add pinging script with dcaro only [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/6 [11:14:03] (03update) 10dcaro: clinic_duty: add pinging script with dcaro only [repos/cloud/wmcs/utils] (add_precomitt) - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/6 [11:26:45] (03open) 10kbach: Proposal: UI changes to linting results [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/27 [11:32:24] (03approved) 10filippo: rotate_clinic_duty: fix clinic duty section name [repos/cloud/wmcs/utils] (add_precomitt) - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/7 (owner: 10dcaro) [11:33:30] (03approved) 10filippo: ci: add pre-commit and gitlab ci [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/8 (owner: 10dcaro) [11:34:45] (03merge) 10dcaro: ci: add pre-commit and gitlab ci [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/8 [11:34:52] (03update) 10dcaro: clinic_duty: add pinging script with dcaro only [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/6 [11:34:56] (03update) 10dcaro: rotate_clinic_duty: fix clinic duty section name [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/7 [11:36:24] (03merge) 10dcaro: rotate_clinic_duty: fix clinic duty section name [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/7 [11:37:00] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission (T432325) [11:37:05] T432325: Make sure dumps-nfs mount/umount is propagated inside Toolforge containers - https://phabricator.wikimedia.org/T432325 [11:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:39:58] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission (T432325) [11:41:09] (03merge) 10filippo: volume-admission: bump to 0.0.87-20260727092249-b2d76db8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1348 (https://phabricator.wikimedia.org/T432325) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:43:20] 10Toolforge (Quota-requests): Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12158309 (10dcaro) >>! In T433184#12157991, @Leaderboard wrote: > If we try 8 - 10 GB instead, (from your experience) is it still likely to be difficult to get the job scheduled? Jus... [11:45:06] !log dcaro@cloudcumin1001 omd START - Cookbook wmcs.vps.create_project for trove-only project omd in eqiad1 (T432818) [11:45:07] dcaro@cloudcumin1001: Unknown project "omd" [11:45:07] dcaro@cloudcumin1001: Did you mean to say "tools.omd" instead? [11:45:08] T432818: Trove-only project request: Postgres database for Toolforge tool - omd - https://phabricator.wikimedia.org/T432818 [11:45:53] (03update) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project omd [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/340 (https://phabricator.wikimedia.org/T432818) [11:45:57] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project omd [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/340 (https://phabricator.wikimedia.org/T432818) [11:47:12] (03approved) 10dcaro: projects: added project omd [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/340 (https://phabricator.wikimedia.org/T432818) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [11:47:22] (03merge) 10dcaro: projects: added project omd [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/340 (https://phabricator.wikimedia.org/T432818) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [11:48:21] !log dcaro@cloudcumin1001 omd END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for trove-only project omd in eqiad1 (T432818) [11:48:21] dcaro@cloudcumin1001: Unknown project "omd" [11:48:21] dcaro@cloudcumin1001: Did you mean to say "tools.omd" instead? [11:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:49:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [11:52:47] 10Toolforge, 06tools-platform-team, 10Data-Services (Quota-requests), 13Patch-For-Review: Trove-only project request: Postgres database for Toolforge tool - omd - https://phabricator.wikimedia.org/T432818#12158347 (10dcaro) 05Open→03Resolved Done! @psubhashis1 you should have now access to the proj... [11:54:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [12:06:02] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers (T432325) [12:06:09] T432325: Make sure dumps-nfs mount/umount is propagated inside Toolforge containers - https://phabricator.wikimedia.org/T432325 [12:06:57] (03update) 10dcaro: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) (owner: 10raymond-ndibe) [12:09:36] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [12:14:26] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Migrate metricsinfra project off of Bullseye - https://phabricator.wikimedia.org/T401813#12158441 (10taavi) 05Open→03Resolved [12:20:32] 10Cloud-VPS, 06tools-infrastructure-team: mediawiki2latex instance crashed in project collection-alt-renderer - https://phabricator.wikimedia.org/T433244#12158450 (10fnegri) [12:22:56] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:23:50] (03update) 10dcaro: clinic_duty: add pinging script with dcaro only [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/6 [12:27:56] RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:31:43] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12158484 (10fnegri) History length is continuing to increase: {F96115202} @magnusmanske there are too many long queries coming from mixnmatch (see list... [12:42:38] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: Investigate systemdunitfailed alerts for clouddb hosts - https://phabricator.wikimedia.org/T432745#12158532 (10fnegri) Similarly to `wmf-pt-kill@x3.service`, on `clouddb1029` I also manually deleted the following failing units: * `wmf_a... [12:42:39] 10Cloud-VPS (Debian Bullseye Deprecation), 10wikimaps-warper: Migrate wikimaps warper from Debian Bullseye to Trixie - https://phabricator.wikimedia.org/T431805#12158533 (10Chippyy) * pushed code fixes * created maps-warper5 instance next: Configure maps-warper5 instance [12:49:21] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: Investigate systemdunitfailed alerts for clouddb hosts - https://phabricator.wikimedia.org/T432745#12158546 (10fnegri) On the newly-setup `clouddb1033`, I disabled the failing unit `prometheus-mysqld-exporter.service`: `lang=shell-sess... [12:58:15] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: Investigate systemdunitfailed alerts for clouddb hosts - https://phabricator.wikimedia.org/T432745#12158573 (10fnegri) 05Resolved→03In progress There's one remaining alert for `check-private-data.service on clouddb1026:9100`. I tried r... [13:02:33] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12158582 (10Arnoldokoth) I re-created the `phabricator.wmcloud.org` proxy to point to the new instance but still doesn't work despite changing `phabr... [13:03:52] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12158585 (10fnegri) > The new patch https://gerrit.wikimedia.org/r/1315819 fixed the issue in cloudd... [13:10:46] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: Investigate systemdunitfailed alerts for clouddb hosts - https://phabricator.wikimedia.org/T432745#12158622 (10Marostegui) I can be deleted, clouddb1026 only holds s1. [13:11:07] (03PS1) 10Majavah: add fake opensearch passwords [labs/private] - 10https://gerrit.wikimedia.org/r/1318131 [13:13:36] (03CR) 10Majavah: [V:03+2 C:03+2] add fake opensearch passwords [labs/private] - 10https://gerrit.wikimedia.org/r/1318131 (owner: 10Majavah) [13:21:51] (03update) 10tlepage: Draft: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) [13:29:07] 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team: Investigate systemdunitfailed alerts for clouddb hosts - https://phabricator.wikimedia.org/T432745#12158668 (10fnegri) > I can be deleted, clouddb1026 only holds s1. It can not be easily deleted, the mapping in Puppet must be modified ([ch... [13:33:30] (03update) 10dcaro: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) (owner: 10raymond-ndibe) [13:38:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:53:03] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance tools-puppetserver-01 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [13:55:25] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12158763 (10Andrew) a:03Andrew [14:01:19] 10Cloud-VPS, 06tools-infrastructure-team: mediawiki2latex instance crashed in project collection-alt-renderer - https://phabricator.wikimedia.org/T433244#12158788 (10dcaro) The machine seems to be hitting the memory limit quite often: ` root@mediawiki2latex:~# dmesg -T | grep 'Out of memory' [Mon Jul 27 12:18:... [14:05:42] 10Cloud-VPS, 06tools-infrastructure-team: mediawiki2latex instance crashed in project collection-alt-renderer - https://phabricator.wikimedia.org/T433244#12158813 (10dhunniger) Hi dcaro, you are very likely right. The memory consumption seems to be the problem. I changed the table rendering library to tabular... [14:08:25] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [14:11:02] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [14:11:09] 10Cloud-VPS, 06tools-infrastructure-team: mediawiki2latex instance crashed in project collection-alt-renderer - https://phabricator.wikimedia.org/T433244#12158850 (10dcaro) You can check also the graphs here without having to ssh: https://grafana-rw.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?orgId=1&from... [14:19:01] 10Cloud-VPS, 06tools-infrastructure-team: mediawiki2latex instance crashed in project collection-alt-renderer - https://phabricator.wikimedia.org/T433244#12158951 (10dhunniger) Thanks a lot for the quick help. I got an idea how to reduce memory usage. I can likely set the table column pre-compilation from mult... [14:29:56] 10Cloud-VPS, 06tools-infrastructure-team: mediawiki2latex instance crashed in project collection-alt-renderer - https://phabricator.wikimedia.org/T433244#12159003 (10dcaro) 05Open→03Resolved a:03dcaro >>! In T433244#12158951, @dhunniger wrote: > Thanks a lot for the quick help. I got an idea how to r... [14:31:05] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#12159012 (10Osuji_pius) **Weekly Internship Report** **Week 5: June 23 – June 28** **Task Progress** - Finalized the remaining frontend implementation for the redesigned CampWiz... [14:36:17] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#12159017 (10Osuji_pius) **Weekly Internship Report** **Week 6: June 30 – July 5** **Task Progress** - PR remains open and awaiting review feedback from mentors. - While waitin... [14:42:24] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#12159040 (10Osuji_pius) **Weekly Internship Report** **Week 7: July 7 – July 12** **Task Progress** - The submitted pull request is still pending review; no new feedback has com... [14:59:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [15:38:19] 06cloud-services-team, 06collaboration-services, 10GitLab (Auth & Access), 06Release-Engineering-Team (Priority Backlog 📥): Consider enforcing 2FA for GitLab users with maintainer/owner permissions - https://phabricator.wikimedia.org/T347295#12159372 (10LSobanski) @brennen This came up in our backlog groom... [15:47:44] 10Toolforge (Quota-requests): Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12159464 (10Leaderboard) Yes that'd be helpful. > k8s side allocates with half of what you request So does it mean that it starts with 4 GB and dynamically grows as the application... [15:52:45] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.roll_reboot_osds [15:54:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [15:55:22] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12159509 (10Dzahn) Can you confirm that the Hiera change actually arrives in the Phabricator config itself in the file system? What error do you get? [15:55:27] PROBLEM - Host cloudcephosd1016 is DOWN: PING CRITICAL - Packet loss = 100% [15:55:42] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [15:56:55] RECOVERY - Host cloudcephosd1016 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [15:57:39] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [16:02:49] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.09% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [16:04:37] 10Toolforge (Quota-requests): Request increased quota for statanalyser Toolforge tool - https://phabricator.wikimedia.org/T433184#12159598 (10dcaro) >>! In T433184#12159464, @Leaderboard wrote: > Yes that'd be helpful. > >> k8s side allocates with half of what you request > > So does it mean that it starts wi... [16:10:11] (03update) 10dcaro: core: cleanup after failing to create job in runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/358 [16:10:34] (03approved) 10dcaro: core: cleanup after failing to create job in runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/358 [16:13:16] (03merge) 10dcaro: core: cleanup after failing to create job in runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/358 [16:16:16] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.553-20260727161329-33451dff [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1349 (https://phabricator.wikimedia.org/T432687) [16:16:24] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.553-20260727161329-33451dff [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1349 (https://phabricator.wikimedia.org/T432687) [16:17:16] 06cloud-services-team, 10Tool-paulina, 10Toolforge, 06tools-platform-team: Automated traffic affects tool performance - https://phabricator.wikimedia.org/T432878#12159705 (10JMeybohm) [16:22:54] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [16:26:51] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) [16:40:01] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:51:29] 06tools-platform-team: [tools,harbor] We are reaching ~80% of object quota for tools, investigate if we have a leak - https://phabricator.wikimedia.org/T433281 (10dcaro) 03NEW [16:53:06] 06tools-platform-team: [tools,harbor] We are reaching ~80% of object quota for tools, investigate if we have a leak - https://phabricator.wikimedia.org/T433281#12159889 (10dcaro) p:05Triage→03Medium a:03dcaro [16:53:09] 06tools-platform-team: [tools,harbor] We are reaching ~80% of object quota for tools, investigate if we have a leak - https://phabricator.wikimedia.org/T433281#12159892 (10dcaro) 05Open→03In progress [16:53:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [16:56:00] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [16:59:07] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:59:11] 06tools-platform-team: [tools,harbor] We are reaching ~80% of object quota for tools, investigate if we have a leak - https://phabricator.wikimedia.org/T433281#12159907 (10dcaro) There's been ~10% growth in the number of artifacts in harbor in the last month (~100 out of 800), so it kinda matches the growth the... [17:04:18] 10Toolforge, 06tools-platform-team, 07OKR-Work: Verify if we can define alert conditions in Prometheus - https://phabricator.wikimedia.org/T432863#12159950 (10fnegri) I spinned up a test app at https://fnegri-test.toolforge.org/ which returns 200 on `/` and returns 500 on `/error`. I verified the PromQL quer... [17:06:01] RESOLVED: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [17:11:48] (03PS3) 10Andrew Bogott: roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) [17:11:48] (03PS1) 10Andrew Bogott: ceph unset_maintenance: don't block if 'noout' or 'norebalance' are set. [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318200 (https://phabricator.wikimedia.org/T427295) [17:12:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [17:14:11] (03approved) 10dcaro: jobs-api: bump to 0.0.553-20260727161329-33451dff [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1349 (https://phabricator.wikimedia.org/T432687) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:14:16] (03merge) 10dcaro: jobs-api: bump to 0.0.553-20260727161329-33451dff [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1349 (https://phabricator.wikimedia.org/T432687) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [17:14:43] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] cleanup resources when failing to create a job - https://phabricator.wikimedia.org/T432687#12159990 (10dcaro) 05In progress→03Resolved [17:15:23] (03CR) 10CI reject: [V:04-1] ceph unset_maintenance: don't block if 'noout' or 'norebalance' are set. [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318200 (https://phabricator.wikimedia.org/T427295) (owner: 10Andrew Bogott) [17:15:28] (03CR) 10CI reject: [V:04-1] roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) (owner: 10Andrew Bogott) [17:20:19] RESOLVED: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.06% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [17:30:05] 06tools-platform-team: [tools,harbor] We are reaching ~80% of object quota for tools, investigate if we have a leak - https://phabricator.wikimedia.org/T433281#12160087 (10dcaro) 05In progress→03Resolved [17:38:58] (03PS4) 10Andrew Bogott: roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) [17:38:58] (03PS1) 10Andrew Bogott: is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 [17:40:49] (03PS2) 10Andrew Bogott: is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 [17:40:49] (03PS5) 10Andrew Bogott: roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) [17:42:27] (03CR) 10CI reject: [V:04-1] roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) (owner: 10Andrew Bogott) [17:43:15] (03CR) 10CI reject: [V:04-1] is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 (owner: 10Andrew Bogott) [17:44:26] (03CR) 10CI reject: [V:04-1] is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 (owner: 10Andrew Bogott) [17:44:32] (03CR) 10CI reject: [V:04-1] roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) (owner: 10Andrew Bogott) [17:50:22] (03update) 10tlepage: Draft: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) [17:53:44] (03PS3) 10Andrew Bogott: is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 [17:53:45] (03PS6) 10Andrew Bogott: roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) [17:55:19] (03PS4) 10Andrew Bogott: is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 [17:55:19] (03PS7) 10Andrew Bogott: roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) [17:58:23] (03update) 10tlepage: Draft: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) [17:58:47] (03CR) 10CI reject: [V:04-1] is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 (owner: 10Andrew Bogott) [17:59:27] (03CR) 10CI reject: [V:04-1] roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) (owner: 10Andrew Bogott) [18:00:01] (03PS5) 10Andrew Bogott: is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 [18:00:01] (03PS8) 10Andrew Bogott: roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) [18:03:42] (03CR) 10CI reject: [V:04-1] roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) (owner: 10Andrew Bogott) [18:03:42] (03CR) 10CI reject: [V:04-1] is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 (owner: 10Andrew Bogott) [18:25:33] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12160311 (10Arnoldokoth) @Dzahn Yes, after changing Hiera and running puppet, that Apache config changes to the value specified in that domain. The... [18:26:13] (03update) 10tlepage: Draft: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) [18:26:56] (03update) 10tlepage: Draft: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) [18:28:58] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [18:30:06] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12160345 (10Dzahn) If the error comes from the haproxy that wmcs operates we should report this as a bug to them. The error sounds like it does. But... [18:38:37] (03PS6) 10Andrew Bogott: is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 [18:38:37] (03PS9) 10Andrew Bogott: roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) [18:41:49] (03CR) 10CI reject: [V:04-1] is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 (owner: 10Andrew Bogott) [18:41:51] (03CR) 10CI reject: [V:04-1] roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) (owner: 10Andrew Bogott) [18:43:19] (03PS7) 10Andrew Bogott: is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 [18:43:19] (03PS10) 10Andrew Bogott: roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) [18:46:47] (03CR) 10CI reject: [V:04-1] is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 (owner: 10Andrew Bogott) [18:47:09] (03CR) 10CI reject: [V:04-1] roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) (owner: 10Andrew Bogott) [18:49:37] (03PS8) 10Andrew Bogott: is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 [18:49:37] (03PS11) 10Andrew Bogott: roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) [18:49:51] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.roll_reboot_osds [18:49:52] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) [18:50:24] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.roll_reboot_osds [18:52:47] (03PS9) 10Andrew Bogott: is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 [18:52:47] (03PS12) 10Andrew Bogott: roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) [18:58:10] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:03:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:07:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:12:55] (03PS13) 10Andrew Bogott: roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) [19:32:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:34:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:40:15] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12160592 (10Arnoldokoth) Yeah, I think the error might be coming from the proxy and not the host. The apache log files did not show anything at the t... [19:57:17] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [20:03:22] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305 (10MusikAnimal) 03NEW [20:03:52] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12160696 (10MusikAnimal) [20:04:08] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12160699 (10MusikAnimal) [20:04:11] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12160698 (10MusikAnimal) [20:04:50] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12160703 (10MusikAnimal) [20:04:51] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12160704 (10MusikAnimal) [20:04:58] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12160705 (10MusikAnimal) [20:05:00] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 10Data-Engineering-Wikistats, and 2 others: Pageviews Analysis 3.0 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12160706 (10MusikAnimal) [20:05:47] 10Tool-Pageviews: Move Pageviews tool to Symfony - https://phabricator.wikimedia.org/T420834#12160708 (10MusikAnimal) [20:05:49] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12160707 (10MusikAnimal) [20:15:40] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12160732 (10Dzahn) I might have ran into something similar before where if you delete a proxy and recreate it it took a while or was cached. But not... [20:24:44] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (publish_continuous_job_to_internet) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) [20:26:02] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (publish_continuous_job_to_internet) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) [21:26:03] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud is about to expire in 27d 23h 58m 16s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [21:31:03] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate Puppet CA: gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud is about to expire in 27d 23h 58m 21s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [21:42:51] 10VPS-project-Codesearch, 06collaboration-services, 13Patch-For-Review: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#12161077 (10Dzahn) [22:33:28] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) [22:42:55] FIRING: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:50:03] (03update) 10raymond-ndibe: kubernetes.py: detect conflict with httproutes created by jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/114 (https://phabricator.wikimedia.org/T431417) [22:50:44] (03update) 10raymond-ndibe: kubernetes.py: detect conflict with httproutes created by jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/114 (https://phabricator.wikimedia.org/T431417) [22:59:20] (03update) 10raymond-ndibe: kubernetes.py: detect conflict with httproutes created by jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/114 (https://phabricator.wikimedia.org/T431417) [23:56:37] 10Toolforge, 06tools-platform-team, 10Toolhub, 07Documentation: Set up a dedicate and official space for documenation of toolforge tools - https://phabricator.wikimedia.org/T433325 (10Ladsgroup) 03NEW