[00:15:39] (03update) 10renovatebot: Update python Docker tag to v3.14.6 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [00:15:40] (03update) 10renovatebot: Update python Docker tag to v3.14.6 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [00:17:48] 10Cloud-Services: Urgent: Backup restore request (Toolforge) - https://phabricator.wikimedia.org/T428830 (10Green_Cardamom) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specifi... [00:19:45] 06cloud-services-team, 10Data-Services, 10Toolforge: Urgent: Backup restore request (Toolforge) - https://phabricator.wikimedia.org/T428830#12007579 (10Green_Cardamom) [00:38:04] 06cloud-services-team, 10Data-Services, 10Toolforge: Urgent: Backup restore request (Toolforge) - https://phabricator.wikimedia.org/T428830#12007604 (10Green_Cardamom) Visually can see: https://tools-static.wmflabs.org/botwikiawk/dashdaily/db/2026/ The numbers run from 001.* to 028.* then jump to 161.* ..... [00:45:56] 06cloud-services-team, 10Data-Services, 10Toolforge: Urgent: Backup restore request (Toolforge) - https://phabricator.wikimedia.org/T428830#12007622 (10Green_Cardamom) [00:46:59] 06cloud-services-team, 10Data-Services, 10Toolforge: Urgent: Backup restore request (Toolforge) - https://phabricator.wikimedia.org/T428830#12007623 (10Green_Cardamom) [01:00:30] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#12007649 (10Osuji_pius) **Weekly Internship Report Week 2: June 1– June 7** **Task progress** - Continued the UI/UX redesign work based on mentor feedback from the previous week.... [01:15:16] (03update) 10renovatebot: Update python Docker tag to v3.14.6 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [01:30:04] 10Tool-centralnotice-banner-editor, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice: Operationalize Outreachy banner editor project - https://phabricator.wikimedia.org/T412454#12007693 (10AKanji-WMF) [02:33:16] 10Tool-wikicontest: Deploy latest WikiContest/WikiEval code and migrate production data - https://phabricator.wikimedia.org/T428838 (10Agamyasamuel) 03NEW [05:21:54] (03open) 10countcount: Fix outdated documentation [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/32 [05:22:32] (03open) 10countcount: Fix outdated documentation [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/19 [05:22:44] (03open) 10countcount: Fix outdated CLAUDE.md: shared CI templates and coverage job [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/23 [05:24:32] (03open) 10countcount: Fix outdated documentation [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/28 [05:24:52] (03open) 10countcount: Fix outdated documentation [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/12 [05:33:20] (03merge) 10countcount: Fix outdated CLAUDE.md: shared CI templates and coverage job [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/23 [05:33:38] (03merge) 10countcount: Fix outdated documentation [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/19 [05:33:54] (03merge) 10countcount: Fix outdated documentation [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/32 [05:34:47] (03merge) 10countcount: Fix outdated documentation [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/28 [05:35:22] (03merge) 10countcount: Fix outdated documentation [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/12 [06:15:22] (03open) 10countcount: Drop separate coverage job; coverage now runs inside rust-test [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/14 [06:16:05] (03open) 10countcount: Run coverage inside rust-test instead of a separate job [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/20 [06:17:25] (03open) 10countcount: Run coverage inside the test job instead of a separate job [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/33 [06:17:59] (03open) 10countcount: Run coverage inside the test job instead of a separate job [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/24 [06:18:18] (03open) 10countcount: Run coverage inside the test job instead of a separate job [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/29 [06:18:24] (03open) 10countcount: Run coverage inside the test jobs instead of separate jobs [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/13 [06:30:19] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12008264 (10KCVelaga_WMF) >>! In T425087#11987928, @KCVelaga_WMF wrote: >>>! In T425087#11987887, @BT... [06:48:20] (03merge) 10countcount: Run coverage inside the test job instead of a separate job [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/29 [06:48:25] (03merge) 10countcount: Run coverage inside the test jobs instead of separate jobs [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/13 [06:48:36] (03merge) 10countcount: Run coverage inside the test job instead of a separate job [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/33 [06:49:44] (03merge) 10countcount: Run coverage inside the test job instead of a separate job [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/24 [06:50:41] 10Tool-Phabricator-bug-status, 10Phabricator: Replace deprecated Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T428851 (10Aklapper) 03NEW [06:51:39] 10Tool-Phabricator-bug-status, 10Phabricator: Replace deprecated Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T428851#12008358 (10Aklapper) [06:54:58] 10Wikibugs, 10Phabricator, 13Patch-For-Review: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#12008371 (10Aklapper) Thanks a lot for the maniphest.info patch, very appreciated! Now I wonder who could review that. Also, that... [06:55:38] (03merge) 10countcount: Run coverage inside rust-test instead of a separate job [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/20 [06:56:18] (03merge) 10countcount: Drop separate coverage job; coverage now runs inside rust-test [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/14 [07:22:21] (03update) 10countcount: tests: serialize DB tests via serial_test groups instead of --test-threads=1 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/21 [07:22:27] (03open) 10countcount: tests: serialize DB tests via serial_test groups instead of --test-threads=1 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/21 [07:22:40] (03update) 10countcount: tests: serialize DB tests via serial_test groups instead of --test-threads=1 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/21 [07:28:17] (03merge) 10countcount: tests: serialize DB tests via serial_test groups instead of --test-threads=1 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/21 [08:53:29] 06cloud-services-team, 10Cloud-VPS: Openstack root disk backups log errors - https://phabricator.wikimedia.org/T428865 (10Volans) 03NEW p:05Triage→03High [08:57:11] 06cloud-services-team, 10Cloud-VPS: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867 (10Volans) 03NEW p:05Triage→03Unbreak! [09:07:30] 06cloud-services-team, 10Cloud-VPS: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12008796 (10Volans) Do we have any concern that after fixing the issue the next round of backups might cause harm to the infrastructure because of the backlog? [09:23:23] 06cloud-services-team, 10Cloud-VPS: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12008873 (10Volans) `cloudbackup1*` hosts doesn't seem to be affected just because they have a leg into the `cloud-private` vlan: ` $ ip route get 185.15.56.161 185.15.56.161 via 172.20.... [09:32:58] 06cloud-services-team, 10Cloud-VPS: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12008912 (10Volans) The systemd unit is clearly failed: ` ● backup_cinder_volumes.service loaded failed failed backu... [09:34:33] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Ditch wmcs-specific alerts for systemdunitdown in favor of production alerts - https://phabricator.wikimedia.org/T428873 (10fgiunchedi) 03NEW [09:34:58] 10Tool-curator: Curator not available for days - https://phabricator.wikimedia.org/T428874 (10PantheraLeo1359531) 03NEW [09:56:14] (03update) 10raymond-ndibe: status: add duration field [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/302 (https://phabricator.wikimedia.org/T401172) [09:56:17] (03approved) 10raymond-ndibe: status: add duration field [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/302 (https://phabricator.wikimedia.org/T401172) [09:56:32] (03merge) 10raymond-ndibe: status: add duration field [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/302 (https://phabricator.wikimedia.org/T401172) [09:56:38] (03update) 10raymond-ndibe: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) [10:07:15] (03update) 10raymond-ndibe: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) [10:08:55] (03update) 10raymond-ndibe: status: add messages to status [repos/cloud/toolforge/jobs-api] (status_add_up_to_date) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/304 (https://phabricator.wikimedia.org/T401172) [10:10:58] (03update) 10raymond-ndibe: status: add out-of-quota detection [repos/cloud/toolforge/jobs-api] (status_add_messages) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/305 (https://phabricator.wikimedia.org/T401172) [10:11:23] (03update) 10raymond-ndibe: status: handle restart messaging and count [repos/cloud/toolforge/jobs-api] (status_add_quota_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/306 (https://phabricator.wikimedia.org/T401172) [10:11:31] (03update) 10raymond-ndibe: status: handle edge cases [repos/cloud/toolforge/jobs-api] (status_add_restarting_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/307 (https://phabricator.wikimedia.org/T401172) [10:12:03] 10Tool-wikimedia-attribution: Create approved brand mark - https://phabricator.wikimedia.org/T426184#12009053 (10Sarai-WMF) [10:12:34] 10Tool-wikimedia-attribution: Create protected gallery page with approved project brand marks - https://phabricator.wikimedia.org/T426184#12009058 (10Sarai-WMF) [10:16:59] (03update) 10raymond-ndibe: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) [10:17:12] (03update) 10raymond-ndibe: status: add messages to status [repos/cloud/toolforge/jobs-api] (status_add_up_to_date) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/304 (https://phabricator.wikimedia.org/T401172) [10:17:20] (03update) 10raymond-ndibe: status: add out-of-quota detection [repos/cloud/toolforge/jobs-api] (status_add_messages) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/305 (https://phabricator.wikimedia.org/T401172) [10:17:30] (03update) 10raymond-ndibe: status: handle restart messaging and count [repos/cloud/toolforge/jobs-api] (status_add_quota_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/306 (https://phabricator.wikimedia.org/T401172) [10:17:39] (03update) 10raymond-ndibe: status: handle edge cases [repos/cloud/toolforge/jobs-api] (status_add_restarting_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/307 (https://phabricator.wikimedia.org/T401172) [10:19:22] (03update) 10raymond-ndibe: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) [10:19:28] (03update) 10raymond-ndibe: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) [10:19:28] (03update) 10raymond-ndibe: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) [10:21:11] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.501-20260611095653-d1683b7a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1291 (https://phabricator.wikimedia.org/T401172) [10:21:19] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.501-20260611095653-d1683b7a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1291 (https://phabricator.wikimedia.org/T401172) [10:22:20] (03update) 10raymond-ndibe: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) [10:27:00] 10Tool-inteGraality: 'missing' groups should be indicated - https://phabricator.wikimedia.org/T251867#12009101 (10JeanFred) [10:30:16] 10Tool-inteGraality: integraality does not handle "No value" groupings - https://phabricator.wikimedia.org/T278101#12009112 (10JeanFred) [10:35:52] 10Toolforge, 06tools-platform-team: Specifying --filelog-stdout or --filelog-stderr requires --filelog - https://phabricator.wikimedia.org/T428354#12009142 (10fnegri) I checked this more carefully, I think we do have an inconsistency that maybe is worth fixing: * Running `toolforge jobs run --command "echo fo... [10:39:11] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12009161 (10fgiunchedi) >>! In T428867#12008912, @Volans wrote: > The systemd unit is clearly failed: > > ` > ● backup_cinder_volumes.service... [10:41:55] 10Toolforge, 06tools-platform-team: Specifying --filelog-stdout or --filelog-stderr requires --filelog - https://phabricator.wikimedia.org/T428354#12009168 (10taavi) >>! In T428354#12007215, @aputhin wrote: > I don't disagree at all and also proposed we re-add it at first, but I missed adding some context abov... [11:29:56] FIRING: SystemdUnitDown: The service unit backup_cinder_volumes.service is in failed status on host cloudbackup1001-dev. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1001-dev - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:46:57] 10Toolforge, 06tools-platform-team, 05Goal: [harbor] Deploy with Helm - https://phabricator.wikimedia.org/T356301#12009423 (10aputhin) [11:47:46] FIRING: JobUnavailable: Reduced availability for job mysql-galera in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:49:56] FIRING: [3x] SystemdUnitDown: The service unit backup_cinder_volumes.service is in failed status on host cloudbackup1001-dev. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:52:46] RESOLVED: JobUnavailable: Reduced availability for job mysql-galera in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:54:56] FIRING: [4x] SystemdUnitDown: The service unit backup_cinder_volumes.service is in failed status on host cloudbackup1001-dev. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:59:56] RESOLVED: [4x] SystemdUnitDown: The service unit backup_cinder_volumes.service is in failed status on host cloudbackup1001-dev. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [12:04:20] 10Cloud-VPS, 06tools-infrastructure-team, 07Security: codfw1dev cloud realm endpoints accessible from cloudbackup1001-dev - https://phabricator.wikimedia.org/T428879#12009545 (10taavi) I checked that the filters are in place for all other cr<->cloudsw links. Closing, and will make public. [12:11:42] FIRING: AlertLintProblem: Linting problems found for HostBGPDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [12:12:44] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12009600 (10Volans) p:05Unbreak!→03Medium With the above patches merged the remote backups are now working again. I've manually triggered the ones from `cloudbac... [12:14:47] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)), 07OKR-Work: MediaWiki REST API appropriately defines array types for parameters - https://phabricator.wikimedia.org/T409113#12009603 (10MGoncalves-WMF) 05Open→03In progress [12:15:32] FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-test-k8s-worker-nfs-9 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:18:38] 10Cloud-VPS, 06tools-infrastructure-team, 07Security: codfw1dev cloud realm endpoints accessible from cloudbackup1001-dev - https://phabricator.wikimedia.org/T428879#12009610 (10cmooney) 05Open→03Resolved [12:25:29] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/43 [12:25:39] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/39 [12:26:59] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1300778 (owner: 10L10n-bot) [12:27:46] 10Tool-curator: Curator not available for days - https://phabricator.wikimedia.org/T428874#12009665 (10DaxServer) I paused it so that database password could be reset. I’ll re-enable it tomorrow when I get home from Berlin. [12:35:32] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-nfs-4 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:35:40] 06cloud-services-team, 10Cloud-VPS: Add monitoring for backy2 openstack backups - https://phabricator.wikimedia.org/T428893 (10Volans) 03NEW [12:35:44] (03update) 10fnegri: core/images.py: fetch all images for a tool [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/309 (https://phabricator.wikimedia.org/T428449) [12:40:32] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-21 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:41:42] RESOLVED: AlertLintProblem: Linting problems found for HostBGPDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [12:42:12] FIRING: AlertLintProblem: Linting problems found for HostBGPDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [12:45:32] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-harbor-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:45:32] (03update) 10fnegri: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [12:45:34] (03update) 10fnegri: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [12:45:36] (03approved) 10fnegri: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [12:46:57] RESOLVED: AlertLintProblem: Linting problems found for HostBGPDown - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [12:47:14] (03merge) 10fnegri: core/images.py: fetch all images for a tool [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/309 (https://phabricator.wikimedia.org/T428449) [12:49:54] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api] allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#12009765 (10Raymond_Ndibe) [12:50:16] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#12009766 (10Raymond_Ndibe) [12:51:03] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.502-20260611124739-a22e5bf8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1291 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428449) [12:51:09] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.502-20260611124739-a22e5bf8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1291 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428449) [12:55:32] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-21 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:55:42] 10Cloud-VPS, 06tools-infrastructure-team: Cloud VPS cinder backy config should ensure a specific number of backups is kept - https://phabricator.wikimedia.org/T428897 (10taavi) 03NEW [12:55:49] !log fnegri@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T428449) [12:55:55] T428449: [jobs-api] is not listing all container images for a tool - https://phabricator.wikimedia.org/T428449 [13:01:39] 10Toolforge (Push-to-Deploy), 06tools-platform-team: support --webservice option in jobs-cli and jobs-api - https://phabricator.wikimedia.org/T428898 (10Raymond_Ndibe) 03NEW [13:07:11] (03update) 10raymond-ndibe: status: add messages to status [repos/cloud/toolforge/jobs-api] (status_add_up_to_date) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/304 (https://phabricator.wikimedia.org/T401172) [13:07:17] (03update) 10raymond-ndibe: status: add out-of-quota detection [repos/cloud/toolforge/jobs-api] (status_add_messages) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/305 (https://phabricator.wikimedia.org/T401172) [13:07:26] (03update) 10raymond-ndibe: status: handle restart messaging and count [repos/cloud/toolforge/jobs-api] (status_add_quota_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/306 (https://phabricator.wikimedia.org/T401172) [13:07:42] (03update) 10raymond-ndibe: status: handle edge cases [repos/cloud/toolforge/jobs-api] (status_add_restarting_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/307 (https://phabricator.wikimedia.org/T401172) [13:08:02] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-etcd-29 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:08:03] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#12009875 (10fnegri) The announcement was sent on May, 20th and the deadline is set for June, 20th: https://lists.wikimedia.org/hyperk... [13:08:14] !log fnegri@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api (T428449) [13:08:20] T428449: [jobs-api] is not listing all container images for a tool - https://phabricator.wikimedia.org/T428449 [13:18:15] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T428449) [13:18:21] T428449: [jobs-api] is not listing all container images for a tool - https://phabricator.wikimedia.org/T428449 [13:20:47] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] is not listing all container images for a tool - https://phabricator.wikimedia.org/T428449#12009943 (10fnegri) @DamianZaremba the patch was deployed and I can now see the image in `toolforge jobs images`: `lang=shell-session tools.cluebotng-... [13:23:02] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-etcd-29 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:25:20] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] is not listing all container images for a tool - https://phabricator.wikimedia.org/T428449#12009975 (10DamianZaremba) Deployment is now succeeding as expected; ` m00m00:component-configs damian$ TARGET_USER=cluebotng-review fab deploy Applyin... [13:25:32] RESOLVED: [4x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-harbor-1 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:31:11] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api (T428449) [13:31:17] T428449: [jobs-api] is not listing all container images for a tool - https://phabricator.wikimedia.org/T428449 [13:32:21] (03update) 10fnegri: jobs-api: bump to 0.0.502-20260611124739-a22e5bf8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1291 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428449) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:32:30] (03merge) 10fnegri: jobs-api: bump to 0.0.502-20260611124739-a22e5bf8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1291 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428449) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:33:17] 06cloud-services-team, 10Toolforge, 06tools-platform-team: components-api failing to connect to internal api - https://phabricator.wikimedia.org/T428214#12010028 (10DamianZaremba) I can't replicate this at the moment on any tool, so it appears to have resolved it's self. It would be good to understand why i... [13:35:01] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27350281127 (https://github.com/cluebotng/component-configs/commits/41e666d399f22121043fa8132a3e8bf6368e5181) [13:35:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [13:36:41] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [13:42:48] 10Tools: cluebotng-review tool stores 180G of backups - https://phabricator.wikimedia.org/T428427#12010057 (10DamianZaremba) Updated cluebotng-review to keep the daily backup on nfs and the intra-daily in object storage with a short retention, nfs usage should now be below 15G. Also moved 36GB of files into obj... [13:49:39] (03approved) 10fnegri: dev: remove makefile [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/53 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:49:41] (03update) 10fnegri: dev: remove makefile [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/53 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:49:43] (03approved) 10fnegri: dev: remove makefile [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/308 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:49:45] (03update) 10fnegri: dev: remove makefile [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/308 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:49:49] (03approved) 10fnegri: dev: cleanup makefile [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/162 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:49:50] (03update) 10fnegri: dev: cleanup makefile [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/162 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:49:52] (03approved) 10fnegri: dev: remove makefile [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/42 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:49:53] (03update) 10fnegri: dev: remove makefile [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/42 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:49:55] (03approved) 10fnegri: dev: remove makefile [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/91 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:49:58] (03update) 10fnegri: dev: remove makefile [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/91 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:50:02] (03approved) 10fnegri: dev: cleanup makefile [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/76 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:50:06] (03update) 10fnegri: dev: cleanup makefile [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/76 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:50:10] (03approved) 10fnegri: dev: remove makefile [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/98 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:50:14] (03update) 10fnegri: dev: remove makefile [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/98 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:50:18] (03update) 10fnegri: dev: remove makefile [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/49 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:50:22] (03approved) 10fnegri: dev: remove makefile [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/49 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:50:26] (03update) 10fnegri: dev: remove makefile [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/42 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:50:35] (03approved) 10fnegri: dev: remove makefile [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/42 (https://phabricator.wikimedia.org/T428698) (owner: 10raymond-ndibe) [13:53:35] (03update) 10raymond-ndibe: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) [13:53:37] (03approved) 10raymond-ndibe: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) [13:53:44] (03update) 10raymond-ndibe: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) [13:54:22] (03approved) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/53 (https://phabricator.wikimedia.org/T428698) [13:54:30] (03merge) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/53 (https://phabricator.wikimedia.org/T428698) [13:54:43] (03update) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/308 (https://phabricator.wikimedia.org/T428698) [13:54:54] (03approved) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/42 (https://phabricator.wikimedia.org/T428698) [13:55:01] (03merge) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/42 (https://phabricator.wikimedia.org/T428698) [13:55:12] (03approved) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/49 (https://phabricator.wikimedia.org/T428698) [13:55:23] (03merge) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/49 (https://phabricator.wikimedia.org/T428698) [13:55:27] (03approved) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/98 (https://phabricator.wikimedia.org/T428698) [13:55:42] (03merge) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/98 (https://phabricator.wikimedia.org/T428698) [13:55:43] (03approved) 10raymond-ndibe: dev: cleanup makefile [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/76 (https://phabricator.wikimedia.org/T428698) [13:55:55] (03merge) 10raymond-ndibe: dev: cleanup makefile [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/76 (https://phabricator.wikimedia.org/T428698) [13:55:56] (03approved) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/91 (https://phabricator.wikimedia.org/T428698) [13:56:09] (03merge) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/91 (https://phabricator.wikimedia.org/T428698) [13:56:17] (03approved) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/42 (https://phabricator.wikimedia.org/T428698) [13:56:27] (03merge) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/42 (https://phabricator.wikimedia.org/T428698) [13:56:54] (03approved) 10raymond-ndibe: dev: cleanup makefile [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/162 (https://phabricator.wikimedia.org/T428698) [13:57:08] (03merge) 10raymond-ndibe: dev: cleanup makefile [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/162 (https://phabricator.wikimedia.org/T428698) [13:57:39] (03merge) 10raymond-ndibe: status: add up_to_date field and API warnings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/303 (https://phabricator.wikimedia.org/T401172) [13:57:51] (03update) 10raymond-ndibe: status: add messages to status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/304 (https://phabricator.wikimedia.org/T401172) [13:58:01] (03approved) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/308 (https://phabricator.wikimedia.org/T428698) [13:58:19] (03update) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/308 (https://phabricator.wikimedia.org/T428698) [13:58:33] (03update) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/308 (https://phabricator.wikimedia.org/T428698) [13:59:01] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.88-20260611135441-3ce40b54 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1292 (https://phabricator.wikimedia.org/T428698) [13:59:09] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.88-20260611135441-3ce40b54 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1292 (https://phabricator.wikimedia.org/T428698) [13:59:10] (03update) 10raymond-ndibe: status: add messages to status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/304 (https://phabricator.wikimedia.org/T401172) [14:01:16] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.99-20260611135619-f0a42178 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1293 (https://phabricator.wikimedia.org/T428698) [14:01:25] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.99-20260611135619-f0a42178 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1293 (https://phabricator.wikimedia.org/T428698) [14:01:47] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-kubeusers: bump to 0.0.200-20260611135640-2c6096d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1294 (https://phabricator.wikimedia.org/T428698) [14:01:47] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: registry-admission: bump to 0.0.78-20260611135520-dff9af97 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1295 (https://phabricator.wikimedia.org/T428698) [14:01:57] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-kubeusers: bump to 0.0.200-20260611135640-2c6096d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1294 (https://phabricator.wikimedia.org/T428698) [14:01:57] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: registry-admission: bump to 0.0.78-20260611135520-dff9af97 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1295 (https://phabricator.wikimedia.org/T428698) [14:03:05] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: volume-admission: bump to 0.0.86-20260611135551-7d41d3e6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1296 (https://phabricator.wikimedia.org/T428698) [14:03:06] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: volume-admission: bump to 0.0.86-20260611135551-7d41d3e6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1296 (https://phabricator.wikimedia.org/T428698) [14:04:22] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: ingress-admission: bump to 0.0.89-20260611135716-2a74371e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1297 (https://phabricator.wikimedia.org/T428698) [14:04:22] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: ingress-admission: bump to 0.0.89-20260611135716-2a74371e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1297 (https://phabricator.wikimedia.org/T428698) [14:04:36] (03merge) 10raymond-ndibe: dev: remove makefile [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/308 (https://phabricator.wikimedia.org/T428698) [14:05:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' (T428549) [14:05:31] (03update) 10raymond-ndibe: status: add messages to status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/304 (https://phabricator.wikimedia.org/T401172) [14:06:08] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix top-level issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428147#12010199 (10AGhirelli-WMF) Notes for estimation: - T427356 (externalDocs support) is now complete, so that item is unblocked - Remaining unblocked wo... [14:06:56] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: envvars-api: bump to 0.0.87-20260611135624-0cf6e072 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1298 (https://phabricator.wikimedia.org/T428698) [14:06:58] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: envvars-api: bump to 0.0.87-20260611135624-0cf6e072 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1298 (https://phabricator.wikimedia.org/T428698) [14:06:59] (03update) 10raymond-ndibe: status: add messages to status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/304 (https://phabricator.wikimedia.org/T401172) [14:07:18] (03update) 10fnegri: status: add messages to status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/304 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [14:07:29] (03approved) 10fnegri: status: add messages to status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/304 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [14:08:13] (03update) 10raymond-ndibe: status: add out-of-quota detection [repos/cloud/toolforge/jobs-api] (status_add_messages) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/305 (https://phabricator.wikimedia.org/T401172) [14:08:43] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix type, media type, and array issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428149#12010206 (10AGhirelli-WMF) Notes for estimation: - Work involves adding missing type fields to response schema properties and fixing... [14:09:22] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.504-20260611140503-3d5701a1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1299 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428698) [14:09:36] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.504-20260611140503-3d5701a1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1299 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428698) [14:10:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [14:11:06] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.216-20260611135728-9fa009a3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1300 (https://phabricator.wikimedia.org/T428698) [14:11:22] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.216-20260611135728-9fa009a3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1300 (https://phabricator.wikimedia.org/T428698) [14:13:16] (03update) 10raymond-ndibe: status: add out-of-quota detection [repos/cloud/toolforge/jobs-api] (status_add_messages) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/305 (https://phabricator.wikimedia.org/T401172) [14:15:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [14:20:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:22:11] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' (T428549) [14:24:07] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet' (T428549) [14:29:44] 06cloud-services-team, 10Toolforge: 2026-06-08 Toolforge NFS cleanup - https://phabricator.wikimedia.org/T428422#12010328 (10taavi) 05Open→03Resolved a:03taavi We're back under 80%. [14:30:15] 10Tools: cluebotng-review tool stores 180G of backups - https://phabricator.wikimedia.org/T428427#12010334 (10taavi) 05Open→03Resolved a:03DamianZaremba Thanks! [14:30:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:41:29] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet' (T428549) [14:42:04] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] is not listing all container images for a tool - https://phabricator.wikimedia.org/T428449#12010389 (10fnegri) 05Open→03Resolved [14:47:49] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12010402 (10bd808) [14:58:05] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2010-dev.codfw.wmnet' (T428549) [15:10:38] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Replace https://os-deprecation.toolforge.org/ with something that handles in-place upgraded hosts - https://phabricator.wikimedia.org/T428919 (10Andrew) 03NEW [15:11:02] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2010-dev.codfw.wmnet' (T428549) [15:18:05] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Replace https://os-deprecation.toolforge.org/ with something that handles in-place upgraded hosts - https://phabricator.wikimedia.org/T428919#12010635 (10fgiunchedi) Following up from the meeting, one source of truth for running/reachable instan... [15:27:20] 10Tools, 06Commons, 07Community-Wishlist: Image Editor for Commons - https://phabricator.wikimedia.org/T426106#12010702 (10mikez-WMF) This is being worked on by @Amdrel with the support of our UX team and [[ https://www.mediawiki.org/wiki/User:Bawolff | Bawolff ]], who just helped review a PR: https://githu... [15:29:23] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix top-level issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428147#12010731 (10AGhirelli-WMF) [15:36:16] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12010765 (10fnegri) After restarting `dimastbkbot`, it's again showing up very frequently in the slow query log: `lang=shell-session fnegri@tools-db-6:... [15:37:52] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix type, media type, and array issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428149#12010766 (10AGhirelli-WMF) [15:49:13] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix top-level issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428147#12010853 (10KineticPelagic) FYI, these are the components where there is an `oas3-unused-component` warning. Each of these components currently also tri... [15:55:58] (03approved) 10raymond-ndibe: status: add messages to status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/304 (https://phabricator.wikimedia.org/T401172) [15:56:18] (03merge) 10raymond-ndibe: status: add messages to status [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/304 (https://phabricator.wikimedia.org/T401172) [15:56:18] (03update) 10fnegri: status: handle edge cases [repos/cloud/toolforge/jobs-api] (status_add_restarting_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/307 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [15:56:18] (03approved) 10fnegri: status: handle edge cases [repos/cloud/toolforge/jobs-api] (status_add_restarting_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/307 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [15:56:26] (03update) 10fnegri: status: handle restart messaging and count [repos/cloud/toolforge/jobs-api] (status_add_quota_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/306 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [15:56:27] (03update) 10raymond-ndibe: status: add out-of-quota detection [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/305 (https://phabricator.wikimedia.org/T401172) [15:56:33] (03approved) 10fnegri: status: handle restart messaging and count [repos/cloud/toolforge/jobs-api] (status_add_quota_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/306 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [15:56:38] (03update) 10fnegri: status: add out-of-quota detection [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/305 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [15:56:42] (03approved) 10fnegri: status: add out-of-quota detection [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/305 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [16:00:41] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.505-20260611155634-2c9cf126 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1299 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428698) [16:00:53] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.505-20260611155634-2c9cf126 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1299 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428698) [16:02:18] (03update) 10raymond-ndibe: status: handle restart messaging and count [repos/cloud/toolforge/jobs-api] (status_add_quota_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/306 (https://phabricator.wikimedia.org/T401172) [16:03:01] (03update) 10raymond-ndibe: status: handle edge cases [repos/cloud/toolforge/jobs-api] (status_add_restarting_detection) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/307 (https://phabricator.wikimedia.org/T401172) [16:03:10] (03approved) 10raymond-ndibe: status: add out-of-quota detection [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/305 (https://phabricator.wikimedia.org/T401172) [16:03:29] (03merge) 10raymond-ndibe: status: add out-of-quota detection [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/305 (https://phabricator.wikimedia.org/T401172) [16:03:40] (03update) 10raymond-ndibe: status: handle restart messaging and count [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/306 (https://phabricator.wikimedia.org/T401172) [16:03:48] (03approved) 10raymond-ndibe: status: handle restart messaging and count [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/306 (https://phabricator.wikimedia.org/T401172) [16:07:23] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.506-20260611160354-c1000ced [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1299 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428698) [16:07:27] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.506-20260611160354-c1000ced [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1299 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428698) [16:08:19] (03merge) 10raymond-ndibe: status: handle restart messaging and count [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/306 (https://phabricator.wikimedia.org/T401172) [16:08:20] (03update) 10raymond-ndibe: status: handle edge cases [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/307 (https://phabricator.wikimedia.org/T401172) [16:11:10] (03approved) 10raymond-ndibe: status: handle edge cases [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/307 (https://phabricator.wikimedia.org/T401172) [16:11:28] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.507-20260611160829-6258daab [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1299 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428698) [16:11:45] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.507-20260611160829-6258daab [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1299 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428698) [16:18:01] (03merge) 10raymond-ndibe: status: handle edge cases [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/307 (https://phabricator.wikimedia.org/T401172) [16:21:50] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.508-20260611161812-37126cfe [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1299 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428698) [16:21:57] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.508-20260611161812-37126cfe [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1299 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428698) [16:33:04] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006.eqiad.wmnet' (T428549) [16:33:04] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2006.eqiad.wmnet' (T428549) [16:37:32] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' (T428549) [16:43:07] 10Cloud-VPS (Quota-requests): Quota increase request for project wlm-it-visual - https://phabricator.wikimedia.org/T427731#12011206 (10Volans) p:05Triage→03Medium [16:47:00] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of VPS project - https://phabricator.wikimedia.org/T428939 (10Andrew) 03NEW [16:48:01] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of osbpo VPS project - https://phabricator.wikimedia.org/T428939#12011236 (10Andrew) [16:49:04] PROBLEM - Host cloudcontrol1006 is DOWN: PING CRITICAL - Packet loss = 100% [16:49:22] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [16:49:38] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of osbpo VPS project - https://phabricator.wikimedia.org/T428939#12011254 (10Andrew) [16:50:10] FIRING: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [16:50:16] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' (T428549) [16:50:17] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:50:22] RECOVERY - Host cloudcontrol1006 is UP: PING OK - Packet loss = 0%, RTA = 0.21 ms [16:51:04] 10Cloud-VPS (Quota-requests): Quota increase request for project wlm-it-visual - https://phabricator.wikimedia.org/T427731#12011262 (10Volans) @Andrew current RAM usage for the two VMs {F87993596} {F87993701} and CPU usage for the wlmitvisual01 instance (the other one is negligible, but might be because not ye... [16:54:22] RESOLVED: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [16:55:10] RESOLVED: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [16:55:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:05:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' (T428549) [17:09:25] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of osbpo VPS project - https://phabricator.wikimedia.org/T428939#12011348 (10Volans) SGTM +1 [17:16:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:16:40] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of osbpo VPS project - https://phabricator.wikimedia.org/T428939#12011381 (10Andrew) Oh, to clarify: I wasn't thinking that this mirror would necessarily be used as part of our cloud-vps deployment at all; I just want to support a project... [17:17:52] PROBLEM - Host cloudcontrol1007 is DOWN: PING CRITICAL - Packet loss = 100% [17:18:10] FIRING: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [17:18:22] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:18:57] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' (T428549) [17:19:02] RECOVERY - Host cloudcontrol1007 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [17:21:17] FIRING: [2x] JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:23:10] RESOLVED: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [17:23:22] RESOLVED: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:26:17] RESOLVED: [2x] JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:36:47] 06cloud-services-team, 10Cloud-VPS: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12011476 (10Volans) The cleanup (that unfortunately was setup to run before the backup and not after) has completed a while ago and the actual backups are running fine so far. Keep monito... [17:39:49] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' (T428549) [17:51:56] PROBLEM - Host cloudcontrol1011 is DOWN: PING CRITICAL - Packet loss = 100% [17:52:10] FIRING: GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [17:52:22] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1011.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:53:23] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet' (T428549) [17:53:26] RECOVERY - Host cloudcontrol1011 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [17:55:41] 10Tool-Phabricator-bug-status: Move away from deprecated `maniphest.query` Conduit API - https://phabricator.wikimedia.org/T418328#12011576 (10bd808) [17:55:42] 10Tool-Phabricator-bug-status, 10Phabricator: Replace deprecated Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T428851#12011578 (10bd808) →14Duplicate dup:03T418328 [17:57:10] RESOLVED: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [17:57:22] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1011.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [18:02:22] RESOLVED: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1011.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [18:03:56] 10Wikibugs, 10Phabricator, 13Patch-For-Review: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#12011605 (10bd808) >>! In T402454#12008371, @Aklapper wrote: > Now I wonder who could review that. I'm sure you did not mean this... [18:14:25] 10Cloud-VPS (Quota-requests), 06Release-Engineering-Team (Radar): Quota increase request for zuul - https://phabricator.wikimedia.org/T428515#12011627 (10Andrew) As I understand it, recent Magnum innovations/discoveries may make this unnecessary. @dduvall if that's incorrect please respond and you will have a... [18:44:02] 06cloud-services-team, 10Toolforge: Buildservice fails - https://phabricator.wikimedia.org/T428951 (10magnusmanske) 03NEW [18:47:00] 10Wikibugs, 10Phabricator, 13Patch-For-Review: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#12011768 (10Aklapper) @bd808: Meh, sorry. No it was not meant like that; I was honestly wondering who may find time at some point,... [18:48:06] 10Tool-Phabricator-bug-status, 10Phabricator: Move away from deprecated `maniphest.query` Conduit API - https://phabricator.wikimedia.org/T418328#12011773 (10Aklapper) [18:52:13] 10Tool-wmf-openapi-linter, 06Test Platform, 03[MWI] FY2025-26 Q4, 07Epic, 07OKR-Work: [5.2.5b Epic] Prepare the linter for use in CI - https://phabricator.wikimedia.org/T422918#12011779 (10HCoplin-WMF) +1 to what @Mooeypoo said. I would like to have flexibility around if and where we start throwing CI bl... [19:00:39] (03open) 10don-vip: Fix jpa/geotools/jts/jackson integration in web profile [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/42 [19:12:55] (03merge) 10don-vip: Fix jpa/geotools/jts/jackson integration in web profile [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/42 [19:16:58] FIRING: JobsEmailerNoEmails: No emails sent in the last hour - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerNoEmails - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsEmailerNoEmails [19:36:58] RESOLVED: JobsEmailerNoEmails: No emails sent in the last hour - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerNoEmails - https://prometheus-alerts.wmcloud.org/?q=alertname%3DJobsEmailerNoEmails [19:49:36] 06cloud-services-team, 10Cloud-VPS: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12011980 (10Volans) Backups on `cloudbackup2003` have started and old backups were kept as expected. New ones so far have been all full backups. [19:57:00] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/43 (owner: 10l10n-bot) [19:57:04] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/43 (owner: 10l10n-bot) [20:08:51] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/39 (owner: 10l10n-bot) [20:08:54] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/39 (owner: 10l10n-bot) [20:55:34] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.508-20260611161812-37126cfe [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1299 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:55:43] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.508-20260611161812-37126cfe [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1299 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:55:56] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.508-20260611161812-37126cfe [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1299 (https://phabricator.wikimedia.org/T401172 https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:55:57] (03update) 10raymond-ndibe: builds-api: bump to 0.0.216-20260611135728-9fa009a3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1300 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:56:00] (03approved) 10raymond-ndibe: builds-api: bump to 0.0.216-20260611135728-9fa009a3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1300 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:56:28] (03update) 10raymond-ndibe: builds-api: bump to 0.0.216-20260611135728-9fa009a3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1300 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:57:19] (03merge) 10raymond-ndibe: builds-api: bump to 0.0.216-20260611135728-9fa009a3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1300 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:57:29] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.87-20260611135624-0cf6e072 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1298 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:57:34] (03update) 10raymond-ndibe: envvars-api: bump to 0.0.87-20260611135624-0cf6e072 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1298 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:57:42] (03approved) 10raymond-ndibe: envvars-api: bump to 0.0.87-20260611135624-0cf6e072 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1298 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:58:23] (03merge) 10raymond-ndibe: envvars-api: bump to 0.0.87-20260611135624-0cf6e072 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1298 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:00:23] (03approved) 10raymond-ndibe: ingress-admission: bump to 0.0.89-20260611135716-2a74371e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1297 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:00:24] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.89-20260611135716-2a74371e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1297 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:00:40] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.89-20260611135716-2a74371e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1297 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:01:06] (03merge) 10raymond-ndibe: ingress-admission: bump to 0.0.89-20260611135716-2a74371e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1297 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:01:24] (03update) 10raymond-ndibe: volume-admission: bump to 0.0.86-20260611135551-7d41d3e6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1296 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:01:30] (03approved) 10raymond-ndibe: volume-admission: bump to 0.0.86-20260611135551-7d41d3e6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1296 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:01:39] (03update) 10raymond-ndibe: volume-admission: bump to 0.0.86-20260611135551-7d41d3e6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1296 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:02:38] (03merge) 10raymond-ndibe: volume-admission: bump to 0.0.86-20260611135551-7d41d3e6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1296 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:03:30] (03update) 10raymond-ndibe: registry-admission: bump to 0.0.78-20260611135520-dff9af97 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1295 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:03:34] (03approved) 10raymond-ndibe: registry-admission: bump to 0.0.78-20260611135520-dff9af97 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1295 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:03:45] (03update) 10raymond-ndibe: registry-admission: bump to 0.0.78-20260611135520-dff9af97 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1295 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:04:28] (03merge) 10raymond-ndibe: registry-admission: bump to 0.0.78-20260611135520-dff9af97 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1295 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:34:14] 10Wikibugs, 10Phabricator, 13Patch-For-Review: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#12012364 (10bd808) Wikibugs has a lot of maintainers in theory: https://toolsadmin.wikimedia.org/tools/id/wikibugs In practice, i... [21:43:48] (03approved) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.200-20260611135640-2c6096d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1294 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:43:50] (03update) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.200-20260611135640-2c6096d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1294 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:43:55] (03update) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.200-20260611135640-2c6096d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1294 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:44:40] (03merge) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.200-20260611135640-2c6096d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1294 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:01:46] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.99-20260611135619-f0a42178 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1293 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:01:55] (03approved) 10raymond-ndibe: api-gateway: bump to 0.0.99-20260611135619-f0a42178 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1293 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:02:05] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.99-20260611135619-f0a42178 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1293 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:02:11] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.88-20260611135441-3ce40b54 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1292 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:02:16] (03approved) 10raymond-ndibe: jobs-emailer: bump to 0.0.88-20260611135441-3ce40b54 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1292 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:02:30] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.88-20260611135441-3ce40b54 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1292 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:03:15] (03merge) 10raymond-ndibe: jobs-emailer: bump to 0.0.88-20260611135441-3ce40b54 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1292 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:03:37] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.99-20260611135619-f0a42178 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1293 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:03:41] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.99-20260611135619-f0a42178 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1293 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:05:12] (03merge) 10raymond-ndibe: api-gateway: bump to 0.0.99-20260611135619-f0a42178 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1293 (https://phabricator.wikimedia.org/T428698) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:58:17] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: remove/cleanup makefiles in projects - https://phabricator.wikimedia.org/T428698#12012639 (10Raymond_Ndibe) 05Open→03Resolved [22:58:18] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-api] make job status an enum, with clearly defined states - https://phabricator.wikimedia.org/T401172#12012641 (10Raymond_Ndibe) 05In progress→03Resolved [23:22:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [23:27:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors