[00:04:49] (03open) 10lokal-profil: Add visualization [toolforge-repos/toolviews] - 10https://gitlab.wikimedia.org/toolforge-repos/toolviews/-/merge_requests/13 [02:37:16] 10Wikibugs, 10Phabricator, 13Patch-For-Review: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#12012937 (10HakanIST) Thanks both for the context. No rush on the review from my side, I can rebase whenever needed. Also, I can h... [06:09:12] 06cloud-services-team, 10Cloud-VPS: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12013155 (10Volans) `cloudbackup2004` backups are still going, currently backing up the `tool-nfs` volume (`17.9% 160.0MB/sØ ETA 14h53m`). I'll keep an eye that the 07:00 UTC timer doesn'... [06:28:40] 10Tool-wmf-openapi-linter, 06Test Platform, 03[MWI] FY2025-26 Q4, 07Epic, 07OKR-Work: [5.2.5b Epic] Prepare the linter for use in CI - https://phabricator.wikimedia.org/T422918#12013165 (10Peter) I'm thinking either if we just swallows the error code when we run it for now (and we still get the output) o... [06:32:04] 10Tool-wmf-openapi-linter, 06Test Platform, 03[MWI] FY2025-26 Q4, 07Epic, 07OKR-Work: [5.2.5b Epic] Prepare the linter for use in CI - https://phabricator.wikimedia.org/T422918#12013166 (10Peter) Something like: ` "api-testing": "spectral YOUR_SETUP || true && mocha --timeout 0 --recursive --parallel te... [06:35:03] 06cloud-services-team, 10Cloud-VPS: wmcs-backup fails to create differential when the data changed too much and fails the whole run - https://phabricator.wikimedia.org/T428995 (10Volans) 03NEW [06:41:59] 06cloud-services-team, 10Cloud-VPS: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12013189 (10Volans) I've manually removed the snapshot for the failed volume so that it should force it to make a full backup and restarted the process on `backup2003`. [08:41:10] (03update) 10aklapper: README: Replace repo mirror URI with canonical URI [toolforge-repos/my-first-flask-oauth-tool] - 10https://gitlab.wikimedia.org/toolforge-repos/my-first-flask-oauth-tool/-/merge_requests/1 (https://phabricator.wikimedia.org/T405596) [08:41:21] (03update) 10aklapper: README: Replace repo mirror URI with canonical URI [toolforge-repos/my-first-flask-oauth-tool] - 10https://gitlab.wikimedia.org/toolforge-repos/my-first-flask-oauth-tool/-/merge_requests/1 (https://phabricator.wikimedia.org/T405596) [08:41:45] (03update) 10aklapper: README: Replace repo mirror URI with canonical URI [toolforge-repos/my-first-flask-oauth-tool] - 10https://gitlab.wikimedia.org/toolforge-repos/my-first-flask-oauth-tool/-/merge_requests/1 (https://phabricator.wikimedia.org/T405596) [08:42:20] (03update) 10aklapper: README: Replace repo mirror URI with canonical URI [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/7 (https://phabricator.wikimedia.org/T405596) [09:41:36] 10Tool-flickr2commons-ng: Flickr2Commons fails to fetch - https://phabricator.wikimedia.org/T429006 (10Effeietsanders) 03NEW [09:52:06] 06cloud-services-team, 10Cloud-VPS: mwopenstackclients.py reliability - https://phabricator.wikimedia.org/T429003#12013704 (10fgiunchedi) [11:11:59] 10Cloud-Services, 06Infrastructure-Foundations, 10netops, 06SRE: Upgrade cloudsw1-e4-eqiad - https://phabricator.wikimedia.org/T429013 (10cmooney) 03NEW p:05Triage→03Medium The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/pr... [11:12:07] 10Cloud-Services, 06Infrastructure-Foundations, 10netops, 06SRE: Upgrade cloudsw1-e4-eqiad - https://phabricator.wikimedia.org/T429013#12013913 (10cmooney) [11:23:21] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops, 06SRE: Upgrade cloudsw1-e4-eqiad - https://phabricator.wikimedia.org/T429013#12013945 (10cmooney) [11:23:38] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops, 06SRE: Upgrade cloudsw1-f4-eqiad - https://phabricator.wikimedia.org/T429014#12013946 (10cmooney) [11:27:00] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)): Support requestBody content examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427360#12013965 (10KineticPelagic) Since this task follows the pattern of {T42... [11:45:00] 06cloud-services-team, 10Cloud-VPS: Openstack root disk backups log errors - https://phabricator.wikimedia.org/T428865#12014021 (10Volans) Partially duplicate of T397105, that was closed without fixing the problem. [11:45:57] 10Toolforge, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#12014029 (10cmooney) 05Open→03Stalled p:05High→03Medium [12:12:14] 06cloud-services-team, 10Cloud-VPS: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12014112 (10Volans) The full run on `cloudbackup2003` failed again for `volume-9ec2e481-26dd-41d1-81bf-24b43abcee12`, due to T428995. I've manually cleaned snapshots for that volume. But... [12:29:34] 06cloud-services-team, 10Cloud-VPS: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12014161 (10Volans) I've extracted the list of volumes from `backy2 ls` for which the last backup is from march, and I passed that to `rbd` to list the snapshots. What I can do is pre-emp... [13:01:32] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1300778 (owner: 10L10n-bot) [13:36:40] 10Tool-curator: Curator not available for days - https://phabricator.wikimedia.org/T428874#12014362 (10PantheraLeo1359531) THank you :) [13:44:23] 10Tools: dimastbkbot tool is making very long db queries and crashing repeatedly - https://phabricator.wikimedia.org/T429027 (10fnegri) 03NEW [13:53:31] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12014412 (10fnegri) Stopped the tool again. Before doing that, I noticed the pod is frequently crashing and restarting: `lang=shell-session tools.dimas... [14:02:30] 06cloud-services-team, 10Cloud-VPS: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12014439 (10Volans) Agreed on irc to clear the unprotected snapshots created by cloudbackup in 2026. Extracted the list with: ` $ while read -r vol; do sudo rbd --pool eqiad1-cinder s... [14:18:00] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12014469 (10fnegri) After stopping `dimastbkbot`, the next issue preventing history log purge was a long transaction from user `s56581`: `lang=mysql Ma... [14:24:15] 06cloud-services-team, 10Toolforge: toolforge webservice logs output has encoding issues and gets truncated compared to kubectl logs - https://phabricator.wikimedia.org/T429028 (10diegodlh) 03NEW [14:55:42] 10Cloud-VPS (Quota-requests): Quota increase request for project wlm-it-visual - https://phabricator.wikimedia.org/T427731#12014603 (10Volans) @Ferdi2005 is my understanding correct that some of the current workload of `wlmitvisual01` will go into the new `wlm-job` host? Because if that's the case it doesn't see... [15:04:33] 06cloud-services-team, 10Data-Services, 10Toolforge: Urgent: Backup restore request (Toolforge) - https://phabricator.wikimedia.org/T428830#12014618 (10Green_Cardamom) OK. I had local but the rsync --delete of local corrupted deleted remote as well.. i need other backups (and get rid of --delete). I migh... [15:06:12] !log volans@cloudcumin1001 osbpo START - Cookbook wmcs.vps.create_project for project osbpo in eqiad1 (T428939) [15:06:14] volans@cloudcumin1001: Unknown project "osbpo" [15:06:14] T428939: Request creation of osbpo VPS project - https://phabricator.wikimedia.org/T428939 [15:06:34] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of osbpo VPS project - https://phabricator.wikimedia.org/T428939#12014619 (10Volans) Creating the project [15:06:54] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project osbpo [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/316 (https://phabricator.wikimedia.org/T428939) [15:07:53] 06cloud-services-team, 10Cloud-VPS: mwopenstackclients.py reliability - https://phabricator.wikimedia.org/T429003#12014625 (10Andrew) This is a classic case of a trivial glue script that has grown way beyond its intended use. I see two ways to move forward here: 1) Decide that managing our own glue library fo... [15:09:44] (03approved) 10andrew: projects: added project osbpo [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/316 (https://phabricator.wikimedia.org/T428939) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [15:10:08] volans@cloudcumin1001 create_project (PID 3004663) is awaiting input [15:11:52] (03merge) 10volans: projects: added project osbpo [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/316 (https://phabricator.wikimedia.org/T428939) (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [15:13:41] !log volans@cloudcumin1001 osbpo END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project osbpo in eqiad1 (T428939) [15:13:43] volans@cloudcumin1001: Unknown project "osbpo" [15:13:43] T428939: Request creation of osbpo VPS project - https://phabricator.wikimedia.org/T428939 [15:16:06] 06cloud-services-team, 10Cloud-VPS (Project-requests), 13Patch-For-Review: Request creation of osbpo VPS project - https://phabricator.wikimedia.org/T428939#12014638 (10Volans) 05Open→03Resolved a:03Volans Project created. @Andrew resolving this task, feel free to add Zigo when the account will be... [15:29:24] 10Cloud-VPS (Project-requests): Request creation of eduwikihubstaging VPS project - https://phabricator.wikimedia.org/T429032 (10Ederporto) 03NEW [15:57:50] (03open) 10apaskulin: Update link to Lift Wing API docs [toolforge-repos/microtask-generator] - 10https://gitlab.wikimedia.org/toolforge-repos/microtask-generator/-/merge_requests/6 [16:12:28] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of osbpo VPS project - https://phabricator.wikimedia.org/T428939#12014786 (10Andrew) Zigo (username 'zigodebian') is now logged in and getting up to speed. [17:13:20] 10Tool-curator: Port Curator to Bun monorepo - https://phabricator.wikimedia.org/T426415#12014975 (10DaxServer) 05In progress→03Resolved [17:14:37] 10Tool-curator: Curator not available for days - https://phabricator.wikimedia.org/T428874#12014977 (10DaxServer) 05Open→03Resolved p:05Triage→03Unbreak! [17:14:49] 10Tool-curator: Curator: Select only not-uploaded images of a sequence - https://phabricator.wikimedia.org/T426494#12014980 (10DaxServer) p:05Triage→03High [17:14:55] 06cloud-services-team, 10Cloud-VPS: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12014981 (10Volans) Cleanup completed, the next backup run will start in less than 2h, hopefully will run all the way. [17:16:12] 10Tool-curator: Curator: Queuing aborts when internet connection is unstable - https://phabricator.wikimedia.org/T426496#12014983 (10DaxServer) p:05Triage→03High [18:09:57] 10Toolforge, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#12015130 (10BLiviero-WMF) Should this ticket still be open? is there any work here still relevant irrespective of ToM project?