[00:53:56] FIRING: ObjectStorageSizeQuotaFull: Object storage quota by 'size' is 100% full for project qrank - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageSizeQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageSizeQuotaFull [00:54:01] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.46% full for project spacemedia - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [01:39:58] (03update) 10raymond-ndibe: [tool-config] handle unset and default arguments consistently [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T402572) [01:54:11] (03update) 10raymond-ndibe: [tool-config] handle unset and default arguments consistently [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T402572) [03:12:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [03:17:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [03:34:33] 10Tool-global-search, 10Discovery-Search (2025.08.15 - 2025.09.05): Global Search displays most search results twice - https://phabricator.wikimedia.org/T391175#11142219 (10MusikAnimal) a:05MusikAnimal→03EBernhardson [04:53:56] FIRING: ObjectStorageSizeQuotaFull: Object storage quota by 'size' is 100% full for project qrank - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageSizeQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageSizeQuotaFull [04:54:01] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.46% full for project spacemedia - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [05:05:43] 10Tool-gawa: [Code] Conception de la page de statistiques - https://phabricator.wikimedia.org/T401767#11142251 (10PenScribe) Pour le TAF de conception (code) de La page de statistiques 2 sont deja terminées : ✅ Des graphes ✅ Une fonctionnalité permettant d'afficher les statistiques par Projet (ex.: Côte d'Iv... [08:02:02] (03update) 10dcaro: registry-admission: bump to 0.0.65-20250902165859-35e89a44 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/944 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:02:19] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [08:13:55] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1183665 (owner: 10L10n-bot) [08:14:00] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [08:17:15] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [08:18:54] 10Tool-gawa: [Code] Conception de la page A PROPOS - https://phabricator.wikimedia.org/T403582 (10poro26) 03NEW [08:19:17] 10Tool-gawa: [Code] Conception de la page A PROPOS - https://phabricator.wikimedia.org/T403582#11142541 (10poro26) a:03poro26 [08:21:35] 10Tool-gawa: [Code] Conception de la page AIDE - https://phabricator.wikimedia.org/T403583 (10poro26) 03NEW [08:23:55] 10Tool-gawa: [Code] Conception de la page STATISTIQUES - https://phabricator.wikimedia.org/T401767#11142561 (10poro26) [08:24:25] 10Tool-gawa: [Code] Conception de la page ACCUEIL - https://phabricator.wikimedia.org/T403438#11142564 (10poro26) [08:26:34] 10Tool-gawa: [Code] Conception de la page EVENEMENTS - https://phabricator.wikimedia.org/T403584 (10poro26) 03NEW [08:28:14] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [08:28:29] 10Tool-gawa: [Documentation] Mise à jour de la description de Tool-gawa - https://phabricator.wikimedia.org/T402289#11142581 (10poro26) [08:29:50] 10Tool-gawa: [Documentation] Mise à jour de la description de Tool-gawa - https://phabricator.wikimedia.org/T402289#11142582 (10poro26) a:05Aklapper→03poro26 [08:30:54] 10Tool-gawa: [Code] Conception de la page ACCUEIL - https://phabricator.wikimedia.org/T403438#11142588 (10poro26) a:05poro26→03Gourebimarc99 [08:34:40] 10Tool-gawa: [Code] Conception de la page EVENEMENTS - https://phabricator.wikimedia.org/T403584#11142608 (10poro26) a:03Gourebimarc99 [08:37:11] 10Tool-gawa: [Code] Conception de la page A PROPOS - https://phabricator.wikimedia.org/T403582#11142627 (10poro26) a:05poro26→03PenScribe [08:39:31] 10Tool-gawa: [Code] Conception de la page AIDE - https://phabricator.wikimedia.org/T403583#11142641 (10poro26) a:05poro26→03Gourebimarc99 [08:45:16] (03approved) 10dcaro: registry-admission: bump to 0.0.65-20250902165859-35e89a44 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/944 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:45:19] (03merge) 10dcaro: registry-admission: bump to 0.0.65-20250902165859-35e89a44 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/944 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:45:25] (03approved) 10dcaro: builds-api: bump to 0.0.198-20250902165321-2b654726 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/950 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:45:30] (03update) 10dcaro: builds-api: bump to 0.0.198-20250902165321-2b654726 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/950 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:45:30] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [08:50:31] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [08:53:28] 10Tools: Spacemedia: object storage quota about to get filled - https://phabricator.wikimedia.org/T403587 (10dcaro) 03NEW [08:54:43] 10Toolforge (Toolforge iteration 24): [harbor,infra] gather stats about object storage qutoa usage and add an alert when tools is getting out of quota - https://phabricator.wikimedia.org/T402932#11142703 (10dcaro) The alerts are up and running: {F65951967} [08:55:02] 10Toolforge (Toolforge iteration 24): [harbor,infra] gather stats about object storage qutoa usage and add an alert when tools is getting out of quota - https://phabricator.wikimedia.org/T402932#11142706 (10dcaro) 05Open→03Resolved a:03dcaro [08:55:40] 10Toolforge (Toolforge iteration 24): [harbor,infra] gather stats about object storage qutoa usage and add an alert when tools is getting out of quota - https://phabricator.wikimedia.org/T402932#11142709 (10dcaro) The dashboard is https://grafana-rw.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-o... [08:55:42] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [09:00:36] 10Tool-spacemedia: Spacemedia: object storage quota about to get filled - https://phabricator.wikimedia.org/T403587#11142747 (10JJMC89) [09:01:08] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [09:05:38] 10Tool-gawa: [Documentation] Rédaction du contenu de la page A PROPOS - https://phabricator.wikimedia.org/T403589 (10poro26) 03NEW [09:09:45] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Enable SSL in Trove MariaDB - Trixie MariaDB client requires SSL but SSL is not enabled in the Trove server - https://phabricator.wikimedia.org/T401861#11142810 (10JJMC89) This also impacts `mariadb-dump` in the Toolforge `mariadb` image. See https://lists.wi... [09:09:55] 10Tool-gawa: [Code] Conception de la page RESULTATS - https://phabricator.wikimedia.org/T403590 (10poro26) 03NEW [09:17:24] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Utility script to get nova server uuid from VM fqdn - https://phabricator.wikimedia.org/T402407#11142856 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Done! [09:19:32] 10Tool-gawa: [Code] Conception de la page ERREUR 404 - https://phabricator.wikimedia.org/T403592 (10poro26) 03NEW [09:28:38] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11142904 (10Tgr) Would be nice to get {T126257} done. Are these a... [09:29:36] (03merge) 10dcaro: builds-api: bump to 0.0.198-20250902165321-2b654726 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/950 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:29:56] (03update) 10dcaro: ingress-admission: bump to 0.0.64-20250902165629-42bf130b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/948 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:30:45] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [09:41:01] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [09:53:23] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11143063 (10Tgr) I haven't verified it but this is my guess of wha... [09:54:50] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11143068 (10Tgr) >>! In T403519#11143063, @Tgr wrote: > PHPSession... [09:59:17] (03open) 10alvindulle: Configure Secret Detection in `.gitlab-ci.yml`, creating this file if it does not already exist [toolforge-repos/wikievents] - 10https://gitlab.wikimedia.org/toolforge-repos/wikievents/-/merge_requests/1 [09:59:41] 10Wikibugs, 10Phabricator: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#11143080 (10Aklapper) p:05Triage→03Low [10:04:56] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [10:12:04] 10Cloud-VPS (Quota-requests), 10Tool-spacemedia: Increase object storage quota for project spacemedia - https://phabricator.wikimedia.org/T386588#11143123 (10Don-vip) [10:16:06] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [10:19:13] 10Cloud-VPS (Quota-requests), 10Tool-spacemedia: Increase object storage quota for project spacemedia - https://phabricator.wikimedia.org/T403602 (10Don-vip) 03NEW [10:20:16] 10Tool-spacemedia: Spacemedia: object storage quota about to get filled - https://phabricator.wikimedia.org/T403587#11143185 (10Don-vip) Thank you David! I've requested a quota increase in T403602. I'm surprised to not be able to see this information in Horizon, is it planned in a future version? [10:44:18] 10Cloud-VPS (Quota-requests), 10Tool-spacemedia: Increase object storage quota for project spacemedia - https://phabricator.wikimedia.org/T403602#11143301 (10dcaro) LGTM +1 [10:49:58] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10Wikilink-Tool: wikilink: Increase cpu + ram + disk quotas - https://phabricator.wikimedia.org/T403609 (10jsn.sherman) 03NEW [10:53:09] 06cloud-services-team, 10Data-Services, 10BetaFeatures, 06Data-Engineering: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11143335 (10Ladsgroup) This is not DBA work. I stay subscribed if you still need one. [11:17:47] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [11:27:24] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11143423 (10LucasWerkmeister) >>! In T403519#11142904, @Tgr wrote:... [11:31:35] (03approved) 10dcaro: ingress-admission: bump to 0.0.64-20250902165629-42bf130b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/948 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:31:37] (03merge) 10dcaro: ingress-admission: bump to 0.0.64-20250902165629-42bf130b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/948 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:32:08] (03update) 10dcaro: volume-admission: bump to 0.0.71-20250902165439-8937b04a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/947 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:32:24] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [11:33:35] (03merge) 10dcaro: api: add `include_unset` parameter to get_job and get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/205 (https://phabricator.wikimedia.org/T402569) [11:34:45] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [11:37:00] 10Tool-spacemedia: Spacemedia: object storage quota about to get filled - https://phabricator.wikimedia.org/T403587#11143445 (10dcaro) > Thank you David! I've requested a quota increase in T403602. I'm surprised to not be able to see this information in Horizon, is it planned in a future version? Horizon is a b... [11:37:14] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [11:38:11] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.411-20250903113356-7dfdec3e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/952 (https://phabricator.wikimedia.org/T402569) [11:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:39:52] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [11:40:39] (03approved) 10dcaro: volume-admission: bump to 0.0.71-20250902165439-8937b04a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/947 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:40:42] (03merge) 10dcaro: volume-admission: bump to 0.0.71-20250902165439-8937b04a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/947 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:40:48] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [11:58:19] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [12:00:32] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [12:15:37] 06cloud-services-team, 10Cloud-VPS: Complete upgrading WMCS bare metal hosts to Trixie - https://phabricator.wikimedia.org/T375217#11143576 (10fgiunchedi) [12:20:24] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [12:22:22] (03approved) 10dcaro: maintain-kubeusers: bump to 0.0.180-20250902162127-0f463217 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/951 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:22:26] (03update) 10dcaro: maintain-kubeusers: bump to 0.0.180-20250902162127-0f463217 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/951 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:22:42] (03update) 10dcaro: envvars-api: bump to 0.0.73-20250901160226-77b2b741 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/949 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:22:44] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [12:22:53] (03merge) 10dcaro: maintain-kubeusers: bump to 0.0.180-20250902162127-0f463217 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/951 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:23:53] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [12:23:56] 06cloud-services-team, 10Cloud-VPS: Complete upgrading WMCS bare metal hosts to Trixie - https://phabricator.wikimedia.org/T375217#11143585 (10fgiunchedi) [12:26:27] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [12:26:58] (03close) 10raymond-ndibe: [tool_router.py] exclude default values from deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/117 (https://phabricator.wikimedia.org/T401648) [12:27:50] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [12:27:58] (03update) 10raymond-ndibe: [tool-config] handle unset and default arguments consistently [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) [12:33:31] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Epic, 10Moderator-Tools-Team (Kanban): hashtags: Replace deprecated Bullseye VM in Cloud VPS - https://phabricator.wikimedia.org/T402056#11143613 (10jsn.sherman) Initial documentation added: https://www.mediawiki.org/wiki/Moderator_Tools... [12:34:37] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Reimage cloudgw hosts to Trixie - https://phabricator.wikimedia.org/T401899#11143617 (10fgiunchedi) I got a clean puppet run on Pontoon with the patches above (and some minor local hacks) for `wmcs::cloudgw` on Trixie. That's not to say everything will... [12:38:31] 06cloud-services-team: PuppetDisabled - https://phabricator.wikimedia.org/T403517#11143636 (10fgiunchedi) p:05Triage→03Low [12:38:39] 06cloud-services-team: PuppetDisabled Puppet disabled on cloudcontrol2005-dev:9100 - https://phabricator.wikimedia.org/T403515#11143643 (10fgiunchedi) p:05Triage→03Low [12:40:47] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [12:42:42] 06cloud-services-team: PuppetDisabled Puppet disabled on cloudcontrol2005-dev:9100 - https://phabricator.wikimedia.org/T403515#11143679 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi no longer true, puppet has been reenabled since ` 2025-09-02T19:24:19.832115+00:00 cloudcontrol2005-dev puppet-agent[4... [12:43:19] 06cloud-services-team: PuppetDisabled - https://phabricator.wikimedia.org/T403517#11143684 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi No longer true, puppet has been reenabled ` 2025-09-02T19:24:19.832115+00:00 cloudcontrol2005-dev puppet-agent[4101022]: Skipping run of Puppet configuration clien... [12:44:01] (03update) 10raymond-ndibe: [cli] add tool config to deployment object [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/58 (https://phabricator.wikimedia.org/T400064) [12:44:03] (03update) 10raymond-ndibe: [cli] add tool config to deployment object [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/58 (https://phabricator.wikimedia.org/T400064) [12:47:10] 06cloud-services-team, 10Cloud-VPS: CloudVPS: create a wrapper script to ease virsh console to VMs - https://phabricator.wikimedia.org/T235061#11143690 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi We have `wmcs.openstack.cloudvirt.vm_console` cookbook now [12:50:27] (03update) 10raymond-ndibe: [config] support port protocol [repos/cloud/toolforge/components-api] (handle_unset_and_default_arguments_consistently) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/119 (https://phabricator.wikimedia.org/T401994) [12:50:38] 06cloud-services-team: Investigate feasibility of Pagerduty integration for Cloud Services - https://phabricator.wikimedia.org/T206145#11143701 (10fgiunchedi) 05Open→03Declined Boldly declining, I doubt Pagerduty road is going anywhere at this point [12:52:53] 06cloud-services-team, 10Cloud-VPS: Cloud VPS: X11 forwarding request failed on channel 0 - https://phabricator.wikimedia.org/T196537#11143720 (10fgiunchedi) 05Open→03Invalid Boldly resolving, feel free to reopen if there's any action on this [12:55:03] 06cloud-services-team, 10Tool-admin: Toolforge Tools listing is now very slow - https://phabricator.wikimedia.org/T198948#11143727 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is loading fast for me nowadays: https://toolhub.wikimedia.org/search?ordering=-modified_date boldly resolving [12:55:49] (03update) 10raymond-ndibe: pre-commit: add check for openapi spec version bump [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/116 (owner: 10dcaro) [12:57:21] (03approved) 10raymond-ndibe: gitlab: move to bookworm images [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/15 (owner: 10dcaro) [12:58:09] (03approved) 10raymond-ndibe: gitlab: move to bookworm images [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/86 (owner: 10dcaro) [12:58:23] (03update) 10raymond-ndibe: gitlab: move to bookworm images [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/86 (owner: 10dcaro) [12:59:00] 06cloud-services-team, 06serviceops, 06SRE: hosts failing puppet compile due to missing secrets - https://phabricator.wikimedia.org/T274392#11143753 (10fgiunchedi) 05Open→03Invalid The drift between public and private private.git will continue to be an issue until we get serious about secrets managem... [13:00:25] 06cloud-services-team, 10Data-Services: Have a cron job delete files that haven't been modified in the last X days / months in /data/scratch - https://phabricator.wikimedia.org/T103084#11143759 (10fgiunchedi) 05Open→03Declined Boldly declining since it doesn't seem to have happened in ten years [13:01:18] (03update) 10raymond-ndibe: logs: use logs-api for logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/121 (owner: 10dcaro) [13:01:55] (03update) 10raymond-ndibe: pre-commit: add golangci-lint check [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/63 (owner: 10dcaro) [13:01:56] (03approved) 10raymond-ndibe: pre-commit: add golangci-lint check [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/63 (owner: 10dcaro) [13:03:34] (03update) 10raymond-ndibe: k8s: upgrade to 1.30 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/241 (https://phabricator.wikimedia.org/T362869) (owner: 10dcaro) [13:04:00] 06cloud-services-team, 10Cloud-VPS, 10observability: Monitor nova-scheduler log for lost contact with compute nodes - https://phabricator.wikimedia.org/T90784#11143801 (10fgiunchedi) 05Open→03Invalid Doesn't look like there's any action to do on this nowadays, boldly resolving [13:06:58] 06cloud-services-team, 10Cloud-VPS: Make sure that the 'secret' repo in self hosted puppetmasters is back-upable - https://phabricator.wikimedia.org/T115177#11143820 (10fgiunchedi) 05Open→03Declined Boldly declining since we haven't done this (nor we should I think), see also https://phabricator.wikime... [13:09:43] 06cloud-services-team, 06Infrastructure-Foundations, 10Puppet-Core, 13Patch-For-Review: puppet: profile::auto_restarts::service: have a way to don't deploy the systemd timers - https://phabricator.wikimedia.org/T336845#11143851 (10fgiunchedi) 05Open→03Declined As far as I know this hasn't caused pr... [13:11:01] 06cloud-services-team, 10Cloud-VPS: systemctl restart keystone doesn't actually restart keystone sometimes - https://phabricator.wikimedia.org/T340127#11143863 (10fgiunchedi) 05Open→03Invalid AFAIK this hasn't been a problem in recent past, boldly resolving and we can reopen as needed [13:16:22] 06cloud-services-team, 10Cloud-VPS, 06Data-Engineering, 06Data-Engineering-Icebox: Report page views for labs instances - https://phabricator.wikimedia.org/T103726#11143884 (10fgiunchedi) 05Open→03Declined Boldly declining, I don't think there's any effort or plan behind this, feel free to reopen a... [13:18:57] (03open) 10raymond-ndibe: [tool home dir] revert change in dir permission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/271 (https://phabricator.wikimedia.org/T403513) [13:19:06] (03update) 10raymond-ndibe: [tool home dir] revert change in dir permission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/271 (https://phabricator.wikimedia.org/T403513) [13:20:22] (03update) 10raymond-ndibe: [tool home dir] revert change in dir permission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/271 (https://phabricator.wikimedia.org/T403513) [13:20:51] (03update) 10raymond-ndibe: [tool home dir] revert change in dir permission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/271 (https://phabricator.wikimedia.org/T403513) [13:27:17] (03approved) 10dcaro: envvars-api: bump to 0.0.73-20250901160226-77b2b741 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/949 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:27:17] 10Cloud-Services, 06SRE, 13Patch-For-Review: Backport sshd with AuthorizedKeysCommand support to Ubuntu precise - https://phabricator.wikimedia.org/T102401#11143917 (10fgiunchedi) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org... [13:27:21] (03update) 10dcaro: envvars-api: bump to 0.0.73-20250901160226-77b2b741 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/949 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:27:42] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [13:28:12] (03merge) 10dcaro: envvars-api: bump to 0.0.73-20250901160226-77b2b741 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/949 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:30:10] 10Quarry: Can't access database in Quarry - https://phabricator.wikimedia.org/T181411#11143928 (10fnegri) 05Invalid→03Resolved This is now working, since {T151158} was resolved last year. Example: https://quarry.wmcloud.org/query/96900 [13:32:03] 06cloud-services-team, 10wikitech.wikimedia.org: improve sync process to wikitech-static - https://phabricator.wikimedia.org/T225621#11143948 (10fgiunchedi) 05Open→03Invalid I believe with our current process (i.e. crawl wikitech to generate a wikitech-static container) this is no longer an issue [13:32:49] 06cloud-services-team, 10wikitech.wikimedia.org, 07Security: wikitech-static pages not being deleted after wikitech pages are - https://phabricator.wikimedia.org/T128548#11143968 (10sbassett) [13:37:46] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [13:39:15] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [13:46:54] 10Cloud-VPS (Quota-requests), 10Tool-spacemedia: Increase object storage quota for project spacemedia - https://phabricator.wikimedia.org/T403602#11144025 (10dcaro) a:03fgiunchedi [13:48:21] 10Cloud-VPS (Quota-requests), 10Tool-spacemedia: Increase object storage quota for project spacemedia - https://phabricator.wikimedia.org/T403602#11144032 (10fgiunchedi) 05Open→03Resolved {{done}} ` root@cloudcontrol1006:~# radosgw-admin quota set --quota-scope=user --uid=spacemedia\$spacemedia --max-... [13:49:28] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [13:58:39] 06cloud-services-team, 10Cloud-VPS, 07Documentation: WMCS VPS docs should describe how to add users to a project - https://phabricator.wikimedia.org/T167816#11144060 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_project#Managing_a_project seems g... [14:01:22] (03approved) 10dcaro: envvars-admission: bump to 0.0.32-20250901160601-7995b656 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/946 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:01:26] (03update) 10dcaro: envvars-admission: bump to 0.0.32-20250901160601-7995b656 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/946 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:01:48] (03merge) 10dcaro: envvars-admission: bump to 0.0.32-20250901160601-7995b656 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/946 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:01:55] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [14:01:55] (03update) 10dcaro: maintain-harbor: bump to 0.0.59-20250901160048-fc2cd60d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/939 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:04:02] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [14:09:41] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [14:11:21] 06cloud-services-team, 10Cloud-VPS: Create a method for forwarding non-HTTP protocols to VMs without public IPs - https://phabricator.wikimedia.org/T178133#11144081 (10fgiunchedi) 05Open→03Declined I'm boldly declining since I don't think we'll be implementing this (nor we should) and OP has a working... [14:13:20] 06cloud-services-team, 10Cloud-VPS: CamelCase vs. VPS instance naming - https://phabricator.wikimedia.org/T176757#11144086 (10fgiunchedi) 05Open→03Declined I doubt we'll be working on this, boldly declining [14:13:43] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [14:25:13] 06cloud-services-team, 06serviceops, 06SRE: hosts failing puppet compile due to missing secrets - https://phabricator.wikimedia.org/T274392#11144118 (10Dzahn) It was just about a social contract to not forget adding fake secrets when adding real secrets. [14:26:53] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06DBA, 13Patch-For-Review: Make watchlist table available as curated foo_p.watchlist_count on labsdb - https://phabricator.wikimedia.org/T59617#11144124 (10fnegri) [14:28:08] (03approved) 10dcaro: maintain-harbor: bump to 0.0.59-20250901160048-fc2cd60d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/939 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:28:10] (03merge) 10dcaro: maintain-harbor: bump to 0.0.59-20250901160048-fc2cd60d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/939 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:28:25] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [14:30:05] (03approved) 10dcaro: dns: Reference floating IP module resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/71 (owner: 10taavi) [14:32:47] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [14:41:31] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [14:46:41] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [14:53:22] (03approved) 10dcaro: builds-builder: bump to 0.0.133-20250901160118-b35bfe0a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/938 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:53:25] (03update) 10dcaro: builds-builder: bump to 0.0.133-20250901160118-b35bfe0a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/938 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:53:47] (03merge) 10dcaro: builds-builder: bump to 0.0.133-20250901160118-b35bfe0a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/938 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:59:44] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:09:39] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [15:12:09] (03update) 10dcaro: jobs-api: bump to 0.0.411-20250903113356-7dfdec3e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/952 (https://phabricator.wikimedia.org/T402569) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:13:02] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:25:22] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:27:11] 06cloud-services-team, 10wikitech.wikimedia.org: improve sync process to wikitech-static - https://phabricator.wikimedia.org/T225621#11144343 (10bd808) >>! In T225621#11143948, @fgiunchedi wrote: > I believe with our current process (i.e. crawl wikitech to generate a wikitech-static container) this is no l... [15:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:07:30] (03open) 10dcaro: d/changelog: bump to 16.1.19 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/126 (https://phabricator.wikimedia.org/T402569) [16:39:20] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [16:41:04] 06cloud-services-team, 10Data-Services: [wikireplicas] slow query runs every hour, but never completes - https://phabricator.wikimedia.org/T403639 (10fnegri) 03NEW [16:47:30] 10VPS-project-Codesearch: Move Codesearch behind OAuth login - https://phabricator.wikimedia.org/T403474#11144791 (10Dzahn) imho the real fix should be in the direction of somehow unifying the approaches used in production and in cloud VPS/toolforge to how we deal with this problem. Because basically every sing... [16:49:04] 10VPS-project-Codesearch: Move Codesearch behind OAuth login - https://phabricator.wikimedia.org/T403474#11144804 (10Dzahn) (the other real fix would be to move tools that we rely on and aren't just testing to actual production.. fully aware that is easier said than done but imho that we basically don't have th... [16:49:44] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [16:50:37] (03approved) 10dcaro: toolsbeta: dns: Add MX records for toolsbeta.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/73 (https://phabricator.wikimedia.org/T394997) (owner: 10taavi) [16:50:59] 10VPS-project-Codesearch: Codesearch down/unreachable (2025-09-02) - https://phabricator.wikimedia.org/T403434#11144816 (10Dzahn) >>! In T403434#11138318, @A_smart_kitten wrote: > Seems like this might now be resolved. Unfortunately, this seems to be a recurring issue as of recently… :[ Yea, basically the A... [16:51:25] (03approved) 10dcaro: dns: Unify emails in SOA records [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/74 (owner: 10taavi) [16:51:44] (03approved) 10dcaro: tools: dns: Drop temporary migration names [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/75 (owner: 10taavi) [16:53:25] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [16:55:39] 06cloud-services-team, 10Cloud-VPS, 06Data-Engineering, 06Data-Engineering-Icebox: Report page views for labs instances - https://phabricator.wikimedia.org/T103726#11144832 (10bd808) It would be reasonably possible to implement this in roughly the same way that https://toolviews.toolforge.org/ works. T... [17:02:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [17:02:54] (03approved) 10dcaro: d/changelog: bump to 16.1.19 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/126 (https://phabricator.wikimedia.org/T402569) [17:03:00] (03merge) 10dcaro: d/changelog: bump to 16.1.19 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/126 (https://phabricator.wikimedia.org/T402569) [17:04:38] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [17:18:48] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [17:20:58] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [17:22:00] 06cloud-services-team, 10Toolforge: [components-api] restart rather than delete/create continuous jobs - https://phabricator.wikimedia.org/T403321#11144934 (10dcaro) Yep, this is a "know" issue on the jobs-api side, thanks for opening the task, will help keep track of it :) [17:25:11] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:34:15] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [17:35:34] (03approved) 10dcaro: jobs-api: bump to 0.0.411-20250903113356-7dfdec3e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/952 (https://phabricator.wikimedia.org/T402569) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:35:38] (03merge) 10dcaro: jobs-api: bump to 0.0.411-20250903113356-7dfdec3e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/952 (https://phabricator.wikimedia.org/T402569) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:21:35] 06cloud-services-team, 10Toolforge: [components-api] restart rather than delete/create continuous jobs - https://phabricator.wikimedia.org/T403321#11145446 (10DamianZaremba) I think the more correct thing in components-api would be to produce a changed image (based on the built digest - T403322) and let jobs-a... [20:41:12] 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T388928#11145747 (10LibUp-bot) A new upstream version of OpenRefine is now available: 3.9.5. * https://github.com/OpenRefine/OpenRefine/releases/tag/3.9.5 [21:08:42] !log bd808@cloudcumin1001 wikitextexp START - Cookbook wmcs.openstack.quota_increase (T403114) [21:08:47] T403114: Quote increase request for wikitextexp - https://phabricator.wikimedia.org/T403114 [21:08:49] !log bd808@cloudcumin1001 wikitextexp END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T403114) [21:10:52] 10Cloud-VPS (Quota-requests), 10Content-Transform-Team (Work In Progress): Quote increase request for wikitextexp - https://phabricator.wikimedia.org/T403114#11145834 (10bd808) @ssastry was not able to make a 600GB volume even though the quota looked like it was available. He and I both were being capped a... [21:12:07] 10Cloud-VPS (Quota-requests), 10Content-Transform-Team (Work In Progress): Quote increase request for wikitextexp - https://phabricator.wikimedia.org/T403114#11145839 (10bd808) Hmmm... still getting this error: `counterexample Error: Unable to create volume. Details VolumeSizeExceedsLimit: Requested volum... [21:22:34] 10Cloud-VPS (Quota-requests), 10Content-Transform-Team (Work In Progress): Quote increase request for wikitextexp - https://phabricator.wikimedia.org/T403114#11145853 (10ssastry) Thanks for trying. For now, I have a good enough workaround. Even though the existing parsing volume attached to ctt-qa-03 on... [23:00:48] 10VPS-Projects, 10Content-Transform-Team (Work In Progress), 07Essential-Work: Set up new cloud VPS server for Content Transform Team Visual Diff testing - https://phabricator.wikimedia.org/T402836#11146111 (10ssastry)