[00:03:04] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.85% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [00:11:18] 06cloud-services-team, 10Toolforge: Your template file contains unknown keys - https://phabricator.wikimedia.org/T417843#11630363 (10Huji) >>! In T417843#11630324, @bd808 wrote: > A service.template matching the intent of the service.manifest format file given would be: > `lang=yaml,name=$HOME/service.temp... [00:19:24] 06cloud-services-team, 10Toolforge: Your template file contains unknown keys - https://phabricator.wikimedia.org/T417843#11630367 (10bd808) >>! In T417843#11630363, @Huji wrote: > So should I edit the file to look like this? The file itself suggests I should not? Yes you should edit your `$HOME/service.te... [00:53:30] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22163899784 (https://github.com/cluebotng/component-configs/commits/0ce51e9bda73cc3ee0df647f7ba8dcfd02eb97e6) [00:53:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [01:02:19] 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11630447 (10Dzahn) 05Open→03In progress [01:02:45] 06cloud-services-team, 06collaboration-services, 06SRE Observability, 13Patch-For-Review: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11630450 (10Dzahn) a:05hnowlan→03Dzahn [04:03:04] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.46% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [06:09:25] 10Tool-curator: Race conditions cause duplicate files to be uploaded - https://phabricator.wikimedia.org/T417682#11630711 (10DaxServer) 05Open→03In progress [08:03:04] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.67% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [09:53:49] 10Wikibugs, 10Gerrit, 06SRE: Wikibugs should ignore `check experimental` messages for operations/puppet - https://phabricator.wikimedia.org/T417866 (10hashar) 03NEW [09:56:37] 10Wikibugs, 10Gerrit, 06SRE: Wikibugs should ignore `check experimental` messages for operations/puppet - https://phabricator.wikimedia.org/T417866#11631180 (10hashar) [12:03:04] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.84% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [12:24:43] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/34 [12:26:36] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1240669 (owner: 10L10n-bot) [12:26:39] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1240670 (owner: 10L10n-bot) [13:12:56] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22183076688 (https://github.com/cluebotng/component-configs/commits/919ebb8860a93b9d071da361cad56448a3b1f2b4) [13:12:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [14:12:41] 06cloud-services-team, 10Data-Services: Query on commonswiki_p takes forever - https://phabricator.wikimedia.org/T417889#11632116 (10fnegri) Probably related to {T415786}: there is currently a big `ALTER TABLE` running on `imagelinks` in the replica clouddb hosts. [14:21:11] 06cloud-services-team, 10Data-Services: Query on commonswiki_p takes forever - https://phabricator.wikimedia.org/T417889#11632142 (10fnegri) 05Open→03Stalled p:05Triage→03Medium Yes, the [patch](https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1222603/7/sql/mysql/patch-imagelinks-pk.sql) is dropping t... [14:21:26] 06cloud-services-team, 10Data-Services: Query on commonswiki_p takes forever - https://phabricator.wikimedia.org/T417889#11632146 (10fnegri) [14:35:31] 10Toolforge (Toolforge iteration 25): [harbor,toolsbeta] for some reason maintain_harbor seems to not be cleaning up toolforge/* images - https://phabricator.wikimedia.org/T417894 (10dcaro) 03NEW [14:43:43] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Toolforge, 13Patch-For-Review: Add tracing to understand Toolforge and CloudVPS usage and dependencies - https://phabricator.wikimedia.org/T399313#11632271 (10fnegri) 05Open→03In progress [14:45:39] 06cloud-services-team, 10Toolforge: Toolforge SSH login: connection closed after publickey authentication - https://phabricator.wikimedia.org/T415239#11632293 (10fnegri) 05In progress→03Invalid Resolving as Invalid. [14:46:03] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [components-api] updated image is not restarted - https://phabricator.wikimedia.org/T417503#11632299 (10fnegri) [14:46:10] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [jobs-api] Allow customizing time to request Loki logs for - https://phabricator.wikimedia.org/T400917#11632301 (10fnegri) [14:59:05] (03update) 10dcaro: runtime: stop calling Image.from_url_or_name in _create_k8s_spec_for_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/266 (https://phabricator.wikimedia.org/T417503) (owner: 10raymond-ndibe) [16:03:04] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 81.05% full for project tools - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [16:23:48] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Epic, 07Essential-Work: OpenAPI linter - performance and dev-ex improvements - https://phabricator.wikimedia.org/T415388#11632817 (10BPirkle) [16:28:25] 10Tool-wsindex, 10Wikisource Reader App, 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11632843 (10Muguro) >>! In T405593#11622185, @Muguro wrote: > **Weekly Internship Report** > > //Week 10: February 9 - February 13// > > > **Overview... [17:01:27] (03update) 10dcaro: runtime: stop calling Image.from_url_or_name in _create_k8s_spec_for_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/266 (https://phabricator.wikimedia.org/T417503) (owner: 10raymond-ndibe) [17:06:21] 10Wikibugs, 10Gerrit, 06SRE: Wikibugs should ignore `check experimental` messages for operations/puppet - https://phabricator.wikimedia.org/T417866#11633002 (10A_smart_kitten) > Wikibugs relay those messages to `#wikimedia-operations` which adds a bit of spam on an already busy enough IRC channel. (cross-li... [18:02:44] (03approved) 10wikigit: Drop specification for /checkuser/v0/temporaryaccount/{name}/revisions/{ids} [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/7 (owner: 10harroyo-wmf) [18:02:49] (03merge) 10wikigit: Drop specification for /checkuser/v0/temporaryaccount/{name}/revisions/{ids} [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/7 (owner: 10harroyo-wmf) [18:07:53] (03approved) 10wikigit: Add Tag Object Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/9 (owner: 10jaredblumer) [18:07:57] (03merge) 10wikigit: Add Tag Object Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/9 (owner: 10jaredblumer) [18:18:15] 10Wikibugs, 10Gerrit, 06SRE: Wikibugs should ignore `check experimental` messages for operations/puppet - https://phabricator.wikimedia.org/T417866#11633432 (10bd808) p:05Triage→03Medium `wikibugs2.gerrit.should_ignore_ci_comment` would be the right place to add filtering logic for this. Here is an exa... [18:19:50] (03open) 10dcaro: functional_tests: added test to check for sha [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1137 (https://phabricator.wikimedia.org/T417503) [18:20:11] (03update) 10dcaro: functional_tests: added test to check for sha [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1137 (https://phabricator.wikimedia.org/T417503) [18:42:13] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (Google Summer of Code (2026)): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11633657 (10Dipanshu1223) Hi @Nokib_Sarkar, I’m a GSoC contributor and was hoping you could share the microtasks link. I’m Dipanshu Vishwakarma — active on Git... [18:47:11] (03update) 10dcaro: runtime: stop calling Image.from_url_or_name in _create_k8s_spec_for_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/266 (https://phabricator.wikimedia.org/T417503) (owner: 10raymond-ndibe) [18:50:06] (03update) 10dcaro: runtime: stop calling Image.from_url_or_name in _create_k8s_spec_for_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/266 (https://phabricator.wikimedia.org/T417503) (owner: 10raymond-ndibe) [18:50:09] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (Google Summer of Code (2026)): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11633670 (10LGoto) @Dipanshu1223 The microtasks are linked in the project description. However, please note that the contribution period does not start until M... [18:50:13] (03update) 10dcaro: runtime: stop calling Image.from_url_or_name in _create_k8s_spec_for_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/266 (https://phabricator.wikimedia.org/T417503) (owner: 10raymond-ndibe) [18:54:22] (03update) 10dcaro: functional_tests: added test to check for sha [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1137 (https://phabricator.wikimedia.org/T417503) [19:04:35] (03open) 10don-vip: Set User-Agent when querying Wikidata [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/28 [19:06:04] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (Google Summer of Code (2026)): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11633772 (10Mostafa_khaled100) ​Hi @Nokib_Sarkar, ​Mustafa here, a React/Next.js developer. I’m really interested in the CampWiz NxT Redesign. ​Since I work he... [19:09:32] (03update) 10dcaro: runtime: stop calling Image.from_url_or_name in _create_k8s_spec_for_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/266 (https://phabricator.wikimedia.org/T417503) (owner: 10raymond-ndibe) [19:20:01] (03update) 10don-vip: Set User-Agent when querying Wikidata [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/28 [19:21:16] (03update) 10dcaro: runtime: stop calling Image.from_url_or_name in _create_k8s_spec_for_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/266 (https://phabricator.wikimedia.org/T417503) (owner: 10raymond-ndibe) [20:08:50] (03merge) 10don-vip: Set User-Agent when querying Wikidata [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/28