[09:19:04] 10GitLab (Infrastructure), 10ChangeProp, 06cloud-services-team, 06collaboration-services, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#10882980 (10hashar) 05Open→03Resolved a:03hashar After chatting with Alexandr... [12:48:26] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10883699 (10Jelto) Total number of artifacts was reduced from 400k to around 100k in T395014. I enabled o... [12:48:40] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10883700 (10Jelto) [13:08:06] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Check GitLab artifact retention time - https://phabricator.wikimedia.org/T395014#10883769 (10Jelto) The cleanup of old artifacts is done, the number of artifacts was reduced from 400k to around 100k. With `... [13:24:15] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018 (10Jelto) 03NEW [13:32:50] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10883933 (10Jelto) p:05Triage→03High The migration of objects is running currently. 25k of the 100k artifacts are migrated already and this... [13:49:26] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10884065 (10Jelto) I stopped the migration, the expected time would be Wed 04 Jun 2025 14:54:15 UTC (so over an hour). I re-tried the failing j... [14:03:23] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10884137 (10Jelto) Unfortunately migrating back to local storage is even slower, estimated date is Wed 04 Jun 2025 17:07:15 UTC. So GitLab CI ar... [14:53:48] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10884348 (10Jelto) Migration is still running, estimated date is 17:07 UTC. Logstash dashboard with some of the 500 requests: https://logstash.... [15:04:10] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10884399 (10Jelto) In https://docs.gitlab.com/administration/object_storage/ I found the information that `host` parameter is deprecated in `git... [15:20:13] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10884482 (10Jelto) I tried different combinations of `endpoint`, `enable_signature_v4_streaming` and `path_style` with no success. I'll continue... [18:38:37] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10885455 (10Jelto) The migration is still running [18:50:37] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10885474 (10Jelto) p:05High→03Medium I disabled object storage again and the issue is fixed. I'll follow up tomorrow a... [19:08:37] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upload of new GitLab ci artifacts fails - https://phabricator.wikimedia.org/T396018#10885527 (10Jdforrester-WMF) Thanks, confirm this has fixed CI / merges for us.