[03:45:33] the lua block looks so much better but i did get it working finally. it's not pretty [03:46:20] https://gitlab.wikimedia.org/dduvall/gitlab-buildkitd-eval/-/blob/main/registry.yaml#L97 [03:56:17] i have realized i don't have enough brain left for understanding this tonight, but nice work [06:44:15] it's slightly more sane now. time to call it quits again [06:45:57] go to bed dan :) [06:54:49] ok :) [06:59:47] i should probably take my own advice [18:30:05] t.hcipriani's doing a state-of-gitlab talk in the infrastructure tools room of the hackathon, if anybody's interested: https://play.workadventu.re/@/wmf/hackathon2022/map [18:31:49] ah, just in time when I had left the room. thanks for the reminder [19:13:39] thanks for saying smart stuff as usual, ya'll [21:38:34] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host gitlab1004.wikimedia.org with OS bullseye [22:06:57] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host gitlab1004.wikimedia.org with OS bullseye completed: - git... [23:10:42] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) >>! In T307142#7941573, @Jelto wrote: > So I'll create a dedicated partman config for GitLab physical hosts, which has a bigger root partition,... [23:12:05] thcipriani: brennen: we have /srv/gitlab-backup and /srv/registry and 425G free on / on gitlab1004. I tried J.elto's partman recipe and it works [23:12:24] (no dedicated docker volume though?) [23:12:52] but the Gerrit patch did not claim it would make one, works as intended [23:13:04] \o/ [23:13:12] is MUCH larger than on gitlab1003 [23:13:16] / is MUCH larger than on gitlab1003 [23:14:06] do we need a docker volume for non-runners? [23:15:11] I am not really sure. I am saying it this way because on Phabricator there was " has a bigger root partition, dedicated volumes for docker, backups and if possible also the registry" [23:15:28] but on the Gerrit patch there was "bigger root volume and additional [23:15:30] volumes for backups and registry" [23:15:37] but it did not mention docker [23:16:02] probably not and that is just for runners [23:17:07] I'm honestly not sure where we stand after a week and change of gitlab-a-thon, so I'll defer to the releng crew, but tentatively this seems fine. A good win that the partman recipe worked! [23:17:19] to answer your question on the email thread though.. reimaging physical hosts is easier than creating more VMs [23:17:22] now [23:17:39] that's a great answer :) [23:18:19] I'll check about the docker volume on Monday as well. [23:18:36] and the we reimage gitlab1003 [23:21:10] does this still mean roughly 2nd week of June as a timeline? [23:22:11] or does verifying the partman recipe give you more confidence? [23:27:35] I don't want to speak for Jelto. but I think this removes the "messing around with the disk and the filesystem on a production instance" part. if gitlab1003 can replace gitlab1001 right away we won't mess with VMs [23:28:03] and being bold and just merging it now ..instead of waiting for reviews next week was an attempt to save us a couple days [23:28:36] so maybe you can subtract 2 days from the estimate [23:30:47] gotcha. I appreciate the boldness :D [23:36:40] :) happy weekend for now