[00:56:05] 10GitLab (Project Migration), 10Abstract Wikipedia team, 10function-orchestrator, 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): Move the mediawiki/services/function-orchestrator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334397 (10Jdforrester-WMF) 05I... [00:56:36] 10GitLab (Project Migration), 10API Platform, 10Anti-Harassment, 10Cloud-Services, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10Jdforrester-WMF) [01:13:47] 10GitLab (Project Migration), 10Abstract Wikipedia team, 10function-evaluator, 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): Move the mediawiki/services/function-evaluator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334398 (10Jdforrester-WMF) OK, other... [12:57:05] 10GitLab (Infrastructure), 10serviceops-collab: Troubleshoot partman config for two additional disks on GitLab hosts - https://phabricator.wikimedia.org/T333674 (10Jelto) Ok thanks @MoritzMuehlenhoff for the context. I'll add a short section in wikitech about how to bootstrap a new instance and the new script.... [18:02:46] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Jelto) [18:10:56] 10GitLab (Infrastructure), 10serviceops-collab: Troubleshoot partman config for two additional disks on GitLab hosts - https://phabricator.wikimedia.org/T333674 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin2002 for host gitlab1003.wikimedia.org with OS bullseye [18:17:06] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) a:03Dzahn [18:50:57] 10GitLab (Infrastructure), 10serviceops-collab: Troubleshoot partman config for two additional disks on GitLab hosts - https://phabricator.wikimedia.org/T333674 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin2002 for host gitlab1003.wikimedia.org with OS bullseye completed: -... [19:48:53] 10GitLab (Infrastructure), 10serviceops-collab: Troubleshoot partman config for two additional disks on GitLab hosts - https://phabricator.wikimedia.org/T333674 (10Jelto) Reimage of `gitlab1003` went fine. The disk names changed again during the reimage (`sda` and `sdc` old disk, `sdb` and `sdd` new disks). Ho... [21:08:22] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) instance name: gitlab-prod-1002.devtools.eqiad1.wikimedia.cloud [21:09:56] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) We should already have this fully automated: ` 66 if $use_acmechief { 67 # Certificates will be available under... [21:11:43] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) ` Apr 20 05:11:03 gitlab-prod-1002 certbot[5595]: Challenge failed for domain gitlab.devtools.wmcloud.org Apr 20 05:11:04 gitl... [21:24:54] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) So.. there seems to be a more general issue here.. since I can't access https://gitlab-prod.wmflabs.org/ (the domain proxy poi... [21:28:36] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) Locally I can talk to nginx on the same IP, on port 80: ` curl 172.16.3.213 Welcome to... [21:30:02] <wikibugs> 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) nginx is instead listening on 172.16.7.146:443 tcp 0 0 172.16.7.146:443 0.0.0.0:* LISTEN... [21:39:11] <wikibugs> 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) I was going to experiment with creating a new proxy, use port 80, us the other IP etc, but I am also getting errors from the p... [23:54:46] <wikibugs> 10GitLab (Project Migration), 10Abstract Wikipedia team, 10function-evaluator, 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): Move the mediawiki/services/function-evaluator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334398 (10Jdforrester-WMF) [23:55:03] <wikibugs> 10GitLab (Project Migration), 10Abstract Wikipedia team, 10function-evaluator, 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): Move the mediawiki/services/function-evaluator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334398 (10Jdforrester-WMF) [23:55:05] <wikibugs> 10GitLab (Project Migration), 10Abstract Wikipedia team, 10function-orchestrator, 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): Move the mediawiki/services/function-orchestrator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334397 (10Jdforrester-WMF)