[03:11:07] 10Continuous-Integration-Config: add Ameisenigel to CI allowlist - https://phabricator.wikimedia.org/T439225#12377658 (10HideonRosie) 05Open→03Resolved [05:58:20] (03PS3) 10Hashar: Provide fresh-node26 [fresh] - 10https://gerrit.wikimedia.org/r/1344729 [05:58:20] (03CR) 10Hashar: "After a night of sleep, I think we can provide fresh-node26 immediately, cut a new release of Fresh and I'll do the refactoring later on. " [fresh] - 10https://gerrit.wikimedia.org/r/1344729 (owner: 10Hashar) [06:40:05] (03open) 10phedenskog: core: Compare tiles with the previous 30 days [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/93 [06:45:48] (03merge) 10phedenskog: core: Compare tiles with the previous 30 days [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/93 [07:09:17] (03PS1) 10Hashar: Prepare 26.09.1 release [fresh] - 10https://gerrit.wikimedia.org/r/1346371 [07:17:39] (03PS4) 10Hashar: Provide fresh-node26 [fresh] - 10https://gerrit.wikimedia.org/r/1344729 [07:17:39] (03PS2) 10Hashar: Prepare 26.09.1 release [fresh] - 10https://gerrit.wikimedia.org/r/1346371 [07:20:26] (03PS3) 10Hashar: Prepare 26.09.1 release [fresh] - 10https://gerrit.wikimedia.org/r/1346371 [07:21:48] (03PS1) 10Hashar: Release 26.09.1 [fresh] - 10https://gerrit.wikimedia.org/r/1346376 [07:24:17] (03CR) 10Hashar: "Fixed: I forgot to add node26 to the installer!" [fresh] - 10https://gerrit.wikimedia.org/r/1344729 (owner: 10Hashar) [07:49:59] (03CR) 10Phedenskog: [C:03+2] Provide fresh-node26 [fresh] - 10https://gerrit.wikimedia.org/r/1344729 (owner: 10Hashar) [08:18:36] (03Merged) 10jenkins-bot: Provide fresh-node26 [fresh] - 10https://gerrit.wikimedia.org/r/1344729 (owner: 10Hashar) [08:18:42] 06Release-Engineering-Team, 06Collaboration-Services, 06SRE, 07ci-test-error: Builds fail on integration-agent-docker-1086 due to expired snapshot.debian.org Release file - https://phabricator.wikimedia.org/T439529#12378071 (10hashar) p:05Triage→03High I have checked the erroring build https://integrat... [08:49:49] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Collaboration-Services, 06SRE, 07ci-test-error: Builds fail on integration-agent-docker-1086 due to expired snapshot.debian.org Release file - https://phabricator.wikimedia.org/T439529#12378271 (10hashar) The `operations/alerts` repos... [08:57:08] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Collaboration-Services, 06SRE, 07ci-test-error: Builds fail on integration-agent-docker-1086 due to expired snapshot.debian.org Release file - https://phabricator.wikimedia.org/T439529#12378293 (10hashar) As an immediate fix I have de... [09:11:56] 10Gerrit, 10phan, 07Upstream: Expose Phan results in Gerrit web UI - https://phabricator.wikimedia.org/T439642#12378346 (10hashar) p:05Triage→03Low [11:11:05] (03open) 10phedenskog: ci-load: Show where builds start and the executor capacity [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/94 [11:11:33] (03merge) 10phedenskog: ci-load: Show where builds start and the executor capacity [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/94 [11:26:11] (03open) 10taavi: Add repos/sre/hiddenparma [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/194 (https://phabricator.wikimedia.org/T439699) [11:27:52] (03update) 10jelto: Draft: digitaocean: use new vpc, egress gateway and isolated_workers in staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/633 (https://phabricator.wikimedia.org/T425441) [11:29:15] (03update) 10jelto: Draft: digitaocean: use new vpc, egress gateway and isolated_workers in staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/633 (https://phabricator.wikimedia.org/T425441) [11:52:52] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Collaboration-Services, 06SRE, 07ci-test-error: Builds fail on integration-agent-docker-1086 due to expired snapshot.debian.org Release file - https://phabricator.wikimedia.org/T439529#12379009 (10MoritzMuehlenhoff) >>! In T439529#123... [11:54:03] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Collaboration-Services, 06SRE, 07ci-test-error: Builds fail on integration-agent-docker-1086 due to expired snapshot.debian.org Release file - https://phabricator.wikimedia.org/T439529#12379010 (10MoritzMuehlenhoff) >>! In T439529#123... [12:21:33] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Cinder: Migrate Wiki repositories pool to Cinder volume - https://phabricator.wikimedia.org/T439711 (10jnuche) 03NEW [12:21:51] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Cinder: Migrate Wiki repositories pool to Cinder volume - https://phabricator.wikimedia.org/T439711#12379152 (10jnuche) [12:23:51] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Cinder: Migrate Wiki repositories pool to Cinder volume - https://phabricator.wikimedia.org/T439711#12379157 (10jnuche) [12:25:12] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Cinder: Migrate Wiki repositories pool to Cinder volume - https://phabricator.wikimedia.org/T439711#12379162 (10jnuche) [12:54:36] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 06Machine-Learning-Team, and 4 others: Set an identifiable User-Agent on WMF tools calling gitlab.wikimedia.org - https://phabricator.wikimedia.org/T437216#12379385 (10Isaac) Just making an acknowledgement from the Research side -- man... [12:57:40] HI THERE [13:07:39] (03open) 10phedenskog: ci-load: Show third builds that could go to a less busy agent [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/95 [13:10:48] (03merge) 10phedenskog: ci-load: Show third builds that could go to a less busy agent [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/95 [13:38:52] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Cinder: Migrate Wiki repositories pool to Cinder volume - https://phabricator.wikimedia.org/T439711#12379655 (10jnuche) [14:25:37] 10Scap, 06SRE Observability: Scap's logstash checker for channels outside "error" and "exception" - https://phabricator.wikimedia.org/T438570#12380054 (10thcipriani) >>! In T438570#12355090, @herron wrote: > I would say `tags: *throttle_warning*` is a reasonable query for the time being, if logstash is on the... [14:36:04] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:48:40] (03merge) 10dancy: Add repos/sre/hiddenparma [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/194 (https://phabricator.wikimedia.org/T439699) (owner: 10taavi) [14:52:40] 06Release-Engineering-Team, 13Patch-For-Review: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419#12380192 (10thcipriani) We spent some time looking into this yesterday and we came away with a few thoughts that I'll summarize: - **can... [14:57:17] 10Scap, 06SRE Observability: Scap's logstash checker for channels outside "error" and "exception" - https://phabricator.wikimedia.org/T438570#12380217 (10herron) >>! In T438570#12380054, @thcipriani wrote: > Hrm. I see that particular tag on 15 `type:mediawiki` log channels over the past week, so I guess not t... [15:02:44] Kokkuri question: I'm looking at a repository (/repos/sre/hiddenparma) that currently unconditionally tests it can build a specific image, so both for commits onto main and for MRs. is there a way I can make that image get published to a the wmf registry on commits to main only without affecting the test runs on MRs, or will I need separate jobs [15:02:44] for those? [15:19:48] (03PS7) 101Veertje: Fix login form: responsive labels and mobile-friendly layout [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 [15:31:53] 06Release-Engineering-Team, 06SRE Observability, 13Patch-For-Review: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419#12380465 (10thcipriani) Tagging in #sre_observability for a question: #sre_observability, is there something that... [15:37:09] 10Scap, 06SRE Observability: Scap's logstash checker for channels outside "error" and "exception" - https://phabricator.wikimedia.org/T438570#12380521 (10thcipriani) Sure, here's the logstash link that shows any `type:mediawiki` channel with `tags:*throttle_warning*` over the last 7 days: https://logstash.wiki... [16:06:45] 10Gerrit, 10phan, 07Upstream: Expose Phan results in Gerrit web UI - https://phabricator.wikimedia.org/T439642#12380687 (10Tacsipacsi) Great idea! Would it be in scope to also find a solution to integrate Phan results in [[https://gitlab.wikimedia.org/help/ci/testing/code_quality.md|GitLab code quality find... [16:18:46] (03PS1) 10Krinkle: zuul: Enable php8.5 job for tools/toolbase, drop php8.2 job [integration/config] - 10https://gerrit.wikimedia.org/r/1346671 (https://phabricator.wikimedia.org/T419082) [16:22:51] (03CR) 10Krinkle: [C:03+2] zuul: Enable php8.5 job for tools/toolbase, drop php8.2 job [integration/config] - 10https://gerrit.wikimedia.org/r/1346671 (https://phabricator.wikimedia.org/T419082) (owner: 10Krinkle) [16:24:10] (03Merged) 10jenkins-bot: zuul: Enable php8.5 job for tools/toolbase, drop php8.2 job [integration/config] - 10https://gerrit.wikimedia.org/r/1346671 (https://phabricator.wikimedia.org/T419082) (owner: 10Krinkle) [16:24:42] 10Scap, 06SRE Observability, 06MediaWiki-Core-Platform-Team (Radar): Scap's logstash checker for channels outside "error" and "exception" - https://phabricator.wikimedia.org/T438570#12380868 (10Krinkle) [16:27:56] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1346671 [16:27:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:36:06] 10Scap, 06SRE Observability, 06MediaWiki-Core-Platform-Team (Radar): Scap's logstash checker for channels outside "error" and "exception" - https://phabricator.wikimedia.org/T438570#12380972 (10herron) Thanks! They come and go, but I'd say there's good signal in that essentially all of those normalized mess... [16:38:48] 06Release-Engineering-Team, 07Essential-Work: Add RelEng clinic duty to deployment calendar so that jouncebot will yell at us about it and we remember to do it - https://phabricator.wikimedia.org/T439747 (10brennen) 03NEW [16:57:37] (03PS1) 10Krinkle: zuul: Enable php8.4,php8.5 job for tools/usage, drop php8.2 job [integration/config] - 10https://gerrit.wikimedia.org/r/1346701 (https://phabricator.wikimedia.org/T419078) [17:01:06] (03CR) 10Krinkle: [C:03+2] zuul: Enable php8.4,php8.5 job for tools/usage, drop php8.2 job [integration/config] - 10https://gerrit.wikimedia.org/r/1346701 (https://phabricator.wikimedia.org/T419078) (owner: 10Krinkle) [17:02:49] (03Merged) 10jenkins-bot: zuul: Enable php8.4,php8.5 job for tools/usage, drop php8.2 job [integration/config] - 10https://gerrit.wikimedia.org/r/1346701 (https://phabricator.wikimedia.org/T419078) (owner: 10Krinkle) [17:03:01] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1346701 [17:03:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:05:33] taavi: Taking a look [17:10:13] taavi: That sounds like a separate job situation. One that uses `.kokkuri:build-and-publish-image`. [17:10:50] dancy: ok, thanks! [17:20:19] (03CR) 10Krinkle: "Acknowledged" [fresh] - 10https://gerrit.wikimedia.org/r/1344729 (owner: 10Hashar) [17:36:11] (03open) 10dancy: digitalocean: Upgrade the DigitalOcean provider to 2.102.0 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/647 (https://phabricator.wikimedia.org/T425441) [17:36:52] (03update) 10dancy: digitalocean: Upgrade the DigitalOcean provider to 2.102.0 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/647 (https://phabricator.wikimedia.org/T425441) [17:36:54] (03update) 10dancy: digitalocean: Upgrade the DigitalOcean provider to 2.102.0 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/647 (https://phabricator.wikimedia.org/T425441) [17:38:40] (03merge) 10dancy: digitalocean: Upgrade the DigitalOcean provider to 2.102.0 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/647 (https://phabricator.wikimedia.org/T425441) [17:39:54] (03PS1) 10Krinkle: zuul: Enable php8.4,php8.5 job for tools/blankpages, drop php8.2 job [integration/config] - 10https://gerrit.wikimedia.org/r/1346728 (https://phabricator.wikimedia.org/T419076) [17:43:29] (03PS1) 10Krinkle: _fresh_versions: Silence stderr noise from firefox command [fresh] - 10https://gerrit.wikimedia.org/r/1346730 [17:43:41] (03CR) 10Krinkle: Pass shellcheck on bin/_fresh_versions (031 comment) [fresh] - 10https://gerrit.wikimedia.org/r/1344765 (https://phabricator.wikimedia.org/T439219) (owner: 10Hashar) [17:43:55] (03CR) 10Krinkle: [C:03+2] "During review stumbled on https://gerrit.wikimedia.org/r/c/fresh/+/1346730, but LGTM." [fresh] - 10https://gerrit.wikimedia.org/r/1346371 (owner: 10Hashar) [17:44:05] (03CR) 10Krinkle: [C:03+2] Release 26.09.1 [fresh] - 10https://gerrit.wikimedia.org/r/1346376 (owner: 10Hashar) [17:44:28] (03CR) 10Krinkle: [C:03+2] zuul: Enable php8.4,php8.5 job for tools/blankpages, drop php8.2 job [integration/config] - 10https://gerrit.wikimedia.org/r/1346728 (https://phabricator.wikimedia.org/T419076) (owner: 10Krinkle) [17:45:07] (03update) 10dancy: Draft: digitaocean: use new vpc, egress gateway and isolated_workers in staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/633 (https://phabricator.wikimedia.org/T425441) (owner: 10jelto) [17:46:07] (03Merged) 10jenkins-bot: zuul: Enable php8.4,php8.5 job for tools/blankpages, drop php8.2 job [integration/config] - 10https://gerrit.wikimedia.org/r/1346728 (https://phabricator.wikimedia.org/T419076) (owner: 10Krinkle) [17:46:36] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1346728 [17:46:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:48:24] (03PS1) 10Krinkle: zuul: Enable php8.4,php8.5 job for tools/orphantalk, drop php8.2 job [integration/config] - 10https://gerrit.wikimedia.org/r/1346733 (https://phabricator.wikimedia.org/T419077) [18:12:51] (03CR) 10Krinkle: [C:03+2] zuul: Enable php8.4,php8.5 job for tools/orphantalk, drop php8.2 job [integration/config] - 10https://gerrit.wikimedia.org/r/1346733 (https://phabricator.wikimedia.org/T419077) (owner: 10Krinkle) [18:14:11] (03Merged) 10jenkins-bot: Prepare 26.09.1 release [fresh] - 10https://gerrit.wikimedia.org/r/1346371 (owner: 10Hashar) [18:14:21] (03Merged) 10jenkins-bot: zuul: Enable php8.4,php8.5 job for tools/orphantalk, drop php8.2 job [integration/config] - 10https://gerrit.wikimedia.org/r/1346733 (https://phabricator.wikimedia.org/T419077) (owner: 10Krinkle) [18:16:12] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1346733 [18:16:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:27:58] (03Merged) 10jenkins-bot: Release 26.09.1 [fresh] - 10https://gerrit.wikimedia.org/r/1346376 (owner: 10Hashar) [18:28:20] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218#12381625 (10brennen) Error on rolling to group0: ` 18:18:12 Executing check 'httpbb /srv/deployment/httpbb-... [18:38:47] (03open) 10dancy: digitalocean: Delete associated resources when the cluster is destroyed [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/648 (https://phabricator.wikimedia.org/T425441) [18:39:37] (03update) 10dancy: digitalocean: Delete associated resources when the cluster is destroyed [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/648 (https://phabricator.wikimedia.org/T425441) [18:39:39] (03update) 10dancy: digitalocean: Delete associated resources when the cluster is destroyed [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/648 (https://phabricator.wikimedia.org/T425441) [18:39:54] (03update) 10dancy: digitalocean: Delete associated resources when the cluster is destroyed [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/648 (https://phabricator.wikimedia.org/T425441) [18:40:16] (03merge) 10dancy: digitalocean: Delete associated resources when the cluster is destroyed [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/648 (https://phabricator.wikimedia.org/T425441) [18:49:14] !log Rebuilding gitlab-cloud-runners (staging) (T425441) [18:49:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:49:17] T425441: gitlab behind CDN - https://phabricator.wikimedia.org/T425441 [19:09:33] (03open) 10dancy: scap analyze-logstash: Make output more informative [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1339 (https://phabricator.wikimedia.org/T435419) [20:05:42] 06Release-Engineering-Team, 06SRE Observability, 13Patch-For-Review: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419#12382111 (10dancy) Update analysis below. It still works the same as before. It just reports more information.... [20:28:34] (03open) 10phedenskog: ci-load: Show when all executors were busy in the last 14 days [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/96 [20:44:12] !log Done rebuilding gitlab-cloud-runners (staging) (T425441) [20:44:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:44:15] T425441: gitlab behind CDN - https://phabricator.wikimedia.org/T425441 [20:47:36] (03merge) 10phedenskog: ci-load: Show when all executors were busy in the last 14 days [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/96 [20:47:39] (03update) 10dancy: digitaocean: use new vpc, egress gateway and isolated_workers in staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/633 (https://phabricator.wikimedia.org/T425441) (owner: 10jelto) [20:47:59] (03update) 10dancy: digitaocean: use new vpc, egress gateway and isolated_workers in staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/633 (https://phabricator.wikimedia.org/T425441) (owner: 10jelto) [20:48:16] (03update) 10dancy: digitaocean: use new vpc, egress gateway and isolated_workers in staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/633 (https://phabricator.wikimedia.org/T425441) (owner: 10jelto) [20:49:12] (03merge) 10dancy: digitaocean: use new vpc, egress gateway and isolated_workers in staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/633 (https://phabricator.wikimedia.org/T425441) (owner: 10jelto) [20:50:17] (03open) 10dancy: digitalocean: Assign project resources with digitalocean_project_resources [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/649 (https://phabricator.wikimedia.org/T425441) [20:50:41] (03update) 10dancy: digitalocean: Assign project resources with digitalocean_project_resources [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/649 (https://phabricator.wikimedia.org/T425441) [20:50:52] (03merge) 10dancy: digitalocean: Assign project resources with digitalocean_project_resources [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/649 (https://phabricator.wikimedia.org/T425441) [20:51:26] (03open) 10dancy: ingress: Use a regional load balancer when the workers are isolated [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/650 (https://phabricator.wikimedia.org/T425441) [20:52:21] (03update) 10dancy: ingress: Use a regional load balancer when the workers are isolated [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/650 (https://phabricator.wikimedia.org/T425441) [20:52:22] (03update) 10dancy: ingress: Use a regional load balancer when the workers are isolated [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/650 (https://phabricator.wikimedia.org/T425441) [20:53:23] (03merge) 10dancy: ingress: Use a regional load balancer when the workers are isolated [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/650 (https://phabricator.wikimedia.org/T425441) [20:53:52] (03open) 10dancy: certs: Use DNS-01 challenges with DigitalOcean DNS [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/651 (https://phabricator.wikimedia.org/T425441) [20:54:11] (03update) 10dancy: certs: Use DNS-01 challenges with DigitalOcean DNS [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/651 (https://phabricator.wikimedia.org/T425441) [20:54:37] (03merge) 10dancy: certs: Use DNS-01 challenges with DigitalOcean DNS [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/651 (https://phabricator.wikimedia.org/T425441) [20:56:15] (03open) 10dancy: cluster.tf: Install pvc-cleaner and external-dns after Istio [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/652 (https://phabricator.wikimedia.org/T425441) [20:56:16] (03update) 10dancy: cluster.tf: Install pvc-cleaner and external-dns after Istio [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/652 (https://phabricator.wikimedia.org/T425441) [20:56:23] (03update) 10dancy: cluster.tf: Install pvc-cleaner and external-dns after Istio [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/652 (https://phabricator.wikimedia.org/T425441) [20:57:18] (03merge) 10dancy: cluster.tf: Install pvc-cleaner and external-dns after Istio [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/652 (https://phabricator.wikimedia.org/T425441) [21:23:33] (03PS4) 10Clare Ming: Add Test Kitchen experiment overrides to the header and popup [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1344123 (https://phabricator.wikimedia.org/T438283) [21:25:27] (03CR) 10Clare Ming: "more local testing with custom text inputs: https://phabricator.wikimedia.org/F104398953#14632" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1344123 (https://phabricator.wikimedia.org/T438283) (owner: 10Clare Ming) [22:28:46] (03open) 10dduvall: examples: Update base images [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/187 [22:30:53] (03update) 10dduvall: examples: Update base images [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/187 [22:36:26] (03update) 10dduvall: examples: Update base images [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/187 [22:37:14] (03merge) 10dduvall: examples: Update base images [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/187 [22:43:31] (03update) 10dduvall: node: Omit devDependencies from `npm dedupe` [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/177 (https://phabricator.wikimedia.org/T433500) (owner: 10jforrester) [23:01:03] (03approved) 10dduvall: node: Omit devDependencies from `npm dedupe` [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/177 (https://phabricator.wikimedia.org/T433500) (owner: 10jforrester) [23:01:13] (03merge) 10dduvall: node: Omit devDependencies from `npm dedupe` [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/177 (https://phabricator.wikimedia.org/T433500) (owner: 10jforrester) [23:02:44] (03open) 10dduvall: version: 1.9.0 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/188 [23:02:46] (03update) 10dduvall: version: 1.9.0 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/188 [23:07:13] (03merge) 10dduvall: version: 1.9.0 [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/188 [23:20:15] 10Scap, 06SRE Observability, 06MediaWiki-Core-Platform-Team (Radar): Scap's logstash checker for channels outside "error" and "exception" - https://phabricator.wikimedia.org/T438570#12383160 (10thcipriani) Recently, the tagged messages seem to happen on the hour (due to traffic). But we start deployment wind...