[00:01:39] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:06:39] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:13:05] <_Gerges> What permissions does a user need to read Logstash? [01:33:35] wmf, nda, or ops https://wikitech.wikimedia.org/wiki/Logstash#Authentication [07:42:03] 10Continuous-Integration-Infrastructure, 06SRE, 10SRE-Access-Requests: Requesting access to `contint-admins`, `contint-docker` for 'Arthur taylor' - https://phabricator.wikimedia.org/T373969#10128736 (10ArthurTaylor) @WMDE-leszek I guess Kara is away right now. Can you approve on her behalf? [08:02:27] 10Continuous-Integration-Infrastructure, 06SRE, 10SRE-Access-Requests: Requesting access to `contint-admins`, `contint-docker`, LDAP `ciadmin` for 'Arthur taylor' - https://phabricator.wikimedia.org/T373969#10128815 (10hashar) [08:12:24] 10Continuous-Integration-Infrastructure, 06SRE, 10SRE-Access-Requests: Requesting access to `contint-admins`, `contint-docker`, LDAP `ciadmin` for 'Arthur taylor' - https://phabricator.wikimedia.org/T373969#10128850 (10WMDE-leszek) I approve on WMDE's behalf. Thank you [10:01:29] 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review: Verify Jenkins Gearman plugin works with Java 17 - https://phabricator.wikimedia.org/T373351#10129219 (10hashar) > gearman-java to be buildable under Java 17 The only fixed needed was Javadoc https://gerrit.wikimedia.org/r/c/integration... [10:35:24] 10GitLab (Pipeline Services MigrationšŸ¤), 06collaboration-services, 06Data-Platform-SRE, 10Wikidata, and 3 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10129316 (10Jelto) [10:40:07] 10GitLab (Pipeline Services MigrationšŸ¤), 06collaboration-services, 06Data-Platform-SRE, 10Wikidata, and 3 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10129320 (10Jelto) A new `wikidata-query-gui` namespace is now available in all... [10:44:11] <_Gerges> AntiComposite: What are the requirements to obtain NDA? [11:53:08] _Gerges: https://wikitech.wikimedia.org/wiki/Volunteer_NDA (but sounds off-topic for this channel?) [11:59:14] <_Gerges> @andre: Sorry, I didn't find a suitable place to ask this question. Thanks [12:00:27] np [13:48:39] 10Deployments, 06Release-Engineering-Team, 10logspam-watch: Make logspam-watch available on deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T372563#10129938 (10Lucas_Werkmeister_WMDE) [13:51:20] 10Release-Engineering-Team (Priority Backlog šŸ“„), 10Scap, 06serviceops, 13Patch-For-Review, 10Sustainability (Incident Followup): scap should check if it is running within a tmux/screen - https://phabricator.wikimedia.org/T361724#10129946 (10Jdforrester-WMF) I just ran into this ā€“ no error, no warning, I... [13:52:20] 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review: Verify Jenkins Gearman plugin works with Java 17 - https://phabricator.wikimedia.org/T373351#10129962 (10hashar) 05Openā†’03Resolved [14:53:12] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team (25Q1 (Julā€“Sep)), 10MW-1.43-notes (1.43.0-wmf.21; 2024-09-03): Beta Cluster orchestrator / evaluator broken, blocking WikiLambda CI (and use of Beta Cluster) - https://phabricator.wikimedia.org/T374242#10130282 (10Jdforrester-WMF) p:05Triageā†’03High [15:35:36] 10MediaWiki-Releasing, 05MW-1.43-release: Branch REL1_43 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T372318#10130562 (10Atieno) a:03Atieno [15:46:12] (03update) 10sandeeps: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 [15:49:01] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Figure out how to provision a Kubernetes cluster using Magnum and OpenTofu - https://phabricator.wikimedia.org/T372498#10130628 (10bd808) >>! In T372498#10127325, @rook wrote: > Looks like what you found in https://github.com/opentofu/opentofu/issues/388 has b... [15:49:45] (03update) 10sandeeps: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 [15:56:12] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Figure out how to provision a Kubernetes cluster using Magnum and OpenTofu - https://phabricator.wikimedia.org/T372498#10130660 (10bd808) >>! In T372498#10127325, @rook wrote: > I'm getting an error requesting that the bucket be generated before the state is s... [16:02:49] (03PS2) 10Effie Mouzeli: background.js: Allow wikitech.wikimedia.org [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1070275 (https://phabricator.wikimedia.org/T371537) [16:11:10] 10Continuous-Integration-Infrastructure, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Requesting access to `contint-admins`, `contint-docker`, LDAP `ciadmin` for 'Arthur taylor' - https://phabricator.wikimedia.org/T373969#10130747 (10Dzahn) [16:13:53] (03CR) 10Effie Mouzeli: background.js: Allow wikitech.wikimedia.org (031 comment) [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1070275 (https://phabricator.wikimedia.org/T371537) (owner: 10Effie Mouzeli) [16:37:14] (03CR) 10Krinkle: background.js: Allow wikitech.wikimedia.org (031 comment) [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1070275 (https://phabricator.wikimedia.org/T371537) (owner: 10Effie Mouzeli) [16:40:39] 10Continuous-Integration-Infrastructure, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Requesting access to `contint-admins`, `contint-docker`, LDAP `ciadmin` for 'Arthur taylor' - https://phabricator.wikimedia.org/T373969#10130953 (10Ladsgroup) a:05andrea.denisseā†’03Ladsgroup This week's clinic... [18:47:19] 10GitLab (Pipeline Services MigrationšŸ¤), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 3 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10131510 (10Gehel) [19:02:25] (03open) 10brennen: prometheus: stub out additionalPrometheusRulesMap [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/411 (https://phabricator.wikimedia.org/T373548) [19:04:35] (03update) 10brennen: prometheus: stub out additionalPrometheusRulesMap [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/411 (https://phabricator.wikimedia.org/T373548) [19:05:01] (03approved) 10dduvall: prometheus: stub out additionalPrometheusRulesMap [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/411 (https://phabricator.wikimedia.org/T373548) (owner: 10brennen) [19:05:47] (03merge) 10brennen: prometheus: stub out additionalPrometheusRulesMap [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/411 (https://phabricator.wikimedia.org/T373548) [19:17:14] (03open) 10brennen: prometheus: bump PVC low disk alert to 90% [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/412 (https://phabricator.wikimedia.org/T370266) [19:17:46] (03approved) 10dduvall: prometheus: bump PVC low disk alert to 90% [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/412 (https://phabricator.wikimedia.org/T370266) (owner: 10brennen) [19:24:24] (03merge) 10brennen: prometheus: bump PVC low disk alert to 90% [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/412 (https://phabricator.wikimedia.org/T370266) [20:18:54] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Figure out how to provision a Kubernetes cluster using Magnum and OpenTofu - https://phabricator.wikimedia.org/T372498#10131841 (10bd808) >>! In T372498#10127325, @rook wrote: > If you're finding other uses of the ssh key please let me know. I have some local... [20:54:30] (03update) 10brennen: prometheus: bump PVC low disk alert to 90% [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/412 (https://phabricator.wikimedia.org/T370266 https://phabricator.wikimedia.org/T373548) [21:11:01] (03update) 10sandeeps: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 [21:11:18] (03update) 10sandeeps: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 [21:11:20] (03update) 10sandeeps: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 [21:11:25] (03update) 10sandeeps: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 [21:11:29] (03update) 10sandeeps: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 [21:15:15] (03update) 10sandeeps: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 [21:17:24] Project beta-scap-sync-world build #171444: 04FAILURE in 2 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/171444/ [21:20:36] Noticed a bug where links with ampsersands in them seem to always break in the Gerrit UI. [21:20:38] e..g. https://gerrit.wikimedia.org/r/c/labs/tools/fileprotectionsync/+/1071364 [21:21:09] I thought maybe it was specific to links that contain certain phrases or domains, but it happens e.g. for https://example.org/foo?a=b&c=d as well, it stops at the first ampersand. [21:21:28] couldn't find any task about it, open or closed (zero matches for "ampersand" and tag:Gerrit) [21:21:30] https://issues.gerritcodereview.com/issues/40003737 [21:21:34] but maybe related to this? [21:28:12] Yippee, build fixed! [21:28:12] Project beta-scap-sync-world build #171445: 09FIXED in 2 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/171445/ [22:41:57] we do have come commentlinks set up in the gerrit config [22:57:39] but that task seems more like a symptom of the same underlying problem with url parsing for commentlinks (maybe). I came up with nothing upstream either. Worth filing a task, probably. [23:25:37] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog šŸ“„): Alerts for disk space issues in GitLab's Digital Ocean cluster - https://phabricator.wikimedia.org/T373548#10132236 (10brennen) Note that you can reach a web UI for Alertmanager directly with: `lang=sh kubectl port-forward svc/kube-... [23:25:53] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog šŸ“„): Alerts for disk space issues in GitLab's Digital Ocean cluster - https://phabricator.wikimedia.org/T373548#10132237 (10brennen)