[00:00:34] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:34] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [02:10:27] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Project-requests): Request creation of deployment_prep_s3 VPS project - https://phabricator.wikimedia.org/T372353#10060292 (10Andrew) +1 workaround ridiculous bug [06:35:25] 06Project-Admins: Create project tag for Metrics Platform - https://phabricator.wikimedia.org/T372368 (10cjming) 03NEW [07:15:23] 06Project-Admins: Create project tag for Metrics Platform - https://phabricator.wikimedia.org/T372368#10060455 (10Aklapper) https://phabricator.wikimedia.org/project/profile/5324/ already exists, so I'd rather call this `#mediawiki-extensions-MetricsPlatform` to avoid confusion? [07:26:58] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Project-requests): Request creation of deployment_prep_s3 VPS project - https://phabricator.wikimedia.org/T372353#10060463 (10dcaro) 05Open→03In progress a:03dcaro [07:27:39] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T366963#10060480 (10Jdlrobson) [07:32:32] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Project-requests): Request creation of deployment_prep_s3 VPS project - https://phabricator.wikimedia.org/T372353#10060499 (10dcaro) Unfortunately, underscores are not valid domain name characters, so the name would have to be something like `deploymentpreps3`, is th... [07:40:22] 10Gerrit, 10Release-Engineering-Team (Radar), 07Voice & Tone: Gerrit: default branch for new projects should be "main" - https://phabricator.wikimedia.org/T279612#10060520 (10ArthurTaylor) 05Declined→03Open [07:51:43] 10Gerrit, 10Release-Engineering-Team (Radar), 07Voice & Tone: Gerrit: default branch for new projects should be "main" - https://phabricator.wikimedia.org/T279612#10060541 (10Aklapper) Per previous comment I'd prefer not to increase entropy, "but some projects ... while other projects ... and there's also ..... [08:01:39] 10Gerrit, 10Release-Engineering-Team (Radar), 07Voice & Tone: Gerrit: default branch for new projects should be "main" - https://phabricator.wikimedia.org/T279612#10060558 (10ArthurTaylor) I could be on board with closing this ticket if there is an alternative ticket that outlines some concrete, pragmatic an... [08:04:53] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 07Voice & Tone: Rename mainline Git branch from "master" to "main" on all WMF-hosted repositories - https://phabricator.wikimedia.org/T281593#10060572 (10Bugreporter) [09:31:21] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Eihel - https://phabricator.wikimedia.org/T372381 (10Eihel) 03NEW [09:49:25] 10Release-Engineering-Team (Priority Backlog 📥), 10Wikimedia-Phabricator-Extensions: Remove misleading warning from security escalation interface - https://phabricator.wikimedia.org/T372335#10060985 (10Aklapper) p:05Triage→03Low a:03Aklapper True, I can confirm after testing locally. [09:58:59] (03open) 10aklapper: Remove misleading warning from security escalation interface [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/39 (https://phabricator.wikimedia.org/T372335) [10:07:27] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Eihel - https://phabricator.wikimedia.org/T372381#10061024 (10Aklapper) 05Open→03Resolved a:03Aklapper Done! [13:05:14] 06Gerrit-Privilege-Requests, 13Patch-For-Review: Give Edesol +2 on Extension:Add_HTML_Meta_and_Title in Gerrit - https://phabricator.wikimedia.org/T372073#10061552 (10Edesol) ok, the hint is in the error log. > PHPCBF CAN FIX THE 46 MARKED SNIFF VIOLATIONS AUTOMATICALLY composer installed "squizlabs/php_cod... [13:11:28] 10GitLab (Project Migration), 06Fundraising-Backlog: Migrate fundraising-dev repo to gitlab - https://phabricator.wikimedia.org/T353374#10061573 (10Aklapper) [13:16:13] 06Project-Admins: Create project tag for Metrics Platform - https://phabricator.wikimedia.org/T372368#10061580 (10Jdforrester-WMF) >>! In T372368#10060455, @Aklapper wrote: > https://phabricator.wikimedia.org/project/profile/5324/ already exists, so I'd rather call this `#mediawiki-extensions-MetricsPlatform` to... [13:17:46] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 07Voice & Tone: Rename mainline Git branch from "master" to "main" on all WMF-hosted repositories - https://phabricator.wikimedia.org/T281593#10061593 (10Jdforrester-WMF) This task is about the proposed/stalled/maybe-cancelled Git... [13:17:57] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 07Voice & Tone: Rename mainline Git branch from "master" to "main" on all WMF-hosted repositories during GitLab migration - https://phabricator.wikimedia.org/T281593#10061594 (10Jdforrester-WMF) [13:38:26] 10Gerrit: Gerrit's syntax highlighting for PHP code breaks when encountering an apostrophe in a // comment in a function call - https://phabricator.wikimedia.org/T372404 (10matmarex) 03NEW [16:07:14] Hi folks! I have a build that consistently fail for a 20 mins timeout https://integration.wikimedia.org/ci/job/tox/1971/console [16:07:34] does it ring a bell? I can't see anything obvious from the errors, I don't think it is related to my code [16:19:36] 10Release-Engineering-Team (Radar), 06collaboration-services, 10Data Pipelines, 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10062265 (10Ottomata) @Stevemunene @BTullis for future upgrades: - Gobblin systemd timer... [17:02:03] 10Release-Engineering-Team (Radar), 06collaboration-services, 10Data Pipelines, 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10062394 (10BTullis) Ah, sorry. That's 100% my fault. This was counter to some previous g... [17:15:37] (03update) 10thcipriani: Release 4.97.0-1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/404 [17:15:39] (03open) 10thcipriani: Release 4.97.0-1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/404 [17:17:23] (03merge) 10thcipriani: Release 4.97.0-1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/404 [19:02:17] (03open) 10dduvall: Initialize /srv/mediawiki-staging submodules [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/82 [19:02:19] (03update) 10dduvall: Initialize /srv/mediawiki-staging submodules [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/82 [19:04:29] (03update) 10dduvall: Initialize /srv/mediawiki-staging submodules [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/82 [19:04:31] (03update) 10dduvall: Initialize /srv/mediawiki-staging submodules [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/82 [19:08:12] 06Project-Admins: Consider archiving #notice, #developer-notice, #sysadmin-notice - https://phabricator.wikimedia.org/T372431 (10Pppery) 03NEW [19:08:29] 06Project-Admins: Consider archiving #notice, #developer-notice, #sysadmin-notice - https://phabricator.wikimedia.org/T372431#10062690 (10Pppery) [19:28:52] 10Release-Engineering-Team (Radar), 06collaboration-services, 10Data Pipelines, 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10062745 (10Ottomata) > sensors should recover cleanly from a restart, so pausing ingesti... [19:32:55] 10Release-Engineering-Team (Radar), 06collaboration-services, 10Data Pipelines, 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10062756 (10Ottomata) We may have another minor problem: In [[ http://localhost:8600/dag... [19:36:05] 10Release-Engineering-Team (Radar), 06collaboration-services, 10Data Pipelines, 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10062763 (10Ottomata) Ah, here is more detail on failed sensor log. It can't actually loa... [19:43:33] 06Project-Admins: Create project tag for Metrics Platform - https://phabricator.wikimedia.org/T372368#10062792 (10cjming) >>! In T372368#10061580, @Jdforrester-WMF wrote: > Maybe the MW extension should be called "Metrics Platform Configuration" for clarity, but `#mediawiki-extensions-MetricsPlatform` is extreme... [19:52:56] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T366963#10062814 (10Jdlrobson) [20:12:33] 10Phabricator (Upstream), 07Upstream: Editing an existing task description which mentions a user should not readd the user as Subscriber - https://phabricator.wikimedia.org/T96464#10062861 (10Aklapper) 05Open→03Stalled a:03Aklapper [20:36:40] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10Wikimedia-Apache-configuration, and 3 others: Apache 2.4.61 throws a 403 Forbidden for links containing %3F - https://phabricator.wikimedia.org/T370110#10062931 (10Aklapper) I'm tempted to resolve this task tagged w... [20:49:34] 10Phabricator: Add a "Thanks" token - https://phabricator.wikimedia.org/T338133#10062974 (10Aklapper) 05Open→03Declined Cleaning the backlog a bit as I do not plan to work on this but please do reopen once someone contributed a patch! For previous examples, see rPHAB78e9024478f48c2523efbb5c4c7a063aee731... [20:53:10] 06Project-Admins: Consider archiving #notice, #developer-notice, #sysadmin-notice - https://phabricator.wikimedia.org/T372431#10062990 (10Aklapper) Yeah overengineering or maybe categorizing for the sake of categorizing (though with good intentions) has its popular moments... [20:58:10] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10Wikimedia-Apache-configuration, and 3 others: Apache 2.4.61 throws a 403 Forbidden for links containing %3F - https://phabricator.wikimedia.org/T370110#10063006 (10Dzahn) `modules/profile/templates/idp/client/httpd-... [21:02:02] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: OpenTofu fails to provision a Magnum managed k8s cluster in deployment-prep - https://phabricator.wikimedia.org/T372365#10063009 (10bd808) Manual cleanup of tofu failure: ` $ sudo wmcs-openstack coe cluster list +--------------------------------------+---------------... [22:14:08] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: OpenTofu fails to provision a Magnum managed k8s cluster in deployment-prep - https://phabricator.wikimedia.org/T372365#10063118 (10bd808) {T332194} looks to have been the same general problem ("Failed to create trustee or trust for Cluster"). Per T332194#8710538 I t... [22:45:43] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10Wikimedia-Apache-configuration, and 3 others: Apache 2.4.61 throws a 403 Forbidden for links containing %3F - https://phabricator.wikimedia.org/T370110#10063137 (10brennen) If this has bitten us anywhere else, I'm n... [22:46:37] 10GitLab (Project Migration), 06Release-Engineering-Team: Create new GitLab project group: sprinthackular2024 - https://phabricator.wikimedia.org/T372443 (10NBaca-WMF) 03NEW [22:48:45] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T366963#10063163 (10Pppery) T372444 seems to be a regression this train. Probably not important enough to be a blocker though. [22:50:41] 10GitLab (Project Migration), 06Release-Engineering-Team: Create new GitLab project group: sprinthackular2024 - https://phabricator.wikimedia.org/T372443#10063167 (10bd808) > to contribute, limit to members of the #sprinthackular2024-all slack channel There is no connection between Developer accounts and the... [23:10:58] 10GitLab (Project Migration), 06Release-Engineering-Team: Create new GitLab project group: sprinthackular2024 - https://phabricator.wikimedia.org/T372443#10063200 (10thcipriani) We could add a static list of folks who are added there now, or just everyone in product and tech? Dumb question: is this an upcomin... [23:12:36] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: OpenTofu fails to provision a Magnum managed k8s cluster in deployment-prep - https://phabricator.wikimedia.org/T372365#10063201 (10bd808) 05Open→03Resolved ` Apply complete! Resources: 3 added, 0 changed, 0 destroyed. ` The need for "Unrestricted (dangerous...