[00:37:26] 10Release-Engineering-Team (Doing 😎), 07OKR-Work: [WE6.2.6] Create design document for Group -1 deployment - https://phabricator.wikimedia.org/T379683#10355963 (10bd808) Today I drafted lists of open questions for testing folks (Quality Services and Test Platform teams) and for SRE folks (Service Operations te... [00:38:13] 10Continuous-Integration-Config, 10WMF-General-or-Unknown, 07Technical-Debt: Drop items we had in production/etc. for wikitech, like LdapAuthentication - https://phabricator.wikimedia.org/T376579#10355965 (10Jdforrester-WMF) [08:32:02] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Quibble: Speed up git clone operation in wmf-quibble job - https://phabricator.wikimedia.org/T374717#10356419 (10hashar) >>! In T374717#10144359, @hashar wrote: > ... > ` > $ git -C ~/projects/mediawiki/extensions/GrowthExperiments branch... [08:38:25] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Quibble: Speed up git clone operation in wmf-quibble job - https://phabricator.wikimedia.org/T374717#10356428 (10hashar) 05Stalled→03Open [09:35:00] Project beta-code-update-eqiad build #523586: 04FAILURE in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/523586/ [09:44:53] Yippee, build fixed! [09:44:53] Project beta-code-update-eqiad build #523587: 09FIXED in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/523587/ [09:55:18] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Quibble: Speed up git clone operation in wmf-quibble job - https://phabricator.wikimedia.org/T374717#10356628 (10hashar) [09:58:20] !log Semi manually converted REL1_19 branch to tags on currently wmf deployed repositories # T380841 [09:58:30] I gotta fix the script :/ [09:58:36] of course stashbot bot is deaed? [09:58:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:59:02] T380841: Convert old release branches to tags - https://phabricator.wikimedia.org/T380841 [10:04:36] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T375664#10356652 (10hashar) I don't know what is up with the listed roadmap. https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar/2024 shows it as n... [10:08:05] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T375664#10356667 (10hashar) [10:12:40] !log Fast forwarded mediawiki/extensions/Math wmf/1.44.0-wmf.5 branch from 0458ea16 to d5076423. That is to apply hotfixes that merged after the branch has been cut # T375664 | T380184 [10:12:51] andre: ^ :) [10:13:01] ah, thanks [10:13:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:13:07] the series of commit was straightforward [10:13:14] T375664: 1.44.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T375664 [10:13:15] so I have "just" pushed the wmf branch to be more recent [10:13:49] jnuche mentioned the nightly job failed due to a toolforge issue [10:13:54] so I guess that is blocking [10:14:45] ah of course [10:14:52] the `!log` does not end up in Phabricator tasks :\ [10:15:34] hmm no it did: https://phabricator.wikimedia.org/T380184#10356674 [10:16:16] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T375664#10356680 (10hashar) //!log Fast forwarded mediawiki/extensions/Math wmf/1.44.0-wmf.5 branch from 0458ea16 to d5076423. That is to apply hotfixes that m... [10:22:20] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T375664#10356722 (10hashar) [10:23:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10356713 (10hashar) 05O... [10:28:09] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T375664#10356741 (10hashar) We have an ongoing issue with toolforge T380844 which breaks the MediaWiki train, notably our automatic jobs that processes the bra... [10:31:20] I have launched https://releases-jenkins.wikimedia.org/job/Branch%20cut%20test%20patches/ in train mode [10:31:29] and it managed to fetch 1.44.0-wmf.5 from the train-blockers tool [10:31:34] so that should complete successfuly [10:43:55] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T375664#10356789 (10hashar) The [[ https://releases-jenkins.wikimedia.org/job/Branch%20cut%20test%20patches/ | Branch cut test patches ]] Jenkins job passed :) [10:44:05] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T375664#10356792 (10hashar) [12:58:29] (03PS1) 10Hashar: Revert "Grant "Rename project" to "Administrators"" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1098021 (https://phabricator.wikimedia.org/T239693) [12:58:50] (03CR) 10Hashar: [V:03+2 C:03+2] Revert "Grant "Rename project" to "Administrators"" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1098021 (https://phabricator.wikimedia.org/T239693) (owner: 10Hashar) [13:07:58] 14Phabricator (2024-10-01): unable to zoom images on mobile - https://phabricator.wikimedia.org/T375671#10357453 (10Aklapper) I also now merged this into upstream: https://we.phorge.it/rP828ea9bca0309b01164af0bee90860bc06abdc0b [13:29:08] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T375664#10357513 (10hashar) **Summary** I have confirmed yesterday `scap clean --delete-gerrit-branch` properly delete old wmf branches (solving T303828). I h... [13:31:30] jnuche: andre: I wrote a quick summary of this mornign events at https://phabricator.wikimedia.org/T375664#10357513 [13:31:39] I guess I should have caught that presync earlier [13:31:41] thanks [13:33:06] hashar: ack, glad you made it through! [13:38:06] 10GitLab (CI & Job Runners), 06collaboration-services, 06serviceops: Create a staging apt repository for CI-based builds of Debian packages - https://phabricator.wikimedia.org/T347004#10357580 (10eoghan) 05Open→03Resolved The pipeline seems to be working correctly, and we've got [[ https://wikitech.w... [13:45:20] (03update) 10hashar: Clear stdin before reading user input [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/468 [13:46:07] that made me remember of that old patch of mine [13:50:39] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [13:51:32] FIRING: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [14:00:39] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [14:01:32] RESOLVED: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [14:11:54] jnuche: if you feel adventurous with reviewing a scap change I have made https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/468/ that is to have scap flush the standard input before invoking input() / asking a question :) [14:12:04] cause: Do you want to continue [y/N]: [14:12:29] would validate N if I ever pressed enter before the interactive question :b [14:13:27] hashar: sure, will take a look in a bit [14:19:45] great, thanks! [14:59:55] 06Project-Admins: Create Phabricator tag for Wikimedia Foundation Office Wiki - https://phabricator.wikimedia.org/T380415#10357933 (10Aklapper) Please see the "file a task" link on https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects for required data [15:46:17] holy m***** f****** g** I found it!!!!!!!!!!!!!!! [15:47:06] so if one uses pygerrit2 with: gerrit_client().put('/projects/X/tags/REL1_19', {'revision': }) [15:47:15] that does not create a tag pointing to [15:47:27] cause the API does not read that or require it to be an application/json or whatever [15:47:45] so one has pass it as a keyword argument: [15:48:07] gerrit_client().put('/projects/X/tags/REL1_19', json={'revision': }) [15:48:10] took me a while [15:58:10] 10Release-Engineering-Team (Priority Backlog 📥), 10Wikimedia-Phabricator-Extensions: Weird indentation on task list (due to empty rendering of deadline subtype tag) - https://phabricator.wikimedia.org/T379376#10358078 (10Aklapper) p:05Triage→03Low a:03Aklapper Code triggering this behavior is custom `has... [16:08:21] (03open) 10hashar: make-release: add a "test" bundle for development [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/137 (https://phabricator.wikimedia.org/T250095 https://phabricator.wikimedia.org/T380841) [16:08:27] * hashar whistles [16:11:07] 10Beta-Cluster-Infrastructure: Re-create poolcounter instance in Beta Cluster (deployment-prep) - https://phabricator.wikimedia.org/T380881 (10Krinkle) 03NEW [16:15:16] (03update) 10hashar: Clear stdin before reading user input [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/468 [16:16:22] (03update) 10hashar: make-release: fix tag conversion of branch [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/137 (https://phabricator.wikimedia.org/T250095 https://phabricator.wikimedia.org/T380841) [16:16:43] (03update) 10hashar: make-release: fix tag conversion of branch [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/137 (https://phabricator.wikimedia.org/T250095 https://phabricator.wikimedia.org/T380841) [16:21:18] (03approved) 10jnuche: Clear stdin before reading user input [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/468 (owner: 10hashar) [16:26:35] (03merge) 10hashar: Clear stdin before reading user input [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/468 [16:26:44] success [16:27:52] (03update) 10hashar: make-release: fix tag conversion of branch [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/137 (https://phabricator.wikimedia.org/T250095 https://phabricator.wikimedia.org/T380841) [19:12:12] 10Phabricator: Reports: Order Over-due Tasks by oldest date instead of task ID - https://phabricator.wikimedia.org/T380907 (10Aklapper) 03NEW p:05Triage→03Low [19:13:19] (03open) 10aklapper: Reports: Order Over-due Tasks by oldest date [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/77 (https://phabricator.wikimedia.org/T380907) [19:29:34] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [19:36:46] 10WikimediaDebug, 10SRE Observability (FY2024/2025-Q2), 07Wikimedia-production-error: On-demand excimer profiling does not work for URLs longer than 255 bytes - https://phabricator.wikimedia.org/T377433#10359218 (10Krinkle) a:05Krinkle→03mszabo [19:37:07] 10WikimediaDebug, 10SRE Observability (FY2024/2025-Q2), 07Wikimedia-production-error: On-demand excimer profiling does not work for URLs longer than 255 bytes - https://phabricator.wikimedia.org/T377433#10359212 (10Krinkle) 05Open→03Resolved a:03Krinkle Verified. https://commons.wikimedia.org/w/ap... [19:39:34] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [19:40:34] (03open) 10aklapper: Task list view: Do not render tag for Deadline tasks without due date [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/42 (https://phabricator.wikimedia.org/T379376) [19:42:05] 10Release-Engineering-Team (Priority Backlog 📥), 10Wikimedia-Phabricator-Extensions, 13Patch-For-Review: Weird indentation on task list (due to empty rendering of deadline subtype tag) - https://phabricator.wikimedia.org/T379376#10359238 (10Aklapper) Digging more into code history related to T379376, it look... [19:48:11] 06Release-Engineering-Team, 06collaboration-services: Digital Ocean-based Cloud Gitlab Runners fail building Varnish - https://phabricator.wikimedia.org/T379110#10359275 (10Dzahn) Since we already know that switching to runners in our own infrastructure fixes the issue I would suggest to just keep doing that (... [19:54:50] 10Phabricator: Phabricator weekly changes email: Include open Deadline type tasks without Due Date - https://phabricator.wikimedia.org/T380915 (10Aklapper) 03NEW p:05Triage→03Low [20:05:05] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review: Phabricator weekly changes email: Include open Deadline type tasks without Due Date - https://phabricator.wikimedia.org/T380915#10359374 (10Aklapper) [20:07:51] 06Release-Engineering-Team, 06collaboration-services: Digital Ocean-based Cloud Gitlab Runners fail building Varnish - https://phabricator.wikimedia.org/T379110#10359376 (10BCornwall) Indeed, I am just setting everything to use wdqs instead of DO on all my projects. I'd prefer our stuff to be default if we're... [20:18:37] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review: Phabricator weekly changes email: Include open Deadline type tasks without Due Date - https://phabricator.wikimedia.org/T380915#10359391 (10Dzahn) @Aklapper Please check your email. I can confirm the query works but I don't... [20:24:07] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Phabricator weekly changes email: Include open Deadline type tasks without Due Date - https://phabricator.wikimedia.org/T380915#10359403 (10Aklapper) 05Open→03Resolved @Dzahn: So fast! Thanks! Test email looks as expected. [20:38:37] 10Release-Engineering-Team (Priority Backlog 📥), 10Wikimedia-Phabricator-Extensions, 13Patch-For-Review: Weird indentation on task list (due to empty rendering of deadline subtype tag) - https://phabricator.wikimedia.org/T379376#10359489 (10Pppery) See also {T335367} [21:12:55] (03merge) 10hashar: make-release: fix tag conversion of branch [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/137 (https://phabricator.wikimedia.org/T250095 https://phabricator.wikimedia.org/T380841) [21:15:01] !log Converted legacy MediaWiki release branches to tags from repositories currently deployed on Wikimedia cluster. REL1_20, REL1_21, REL1_22, REL1_23, REL1_24, REL1_25, REL1_26, REL1_27, REL1_28, REL1_29, REL1_30, REL1_31, REL1_32, REL1_33, REL1_34, REL1_35, REL1_36, REL1_37, REL1_38 # T380841 [21:15:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:15:04] T380841: Convert old release branches to tags - https://phabricator.wikimedia.org/T380841 [21:15:17] I am surprised that did not get truncated [21:19:36] 10Scap: Selecting N for "backport the change" doesn't seem to work/exit - https://phabricator.wikimedia.org/T380924 (10Reedy) 03NEW [21:21:26] 10Scap: Selecting N for "backport the change" doesn't seem to work/exit - https://phabricator.wikimedia.org/T380924#10359624 (10Reedy) [21:21:57] Reedy: try n ? [21:22:10] or just return [21:22:17] I used ctrl + c :P [21:22:20] Backport the changes? [y/N]: [21:22:26] if if it's case sensitive... :P [21:22:28] ^C [21:29:22] 10Scap: Selecting N for "backport the change" doesn't seem to work/exit - https://phabricator.wikimedia.org/T380924#10359665 (10hashar) p:05Triage→03Low [21:29:28] 10Scap: Selecting N for "backport the change" doesn't seem to work/exit - https://phabricator.wikimedia.org/T380924#10359664 (10hashar) That is in `scap/interaction.py` ` lang=python YES_NO_CHOICES = {"Yes": "y", "No": "n"} if choices == bool: choices = YES_NO_CHOICES if resp in... [21:29:36] Reedy: do we have a runbook to mark a release branch end of life? [21:29:47] AFAIK no [21:29:59] It's mostly just turning off CI [21:29:59] some minutes ago I have converted all the REL1_20 .. REL1_38 branches to tags [21:31:17] well [21:31:44] we have survived with ten + years of legacy branches [21:31:54] so I guess we can survive an additional decade [21:32:08] and one will find some doc in T380841 :-] [21:32:08] T380841: Convert old release branches to tags - https://phabricator.wikimedia.org/T380841 [21:37:58] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Quibble: Speed up git clone operation in wmf-quibble job - https://phabricator.wikimedia.org/T374717#10359685 (10hashar) a:03hashar [21:38:04] I guess I had enough for today [21:38:15] have a good night all [21:50:59] 06Release-Engineering-Team, 10WMF-General-or-Unknown: wgBetaFeaturesAllowList lists to ask Greg G - https://phabricator.wikimedia.org/T380929 (10Reedy) 03NEW [21:51:04] 06Release-Engineering-Team, 10WMF-General-or-Unknown: wgBetaFeaturesAllowList lists to ask Greg G - https://phabricator.wikimedia.org/T380929#10359750 (10Reedy) p:05Triage→03Low [21:56:05] 06Release-Engineering-Team, 10Scap, 06collaboration-services, 13Patch-For-Review: scap / puppet dependency issue on reimaged phabricator hosts - https://phabricator.wikimedia.org/T378769#10359777 (10Dzahn) Thank you, @jnuche !:) Per IRC discussion and gerrit. The patch above looks good to me! I agree scap... [22:50:02] maintenance-disconnect-full-disks build 653017 integration-agent-docker-1046 (/: 28%, /srv: 100%, /var/lib/docker: 38%): OFFLINE due to disk space [22:55:02] maintenance-disconnect-full-disks build 653018 integration-agent-docker-1046 (/: 28%, /srv: 46%, /var/lib/docker: 35%): RECOVERY disk space OK [23:04:52] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 06SRE, 10SRE-swift-storage: Migrate gitlab storage to apus - https://phabricator.wikimedia.org/T378922#10360018 (10Dzahn) I ran across this https://bacula.org/whitepapers/CloudBackup.pdf (e.g. `we included an S3 driver that is compatible wit... [23:07:17] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus - https://phabricator.wikimedia.org/T378922#10360019 (10Dzahn) [23:08:03] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10360020 (10Dzahn) [23:56:00] kostajh: I just checked on the wmf/next build from last night. The parallel work on wmf-quibble-core-vendor-mysql-php74 helped a bunch. It ran 12 minutes faster than the historical average. We are now seeing wmf-quibble-selenium-php74 as the slowest suite (18m) with mediawiki-quibble-vendor-postgres-php74 close behind at 17m. [23:56:15] More at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1097575?tab=checks is you're interested.