[01:20:03] maintenance-disconnect-full-disks build 680691 integration-agent-docker-1045 (/: 25%, /srv: 97%, /var/lib/docker: 24%): OFFLINE due to disk space [01:25:03] maintenance-disconnect-full-disks build 680692 integration-agent-docker-1045 (/: 25%, /srv: 20%, /var/lib/docker: 23%): RECOVERY disk space OK [07:28:41] 06Release-Engineering-Team, 06collaboration-services: phabricator.wmcloud.org (test instance) is broken - https://phabricator.wikimedia.org/T387699 (10LSobanski) 03NEW [07:34:48] 06Release-Engineering-Team, 06collaboration-services: phabricator.wmcloud.org (test instance) is broken - https://phabricator.wikimedia.org/T387699#10594897 (10Anoop) β†’14Duplicate dup:03T387619 [10:05:50] (03open) 10hashar: Release 4.139.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/672 [10:08:16] (03merge) 10hashar: Release 4.139.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/672 [11:12:56] (03close) 10aklapper: Don't duplicate code from Security extension [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/83 (https://phabricator.wikimedia.org/T382034) [11:14:37] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Wikimedia-Phabricator-Extensions, 07Technical-Debt: Consolidate code to check ACL membership of acting user - https://phabricator.wikimedia.org/T382034#10595551 (10Aklapper) 05Openβ†’03Invalid Unneeded now due to https://phabricator.wi... [11:38:48] 10Gerrit, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Remove explicit enablement of G1 garbage collector for Gerrit - https://phabricator.wikimedia.org/T387223#10595592 (10hashar) @thcipriani `G1NewSizePercent` and `+UseStringDeduplication` are the ones I have on my radar. Tha... [12:01:25] (03PS1) 10JMeybohm: Revert^2 "helm-linter: Update helm and helmfile" [integration/config] - 10https://gerrit.wikimedia.org/r/1124077 (https://phabricator.wikimedia.org/T387376) [12:02:01] hashar: would you be so kind? ^^ [12:19:47] jayme: sure!! :) [12:20:27] (03CR) 10Hashar: [C:03+2] Revert^2 "helm-linter: Update helm and helmfile" [integration/config] - 10https://gerrit.wikimedia.org/r/1124077 (https://phabricator.wikimedia.org/T387376) (owner: 10JMeybohm) [12:22:23] (03Merged) 10jenkins-bot: Revert^2 "helm-linter: Update helm and helmfile" [integration/config] - 10https://gerrit.wikimedia.org/r/1124077 (https://phabricator.wikimedia.org/T387376) (owner: 10JMeybohm) [12:25:07] jayme: 2025-03-03 12:24:24 [docker-pkg-build] INFO - helm-diff : Depends: libc6 (>= 2.34) but 2.31-13+deb11u8 is to be installed [12:25:30] the base image is bullseye [12:26:18] and bookworm/stable only has 2.36 [12:27:08] what I don't get is that CI should be catching those [12:33:16] (03approved) 10jnuche: trigger_docpub.py: Handle missing Location header better [repos/releng/docpub] - 10https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/27 (owner: 10dancy) [12:58:15] hashar: oops, let me drop the dependencies. Those are statically compiles binaries anyways [12:59:03] ah [12:59:18] also the CI job should probably have caught it [12:59:29] or at least I don't know how 2.34 was added [12:59:51] well, we build on bookworm - so CI can't catch that [13:26:06] rebuild is running [13:32:37] (03open) 10oacorn: Fix port mapping logic [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/607 (https://phabricator.wikimedia.org/T387729) [13:35:40] hashar: should be fixed now. Could you try again please? [13:36:07] trying [13:37:12] 2025-03-03 13:36:50 [docker-pkg-build] INFO - helm311 : Conflicts: helm3 (< 3.11.3-3) (drivers.py:106) [13:37:12] 2025-03-03 13:36:50 [docker-pkg-build] INFO - E: Unable to correct problems, you have held broken packages. [13:37:15] eheh progress! : [13:37:16] ) [13:42:34] jayme: so yeah still broken but differently this time! [13:42:52] hmpf [13:43:49] docker run --rm -it --entrypoint=bash --user=root docker-registry.wikimedia.org/releng/ci-bullseye -c 'apt update && apt install helm311 helm317 helmfile helm-diff' [13:43:51] that is the repro [13:46:24] but if I install helmfile it says: [13:46:25] The following additional packages will be installed: [13:46:26] helm-diff (3.10.0-2) [13:46:26] helm3 (3.11.3-1) [13:46:40] and I can them install helm317 [13:47:03] but when installing helm311 it says it will remove helm-diff helm3 helmfile [13:47:04] :) [13:47:16] so I guess there is some magic annoyance between helm311 and helm3 [13:47:39] I think it's because helm-diff depends on helm and neither of the helmXXX packages provide helm [13:48:08] ahh [13:48:20] 10GitLab (CI & Job Runners): https://gitlab.wikimedia.org/repos/test-platform/catalyst/ci-charts buildkit failure - https://phabricator.wikimedia.org/T386947#10595974 (10jnuche) Used https://gitlab.wikimedia.org/repos/test-platform/catalyst/ci-charts/-/commit/fea7ab56a5336582d581e836bb7f4c271b78c967 to trigger h... [13:54:01] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T386214#10595998 (10mszabo) ##### Risky Patch! πŸš‚πŸ”₯ * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1123674, https://gerrit.wikimedia.org/r/c... [14:07:49] 06Gerrit-Privilege-Requests: Request membership in toollabs-trusted, wmcs-trusted group for chuckonwumelu - https://phabricator.wikimedia.org/T387584#10596045 (10aborrero) 05Openβ†’03Resolved a:03aborrero done! [14:17:01] hashar: okay...now it works, promise πŸ˜‡ [14:47:00] * hashar got lost in Java garbage collection [14:47:40] * andre passes another trashcan [14:48:37] jayme: 2025-03-03 14:48:06 [docker-pkg-build] INFO - update-alternatives: error: alternative /usr/bin/helm311 for helm not registered; not setting [14:48:37] :) [14:48:53] which I guess is in the Dockerfile.template [14:49:29] aw jeez! [14:51:26] ah yeah [14:51:31] jayme: yeah they are missing dots [14:51:40] update-alternatives --query helm3 [14:51:50] tells me Alternative: /usr/bin/helm3.11 [14:51:50] Alternative: /usr/bin/helm3.17 [14:51:51] :) [14:52:18] yeah...ofc. let me fix that and try to build locally before I get to you again [14:52:58] of course it does not help we don't have CI for patches made to docker-pkg definitions :b [14:53:45] is does also not help that I did not test my shit πŸ™ˆ [15:01:37] oh I am not the one that would blame you on that ! :b [15:01:37] (03PS1) 10JMeybohm: helm-linter: Fix update-alternatives calls [integration/config] - 10https://gerrit.wikimedia.org/r/1124115 [15:02:18] (03CR) 10Hashar: [C:03+2] helm-linter: Fix update-alternatives calls [integration/config] - 10https://gerrit.wikimedia.org/r/1124115 (owner: 10JMeybohm) [15:02:23] local docker-pkg call passes for that one [15:02:35] awesome! [15:02:52] I think I suggested helm311 / dotless version when I did the code review last week [15:02:52] we'll see :D [15:03:16] yeah, well...I'm the one who copied it without checking :D [15:03:36] 10Gerrit, 06collaboration-services, 10Projects-Cleanup, 10Wikidata, 10Wikidata Query UI: Archive wikidata/query/gui Gerrit repository - https://phabricator.wikimedia.org/T387199#10596380 (10Nikerabbit) It seems translatewiki was not informed: {T387741} [15:03:43] I need to setup another Gerrit just to code review the code reviews [15:03:59] * hashar looks at ChatGPT [15:04:05] (03Merged) 10jenkins-bot: helm-linter: Fix update-alternatives calls [integration/config] - 10https://gerrit.wikimedia.org/r/1124115 (owner: 10JMeybohm) [15:08:17] Successfully published image docker-registry.discovery.wmnet/releng/helm-linter:0.7.0 [15:08:42] jayme: congratulations, should I switch the Jenkins job to it? [15:10:05] (03PS1) 10Hashar: jjb: update helm-lint job to updated helm image [integration/config] - 10https://gerrit.wikimedia.org/r/1124118 (https://phabricator.wikimedia.org/T387376) [15:12:46] hashar: yes please [15:36:06] (03CR) 10Hashar: [C:03+2] jjb: update helm-lint job to updated helm image [integration/config] - 10https://gerrit.wikimedia.org/r/1124118 (https://phabricator.wikimedia.org/T387376) (owner: 10Hashar) [15:36:12] jayme: I have updated the job! [15:37:17] hashar: great, thanks! I'll recheck something [15:38:30] (03Merged) 10jenkins-bot: jjb: update helm-lint job to updated helm image [integration/config] - 10https://gerrit.wikimedia.org/r/1124118 (https://phabricator.wikimedia.org/T387376) (owner: 10Hashar) [16:03:57] hashar: looks good apart from CI now failing (because validation is better maybe) - I'll go check [16:14:24] jayme: cool! You can try with an empty change as well [17:13:15] (03merge) 10dancy: trigger_docpub.py: Handle missing Location header better [repos/releng/docpub] - 10https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/27 [17:13:28] 06Release-Engineering-Team, 06Security-Team, 07SecTeam-Processed: Clarify what is the security patch deployment procedure - https://phabricator.wikimedia.org/T384439#10597088 (10sbassett) 05In progressβ†’03Resolved a:03sbassett Since there's been no other updates or feedback provided here, I'm going... [17:17:45] 10GitLab (CI & Job Runners): https://gitlab.wikimedia.org/repos/test-platform/catalyst/ci-charts buildkit failure - https://phabricator.wikimedia.org/T386947#10597117 (10dancy) 05Openβ†’03Resolved a:03dancy Thanks for the notes @jnuche. Closing this ticket. [17:23:18] (03PS1) 10JMeybohm: helm-linter: Bump version to update helmfile and helm-diff [integration/config] - 10https://gerrit.wikimedia.org/r/1124153 [17:25:52] (03PS2) 10JMeybohm: helm-linter: Bump version to update helmfile and helm-diff [integration/config] - 10https://gerrit.wikimedia.org/r/1124153 (https://phabricator.wikimedia.org/T387376) [17:25:55] (03PS1) 10JMeybohm: jjb: update helm-lint job to updated helm image [integration/config] - 10https://gerrit.wikimedia.org/r/1124154 (https://phabricator.wikimedia.org/T387376) [17:26:23] hashar: helmfile package not yet ready, 5min or so :p [17:26:58] jayme: sure poke me when ready :] [17:27:26] (03CR) 10CI reject: [V:04-1] jjb: update helm-lint job to updated helm image [integration/config] - 10https://gerrit.wikimedia.org/r/1124154 (https://phabricator.wikimedia.org/T387376) (owner: 10JMeybohm) [17:27:34] I am following up on some code review for a web browser extension so I am not going to vanish anytime soon :) [17:30:46] 06Project-Admins, 06Security-Team, 10Vulnerability Management, 07SecTeam-Processed, 07Security: Proposed Security Tags for Phabricator - https://phabricator.wikimedia.org/T387508#10597200 (10sbassett) 05Openβ†’03In progress [17:32:59] (03update) 10dduvall: templates: Create GitLab CI component [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/120 [17:33:15] (03update) 10dduvall: templates: Create GitLab CI component [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/120 [17:33:43] (03update) 10dduvall: blubber.yaml: Include GitLab release-cli in our image [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/121 [17:33:45] (03update) 10dduvall: blubber.yaml: Include GitLab release-cli in our image [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/121 [17:37:21] hashar: done [17:37:41] if you're still around :) ... building helmfile is a bit slow :/ [17:39:09] (03update) 10dancy: templates: Create GitLab CI component [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/120 (owner: 10dduvall) [17:39:24] (03approved) 10dancy: templates: Create GitLab CI component [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/120 (owner: 10dduvall) [17:39:47] (03approved) 10dancy: blubber.yaml: Include GitLab release-cli in our image [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/121 (owner: 10dduvall) [17:39:48] (03PS2) 10JMeybohm: jjb: update helm-lint job to updated helm image [integration/config] - 10https://gerrit.wikimedia.org/r/1124154 (https://phabricator.wikimedia.org/T387376) [17:49:53] bd808: I am testing the xde bug extension :) [17:50:02] jayme: yes I am around [17:52:11] (03CR) 10Hashar: [C:03+1] "All good, thanks for the follow up. I am giving it a quick round of testing with the Debian Firefox ESR. Thank you for the migration!" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) (owner: 10BryanDavis) [17:55:31] hashar: would love another helm-lint build - but tomorrow is fine as well [17:55:42] yeah [17:55:59] I am poor at multi tasking it seems [17:56:09] or I need more computers in order to easily context switch between tasks :b [17:56:21] (03CR) 10Hashar: [C:03+2] helm-linter: Bump version to update helmfile and helm-diff [integration/config] - 10https://gerrit.wikimedia.org/r/1124153 (https://phabricator.wikimedia.org/T387376) (owner: 10JMeybohm) [17:56:33] eheh [17:57:57] (03Merged) 10jenkins-bot: helm-linter: Bump version to update helmfile and helm-diff [integration/config] - 10https://gerrit.wikimedia.org/r/1124153 (https://phabricator.wikimedia.org/T387376) (owner: 10JMeybohm) [17:58:01] we should grant you the appropriate accesses so you can do it on your own [17:58:15] though I am happy to do it :) [17:59:27] jayme: it is building [17:59:50] what would rob me from a big chunk of joyful interaction with you :D [17:59:58] touchΓ© [18:00:08] :-] [18:00:32] it is pushing [18:01:01] ah that remembers me about Docker being terrible at pushing layers to the registry [18:01:24] jayme: job updated! [18:02:10] hashar: cool, thanks [18:04:25] (03CR) 10Hashar: [C:03+2] jjb: update helm-lint job to updated helm image [integration/config] - 10https://gerrit.wikimedia.org/r/1124154 (https://phabricator.wikimedia.org/T387376) (owner: 10JMeybohm) [18:05:44] (03Merged) 10jenkins-bot: jjb: update helm-lint job to updated helm image [integration/config] - 10https://gerrit.wikimedia.org/r/1124154 (https://phabricator.wikimedia.org/T387376) (owner: 10JMeybohm) [18:09:29] (03CR) 10Hashar: [C:03+2] "Looks good to me in Chromium 133 and Firefox-esr 128.6.0 \o/" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) (owner: 10BryanDavis) [18:09:53] (03Merged) 10jenkins-bot: Convert to Manifest v3 [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1108864 (https://phabricator.wikimedia.org/T312694) (owner: 10BryanDavis) [18:10:20] bd808: merged \o/ [18:10:30] thank you for the **heavy** lifting [18:15:26] jayme: that poor helm-lint looks stuck at https://integration.wikimedia.org/ci/job/helm-lint/23279/console [18:15:27] :( [18:16:00] 3586712 ? Sl 0:00 \_ docker run --volume /srv/jenkins/workspace/helm-lint/src:/src --volume /srv/jenkins/workspace/helm-lint/cache:/cache --volume /srv/jenkins/workspace/helm-lint/log:/log --security-opt seccomp=unconfined --init --rm --label jenkins.job=helm-lint --label jenkins.build=23279 --env-file /dev/fd/63 docker-registry.wikimedia.org/releng/helm-linter:0.7.1 [18:16:00] 3586720 ? Z 0:00 \_ [bash] [18:16:06] not sure how helpful that is :b [18:16:11] but there is a ZOMBIE! [18:16:18] hashar: thanks for the review and the good questions. I will try to get the packaging done today. [18:17:04] jayme: oh it eventually has completed and reports on deployment after being quiet for ~ 12 minutes :) [18:17:34] bd808: that was a challenging one opening at least three different can of worms. But I had fun [18:17:45] hashar: yeah...it takes a quite a while for a full CI run these days [18:17:55] bd808: kudos on doing the migration, that must have been a few hours of digging in various docs [18:18:14] jayme: at least it seems to be working [18:18:33] don't jinx it [18:19:36] oh we have a law for that in france: "Loi de l'emmerdement maximum" (literally: law of maximum enshitification) [18:19:42] aka Murphy law :) [18:20:10] maybe the deployments [18:20:12] err [18:20:16] 00:17:47.182 Finished: SUCCESS [18:20:18] jayme: hurrah [18:20:32] hashar: it was a day or two of web searches for sure. And then the inevitable discoveries of things that weren't in the docs I had been able to find. :) [18:20:52] yay [18:20:58] thanks for your patience [18:22:24] jayme: anything else needed or can I wrap up? ;) [18:23:38] bd808: and about the icon the fun thing is Webkit already has the icon_variants entry in manifest v3 https://bugs.webkit.org/show_bug.cgi?id=279135 [18:23:59] hashar: you're free to go from my end πŸ˜‚ [18:24:23] jayme: awesome! thank you for having taken care of CI / helm diff and all the Debian packaging! [18:27:58] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T386214#10597611 (10hashar) Hi I am running the train this week and given I am based in Europe I will run it at 09:00–11:00 UTC (10am CET). Thank you @mszabo... [18:29:23] dinner time [18:43:31] 06Project-Admins, 06Security-Team, 10Vulnerability Management, 07SecTeam-Processed, 07Security: Proposed Security Tags for Phabricator - https://phabricator.wikimedia.org/T387508#10597755 (10Aklapper) > In addition to the existing Vuln-* tags, the following new categories are suggested: I'd say feel fre... [18:52:13] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [sa.lelya] - https://phabricator.wikimedia.org/T383007#10597830 (10Aklapper) 05Openβ†’03Declined No reply; closing. [19:24:31] (03merge) 10dduvall: templates: Create GitLab CI component [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/120 [19:24:55] (03update) 10dduvall: blubber.yaml: Include GitLab release-cli in our image [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/121 [19:27:35] (03merge) 10dduvall: blubber.yaml: Include GitLab release-cli in our image [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/121 [19:43:40] 10Continuous-Integration-Config: Several recent slow (>15 minute) helm-lint job runs - https://phabricator.wikimedia.org/T387781#10598025 (10dancy) [19:45:47] 10Continuous-Integration-Config: Several recent slow (>15 minute) helm-lint job runs - https://phabricator.wikimedia.org/T387781#10598035 (10dancy) @JMeybohm I don't know if your change https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1123006 is the caused of the slow `helm-lint` runs but I'm wond... [20:15:17] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T386214#10598111 (10Tgr) ##### Risky Patch! πŸš‚πŸ”₯ * **Change**: {d8f6bd4b6bd037779dbc5f30577602fc98fafdca}, {824bea33b28d3a9c6a49f8d9954dcc5fcc73d83e} * **Summ... [20:18:30] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-on-K8s, 06serviceops, 06SRE: Automated validation of mediawiki-multiversion images - https://phabricator.wikimedia.org/T288629#10598113 (10dancy) >>! In T288629#10582102, @JMeybohm wrote: > I stumbled upon this again recently and I think the current con... [20:20:03] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-on-K8s, 06serviceops, 06SRE: Automated validation of mediawiki-multiversion images - https://phabricator.wikimedia.org/T288629#10598121 (10dancy) 05Openβ†’03Resolved a:03dancy ` After the build process creates the restricted mediawiki-multiversi... [20:52:25] (03PS7) 10Jforrester: Zuul: Switch all wmf-quibble gate jobs from 7.4 to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1115147 (https://phabricator.wikimedia.org/T320818) [20:52:30] (03PS8) 10Jforrester: Zuul: Switch all wmf-quibble gate jobs from 7.4 to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1115147 (https://phabricator.wikimedia.org/T320818) [20:52:36] (03CR) 10Jforrester: [C:03+2] "Let's do it." [integration/config] - 10https://gerrit.wikimedia.org/r/1115147 (https://phabricator.wikimedia.org/T320818) (owner: 10Jforrester) [20:54:31] (03Merged) 10jenkins-bot: Zuul: Switch all wmf-quibble gate jobs from 7.4 to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1115147 (https://phabricator.wikimedia.org/T320818) (owner: 10Jforrester) [21:05:54] 10Continuous-Integration-Config, 13Patch-For-Review, 07PHP 8.1 support: Configure CI to test the Wikimedia production gate code against PHP 8.1 instead of PHP 7.4 - https://phabricator.wikimedia.org/T320818#10598256 (10Jdforrester-WMF) 05Openβ†’03Resolved Now live; prod is at 30%, it's time. [21:31:50] (03PS1) 10BryanDavis: v3.0.0: Update changelog [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1124189 [21:32:57] (03CR) 10BryanDavis: [C:03+2] v3.0.0: Update changelog [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1124189 (owner: 10BryanDavis) [21:33:18] (03Merged) 10jenkins-bot: v3.0.0: Update changelog [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1124189 (owner: 10BryanDavis) [22:05:11] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10WikimediaDebug: Support WebExtensions Manifest v3 - https://phabricator.wikimedia.org/T312694#10598440 (10bd808) v3.0.0 submitted to Chrome Web Store for review and publication: {F58586152, size=full} [22:33:37] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10WikimediaDebug: Support WebExtensions Manifest v3 - https://phabricator.wikimedia.org/T312694#10598579 (10bd808) v3.0.0 uploaded to https://addons.mozilla.org/ [22:40:50] 06Release-Engineering-Team, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796 (10Dzahn) 03NEW [22:43:04] 06Release-Engineering-Team, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10598664 (10Dzahn) Also the part that if the only notification is an IRC line on the -operations channel it is not easily noticed anymore nowadays. Might it be better if that was an email... [22:45:32] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10WikimediaDebug: Support WebExtensions Manifest v3 - https://phabricator.wikimedia.org/T312694#10598671 (10bd808) 05Openβ†’03In progress [22:55:22] 10Scap: scap confused when deploying a patch with a cross-repo dependency - https://phabricator.wikimedia.org/T387798 (10Tgr) 03NEW [22:55:27] 10WikimediaDebug, 06MediaWiki-Platform-Team, 10Observability-Tracing: Add support for request tracing to WikimediaDebug browser extension - https://phabricator.wikimedia.org/T340573#10598730 (10bd808) >>! In T340573#9288404, @CDanis wrote: > I'll need to get back to you about the URL. We don't have jaeger-u... [22:56:03] 10Scap: scap confused when deploying a patch with a cross-repo dependency - https://phabricator.wikimedia.org/T387798#10598736 (10Tgr) {T353720} is a similar task, not sure if it's the same issue. [23:05:06] 10WikimediaDebug, 10Tool-schedule-deployment: Allow schedule-deployment to determine if WikimediaDebug browser add-on is installed to simplify end user instructions - https://phabricator.wikimedia.org/T367213#10598753 (10bd808) [23:06:11] 10Gerrit, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Remove explicit enablement of G1 garbage collector for Gerrit - https://phabricator.wikimedia.org/T387223#10598754 (10thcipriani) >>! In T387223#10595592, @hashar wrote: > @thcipriani `G1NewSizePercent` and `+UseStringDedupl... [23:24:50] 10Release-Engineering-Team (Radar), 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10598803 (10thcipriani) > Somehow needs cleaning up but since it's not OS but actual deployment data the question is what can be deleted. Probably old mw versions.. Old MW versions... [23:37:25] 10WikimediaDebug: Extract and display hostname from server header - https://phabricator.wikimedia.org/T340727#10598839 (10bd808) We don't currently vary display on the popup.html add-on control screen based on the active tab's content other than toggling `hidden: true` for the `.warning` css class based on the `... [23:41:55] 10Release-Engineering-Team (Radar), 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10598845 (10Dzahn) Yea, those are mediawiki-multiversion images. Some are 8GB. Example: ` docker-registry.discovery.wmnet/restricted/mediawiki-multiversion 2025-02-19-01... [23:42:29] 10Release-Engineering-Team (Radar), 06serviceops-radar, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10598848 (10Dzahn) [23:45:34] 10Release-Engineering-Team (Radar), 06serviceops-radar, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10598850 (10Dzahn) p:05Triageβ†’03High With about 8.8GB space left and those images that can also be about 8.8G and sometimes multiple images per day ...I am... [23:45:51] 10WikimediaDebug: Extract and display hostname from server header - https://phabricator.wikimedia.org/T340727#10598852 (10Reedy) p:05Triageβ†’03Low [23:45:57] thcipriani: maybe we should stop deploys [23:46:28] seems like one more image could mean 0 bytes left [23:46:41] at least it's a separate partition though [23:48:22] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Seen), 06Data-Engineering: Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10598866 (10thcipriani) >>! In T384364#10527963, @Ottomata wrote: > Related: > {T367322} > {T379187} > > @... [23:52:59] 10WikimediaDebug, 07Upstream: Display better error message in XHGui for empty profile - https://phabricator.wikimedia.org/T259950#10598880 (10bd808) 05Openβ†’03Resolved > Example: https://performance.wikimedia.org/xhgui/?url=5287f7a7-07fe-4bac-bc07-d4f23eeb607b 4 years later this example URL displays "Y... [23:56:16] mutante: we do build three images per deploy. I'd imagine that the latest image build only needs the latest build (there's some rsync magic it uses) and so the older images are superfluous, but I'm not sure enough to untag and do a garbage collection :\ [23:57:57] I also wonder if we even need to keep _any_ images on disk or if the build script will pull down what it needs... [23:58:03] thcipriani: yep, it seems this needs some new kind of automated cleanup.. since mw moved to k8s [23:58:41] added serviceops-radar [23:59:12] imho it could also benefit from a different type of notification for the monitoring alert [23:59:37] it feels like "just a line on -operations" used to work but not really anymore