[00:00:36] <wikibugs>	 10Beta-Cluster-Infrastructure: Beta cluster log spam: Echo: Couldn't resolve host name - https://phabricator.wikimedia.org/T383588#12074145 (10bd808) 05Open→03Declined Not happening at the moment and 18 months old.
[00:06:55] <wikibugs>	 10Scap, 07Beta-Cluster-reproducible: scap log doesn't always provide enough information on an error - https://phabricator.wikimedia.org/T377866#12074149 (10bd808)
[00:08:46] <wikibugs>	 10Beta-Cluster-Infrastructure: "Error: Class '(random class)' not found" logspam in beta - https://phabricator.wikimedia.org/T371507#12074152 (10bd808) 05Open→03Declined Not happening currently and 2 years old.
[00:52:16] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikimediaMaintenance, 10Wikimedia-Interwiki-links: Beta pages with interwiki URLs redirect to production - https://phabricator.wikimedia.org/T355279#12074232 (10bd808) This actually seems to be only partially problematic. In https://meta.wikimedia.beta.wm...
[02:45:36] <hashar>	 !log gerrit: on gerrit2003 deleted /srv/gerrit/java_pid3520115.hprof  (the JVM apparently died at some point, I assume due to heavy crawling)
[02:45:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[03:16:15] <wikibugs>	 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12074313 (10hashar) ` +----------------------------------------------------+-------------------+ | Name                                               | lstat_size(LStat) | +---------...
[03:42:13] <wikibugs>	 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12074321 (10hashar) Actions =========  Compact the DB -------------  We might want to raise `-Dh2.maxCompactTime=15000` to some higher limit with the drawback it would delay Gerrit s...
[03:42:52] <wikibugs>	 10Gerrit, 13Patch-For-Review: Decide if Gerrit's indices should get backed up - https://phabricator.wikimedia.org/T257744#12074326 (10hashar) Some discussion happened on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306166/comments/5b48b203_440fd774 which should be captured here.
[06:02:52] <wikibugs>	 10GitLab, 06collaboration-services: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563#12074422 (10Jelto) 05Open→03Declined >>! In T430563#12066958, @Dzahn wrote: > Oh, well, the part that we see "HTTP 502: Waiting for GitLab to boot" on replica-b is unexpected,...
[06:05:35] <wikibugs>	 10GitLab, 06collaboration-services: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563#12074424 (10Jelto) 05Declined→03Open a:03Jelto > We could however use a different robots.txt. But it looks like this is not supported by default and requires some refactoring.  I...
[06:59:50] <wikibugs>	 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12074470 (10jcrespo) I am happy to report that right now backups have been reduced by an order of magnitude, enough to give us time to decide on actionables/followups in a non unbrea...
[07:05:02] <wikibugs>	 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12074489 (10jcrespo) RE: @hashar "when the problem started?" I'd say the last ongoing issue started on 2026-04-12, when backups went from 16 to 34 GB, but that was already too big.
[07:39:23] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Gerrit: network errors in CI - https://phabricator.wikimedia.org/T429749#12074565 (10ABran-WMF) With [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1302834 | more local cache ]] o...
[07:41:21] <wikibugs>	 10Gerrit (Gerrit 3.12), 06collaboration-services: Compact Gerrit H2 databases to reclaim disk - https://phabricator.wikimedia.org/T430774 (10ABran-WMF) 03NEW
[07:41:38] <wikibugs>	 10Gerrit (Gerrit 3.12), 06collaboration-services: Compact Gerrit H2 databases to reclaim disk - https://phabricator.wikimedia.org/T430774#12074587 (10ABran-WMF)
[07:41:39] <wikibugs>	 10Gerrit, 06collaboration-services: Investigate Gerrit root disk usage and logging - https://phabricator.wikimedia.org/T425667#12074588 (10ABran-WMF)
[07:42:18] <wikibugs>	 10Gerrit (Gerrit 3.12), 06collaboration-services: Compact Gerrit H2 databases to reclaim disk - https://phabricator.wikimedia.org/T430774#12074589 (10ABran-WMF) p:05Triage→03Medium a:05ABran-WMF→03None
[07:43:46] <wikibugs>	 10Gerrit, 06collaboration-services: Validate Gerrit backup recoverability after the fileset exclusions - https://phabricator.wikimedia.org/T430775 (10ABran-WMF) 03NEW
[07:44:16] <wikibugs>	 10Gerrit, 06collaboration-services: Validate Gerrit backup recoverability after the fileset exclusions - https://phabricator.wikimedia.org/T430775#12074610 (10ABran-WMF)
[07:44:19] <wikibugs>	 10Gerrit, 13Patch-For-Review: Decide if Gerrit's indices should get backed up - https://phabricator.wikimedia.org/T257744#12074611 (10ABran-WMF)
[07:49:16] <wikibugs>	 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12074620 (10ABran-WMF) >>! In T411583#12074470, @jcrespo wrote: > I would encourage to keep working on this as if right now were are not going to break soon due to disk space issues,...
[07:51:13] <wikibugs>	 10Gerrit, 06collaboration-services: Validate Gerrit backup recoverability after the fileset exclusions - https://phabricator.wikimedia.org/T430775#12074630 (10jcrespo) My only input is that I would delay starting this until all other fixes are in place, to save time and just test it once.
[07:53:06] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Compact Gerrit H2 databases to reclaim disk - https://phabricator.wikimedia.org/T430774#12074640 (10ABran-WMF)
[07:54:25] <wikibugs>	 (03CR) 10Matthias Mullie: [C:03+1] Zuul: Remove PageImages from MMV test dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1306794 (owner: 10Eric Gardner)
[08:10:33] <wikibugs>	 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Request for project "User-tfogli" for organizing my personal workboard - https://phabricator.wikimedia.org/T430128#12074692 (10tappof) Thanks!
[08:47:21] <wikibugs>	 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T423918#12074793 (10Aklapper)
[08:48:06] <wikibugs>	 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T423918#12074797 (10Aklapper) Rolling back wmf.9 from group2 to group1 due to {T430778} creating a large amount of exceptions on wikidata.org only.
[13:38:40] <wikibugs>	 (03PS1) 10Hashar: jjb: add timestamps to console of PCC jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1306929
[13:39:12] <hashar>	 !log integration: added timestamping to operations-puppet-catalog-compiler and operations-puppet-catalog-compiler-puppet7-test jobs
[13:39:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:39:28] <wikibugs>	 (03CR) 10Hashar: [C:03+2] jjb: add timestamps to console of PCC jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1306929 (owner: 10Hashar)
[13:41:26] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: add timestamps to console of PCC jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1306929 (owner: 10Hashar)
[13:50:27] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Horizon, 10Openstack-Magnum, and 2 others: Can't retrieve Magnum cluster certificate and config via Horizon - https://phabricator.wikimedia.org/T430680#12076122 (10Andrew) 05Open→03Resolved
[13:52:34] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Muguro Levy - https://phabricator.wikimedia.org/T430824 (10Muguro) 03NEW
[13:56:28] <wikibugs>	 10GitLab, 06collaboration-services, 13Patch-For-Review: Gitlab replica-b being indexed by Duck Duck Go - https://phabricator.wikimedia.org/T430563#12076211 (10Jelto) 05Open→03Resolved The replicas use a restricted `robots.txt` now and should no longer be indexed. Thank you for reporting the issue! I'...
[15:02:53] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Muguro Levy - https://phabricator.wikimedia.org/T430824#12076693 (10brennen) a:03brennen
[15:05:05] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.47.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T430829#12076708 (10thcipriani) p:05Triage→03Medium a:03hashar
[15:05:29] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.47.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T430830#12076719 (10thcipriani) p:05Triage→03Medium a:03dancy
[15:06:40] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.47.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T430831#12076724 (10thcipriani) p:05Triage→03Medium a:03dduvall
[15:07:14] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.47.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T430832#12076729 (10thcipriani) p:05Triage→03Medium a:03jnuche
[15:07:46] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.47.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T430833#12076740 (10thcipriani) p:05Triage→03Medium a:03jeena
[15:08:12] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834#12076748 (10thcipriani) p:05Triage→03Medium a:03brennen
[15:12:52] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Muguro Levy - https://phabricator.wikimedia.org/T430824#12076788 (10Muguro) Hi @brennen ,   I have linked my Phabricator account and still getting the following when I try to log in  **Your account is pending appr...
[15:14:24] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T430835#12076797 (10thcipriani) p:05Triage→03Medium a:03Aklapper
[15:15:00] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T430836#12076810 (10thcipriani) p:05Triage→03Medium a:03hashar
[15:15:39] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T430837#12076818 (10thcipriani) p:05Triage→03Medium a:03dancy
[15:16:19] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T430838#12076825 (10thcipriani) p:05Triage→03Medium a:03dduvall
[15:16:52] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T430839#12076838 (10thcipriani) p:05Triage→03Medium a:03jnuche
[15:17:58] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 10Catalyst (Luka Ijo Pimeja Jan), 05MW-1.46-release, 13Patch-For-Review: Cannot select REL1_46 on PatchDemo - https://phabricator.wikimedia.org/T425165#12076845 (10brennen) 05In progress→03Resolved Currently:  {F91454680}
[15:20:27] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Muguro Levy - https://phabricator.wikimedia.org/T430824#12076872 (10Aklapper) Hi, please wait for a short moment and try again: > At this point, the approval bot should automatically approve your account **on its...
[15:30:01] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for Muguro Levy - https://phabricator.wikimedia.org/T430824#12076897 (10brennen) 05Open→03Resolved p:05Triage→03Medium Looks good now. Happy hacking!
[15:37:27] <wmf-insecte>	 Yippee, build fixed!
[15:37:28] <wmf-insecte>	 Project mediawiki-core-doxygen build #21374: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21374/
[15:39:59] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12076938 (10Dzahn) please try again now
[16:07:56] <wikibugs>	 10Scap: Scap / Spiderpig should not warn about i18n rebuild whenever extension.json is changed - https://phabricator.wikimedia.org/T430750#12077056 (10dancy)
[16:07:58] <wikibugs>	 10Scap: scap backport should warn if it knows it will take a long time - https://phabricator.wikimedia.org/T397089#12077057 (10dancy)
[16:08:24] <wikibugs>	 (03open) 10dancy: backport: Remove extension.json from L10N_REBUILD_PATH_PATTERNS [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1212 (https://phabricator.wikimedia.org/T397089 https://phabricator.wikimedia.org/T430750)
[16:08:26] <wikibugs>	 (03open) 10ejegg: Update fundraising ML image to use WSGI [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/109
[16:18:01] <wikibugs>	 (03update) 10dancy: backport: Remove extension.json from L10N_REBUILD_PATH_PATTERNS [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1212 (https://phabricator.wikimedia.org/T397089 https://phabricator.wikimedia.org/T430750)
[16:18:06] <wikibugs>	 (03update) 10dancy: backport: Remove extension.json from L10N_REBUILD_PATH_PATTERNS [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1212 (https://phabricator.wikimedia.org/T397089 https://phabricator.wikimedia.org/T430750)
[16:18:52] <wikibugs>	 (03merge) 10dancy: backport: Remove extension.json from L10N_REBUILD_PATH_PATTERNS [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1212 (https://phabricator.wikimedia.org/T397089 https://phabricator.wikimedia.org/T430750)
[16:19:42] <wikibugs>	 (03open) 10dancy: Release 4.271.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1213
[16:22:21] <wikibugs>	 (03merge) 10dancy: Release 4.271.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1213
[16:31:27] <wikibugs>	 10Scap: Scap / Spiderpig should not warn about i18n rebuild whenever extension.json is changed - https://phabricator.wikimedia.org/T430750#12077208 (10dancy) 05Open→03Resolved a:03dancy Fixed in scap 4.271.0
[16:58:03] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services, 13Patch-For-Review: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12077464 (10hashar) I have confirmed the port 25000 works!  Unfortunatel...
[17:52:18] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for Muguro Levy - https://phabricator.wikimedia.org/T430824#12077815 (10Muguro) yes yes, all is well. Thank you
[18:51:40] <wikibugs>	 (03update) 10ejegg: Update fundraising ML image to use WSGI [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/109
[18:51:47] <wikibugs>	 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T423918#12077985 (10Aklapper) Proceeded with the train again to group1 after `@Jgiannelos` kindly deployed two fixes for T430778. After deplying to gr...
[18:53:28] <andre>	 brennen: Train moved forward and looks okay. Status in https://phabricator.wikimedia.org/T423918#12077985. FYI I wrote in https://phabricator.wikimedia.org/T430778#12077961 to contact you if for any reason the new problem which appeared justified a rollback (definitely does not look like, so far). Thanks!
[18:58:27] <wikibugs>	 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T423918#12078003 (10Aklapper)
[19:15:15] <brennen>	 andre: thanks!  will keep an eye on things.
[19:17:10] <wikibugs>	 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T423918#12078042 (10brennen)
[19:53:44] <wikibugs>	 (03update) 10milazg: Rules: allow valid media types and handle polymorphic properties [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/14 (https://phabricator.wikimedia.org/T428149)
[19:56:46] <wikibugs>	 (03update) 10milazg: Rules: allow valid media types and handle polymorphic properties [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/14 (https://phabricator.wikimedia.org/T428149)
[20:17:10] <wikibugs>	 (03update) 10milazg: Rules: allow valid media types and handle polymorphic properties [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/14 (https://phabricator.wikimedia.org/T428149)
[20:17:57] <wikibugs>	 (03CR) 10Hashar: [C:03+2] Zuul: Remove PageImages from MMV test dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1306794 (owner: 10Eric Gardner)
[20:19:40] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: Remove PageImages from MMV test dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1306794 (owner: 10Eric Gardner)
[20:42:24] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T430832#12078144 (10thcipriani) a:05jnuche→03jeena
[20:42:41] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T430833#12078146 (10thcipriani) a:05jeena→03jnuche
[20:42:55] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T430834#12078149 (10thcipriani)
[21:13:48] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate: Beta cluster is not working due to error in Translate\TtmServer\ServiceCreationFailure - https://phabricator.wikimedia.org/T430882 (10aude) 03NEW
[21:46:57] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate: Beta cluster is not working due to error in Translate\TtmServer\ServiceCreationFailure - https://phabricator.wikimedia.org/T430882#12078316 (10dancy) Probably related to https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1305062
[21:51:55] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate: Beta cluster is not working due to error in Translate\TtmServer\ServiceCreationFailure - https://phabricator.wikimedia.org/T430882#12078323 (10dancy) The change made in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1305062 appear...