[01:39:32] <wikibugs>	 (03open) 10rzl: deployment-calendar: Moritz takes over for Reuven in the Puppet window [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/173
[01:41:33] <rzl>	 ^ that's ready for any kind relenger to merge at your convenience, either if you're around late or when you see it in the morning :)
[06:24:39] <wmf-insecte>	 Project beta-code-update-eqiad build #548693: 04FAILURE in 1 min 39 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/548693/
[06:30:39] <wmf-insecte>	 Project mediawiki-core-doxygen build #10560: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/10560/
[06:35:05] <wmf-insecte>	 Yippee, build fixed!
[06:35:05] <wmf-insecte>	 Project beta-code-update-eqiad build #548694: 09FIXED in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/548694/
[06:59:03] <wikibugs>	 10Continuous-Integration-Config, 10banana-checker, 10MediaWiki-Core-Tests, 06translatewiki.net, and 2 others: Should banana-checker tests be added to gate-and-submit-l10n? - https://phabricator.wikimedia.org/T394681#10837866 (10A_smart_kitten)
[07:20:40] <wikibugs>	 10Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T392172#10837886 (10Aklapper)
[07:29:51] <wmf-insecte>	 Yippee, build fixed!
[07:29:51] <wmf-insecte>	 Project mediawiki-core-doxygen build #10561: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/10561/
[07:35:56] <wikibugs>	 10Continuous-Integration-Infrastructure, 10LDAP-Access-Requests, 10SRE-Access-Requests: Grant Jenkins admin rights to Peter Hedenskog (QTE) - https://phabricator.wikimedia.org/T394749 (10hashar) 03NEW
[07:36:26] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 10LDAP-Access-Requests, 10SRE-Access-Requests: Grant Jenkins admin rights to Peter Hedenskog (QTE) - https://phabricator.wikimedia.org/T394749#10837923 (10hashar)
[07:47:36] <wikibugs>	 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10837944 (10Jelto) GitLab 18 is released https://about.gitlab.com/releases/2025/05/15/gitlab-18-0-released/
[07:48:30] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 10LDAP-Access-Requests, 10SRE-Access-Requests, 13Patch-For-Review: Grant Jenkins admin rights to Peter Hedenskog (QTE) - https://phabricator.wikimedia.org/T394749#10837946 (10hashar) + @thcipriani as the manager approving shell access / contint-admins (...
[09:46:30] <wikibugs>	 (03PS1) 10Hashar: wm-zuul-status: reset current checks [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1148286 (https://phabricator.wikimedia.org/T394485)
[09:46:44] <jgleeson>	 crossposting from wikimedia-ops: Hi folks, I'm not familiar with the backporting process, and I am trying to help push out a UBN for central-notice, which powers our fundraising banners. Jdlrobson has advised me to submit the patch via https://schedule-deployment.toolforge.org/ to be released as a backport, but when I submit the patch number, I get "Only unmerged changes can be backported.". I'm not sure how to proceed. Do I need 
[09:46:44] <jgleeson>	 to create a new unmerged patch from the approved and merged fix? Here's the patch chain we'd like to deploy https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralNotice/+/1147911/1. Any help would be greatly appreciated! 
[09:48:56] <wikibugs>	 (03CR) 10Hashar: [C:03+2] wm-zuul-status: reset current checks [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1148286 (https://phabricator.wikimedia.org/T394485) (owner: 10Hashar)
[09:49:27] <wikibugs>	 (03Merged) 10jenkins-bot: wm-zuul-status: reset current checks [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1148286 (https://phabricator.wikimedia.org/T394485) (owner: 10Hashar)
[09:58:02] <wikibugs>	 10Gerrit, 13Patch-For-Review: Incorrect "CI has completed checks" popup appears when navigating from a change with tests in progress to one with no tests in progress - https://phabricator.wikimedia.org/T394485#10838252 (10hashar) @A_smart_kitten thank you for the detailed report! Indeed `curChecks` was not cle...
[10:01:14] <Dreamy_Jazz>	 jgleeson: I think you need to create backports in one of the wmf.X branches.
[10:01:37] <Dreamy_Jazz>	 Specifically the branch(es) for the versions having the issues listed at https://versions.toolforge.org/
[10:01:47] <taavi>	 Dreamy_Jazz: already being discussed on -operations
[10:02:02] <Dreamy_Jazz>	 Thanks. Didn't see that.
[10:03:55] <jgleeson>	 thanks Dreamy_Jazz!
[10:05:07] <wmf-insecte>	 maintenance-disconnect-full-disks build 703252 integration-agent-docker-1052 (/: 25%, /srv: 18%, /var/lib/docker: 100%): OFFLINE due to disk space
[10:20:06] <wmf-insecte>	 maintenance-disconnect-full-disks build 703255 integration-agent-docker-1052 (/: 25%, /srv: 18%, /var/lib/docker: 100%): still OFFLINE due to disk space
[10:35:48] <wikibugs>	 06Project-Admins, 06Traffic: Create a #HaproxyKafka tag - https://phabricator.wikimedia.org/T394758 (10Fabfur) 03NEW
[10:37:49] <wikibugs>	 06Project-Admins, 10HaproxyKafka, 06Traffic: Create a #HaproxyKafka tag - https://phabricator.wikimedia.org/T394758#10838376 (10Peachey88) 05Open→03Resolved a:03Peachey88 Project created. Please feel free to encourage users to watch the project etc :).
[10:45:09] <wmf-insecte>	 maintenance-disconnect-full-disks build 703260 integration-agent-docker-1052 (/: 25%, /srv: 18%, /var/lib/docker: 100%): still OFFLINE due to disk space
[10:53:12] <wikibugs>	 (03open) 10cgoubert: mwscript-mwcron: Add some logging [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/174
[11:04:31] <wikibugs>	 (03approved) 10oblivian: Allow deployments to specify custom k8s clusters [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/795 (https://phabricator.wikimedia.org/T388761) (owner: 10swfrench)
[11:08:11] <wikibugs>	 10Gerrit: gerrit.wikimedia.org: Verification Email never recieved. - https://phabricator.wikimedia.org/T394762 (10TJungHW) 03NEW
[11:10:33] <wmf-insecte>	 maintenance-disconnect-full-disks build 703265 integration-agent-docker-1052 (/: 25%, /srv: 18%, /var/lib/docker: 100%): still OFFLINE due to disk space
[11:13:33] <wikibugs>	 10Gerrit: gerrit.wikimedia.org: Verification Email never received - https://phabricator.wikimedia.org/T394762#10838523 (10Aklapper)
[11:14:28] <wikibugs>	 10Gerrit, 10Bitu, 06Infrastructure-Foundations: gerrit.wikimedia.org: Verification Email never received - https://phabricator.wikimedia.org/T394762#10838526 (10Reedy)
[11:15:04] <wmf-insecte>	 maintenance-disconnect-full-disks build 703266 integration-agent-docker-1061 (/: 25%, /srv: 32%, /var/lib/docker: 98%): OFFLINE due to disk space
[11:20:07] <wmf-insecte>	 maintenance-disconnect-full-disks build 703267 integration-agent-docker-1061 (/: 25%, /srv: 26%, /var/lib/docker: 98%): RECOVERY disk space OK
[11:20:19] <wikibugs>	 10Phabricator: Please remove an MFA factor from my Phabricator account - https://phabricator.wikimedia.org/T394763 (10SomeRandomDeveloper) 03NEW
[11:22:22] <wikibugs>	 (03merge) 10cgoubert: mwscript-mwcron: Add some logging [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/174
[11:30:10] <wikibugs>	 (03update) 10cgoubert: mwscript-mwcron: Add some logging [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/174
[11:35:05] <wmf-insecte>	 maintenance-disconnect-full-disks build 703270 integration-agent-docker-1052 (/: 25%, /srv: 18%, /var/lib/docker: 100%): still OFFLINE due to disk space
[11:46:05] <wikibugs>	 10Phabricator, 06Security-Team: Please remove an MFA factor from my Phabricator account - https://phabricator.wikimedia.org/T394763#10838651 (10Aklapper) Hi, uh I once also ran into setting up two MFA factors... I am going to file an upstream ticket to improve the UI to make this clearer.  These reset requests...
[11:52:23] <wikibugs>	 10Gerrit, 10Bitu, 06Infrastructure-Foundations: gerrit.wikimedia.org: Verification Email never received - https://phabricator.wikimedia.org/T394762#10838657 (10SLyngshede-WMF) p:05Triage→03High
[11:54:02] <wikibugs>	 10Gerrit, 10Bitu, 06Infrastructure-Foundations: gerrit.wikimedia.org: Verification Email never received - https://phabricator.wikimedia.org/T394762#10838663 (10SLyngshede-WMF) Hi, I don't see the signup request anywhere, so I can't resend the email.  Can you provide me with the username you tried to create?...
[11:54:48] <wikibugs>	 10Gerrit, 10Bitu, 06Infrastructure-Foundations: gerrit.wikimedia.org: Verification Email never received - https://phabricator.wikimedia.org/T394762#10838665 (10SLyngshede-WMF) a:03SLyngshede-WMF
[11:56:37] <wikibugs>	 10Phabricator, 06Security-Team: Please remove an MFA factor from my Phabricator account - https://phabricator.wikimedia.org/T394763#10838667 (10SomeRandomDeveloper) Hi, thank you for the quick response. Unfortunately, nobody in the MW community knows how I look like, because I am very careful about sharing my...
[12:00:45] <wmf-insecte>	 maintenance-disconnect-full-disks build 703275 integration-agent-docker-1052 (/: 25%, /srv: 18%, /var/lib/docker: 100%): still OFFLINE due to disk space
[12:15:37] <wikibugs>	 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, 06SRE, 10vm-requests: codfw: 1VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10838820 (10Jelto) Thanks @Dzahn for picking this up. I left a comment in [operations/puppet/+/1147855](https://gerri...
[12:25:05] <wmf-insecte>	 maintenance-disconnect-full-disks build 703280 integration-agent-docker-1052 (/: 25%, /srv: 18%, /var/lib/docker: 100%): still OFFLINE due to disk space
[12:30:19] <wikibugs>	 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10838876 (10Jelto) We’ve decided to start by switching `s3://gitlab-artifacts` to object storage. Since a...
[12:30:39] <wikibugs>	 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10838888 (10Jelto)
[12:50:05] <wmf-insecte>	 maintenance-disconnect-full-disks build 703285 integration-agent-docker-1052 (/: 25%, /srv: 18%, /var/lib/docker: 100%): still OFFLINE due to disk space
[13:00:35] <wikibugs>	 (03CR) 10Hashar: [C:04-1] "In a decade you are probably the first one trying to run those shell scripts on a system that lack `/bin/bash`, or others did a symlink si" [integration/config] - 10https://gerrit.wikimedia.org/r/1147819 (owner: 10BryanDavis)
[13:07:20] <wikibugs>	 10Phabricator, 06Security-Team: Please remove an MFA factor from my Phabricator account - https://phabricator.wikimedia.org/T394763#10839092 (10Aklapper) Garrr, Phabricator doesn't allow removing only one of two 2FA using the very same "provider" (TOTP)
[13:10:20] <wikibugs>	 (03CR) 10Hashar: [C:04-1] "I had that discussion a few times and I am not willing to support bash v3. Please install bash using Homebrew or whatever package manager/" [integration/config] - 10https://gerrit.wikimedia.org/r/1147820 (owner: 10BryanDavis)
[13:11:33] <Krinkle>	 is there a Prometheus or other Grafana-native equivalent to "MW deploy" annotations from Scap?
[13:11:41] <Krinkle>	 Looks like those no longer work now since Graphite is read-only
[13:12:16] <Krinkle>	 Aha https://wikitech.wikimedia.org/wiki/Grafana#Show_deployments
[13:12:22] <Krinkle>	 Looks like there's a Loki equivalent already!
[13:15:17] <wmf-insecte>	 maintenance-disconnect-full-disks build 703290 integration-agent-docker-1052 (/: 25%, /srv: 18%, /var/lib/docker: 100%): still OFFLINE due to disk space
[13:33:30] <James_F>	 Krinkle: Ah, yes quibble-bullseye-php81-coverage. I was hoping to land that in March, but of course the prod migration was more cautious.
[13:33:51] <Krinkle>	 Found it, thanks!
[13:34:24] <James_F>	 https://gerrit.wikimedia.org/r/c/integration/config/+/1127083 is the base of all the changes, but we can't do that until we're OK with PHP 7.4-incompatibility landing.
[13:34:38] <James_F>	 Which I imagine SRE would not like with cron jobs. :-)
[13:35:41] <Krinkle>	 ack, there's a number of stages kinds of things: having general stuff that can use either version switch to php81 so that "dropping 74 *could* land in core, even if we don't yet". This e.g. includes docgen and coverage report jobs, and for stuff that runs in multiple versions, to ensure a php81 is included.
[13:35:54] <Krinkle>	 then there's switching the default/test pipeline stuff
[13:36:03] <James_F>	 Yeah, but for some things moving doxygen to php81 might change the output.
[13:36:06] <Krinkle>	 then there's removing 74 from multi-job stuff
[13:36:21] <James_F>	 See e.g. the phan change we landed yesterday, which was mostly a no-op but a handful of repos had different results.
[13:36:39] <Krinkle>	 I think we should start switching some stuff that's single-version but not load-bearing in terms of enforcement to php8.1., e.g. mwext coverage, as an example.
[13:36:42] <James_F>	 Maybe we should do that anyway.
[13:37:13] <Krinkle>	 since  mwext patches still run with quibble 74 and 81 for the main phpunit and linting etc
[13:37:54] <James_F>	 Yeah…
[13:37:58] <James_F>	 Let me fiddle.
[13:38:37] <Krinkle>	 https://gerrit.wikimedia.org/r/c/integration/config/+/1127116/ jjb: Provide php81-compile-coverage-publish
[13:38:41] <Krinkle>	 https://gerrit.wikimedia.org/r/c/integration/config/+/1127117 Docker: Provide quibble-bullseye-php81-coverage
[13:38:45] <Krinkle>	 these LGTM to land first
[13:38:46] <James_F>	 Yup.
[13:39:05] <Krinkle>	 is something up with gerrit bot or I have accidentally ignored it?
[13:39:28] <Krinkle>	 https://wm-bot.wmflabs.org/libera_logs/%23wikimedia-releng/20250520.txt looks like it's not just me
[13:39:58] <James_F>	 Yeah, WIP patches don't get shown.
[13:40:01] <wikibugs>	 (03CR) 10Krinkle: "ignore me, testing wikibugs bot" [integration/config] - 10https://gerrit.wikimedia.org/r/1147820 (owner: 10BryanDavis)
[13:40:03] <wmf-insecte>	 maintenance-disconnect-full-disks build 703295 integration-agent-docker-1052 (/: 25%, /srv: 18%, /var/lib/docker: 100%): still OFFLINE due to disk space
[13:40:10] <Krinkle>	 OK, well that one worked. weird.
[13:40:45] <Krinkle>	 and then the next one is missing again, e.g. me rebasing https://gerrit.wikimedia.org/r/c/integration/config/+/1127117
[13:40:55] <Krinkle>	 anyway, I'm diffing that now against 74 to review
[13:42:56] <wikibugs>	 (03PS4) 10Krinkle: Docker: Provide quibble-bullseye-php81-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1127117 (https://phabricator.wikimedia.org/T328921) (owner: 10Jforrester)
[13:43:46] <wikibugs>	 (03CR) 10Krinkle: [C:03+1] "Ha, even enforced by tests. Nice!" [integration/config] - 10https://gerrit.wikimedia.org/r/1127117 (https://phabricator.wikimedia.org/T328921) (owner: 10Jforrester)
[13:44:59] <wikibugs>	 (03PS5) 10Jforrester: Docker: Provide quibble-bullseye-php81-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1127117
[13:45:14] <wmf-insecte>	 maintenance-disconnect-full-disks build 703296 integration-agent-docker-1061 (/: 25%, /srv: 32%, /var/lib/docker: 98%): OFFLINE due to disk space
[13:45:43] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Docker: Provide quibble-bullseye-php81-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1127117 (owner: 10Jforrester)
[13:47:46] <wikibugs>	 (03Merged) 10jenkins-bot: Docker: Provide quibble-bullseye-php81-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1127117 (owner: 10Jforrester)
[13:49:43] <James_F>	 !log Docker: Provide quibble-bullseye-php81-coverage
[13:49:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:52:52] <James_F>	 Krinkle: Hmmmm.
[13:53:00] <James_F>	 First test run of composer-coverage-patch fails: https://integration.wikimedia.org/ci/job/composer-coverage-patch/2645/console
[13:55:03] <James_F>	 Do we need to do a new release of phpunit-patch-coverage?
[13:55:16] <Reedy>	 after updating it
[13:55:19] <Reedy>	 well, clover diff too
[13:55:24] <wmf-insecte>	 maintenance-disconnect-full-disks build 703298 integration-agent-docker-1061 (/: 25%, /srv: 13%, /var/lib/docker: 95%): RECOVERY disk space OK
[13:55:26] <James_F>	 Meh.
[13:55:40] <Reedy>	 that's been on ^5 since Dec 2022
[13:55:50] * Reedy makes a patch
[13:56:53] <Reedy>	 https://gerrit.wikimedia.org/r/c/mediawiki/libs/CloverDiff/+/1148344
[13:56:57] <Reedy>	 see if that's overly optimistic
[13:57:28] <James_F>	 Fails.
[13:57:53] <James_F>	 Oh, you do need to terminate the string. :-)
[13:58:01] <wikibugs>	 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, 06SRE, 10vm-requests: codfw: 1VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10839331 (10MoritzMuehlenhoff) >>! In T393873#10838820, @Jelto wrote: > Thanks @Dzahn for picking this up. I left a c...
[13:58:09] <Reedy>	 Don't want anyone quoting me
[13:58:10] <Reedy>	 https://github.com/wikimedia/mediawiki-libs-CloverDiff/compare/2.0.1...master
[13:58:21] <James_F>	 Ack.
[13:58:30] <Reedy>	 can probably go out as 2.0.2 then...
[13:58:33] <James_F>	 Yes.
[14:00:29] <Reedy>	 phpunit-patch-coverage has a loose ^2.0.0
[14:00:41] <James_F>	 So no explicit bump needed, but we might as well.
[14:00:51] <James_F>	 Given we're going to have to roll a new Docker CI image.
[14:00:54] <Reedy>	 https://github.com/wikimedia/mediawiki-tools-phpunit-patch-coverage/compare/0.0.14...master
[14:01:15] <Reedy>	 lets have a quick look if there's anything else we should be updating...
[14:01:31] <James_F>	 FFS, stuck in CI behind spicerack.
[14:04:05] <James_F>	 Reedy: Worth allowing symfony/console ^7 in phpunit-patch-coverage's composer, for future-compat? And the other symfony libs.
[14:04:40] <Reedy>	 yeah... lets drop some of those older ones too
[14:05:04] <wmf-insecte>	 maintenance-disconnect-full-disks build 703300 integration-agent-docker-1052 (/: 25%, /srv: 18%, /var/lib/docker: 100%): still OFFLINE due to disk space
[14:05:04] <wmf-insecte>	 maintenance-disconnect-full-disks build 703300 integration-agent-docker-1061 (/: 25%, /srv: 26%, /var/lib/docker: 99%): OFFLINE due to disk space
[14:05:10] <James_F>	 Ack.
[14:05:16] <James_F>	 But first, we need CI to land.
[14:05:25] <James_F>	 And then I'll push a new tag for CloverDiff
[14:06:03] <Reedy>	 I'm guessing there's little to no value bumping phpunit to .23 in cloverdiff
[14:06:07] <James_F>	 Also, why is https://libup.wmcloud.org/r/labs/tools/coverme using legoktm/clover-diff and not wikimedia/clover-diff? Tsk.
[14:06:22] <Reedy>	 are they just redirects in practice?
[14:07:29] <James_F>	 https://packagist.org/packages/legoktm/clover-diff says to use the wikimedia one, but the tool hasn't switched.
[14:07:32] <James_F>	 I'll write a patch.
[14:09:17] <Reedy>	 AFK for a little bit, got an electrician over
[14:09:22] <James_F>	 https://gerrit.wikimedia.org/r/c/labs/tools/coverme/+/1148349
[14:10:04] <wmf-insecte>	 maintenance-disconnect-full-disks build 703301 integration-agent-docker-1061 (/: 25%, /srv: 25%, /var/lib/docker: 97%): RECOVERY disk space OK
[14:23:30] <hashar>	 ID						RECLAIMABLE	SIZE		LAST ACCESSED
[14:23:30] <hashar>	 e6es6jxjc9j7wacjx3xwhn12a*              	true 		13.89GB   	4 hours ago
[14:23:36] <hashar>	 so hmm something is large :/
[14:25:03] <wmf-insecte>	 maintenance-disconnect-full-disks build 703304 integration-agent-docker-1052 (/: 25%, /srv: 18%, /var/lib/docker: 39%): RECOVERY disk space OK
[14:25:54] <wikibugs>	 (03PS1) 10Jforrester: Docker: [quibble-bullseye-php74-coverage] Upgrade phpunit-patch-coverage to 0.0.15 [integration/config] - 10https://gerrit.wikimedia.org/r/1148356
[14:26:00] <hashar>	 what I don't get is `docker build du` tells me it uses 19.66GB.    `df -h /var/lib/docker` tells me 37G used
[14:26:34] <James_F>	 inodes or whatever poor packing of lots of small files?
[14:26:42] <hashar>	 and after a `docker buildx prune` df is down to 4.2G used
[14:26:47] <hashar>	 so that is 33GB freed
[14:26:49] <James_F>	 Hmmmmmm.
[14:26:55] <hashar>	 which does not match with the 19.66GB reported by Docker
[14:27:19] <hashar>	 anyway some repos generate large layers
[14:27:26] <hashar>	 and I guess the instance can simply use more disk
[14:27:42] <hashar>	 maybe inodes consume disk yeah
[14:27:55] <hashar>	 I mean Docker would report the size
[14:28:01] <hashar>	 while df reports blocks used
[14:28:05] <wikibugs>	 (03PS2) 10Jforrester: Docker: [quibble-bullseye-php74-coverage] Bump phpunit-patch-coverage to 0.0.15 [integration/config] - 10https://gerrit.wikimedia.org/r/1148356
[14:28:08] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Docker: [quibble-bullseye-php74-coverage] Bump phpunit-patch-coverage to 0.0.15 [integration/config] - 10https://gerrit.wikimedia.org/r/1148356 (owner: 10Jforrester)
[14:28:08] <hashar>	 yeah that would make sense
[14:28:36] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Docker: [quibble-bullseye-php74-coverage] Bump phpunit-patch-coverage to 0.0.15 [integration/config] - 10https://gerrit.wikimedia.org/r/1148356 (owner: 10Jforrester)
[14:28:37] <James_F>	 Is node_modules or something using thousands of tiny files "for efficiency"?
[14:28:50] <hashar>	 !log integration: cleared Docker build cache on integration-agent-docker-1052 and integration-agent-docker-1061
[14:28:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:30:28] <wikibugs>	 (03Merged) 10jenkins-bot: Docker: [quibble-bullseye-php74-coverage] Bump phpunit-patch-coverage to 0.0.15 [integration/config] - 10https://gerrit.wikimedia.org/r/1148356 (owner: 10Jforrester)
[14:30:43] <James_F>	 !log Docker: [quibble-bullseye-php74-coverage] Bump phpunit-patch-coverage to 0.0.15
[14:30:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:32:36] <wikibugs>	 (03PS7) 10Jforrester: jjb: Migrate jobs silently using PHP 7.4 to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1127118 (https://phabricator.wikimedia.org/T328921)
[14:34:39] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1127118 (https://phabricator.wikimedia.org/T328921) (owner: 10Jforrester)
[14:35:51] <James_F>	 Krinkle: OK, DonationInterface's CI no longer fails due to composer libraries, but instead due to something else: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/28596/console
[14:38:13] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Migrate jobs silently using PHP 7.4 to 8.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1127118 (https://phabricator.wikimedia.org/T328921) (owner: 10Jforrester)
[14:43:11] <wikibugs>	 (03update) 10dancy: deployment-calendar: Moritz takes over for Reuven in the Puppet window [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/173 (owner: 10rzl)
[14:48:46] <wikibugs>	 (03PS1) 10Jforrester: jjb: Switch integration-quibble-fullrun…opensearch to php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1148364
[14:49:55] <MatmaRex>	 i'm seeing a weird error: https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php81/11398/consoleFull
[14:50:12] <MatmaRex>	 MediaWiki\Extension\Notifications\Test\NotificationStructureTest::testNotificationTypes with data set #20 ('flow-thank', array('edit-thank', 'positive', 'message', 'MediaWiki\Extension\Thanks\Ec...nModel', array(true, true)))
[14:50:12] <MatmaRex>	 Error: Class "Flow\Notifications\FlowPresentationModel" not found
[14:50:31] <MatmaRex>	 any changes to the CI config of Flow/Echo/Thanks recently?
[14:50:36] <James_F>	 Krinkle: Stack now just at https://gerrit.wikimedia.org/r/q/topic:%22drop-php74-from-ci%22
[14:50:57] <James_F>	 MatmaRex: We switched phan over to PHP 8.1 yesterday, but it shouldn't have changed regular CI.
[14:51:30] <James_F>	 MatmaRex: OTOH, https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php81/11398/consoleFull doesn't show Flow in the zuul clone list. Did it definitely work beforehand?
[14:52:21] <MatmaRex>	 yes
[14:52:24] <James_F>	 I don't think we removed Flow as a dependency recently.
[14:52:25] <MatmaRex>	 hmm, or maybe it's becaue of https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Thanks/+/1147082
[14:52:33] <James_F>	 Ooooh.
[14:52:58] <James_F>	 So a different code path that actually tries to create the class?
[14:53:24] <James_F>	 EchoFlowThanksPresentationModel previously wasn't actually instantiated?
[14:54:21] <wikibugs>	 (03update) 10swfrench: Switch mw-(debug|web) to 8.1 images [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/138 (https://phabricator.wikimedia.org/T391057)
[14:54:36] <wikibugs>	 (03CR) 10BryanDavis: "My system has /bin/bash, it is just not a version of bash that works with the non-portable bashisms used in this particular script." [integration/config] - 10https://gerrit.wikimedia.org/r/1147819 (owner: 10BryanDavis)
[14:54:38] <James_F>	 Should we just drop the Flow-thanks feature, given it's being sunsetted?
[14:55:04] <Reedy>	 I guess the guarding for ExtensionRegistry::getInstance()->isLoaded( 'Flow' ) disappears
[14:55:29] <James_F>	 Yeah. Maybe we should put that into EchoFlowThanksPresentationModel?
[14:56:23] <MatmaRex>	 eh
[14:56:40] <MatmaRex>	 i think it's more that we didn't test notif types defined in onBeforeCreateEchoEvent() at all
[14:56:48] <James_F>	 Plausible.
[14:56:59] <MatmaRex>	 but we test the notif types defined in the Notifications attribute, by accident
[14:57:01] <James_F>	 (Code? Untested?! In *our* production?)
[14:57:27] <wikibugs>	 (03PS4) 10Jforrester: jjb: Provide php81-compile-coverage-publish [integration/config] - 10https://gerrit.wikimedia.org/r/1127116
[14:57:28] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] jjb: Provide php81-compile-coverage-publish [integration/config] - 10https://gerrit.wikimedia.org/r/1127116 (owner: 10Jforrester)
[14:57:44] <wikibugs>	 (03update) 10swfrench: Switch mw-(debug|web) to 8.1 images [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/138 (https://phabricator.wikimedia.org/T391057)
[14:57:56] <MatmaRex>	 anyway, looks like it's my fault. i'll figure out a way to fix this
[14:57:58] <wikibugs>	 (03CR) 10Jforrester: "@hashar, I think this is OK to go now, but I wanted you to confirm. :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/1148364 (owner: 10Jforrester)
[15:00:11] <wikibugs>	 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, 06SRE, and 2 others: codfw: 1VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10839625 (10Dzahn) >>! In T393873#10838820, @Jelto wrote: > I don't think we can use `nftables` because we most likely u...
[15:00:22] <wikibugs>	 (03update) 10dancy: Draft: spiderpig: Handle simple add/drop of table columns [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/817
[15:01:29] <wikibugs>	 (03CR) 10Hashar: [C:03+1] "That job is for OpenSearch integration testing: T386691 .  We had to stop triggering it cause some tests were broken, but it should surely" [integration/config] - 10https://gerrit.wikimedia.org/r/1148364 (owner: 10Jforrester)
[15:01:40] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Provide php81-compile-coverage-publish [integration/config] - 10https://gerrit.wikimedia.org/r/1127116 (owner: 10Jforrester)
[15:07:42] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] jjb: Switch integration-quibble-fullrun…opensearch to php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1148364 (owner: 10Jforrester)
[15:08:19] <hashar>	 James_F: 🏆 !!
[15:08:31] <James_F>	 hashar: Gotta catch them all. :-)
[15:08:59] <hashar>	 and eventually in the end you would be able to claim victory!
[15:09:06] <James_F>	 Well, let's not go that far.
[15:09:12] <James_F>	 "Avoided defeat."
[15:09:36] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Switch integration-quibble-fullrun…opensearch to php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1148364 (owner: 10Jforrester)
[15:10:23] * hashar whistles
[15:13:45] <wikibugs>	 (03PS1) 10Dreamy Jazz: Zuul: [mediawiki/extensions/IPInfo] Add GlobalPreferences dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1148372 (https://phabricator.wikimedia.org/T298977)
[15:13:49] <wikibugs>	 (03open) 10brennen: Merge upstream 2024.35 into wmf/stable [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/91 (https://phabricator.wikimedia.org/T370266)
[15:14:52] <wikibugs>	 (03PS2) 10Dreamy Jazz: Zuul: [mediawiki/extensions/IPInfo] Add GlobalPreferences dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1148372 (https://phabricator.wikimedia.org/T298977)
[15:19:54] <wikibugs>	 (03update) 10jforrester: deployment-calendar: Moritz takes over for Reuven in the Puppet window [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/173 (owner: 10rzl)
[15:20:03] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 06Traffic: beta cluster: profile::cache::varnish::frontend needs to reload varnish-frontend.service when /etc/varnish/blocked-nets.inc.vcl changes - https://phabricator.wikimedia.org/T358887#10839729 (10ssingh) Hi @bd808: we discussed this in the Tra...
[15:20:59] <wikibugs>	 (03approved) 10jforrester: deployment-calendar: Moritz takes over for Reuven in the Puppet window [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/173 (owner: 10rzl)
[15:21:02] <wikibugs>	 (03merge) 10jforrester: deployment-calendar: Moritz takes over for Reuven in the Puppet window [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/173 (owner: 10rzl)
[15:24:36] <rzl>	 James_F: thank you! and thanks for the manual edit, that was on my list for today :)
[15:24:45] <James_F>	 rzl: No worries at all. 
[15:32:09] <MatmaRex>	 James_F: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Thanks/+/1148375
[15:32:56] <James_F>	 MatmaRex: I can't wait for us to delete all the SD code.
[15:33:27] <MatmaRex>	 yeah. but you're the only person calling it SD ;)
[15:33:42] <James_F>	 I can't wait for us to delete all the StructuredFlow.
[15:33:43] <James_F>	 Happy?
[15:35:19] <MatmaRex>	 every time we rename something, my face goes deeper into my palm
[15:36:07] <James_F>	 StructuredPalm.
[15:37:15] <wikibugs>	 10Continuous-Integration-Config, 10Testing Support, 13Patch-For-Review, 10Test-Platform (sesa): Only alert when selenium-daily-{sitename}-{project} tests fail on consecutive runs - https://phabricator.wikimedia.org/T394551#10839872 (10zeljkofilipin)
[15:38:02] <wmf-insecte>	 Project mwcore-phpunit-coverage-master build #4343: 04FAILURE in 38 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/4343/
[15:44:43] <wikibugs>	 (03PS2) 10BryanDavis: support scripts: Use `#!/usr/bin/env bash` [integration/config] - 10https://gerrit.wikimedia.org/r/1147819
[15:46:23] <wikibugs>	 (03CR) 10BryanDavis: "If Apple had removed bash entirely what would be the fix here? Would you be willing to see a fully posix rewrite of this script so that it" [integration/config] - 10https://gerrit.wikimedia.org/r/1147820 (owner: 10BryanDavis)
[15:46:42] <wikibugs>	 (03PS2) 10BryanDavis: utils/diff2changelog: Use tests compatible with bash 3.x [integration/config] - 10https://gerrit.wikimedia.org/r/1147820
[15:47:18] <andre>	 brennen: whoah thanks <3
[15:47:51] <Reedy>	 16:37:55 + clover-edit /workspace/log/clover.xml --name 'MediaWiki core' --remove-full-info --save /workspace/cover/clover.xml
[15:47:51] <Reedy>	 16:37:56 Traceback (most recent call last):
[15:47:51] <Reedy>	 16:37:56   File "/usr/local/bin/clover-edit", line 52, in <module>
[15:47:51] <Reedy>	 16:37:56     main()
[15:47:51] <Reedy>	 16:37:56   File "/usr/local/bin/clover-edit", line 38, in main
[15:47:52] <Reedy>	 16:37:56     project = root.getchildren()[0]
[15:47:54] <Reedy>	 16:37:56 AttributeError: 'xml.etree.ElementTree.Element' object has no attribute 'getchildren'
[15:47:56] <Reedy>	 Did we break it...
[15:48:15] <James_F>	 Oh dear.
[15:48:35] <Reedy>	 Considering there were no code changes...
[15:49:31] <wikibugs>	 (03CR) 10Bartosz Dziewoński: "Y'all could rewrite it in a real programming language ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/1147820 (owner: 10BryanDavis)
[15:49:45] <wikibugs>	 (03open) 10dancy: JobLog.vue: Add tooltip for "Show sensitive information" checkbox [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/818 (https://phabricator.wikimedia.org/T393889)
[15:49:47] <wikibugs>	 (03update) 10dancy: JobLog.vue: Add tooltip for "Show sensitive information" checkbox [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/818 (https://phabricator.wikimedia.org/T393889)
[15:49:59] <Reedy>	 line 38
[15:50:00] <Reedy>	 >	echo "Error, unable to find composer autoloader.\n";
[15:50:13] <Reedy>	 https://github.com/wikimedia/mediawiki-libs-CloverDiff/blob/master/bin/clover-diff
[15:50:15] * Reedy squints
[15:50:40] <Reedy>	 >COPY clover-edit.py /usr/local/bin/clover-edit
[15:50:40] <Reedy>	 oh
[15:51:55] <Reedy>	 https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/4343/artifact/log/clover.xml 14.44MiB
[15:55:28] <Reedy>	 the previous one is similar, soo...
[15:58:27] <Reedy>	 James_F: Any chance we've bundled different python-y stuffs in the update?
[15:58:46] <James_F>	 Reedy: Yes. buster -> bullseye would have happened in the 74 -> 81 upgrade. :-(
[15:58:52] <James_F>	 So py2 -> py3?
[15:58:55] <James_F>	 Or whatever.
[15:59:01] <James_F>	 Eurgh, I hate Python environments.
[15:59:11] * Reedy eyes legoktm who is not here
[15:59:15] <wikibugs>	 10Gerrit, 10Bitu, 06Infrastructure-Foundations: gerrit.wikimedia.org: Verification Email never received - https://phabricator.wikimedia.org/T394762#10839977 (10TJungHW) Hi, It seems to got deleted:  TJungHW is the Username.
[15:59:19] <Reedy>	 I'm guessing it's python upgrade and/or losing some libraries
[15:59:36] <James_F>	 Les absents ont toujours tort
[15:59:42] <hashar>	 carrément
[15:59:47] <hashar>	 clover diff is hmm
[15:59:58] <hashar>	 just keep in on 7.4 for now until that is figured out
[16:00:27] <Reedy>	 The previous clover file is similar enough, so I don't think we've magically broken the output of it
[16:00:57] <Reedy>	 https://docs.python.org/3.8/library/xml.etree.elementtree.html#xml.etree.ElementTree.Element.getchildren
[16:01:03] <Reedy>	 >https://docs.python.org/3.8/library/xml.etree.elementtree.html#xml.etree.ElementTree.Element.getchildren
[16:01:05] <Reedy>	 ffs
[16:01:10] <Reedy>	 >    Deprecated since version 3.2, will be removed in version 3.9: Use list(elem) or iteration.
[16:01:13] <Reedy>	 almost certainly realted
[16:01:18] <James_F>	 Helpful.
[16:01:22] <taavi>	 buster -> bullseye is 3.7 -> 3.9
[16:01:29] <James_F>	 :-(
[16:01:36] <hashar>	 where is that code?
[16:01:45] <Reedy>	 there's 3 copies of it in integration-confiug
[16:01:59] * hashar blames Docker
[16:02:04] <Reedy>	 but dockerfiles/quibble-bullseye-php81-coverage/clover-edit.py
[16:02:27] <Reedy>	 shouldn't be too bad to fix
[16:02:29] * Reedy looks
[16:02:40] <hashar>	 hopefully yes
[16:03:02] <hashar>	 the whole coverage diff system could use an overhaul
[16:03:08] <James_F>	 Or deletion.
[16:03:16] <Reedy>	 hashar just licked the cookie
[16:03:17] <hashar>	 it has several glitches here and well and thhere are multiple stacks/scripts involved
[16:03:18] <hashar>	 or yes
[16:03:25] <Reedy>	 one thing at a time
[16:03:31] <hashar>	 if nobody actually uses those coverage diff:  decommission it
[16:03:39] <James_F>	 I use them.
[16:03:42] <hashar>	 but I don't have any metrics about their usage
[16:03:45] <James_F>	 But not enough to care to maintain the system.
[16:03:45] * hashar bans James_F 
[16:03:55] <Reedy>	 They're not as useful as they could be
[16:03:55] <hashar>	 seirously
[16:03:55] <Reedy>	 but yes
[16:03:59] <hashar>	 maybe nobody reads them
[16:04:28] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] "Confirmed that this works great, thank you." [integration/config] - 10https://gerrit.wikimedia.org/r/1147819 (owner: 10BryanDavis)
[16:04:38] <hashar>	 anyway the python script should be easy to adjust
[16:04:45] <hashar>	 I am off, I gotta prepare dinner
[16:05:49] <wikibugs>	 (03PS1) 10Reedy: quibble-bullseye-php81-coverage: Update clover-edit.py for python 3.9 in bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1148380
[16:06:25] <James_F>	 Reedy: Should I just take that as working? :-) But will need the image to be tagged and re-built.
[16:06:40] <wikibugs>	 (03PS2) 10Reedy: quibble-bullseye-php81-coverage: Update clover-edit.py for python 3.9 in bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1148380
[16:06:48] <Reedy>	 I mean you can, but let me at least provisionally test it
[16:06:49] <taavi>	 i do use them as well
[16:07:06] <Reedy>	 `clover-edit /workspace/log/clover.xml --name 'MediaWiki core' --remove-full-info --save /workspace/cover/clover.xml`
[16:07:09] <Reedy>	 should vaguely work
[16:08:56] <Reedy>	 heh, there's another getchildren call too
[16:08:59] <wikibugs>	 (03Merged) 10jenkins-bot: support scripts: Use `#!/usr/bin/env bash` [integration/config] - 10https://gerrit.wikimedia.org/r/1147819 (owner: 10BryanDavis)
[16:09:14] <wikibugs>	 (03CR) 10CI reject: [V:04-1] quibble-bullseye-php81-coverage: Update clover-edit.py for python 3.9 in bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1148380 (owner: 10Reedy)
[16:09:19] <Reedy>	 pfft
[16:09:42] <wikibugs>	 (03PS3) 10Reedy: quibble-bullseye-php81-coverage: Update clover-edit.py for python 3.9 in bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1148380
[16:09:55] <Reedy>	 that now runs without error
[16:10:23] <Reedy>	 well, bar the long line
[16:10:27] <James_F>	 Reedy: Doesn't error, or actually works?
[16:11:34] <Reedy>	 -rw-rw-r--   1 reedy reedy 15142384 May 20 16:37 clover.xml
[16:11:34] <Reedy>	 -rw-rw-r--   1 reedy reedy      356 May 20 17:09 clover.xml.new
[16:11:44] <Reedy>	 LGTM
[16:11:52] <wikibugs>	 (03CR) 10CI reject: [V:04-1] quibble-bullseye-php81-coverage: Update clover-edit.py for python 3.9 in bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1148380 (owner: 10Reedy)
[16:11:57] <Reedy>	 per the comment in the script
[16:11:58] <Reedy>	 >All we want to keep is the final <metrics> tag
[16:12:05] <Reedy>	 we get
[16:12:05] <Reedy>	 <coverage generated="1747755423">
[16:12:05] <Reedy>	   <project timestamp="1747755423" name="MediaWiki core">
[16:12:05] <Reedy>	     <metrics files="3110" loc="721048" ncloc="480270" classes="2260" methods="22102" coveredmethods="8678" conditionals="0" coveredconditionals="0" statements="239829" coveredstatements="105780" elements="261931" coveredelements="114458" />
[16:12:05] <Reedy>	   </project>
[16:12:06] <Reedy>	 </coverage>
[16:12:10] <James_F>	 Ack.
[16:12:12] <wikibugs>	 (03PS4) 10Reedy: quibble-bullseye-php81-coverage: Update clover-edit.py for python 3.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1148380
[16:18:49] <kart_>	 Is this failure known issue? https://integration.wikimedia.org/ci/job/machinetranslation-pipeline-test/564/console or something else?
[16:18:55] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Epic: Use infrastructure as code techniques to rebuild the Beta Cluster - https://phabricator.wikimedia.org/T394316#10840100 (10bd808) https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/tree/main/tofu-provisioning is worth looking at to see if it would be reasonable to u...
[16:22:17] <wikibugs>	 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10840115 (10Dzahn)
[16:22:24] <wikibugs>	 (03PS5) 10Jforrester: Docker: [quibble-bullseye-php81-coverage]: Fix clover-edit for py39 [integration/config] - 10https://gerrit.wikimedia.org/r/1148380 (owner: 10Reedy)
[16:25:40] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Docker: [quibble-bullseye-php81-coverage]: Fix clover-edit for py39 [integration/config] - 10https://gerrit.wikimedia.org/r/1148380 (owner: 10Reedy)
[16:26:12] <Reedy>	 kart_: Is it persistent or a one off?
[16:26:51] <wikibugs>	 (03Merged) 10jenkins-bot: Docker: [quibble-bullseye-php81-coverage]: Fix clover-edit for py39 [integration/config] - 10https://gerrit.wikimedia.org/r/1148380 (owner: 10Reedy)
[16:27:18] <James_F>	 !log Docker: [quibble-bullseye-php81-coverage]: Fix clover-edit for py39
[16:27:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:33:13] <wikibugs>	 (03PS1) 10Jforrester: jjb: Switch jobs using quibble-bullseye-php81-coverage to 1.14.1-s2 [integration/config] - 10https://gerrit.wikimedia.org/r/1148383
[16:33:32] <wikibugs>	 06Project-Admins, 06Release-Engineering-Team, 10MediaWiki-extensions-General, 10MediaWiki-General, and 2 others: Create a security pre-release Phabricator policy manageable by the Security Team - https://phabricator.wikimedia.org/T393403#10840171 (10sbassett) >>! In T393403#10836678, @Aklapper wrote: >> If...
[16:35:50] <MatmaRex>	 another unusual failure, this one doesn't seem like my fault: "mkdir: cannot create directory ‘log’: Permission denied" and a bunch of other permission errors  https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83/1155/console
[16:36:45] <wikibugs>	 (03PS2) 10Jforrester: jjb: Switch jobs using quibble-bullseye-php81-coverage to 1.14.1-s2 [integration/config] - 10https://gerrit.wikimedia.org/r/1148383
[16:36:54] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1148383 (owner: 10Jforrester)
[16:38:30] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Switch jobs using quibble-bullseye-php81-coverage to 1.14.1-s2 [integration/config] - 10https://gerrit.wikimedia.org/r/1148383 (owner: 10Jforrester)
[16:38:59] <wikibugs>	 10Phabricator, 06Security-Team: Please remove an MFA factor from my Phabricator account - https://phabricator.wikimedia.org/T394763#10840187 (10sbassett) Since @SomeRandomDeveloper seems fairly privacy conscious (that's great!) and doesn't know anyone at the WMF personally, I don't think video, etc. verificati...
[16:39:10] <James_F>	 Reedy: Does that work now.
[16:39:12] <James_F>	 ?
[16:39:37] <Reedy>	 I haven't re-run the job (as it's the twice daily one?)
[16:39:42] <Reedy>	 I can do it now
[16:39:47] <James_F>	 Eh. Let's wait.
[16:40:06] <kart_>	 Reedy: failing since last 4 rechecks
[16:42:20] <kart_>	 and more: https://integration.wikimedia.org/ci/job/machinetranslation-pipeline-test/565/
[16:43:55] <Reedy>	 looks like it worked earlier today ~0500 and failed ~1300
[16:44:02] <wikibugs>	 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10840208 (10Dzahn) after today's meetings with SRE-collab/releng/zuul author we update this request to 3 VMs p...
[16:44:29] <Reedy>	 I don't see it (obviously) being related to any deploys today
[16:45:54] <wikibugs>	 (03merge) 10dancy: Switch mw-(debug|web) to 8.1 images [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/138 (https://phabricator.wikimedia.org/T391057) (owner: 10swfrench)
[16:46:21] <Reedy>	 Wonder if jenkins just needs restarting because java is unhappy
[16:46:47] <Reedy>	 hashar: around?
[16:48:47] <wikibugs>	 10Phabricator: Please remove an MFA factor from my Phabricator account - https://phabricator.wikimedia.org/T394763#10840243 (10Aklapper) p:05Triage→03Medium a:03Aklapper Thanks a lot for the guidance (also noted for next time)! I'll follow up with SomeRandomDeveloper out-of-band.
[16:53:38] <wikibugs>	 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10840269 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host z...
[16:54:25] * Krinkle is an active user of the coverage diff jobs as well.
[16:57:52] <wikibugs>	 (03open) 10dancy: Install k9s v0.50.6 using the .deb package. [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/139
[16:57:54] <wikibugs>	 (03update) 10dancy: Install k9s v0.50.6 using the .deb package. [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/139
[16:58:46] <wikibugs>	 (03merge) 10dancy: Install k9s v0.50.6 using the .deb package. [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/139
[17:05:37] <wmf-insecte>	 Project beta-scap-sync-world build #207341: 04FAILURE in 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/207341/
[17:06:20] <Reedy>	 well, that broke stuff
[17:06:26] <Reedy>	 ffs
[17:07:51] <dancy>	 Reedy: I'm collecting info about the mobileapps-pipeline-test issue
[17:07:58] <Reedy>	 ty!
[17:08:03] <wikibugs>	 10Scap, 10ConfirmEdit (CAPTCHA extension), 10WMF-General-or-Unknown: Unable to have multiple captcha implementations in extension-list - https://phabricator.wikimedia.org/T394814 (10Reedy) 03NEW
[17:09:13] <wikibugs>	 10Scap, 10ConfirmEdit (CAPTCHA extension), 10WMF-General-or-Unknown, 13Patch-For-Review, 07Wikimedia-production-error: Unable to have multiple captcha implementations in extension-list - https://phabricator.wikimedia.org/T394814#10840424 (10Jdforrester-WMF)
[17:09:56] <wikibugs>	 10Scap, 10ConfirmEdit (CAPTCHA extension), 10WMF-General-or-Unknown, 13Patch-For-Review, 07Wikimedia-production-error: Unable to have multiple captcha implementations in extension-list - https://phabricator.wikimedia.org/T394814#10840430 (10Reedy) p:05Triage→03High
[17:10:37] <taavi>	 dancy: I've seen similar errors happen after JRE upgrades if you don't restart the service afterwards
[17:11:04] <dancy>	 Thanks.  I'll give that a try first
[17:12:50] <wikibugs>	 10Continuous-Integration-Infrastructure: mobileapps-pipeline-test job failing - https://phabricator.wikimedia.org/T394817 (10dancy) 03NEW
[17:13:42] <dancy>	 !log Restarting Jenkins on contint1002
[17:13:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:15:55] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 10SRE-Access-Requests: create new admin group for "zuul devs" - https://phabricator.wikimedia.org/T394819 (10Dzahn) 03NEW
[17:16:34] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 10SRE-Access-Requests: create new admin group for "zuul devs" - https://phabricator.wikimedia.org/T394819#10840512 (10Dzahn) Who, besides James and global roots, should have access to zuul(3) VMs?
[17:16:39] <wikibugs>	 10Continuous-Integration-Infrastructure: mobileapps-pipeline-test job failing - https://phabricator.wikimedia.org/T394817#10840513 (10dancy) 05Open→03Resolved a:03dancy Resolved by restarting Jenkins on `contint1002.wikimedia.org`.
[17:17:06] <wmf-insecte>	 Yippee, build fixed!
[17:17:06] <wmf-insecte>	 Project beta-code-update-eqiad build #548758: 09FIXED in 3 min 6 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/548758/
[17:18:24] <wikibugs>	 10Continuous-Integration-Infrastructure: mobileapps-pipeline-test job failing - https://phabricator.wikimedia.org/T394817#10840521 (10dancy) This was the same as T386755
[17:18:57] <wmf-insecte>	 Yippee, build fixed!
[17:18:57] <wmf-insecte>	 Project beta-scap-sync-world build #207342: 09FIXED in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/207342/
[17:19:12] <dancy>	 Reedy: We should be back in action now.
[17:19:20] <Reedy>	 kart_: ^^
[17:23:18] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10840540 (10Dzahn)
[17:27:42] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10840552 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rei...
[17:27:50] <wikibugs>	 10Scap, 10ConfirmEdit (CAPTCHA extension), 10WMF-General-or-Unknown, 13Patch-For-Review, 07Wikimedia-production-error: Unable to have multiple captcha implementations in extension-list - https://phabricator.wikimedia.org/T394814#10840553 (10Reedy)
[17:28:14] <wikibugs>	 10Scap, 10ConfirmEdit (CAPTCHA extension), 10WMF-General-or-Unknown, 13Patch-For-Review, 07Wikimedia-production-error: Unable to have multiple captcha implementations in extension-list - https://phabricator.wikimedia.org/T394814#10840556 (10Reedy)
[17:32:59] <wikibugs>	 10Scap, 10ConfirmEdit (CAPTCHA extension), 10WMF-General-or-Unknown, 13Patch-For-Review, 07Wikimedia-production-error: Unable to have multiple captcha implementations in extension-list - https://phabricator.wikimedia.org/T394814#10840578 (10Reedy) We can/could work around this by listing the hCaptcha `Me...
[17:41:16] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10840589 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rei...
[17:45:28] <wmf-insecte>	 maintenance-disconnect-full-disks build 703344 integration-agent-docker-1051 (/: 25%, /srv: 95%, /var/lib/docker: 88%): OFFLINE due to disk space
[17:50:32] <wmf-insecte>	 maintenance-disconnect-full-disks build 703345 integration-agent-docker-1051 (/: 25%, /srv: 56%, /var/lib/docker: 73%): RECOVERY disk space OK
[17:51:14] <bd808>	 !log Open CDN edge blocks to allow traffic from 190.217.20.32/28
[17:51:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:55:16] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10840651 (10Dzahn)
[17:58:12] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10840660 (10Dzahn)
[17:58:41] <wikibugs>	 10Continuous-Integration-Config, 10CommonPasswords, 07ci-test-error: Tests on mediawiki/libs/CommonPasswords failing with timeouts - https://phabricator.wikimedia.org/T394821 (10Umherirrender) 03NEW
[17:59:02] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10840672 (10Dzahn)
[18:14:39] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 3 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10840708 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rei...
[18:18:38] <musikanimal>	 hi! https://integration.wikimedia.org/zuul/#q=1141132 appears to be stuck. Commands from Gerrit don't seem to do anything. How can get it unstuck? :)
[18:19:46] <James_F>	 musikanimal: Aborting runs can do that, sadly.
[18:19:51] <James_F>	 musikanimal: Gone now.
[18:20:12] <musikanimal>	 thank you! and noted about aborting builds
[18:20:21] <James_F>	 Np.
[18:39:16] <wikibugs>	 06Release-Engineering-Team, 06Java-Scala-Standardization: Renew Gitlab Maven CI_RELEASE_TOKEN - https://phabricator.wikimedia.org/T394827 (10JAllemandou) 03NEW
[18:45:02] <Reedy>	 >19:36:45 gzip: log/mw-debug-cli.log.gz: No space left on device
[18:45:22] <Reedy>	 integration-agent-docker-1049
[18:52:57] <gehel>	 thcipriani: Hey! We have CI tokens on Gitlab that will expire soon. Could you help renew them? And do you have a process to automate it? T394827
[18:52:57] <stashbot>	 T394827: Renew Gitlab Maven CI_RELEASE_TOKEN - https://phabricator.wikimedia.org/T394827
[19:00:16] <brennen>	 gehel: one note here is that we disabled expiry for new tokens, at least
[19:01:42] <brennen>	 https://phabricator.wikimedia.org/T385930
[19:02:32] <gehel>	 That might at least make this simpler in the future!
[19:03:01] <wikibugs>	 06Release-Engineering-Team, 06Java-Scala-Standardization: Renew Gitlab Maven CI_RELEASE_TOKEN - https://phabricator.wikimedia.org/T394827#10841007 (10Gehel) It looks like we now allow tokens to not expire: T385930
[19:07:30] <wikibugs>	 (03update) 10dancy: JobLog.vue: Add tooltip for "Show sensitive information" checkbox [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/818 (https://phabricator.wikimedia.org/T393889)
[19:11:00] <thcipriani>	 gehel: thanks for task, taking a look
[19:14:51] <thcipriani>	 brennen: would the gitlab-rake gitlab:tokens:edit work to remove an expiry date on an existing token for the maven group?
[19:14:53] <brennen>	 thcipriani, gehel: is this gitlab-release-token in jenkins?
[19:15:01] <brennen>	 thcipriani: we can try it
[19:15:41] <thcipriani>	 brennen: it's the soon-to-expire token here: https://gitlab.wikimedia.org/groups/repos/maven/-/settings/access_tokens
[19:15:48] <wmf-insecte>	 maintenance-disconnect-full-disks build 703362 integration-agent-docker-1042 (/: 25%, /srv: 100%, /var/lib/docker: 86%): OFFLINE due to disk space
[19:16:06] <thcipriani>	 (which you need to be an admin or a project member to see)
[19:16:20] <brennen>	 yeah, found that bit - just couldn't remember where else it's stored
[19:16:29] <brennen>	 but let me see if i can find and reset it from the CLI
[19:16:48] <brennen>	 (the earlier bulk operation doesn't seem to have worked for at least some stuff, so we'll see)
[19:18:35] <wikibugs>	 10Release-Engineering-Team (Doing 😎), 06Java-Scala-Standardization: Renew Gitlab Maven CI_RELEASE_TOKEN - https://phabricator.wikimedia.org/T394827#10841146 (10thcipriani) a:03brennen Assigning to brennen to look at for the moment, per IRC:  ` 19:14:51     +thcipriani brennen: would the gitlab-rake gitlab:to...
[19:18:58] <brennen>	 i love how gitlab's display logic thinks july 18th is "in 1 month".  just use exact dates where exact dates matter!
[19:19:35] <thcipriani>	 :)
[19:19:42] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10841161 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rei...
[19:19:57] <thcipriani>	 self destruct in about an hour
[19:20:01] <brennen>	 haha
[19:20:03] <wmf-insecte>	 maintenance-disconnect-full-disks build 703363 integration-agent-docker-1042 (/: 25%, /srv: 40%, /var/lib/docker: 67%): RECOVERY disk space OK
[19:20:05] <dancy>	 haha
[19:20:33] <brennen>	 i imagine this scene left on the editing room floor for _galaxy quest_.
[19:20:44] <dancy>	 brennen: Search for "Time preferences" in https://gitlab.wikimedia.org/-/profile/preferences
[19:20:53] <dancy>	 I have previously felt this pain.
[19:20:58] <brennen>	 omg
[19:21:13] <brennen>	 thanks dancy, my life just improved by a tiny but meaningful increment.
[19:22:46] <brennen>	 also of interest here is that you can change the default homepage to an activity feed instead of an entirely useless list of projects.
[19:23:06] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: allow new zuul executor VMs in prod to talk to cloud VPS - https://phabricator.wikimedia.org/T394838 (10Dzahn) 03NEW
[19:39:02] <wikibugs>	 10Release-Engineering-Team (Doing 😎), 06Java-Scala-Standardization: Renew Gitlab Maven CI_RELEASE_TOKEN - https://phabricator.wikimedia.org/T394827#10841238 (10bd808) The potentially easier action than database updates that we have been taking with other PATs is issuing a new non-expiring token now that the up...
[19:47:45] <wikibugs>	 10Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: Separate Gerrit https and ssh/git hostnames - https://phabricator.wikimedia.org/T394271#10841273 (10Dzahn) >>! In T394271#10822942, @bd808 wrote: > Naming is always tricky, but I wonder if putting `ssh` in the hostname would be helpful...
[19:51:51] <wikibugs>	 10Continuous-Integration-Config, 10Testing Support, 13Patch-For-Review, 10Test-Platform (sesa): Only alert when selenium-daily-{sitename}-{project} tests fail on consecutive runs - https://phabricator.wikimedia.org/T394551#10841284 (10Etonkovidova) >>! In T394551#10831581, @bd808 wrote: > The `selenium-dai...
[19:52:20] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10841286 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rei...
[19:56:06] <wikibugs>	 10Continuous-Integration-Config, 10Testing Support, 13Patch-For-Review, 10Test-Platform (sesa): Only alert when selenium-daily-{sitename}-MediaWiki tests fail on consecutive runs - https://phabricator.wikimedia.org/T394551#10841287 (10bd808)
[19:58:37] <wikibugs>	 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services: Consider disabling personal access token forced expiration - https://phabricator.wikimedia.org/T385930#10841299 (10brennen) I see my mistake here: The `gitlab:tokens:analyze` and `gitl...
[19:59:44] <wikibugs>	 10Release-Engineering-Team (Doing 😎), 06Java-Scala-Standardization: Renew Gitlab Maven CI_RELEASE_TOKEN - https://phabricator.wikimedia.org/T394827#10841318 (10brennen) 05Open→03Resolved > The potentially easier action than database updates that we have been taking with other PATs is issuing a new non-...
[20:03:25] <wikibugs>	 10Continuous-Integration-Config, 10Testing Support, 13Patch-For-Review, 10Test-Platform (sesa): Only alert when selenium-daily-{sitename}-MediaWiki tests fail on consecutive runs - https://phabricator.wikimedia.org/T394551#10841337 (10bd808) I have renamed this task to be about the `selenium-daily-beta-Med...
[20:08:30] <wikibugs>	 10Continuous-Integration-Config, 10Testing Support, 13Patch-For-Review, 10Test-Platform (sesa): Only alert when selenium-daily-{sitename}-MediaWiki tests fail on consecutive runs - https://phabricator.wikimedia.org/T394551#10841349 (10vaughnwalters) >>! In T394551#10841337, @bd808 wrote: > I have renamed t...
[20:22:13] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: request mariadb database for zuul3 - https://phabricator.wikimedia.org/T394844 (10Dzahn) 03NEW
[20:26:10] <wikibugs>	 10Phabricator: Restrict visibility of form 91 to #trusted-contributors - https://phabricator.wikimedia.org/T394845 (10Pppery) 03NEW
[20:32:43] <wikibugs>	 06Project-Admins, 06Release-Engineering-Team, 10MediaWiki-extensions-General, 10MediaWiki-General, and 2 others: Create a security pre-release Phabricator policy manageable by the Security Team - https://phabricator.wikimedia.org/T393403#10841465 (10Aklapper) I don't think we have docs because it's very un...
[20:37:56] <wikibugs>	 10Scap, 10ConfirmEdit (CAPTCHA extension), 10WMF-General-or-Unknown, 13Patch-For-Review, 07Wikimedia-production-error: Unable to have multiple captcha implementations in extension-list - https://phabricator.wikimedia.org/T394814#10841481 (10Krinkle) I wouldn't object to that personally.  Long-term that m...
[20:40:59] <wmcs-alerts>	 FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[20:42:12] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Doing 😎): Please remove an MFA factor from my Phabricator account - https://phabricator.wikimedia.org/T394763#10841492 (10Aklapper) 05Open→03Resolved Verified the request, TOTP stripped and asked the user to set it up again.
[20:44:32] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Doing 😎): Restrict visibility of form 91 to #trusted-contributors ("Request creation of a new Wiki") - https://phabricator.wikimedia.org/T394845#10841496 (10Aklapper) a:03Aklapper
[20:47:13] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Doing 😎): Restrict visibility of form 91 to #trusted-contributors ("Request creation of a new Wiki") - https://phabricator.wikimedia.org/T394845#10841501 (10Aklapper) 05Open→03Resolved p:05Triage→03Low Yeah, that makes sense. Thanks for that idea! Folks mi...
[20:49:20] <wikibugs>	 10Scap, 10ConfirmEdit (CAPTCHA extension), 10WMF-General-or-Unknown, 13Patch-For-Review, 07Wikimedia-production-error: Unable to have multiple captcha implementations in extension-list - https://phabricator.wikimedia.org/T394814#10841508 (10Jdforrester-WMF) >>! In T394814#10841481, @Krinkle wrote: > I wo...
[20:51:11] <wikibugs>	 (03CR) 10Tchanders: [C:03+1] Zuul: [mediawiki/extensions/IPInfo] Add GlobalPreferences dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1148372 (https://phabricator.wikimedia.org/T298977) (owner: 10Dreamy Jazz)
[20:51:21] <wikibugs>	 10Continuous-Integration-Config, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, 05MW-1.39-release: GrowthExperiments fatal on REL1_39 - https://phabricator.wikimedia.org/T394846#10841525 (10Reedy)
[20:53:59] <wikibugs>	 10Continuous-Integration-Config, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, 05MW-1.39-release, 13Patch-For-Review: GrowthExperiments fatal on REL1_39 - https://phabricator.wikimedia.org/T394846#10841548 (10Jdforrester-WMF) Oh, sorry, I thought we'd already landed all these 8.1 fixes into major ext...
[21:09:04] <wikibugs>	 10Continuous-Integration-Config, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, 05MW-1.39-release, and 2 others: GrowthExperiments fatal on REL1_39 - https://phabricator.wikimedia.org/T394846#10841655 (10Reedy)
[21:09:46] <wikibugs>	 06Project-Admins, 06Release-Engineering-Team, 10MediaWiki-extensions-General, 10MediaWiki-General, and 2 others: Create a security pre-release Phabricator policy manageable by the Security Team - https://phabricator.wikimedia.org/T393403#10841657 (10Mstyles) @Aklapper if you could create the ACL with the s...
[21:10:47] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10841661 (10Dzahn)
[21:12:00] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10841664 (10Dzahn)
[21:16:58] <wikibugs>	 10Continuous-Integration-Config, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, 10ci-test-error (WMF-deployed Build Failure), and 3 others: GrowthExperiments fatal on REL1_39 - https://phabricator.wikimedia.org/T394846#10841697 (10Reedy)
[21:22:28] <wikibugs>	 10Continuous-Integration-Config, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, 10ci-test-error (WMF-deployed Build Failure), and 3 others: GrowthExperiments fatal on REL1_39 - https://phabricator.wikimedia.org/T394846#10841754 (10Reedy)
[21:32:32] <wikibugs>	 (03open) 10rzl: mwscript-mwcron: Make logging optional [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/175
[21:34:26] <wikibugs>	 10Scap (SpiderPig 🕸️): SpiderPig: Give folks the option to rollback changes - https://phabricator.wikimedia.org/T394858 (10thcipriani) 03NEW
[21:43:11] <wikibugs>	 10Beta-Cluster-Infrastructure: Unbreak deployment-prep config-master - https://phabricator.wikimedia.org/T394803#10841875 (10bd808) https://wikitech.wikimedia.org/wiki/Config-master.wikimedia.org leads me to believe this is about conftool and etcd ultimately. Does that make this a duplicate of {T278007}?
[21:49:32] <wikibugs>	 10Continuous-Integration-Config, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, 10ci-test-error (WMF-deployed Build Failure), and 3 others: GrowthExperiments fatal on REL1_39 - https://phabricator.wikimedia.org/T394846#10841896 (10Reedy) 05Open→03Resolved a:03Reedy
[21:51:03] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Dumps-Generation: mediawiki scripts fail on new buster image in deployment-prep - https://phabricator.wikimedia.org/T273089#10841901 (10bd808) 05Open→03Resolved `lang=shell-session bd808@deployment-snapshot05:~$ sudo -u dumpsgen php /srv/mediawiki/multiversion/MWScript....
[21:52:38] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure: beta-scap-sync-world failure — SSL peer certificate or SSH remote key was not OK - https://phabricator.wikimedia.org/T338495#10841905 (10bd808) 05Open→03Resolved a:03jbond
[21:53:32] <wikibugs>	 10Beta-Cluster-Infrastructure: Unbreak deployment-prep config-master - https://phabricator.wikimedia.org/T394803#10841907 (10bd808) `role::config_master` does not seem to be applied to any deployment-prep hosts per https://openstack-browser.toolforge.org/puppetclass/
[22:03:36] <wikibugs>	 (03update) 10swfrench: Draft: make-container-image: stop building the "publish" flavour [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/172 (https://phabricator.wikimedia.org/T391057)
[22:04:53] <wikibugs>	 (03update) 10swfrench: make-container-image: stop building the "publish" flavour [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/172 (https://phabricator.wikimedia.org/T391057)
[22:05:09] <wikibugs>	 (03close) 10aklapper: Add temporary debug output for T374441 [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/74 (https://phabricator.wikimedia.org/T374441)
[22:05:58] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10841941 (10Dzahn)
[22:08:01] <wikibugs>	 10Scap, 10ConfirmEdit (CAPTCHA extension), 10WMF-General-or-Unknown, 13Patch-For-Review, 07Wikimedia-production-error: Unable to have multiple captcha implementations in extension-list - https://phabricator.wikimedia.org/T394814#10841947 (10Reedy) We already specifically define it in wmf-config (`$wgCapt...
[22:15:35] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10841950 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rei...
[22:34:53] <wikibugs>	 (03update) 10swfrench: Allow deployments to specify custom k8s clusters [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/795 (https://phabricator.wikimedia.org/T388761)
[22:36:21] <wikibugs>	 (03update) 10swfrench: Allow deployments to specify custom k8s clusters [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/795 (https://phabricator.wikimedia.org/T388761)
[22:47:40] <wikibugs>	 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10842034 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rei...
[22:54:45] <wikibugs>	 10Scap (SpiderPig 🕸️): SpiderPig: Give folks the option to rollback changes - https://phabricator.wikimedia.org/T394858#10842038 (10thcipriani) > Is this only an option for SpiderPig? Or should we prompt to do this as part of scap commands run on the command line, e.g., scap backport, scap sync-world?  Noting th...
[23:07:56] <wikibugs>	 (03update) 10swfrench: mwscript-mwcron: Make logging optional [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/175 (owner: 10rzl)
[23:41:07] <wikibugs>	 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Beta cluster IP block page should not point to noc@wikimedia.org - https://phabricator.wikimedia.org/T393404#10842185 (10bd808) I have created https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Blocked_help and a redirect at https://wikitech.wik...