[00:30:00] PROBLEM - jenkins_service_running on contint1003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [06:55:44] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316#12084088 (10Jelto) [08:16:58] (03merge) 10taavi: phab: Replace deprecated endpoints [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/116 (https://phabricator.wikimedia.org/T427620) [08:22:31] 10Phabricator, 10LibUp, 13Patch-For-Review: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T427620#12084285 (10taavi) 05Open→03Resolved a:03taavi [08:30:23] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Zuul web API calls are cached - https://phabricator.wikimedia.org/T430462#12084347 (10hashar) The Zuul web app does not emit any cache control headers and our edge thus end up caching everything. Turning off caching is a good first ste... [08:32:11] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review, 07User-notice: Sunsetting mirrors.wikimedia.org - https://phabricator.wikimedia.org/T416707#12084350 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2003 for hosts: `mirror1001.wikime... [08:34:23] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review, 07User-notice: Sunsetting mirrors.wikimedia.org - https://phabricator.wikimedia.org/T416707#12084376 (10MoritzMuehlenhoff) [08:37:51] dancy: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1307322/ and stack to fix and add some actual testing. :-( [09:09:35] 10Phabricator, 10LibUp: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T427620#12084521 (10Aklapper) Thank you so much! [09:27:35] 10Continuous-Integration-Config, 10phpunit-patch-coverage: Some mwext-phpunit-coverage-patch runs failing with `Failed to open stream: No such file or directory in /opt/phpunit-patch-coverage/vendor/mediawiki/phpunit-patch-coverage/src/CheckCommand.php` - https://phabricator.wikimedia.org/T423213#12084647 (... [09:47:12] 10Phabricator, 10Test Kitchen, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07OKR-Work, 13Patch-For-Review: Add phabricator to the kubernetes service mesh - https://phabricator.wikimedia.org/T430024#12084750 (10Gehel) [09:47:39] (03PS1) 10Jforrester: Docker: [quibble-coverage] Update phpunit-patch-coverage to 0.0.18 [integration/config] - 10https://gerrit.wikimedia.org/r/1307364 (https://phabricator.wikimedia.org/T423987) [09:47:53] (03PS1) 10Jforrester: jjb: Switch coverage jobs to phpunit-patch-coverage to 0.0.18 [integration/config] - 10https://gerrit.wikimedia.org/r/1307365 [09:50:01] (03CR) 10Jforrester: [C:03+2] Docker: [quibble-coverage] Update phpunit-patch-coverage to 0.0.18 [integration/config] - 10https://gerrit.wikimedia.org/r/1307364 (https://phabricator.wikimedia.org/T423987) (owner: 10Jforrester) [09:50:29] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): No Puppet resources found on instance deployment-cirrussearch15 on project deployment-prep - https://phabricator.wikimedia.org/T430217#12084826 (10Gehel) [09:50:47] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): No Puppet resources found on instance deployment-cirrussearch13 on project deployment-prep - https://phabricator.wikimedia.org/T428822#12084832 (10Gehel) [09:53:12] !log Docker: [quibble-coverage] Update phpunit-patch-coverage to 0.0.18, for T423987 and T425807 [09:53:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:53:18] T423987: fix any preg_quote() functions missing a 2nd parameter if the delimiter of the affected RegEx is not automatically escaped - https://phabricator.wikimedia.org/T423987 [09:53:18] T425807: PHP Fatal error: Uncaught TypeError: explode(): Argument #2 ($string) must be of type string, false given in CheckCommand.php:153 - https://phabricator.wikimedia.org/T425807 [09:53:33] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): No Puppet resources found on instance deployment-cirrussearch14 on project deployment-prep - https://phabricator.wikimedia.org/T424100#12084875 (10Gehel) [09:53:47] (03Merged) 10jenkins-bot: Docker: [quibble-coverage] Update phpunit-patch-coverage to 0.0.18 [integration/config] - 10https://gerrit.wikimedia.org/r/1307364 (https://phabricator.wikimedia.org/T423987) (owner: 10Jforrester) [09:56:47] (03PS2) 10Jforrester: jjb: Switch coverage jobs to phpunit-patch-coverage to 0.0.18 [integration/config] - 10https://gerrit.wikimedia.org/r/1307365 (https://phabricator.wikimedia.org/T423987) [09:59:22] (03CR) 10Jforrester: [C:03+2] jjb: Switch coverage jobs to phpunit-patch-coverage to 0.0.18 [integration/config] - 10https://gerrit.wikimedia.org/r/1307365 (https://phabricator.wikimedia.org/T423987) (owner: 10Jforrester) [09:59:30] (03Merged) 10jenkins-bot: jjb: Switch coverage jobs to phpunit-patch-coverage to 0.0.18 [integration/config] - 10https://gerrit.wikimedia.org/r/1307365 (https://phabricator.wikimedia.org/T423987) (owner: 10Jforrester) [09:59:53] 10Continuous-Integration-Config, 10MediaWiki-General, 10phpunit-patch-coverage: MediaWiki code coverage reports include coverage for the Vector skin - https://phabricator.wikimedia.org/T420906#12085061 (10Jdforrester-WMF) [10:03:54] (03CR) 10Jforrester: "Let's give this a go now I have a free weekend." [integration/config] - 10https://gerrit.wikimedia.org/r/1248853 (owner: 10Jforrester) [10:03:59] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/TestKitchen] Don't drop from release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1248853 [10:04:07] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/TestKitchen] Don't drop from release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1248853 (owner: 10Jforrester) [10:05:42] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/TestKitchen] Don't drop from release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1248853 (owner: 10Jforrester) [10:06:25] !log Zuul: [mediawiki/extensions/TestKitchen] Don't drop from release branches [10:06:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:29:49] (03PS1) 10Jforrester: Zuul: Disable all browser tests on release branches except Wikibase's [integration/config] - 10https://gerrit.wikimedia.org/r/1307374 (https://phabricator.wikimedia.org/T430415) [10:31:20] (03CR) 10CI reject: [V:04-1] Zuul: Disable all browser tests on release branches except Wikibase's [integration/config] - 10https://gerrit.wikimedia.org/r/1307374 (https://phabricator.wikimedia.org/T430415) (owner: 10Jforrester) [10:47:06] (03PS2) 10Jforrester: Zuul: Disable all browser tests on release branches except Wikibase's [integration/config] - 10https://gerrit.wikimedia.org/r/1307374 (https://phabricator.wikimedia.org/T430415) [10:59:18] 06Project-Admins: Consider archiving #server-side-upload-request - https://phabricator.wikimedia.org/T428508#12085266 (10Aklapper) I'd prefer to see https://commons.wikimedia.org/wiki/Help:Server-side_upload updated accordingly before archiving this Phabricator project. [11:15:21] 10Gerrit, 06collaboration-services: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12085288 (10jcrespo) I confirm this is very sustainable now: ` 725244 Incr 1,817 84.76 M OK 03-Jul-26 11:00 gerrit2002.wikimedia.org-Hourly-Tue-ReposEqiad-gerrit-repo-data 725243 In... [11:17:44] (03PS1) 10Jforrester: Zuul: Make the in-mediawiki-tarball template real [integration/config] - 10https://gerrit.wikimedia.org/r/1307400 [11:17:44] (03PS1) 10Jforrester: zuul-mw-jobs-runner: Don't run vendor jobs on release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1307401 [11:17:44] (03PS1) 10Jforrester: zuul-mw-jobs-runner: Allow filtering to tarball jobs, too [integration/config] - 10https://gerrit.wikimedia.org/r/1307402 [11:20:09] (03CR) 10Jforrester: [C:03+2] Zuul: Make the in-mediawiki-tarball template real [integration/config] - 10https://gerrit.wikimedia.org/r/1307400 (owner: 10Jforrester) [11:20:15] (03CR) 10Jforrester: [C:03+2] zuul-mw-jobs-runner: Don't run vendor jobs on release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1307401 (owner: 10Jforrester) [11:20:20] (03CR) 10Jforrester: [C:03+2] zuul-mw-jobs-runner: Allow filtering to tarball jobs, too [integration/config] - 10https://gerrit.wikimedia.org/r/1307402 (owner: 10Jforrester) [11:21:53] (03Merged) 10jenkins-bot: Zuul: Make the in-mediawiki-tarball template real [integration/config] - 10https://gerrit.wikimedia.org/r/1307400 (owner: 10Jforrester) [11:21:57] (03Merged) 10jenkins-bot: zuul-mw-jobs-runner: Don't run vendor jobs on release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1307401 (owner: 10Jforrester) [11:22:04] (03Merged) 10jenkins-bot: zuul-mw-jobs-runner: Allow filtering to tarball jobs, too [integration/config] - 10https://gerrit.wikimedia.org/r/1307402 (owner: 10Jforrester) [11:22:07] !log Zuul: Make the in-mediawiki-tarball template real [11:22:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:46:58] 10Continuous-Integration-Config, 10Diffusion, 10Phabricator: integration-agent-docker machines excessively pull some Wikibase related Git repos in Diffusion - https://phabricator.wikimedia.org/T349921#12085364 (10Aklapper) ` mysql:phstats@m3-slave.eqiad.wmnet [phabricator_maniphest]> SELECT COUNT(r.name) AS... [11:54:53] (03CR) 10Jforrester: [C:03+2] "> Hi James! Sorry for the delay! I missed this conversation." [integration/config] - 10https://gerrit.wikimedia.org/r/1248853 (owner: 10Jforrester) [12:08:24] (03update) 10aklapper: Bot comment styles improvements from the hackathon [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/115 (https://phabricator.wikimedia.org/T416964 https://phabricator.wikimedia.org/T420522) (owner: 10matmarex) [12:14:15] (03merge) 10aklapper: Bot comment styles improvements from the hackathon [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/115 (https://phabricator.wikimedia.org/T416964 https://phabricator.wikimedia.org/T420522) (owner: 10matmarex) [12:16:04] 10Phabricator (phabricator-next), 07CSS, 07Mobile: Simplified patch status comments have inconsistent margin on mobile - https://phabricator.wikimedia.org/T416964#12085451 (10Aklapper) [12:16:14] 10Phabricator (phabricator-next): Hovering over long truncated strings may result in some vertical movement - https://phabricator.wikimedia.org/T420522#12085453 (10Aklapper) [12:17:52] (03update) 10aklapper: Diffusion: Use PhorgeRestrictableInteractionInterface to disable some UI [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/123 (https://phabricator.wikimedia.org/T418578) [12:18:15] (03merge) 10aklapper: Diffusion: Use PhorgeRestrictableInteractionInterface to disable some UI [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/123 (https://phabricator.wikimedia.org/T418578) [12:18:42] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎): Port custom code in DiffusionCommitController to use upstream PhorgeRestrictableInteractionInterface - https://phabricator.wikimedia.org/T418578#12085461 (10Aklapper) [13:09:34] (03CR) 10Santiago Faci: "> Ah, interesting. Note that I don't see any sign-offs from Release Engineering (or Testing Platform or SRE or anyone else) about the impa" [integration/config] - 10https://gerrit.wikimedia.org/r/1248853 (owner: 10Jforrester) [13:11:23] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12085622 (10hashar) On https://zuul.wikimedia.org/openapi the API parameters can be filed with spaces or slashes and given they are stri... [13:25:19] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12085675 (10hashar) >>! In T431003#12082436, @Dzahn wrote: > When we use `AllowEncodedSlashes NoDecode` to ensure... [15:06:05] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/WikiLambda] Add CirrusSearch dep [integration/config] - 10https://gerrit.wikimedia.org/r/1307465 [15:06:54] 10Scap: scap deploy --init on deployment server fails on first puppet run - https://phabricator.wikimedia.org/T257317#12086024 (10Raine) Guess what: also blocking T418262 and by extension T423714. I'm trying to actually understand what needs to happen to resolve it, and then ideally fix it in puppet... [15:07:59] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikiLambda] Add CirrusSearch dep [integration/config] - 10https://gerrit.wikimedia.org/r/1307465 (owner: 10Jforrester) [15:09:44] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikiLambda] Add CirrusSearch dep [integration/config] - 10https://gerrit.wikimedia.org/r/1307465 (owner: 10Jforrester) [15:09:57] !log Zuul: [mediawiki/extensions/WikiLambda] Add CirrusSearch dep [15:09:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:20:23] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/WikiLambda] Add Elastica dep too [integration/config] - 10https://gerrit.wikimedia.org/r/1307472 [15:22:52] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T430829#12086041 (10Jdforrester-WMF) [15:23:13] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikiLambda] Add Elastica dep too [integration/config] - 10https://gerrit.wikimedia.org/r/1307472 (owner: 10Jforrester) [15:25:02] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikiLambda] Add Elastica dep too [integration/config] - 10https://gerrit.wikimedia.org/r/1307472 (owner: 10Jforrester) [15:25:11] !log Zuul: [mediawiki/extensions/WikiLambda] Add Elastica dep too [15:25:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:42:46] 10Scap: scap deploy --init on deployment server fails on first puppet run - https://phabricator.wikimedia.org/T257317#12086101 (10Raine) >>! In T257317#12086024, @Raine wrote: > Guess what: also blocking T418262 and by extension T423714. I'm trying to actually understand what needs to happen to resolve it, and t... [17:16:01] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12086289 (10AlexisJazz) >>! In T420833#12082393, @bd808 wrote: > > Every existing automation against Beta cluster would fail. Bot wave would probably d... [19:59:24] 06Project-Admins: Create project tag for standart thumbnail sizes bugs - https://phabricator.wikimedia.org/T431143 (10Nemoralis) 03NEW [22:43:34] (03PS1) 10Hashar: zuul: keep logs for up to 2 days (instead of 5 minutes) [integration/config] - 10https://gerrit.wikimedia.org/r/1307516 [22:46:41] (03CR) 10Hashar: [C:03+2] zuul: keep logs for up to 2 days (instead of 5 minutes) [integration/config] - 10https://gerrit.wikimedia.org/r/1307516 (owner: 10Hashar) [22:48:19] (03Merged) 10jenkins-bot: zuul: keep logs for up to 2 days (instead of 5 minutes) [integration/config] - 10https://gerrit.wikimedia.org/r/1307516 (owner: 10Hashar)