[00:04:17] >23:56:50 [10 hits] PHP Warning: require_once(/srv/mediawiki/php-master/includes/libs/rdbms/database/utils/CriticalSessionInfo.php): Failed to open stream: No such file or directory [00:04:38] but [00:04:40] > rename includes/libs/rdbms/{database => Database}/Utils/CriticalSessionInfo.php (100%) [00:04:45] I guess it'll sort itself out [00:05:05] slightly surprised they've not broken more things [00:07:04] Yippee, build fixed! [00:07:04] Project beta-scap-sync-world build #231165: 09FIXED in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231165/ [00:25:09] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T408271#11347541 (10Wargo) [00:43:21] I'm pretty sure that https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/10070/ is stuck. runnig for 33 minutes, last log output ~28 minutes ago. [00:44:12] Do we kill stuck selenium tests these days? I honestly can't remember. [00:46:44] !log Cancelled stuck "quibble-with-gated-extensions-selenium-php81 #10070" job [00:46:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:49:19] thcipriani: zuul queues are "holy shit" levels of backed up. Not sure of a cause at the moment. [00:57:00] lots of long patch chains [00:57:07] and some backlog from a lot of failures from earlier [00:57:16] lots of things being merged [00:58:58] * bd808 dreams of flakey test detect and disable automation [01:05:19] also browser tests are sloooooow [01:31:20] !log Unblock 197.14.24.0/21 (T409197) [01:31:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:36:17] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-cxserver02 on project deployment-prep - https://phabricator.wikimedia.org/T409254#11347692 (10bd808) `COUNTEREXAMPLE Error: Could not prefetch package provider 'apt': Execution of '/usr/bin/apt-mark showmanual' returned 100: E... [01:44:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-docker-cxserver02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [01:47:34] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-cxserver02 on project deployment-prep - https://phabricator.wikimedia.org/T409254#11347712 (10bd808) 05Openβ†’03Resolved a:03bd808 This was weird. The first time I ran puppet I got the error message in T409254#11347692... [02:13:09] 06Release-Engineering-Team (Radar), 06collaboration-services, 06SRE, 06Traffic, 13Patch-For-Review: Deploy a TCP proxy across all DCs - https://phabricator.wikimedia.org/T408532#11347743 (10Dzahn) We had some strange results when trying to debug this together. So I ended up testing every combination betw... [06:21:07] (03PS1) 10Kevin Bazira: inference-services: update CI pipeline jobs for revertrisk-wikidata model-server [integration/config] - 10https://gerrit.wikimedia.org/r/1202379 (https://phabricator.wikimedia.org/T406179) [06:30:27] 10GitLab, 06collaboration-services, 06Infrastructure-Foundations: Continuous breakages of apt-staging - https://phabricator.wikimedia.org/T409253#11347994 (10Joe) >>! In T409253#11343878, @LSobanski wrote: > As an aside, this task doesn''t meet Unbreak Now criteria as defined in https://www.mediawiki.org/wik... [06:36:17] 10GitLab, 06collaboration-services, 06Infrastructure-Foundations: Continuous breakages of apt-staging - https://phabricator.wikimedia.org/T409253#11347996 (10Joe) As my closing 2 cents about this whole system: * A system should report correctly what is blocking it and what is erroring out. The apt-staging... [08:48:24] 10Continuous-Integration-Config, 06Release-Engineering-Team (Doing 😎): Jenkins: In the integration project in WMCS, test out larger workers for CI - https://phabricator.wikimedia.org/T383254#11348421 (10hashar) Brennen and I exchanged about it yesterday. We currently have instances with 8 virtual CPUS, I based... [08:53:50] 10Continuous-Integration-Config, 06Release-Engineering-Team (Doing 😎): Jenkins: In the integration project in WMCS, test out larger workers for CI - https://phabricator.wikimedia.org/T383254#11348440 (10hashar) As a side note, I think it is possible to change the flavor of an instance which saves the hassle of... [08:54:44] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Regression, 07Upstream: The 'Other Assignee' task field is unsearchable using the web UI - https://phabricator.wikimedia.org/T408524#11348442 (10A_smart_kitten) (per upstream task description) [08:55:38] (03CR) 10Hashar: [C:03+2] inference-services: update CI pipeline jobs for revertrisk-wikidata model-server [integration/config] - 10https://gerrit.wikimedia.org/r/1202379 (https://phabricator.wikimedia.org/T406179) (owner: 10Kevin Bazira) [08:57:35] (03Merged) 10jenkins-bot: inference-services: update CI pipeline jobs for revertrisk-wikidata model-server [integration/config] - 10https://gerrit.wikimedia.org/r/1202379 (https://phabricator.wikimedia.org/T406179) (owner: 10Kevin Bazira) [08:59:18] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1202058 "Zuul: use 2 spaces indent in YAML deps files" and https://gerrit.wikimedia.org/r/c/integration/config/+/1202379 "inference-services: update CI pipeline jobs for revertrisk-wikidata model-server" [08:59:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:07:02] Project beta-scap-sync-world build #231217: 04FAILURE in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231217/ [09:14:34] Yippee, build fixed! [09:14:34] Project beta-scap-sync-world build #231218: 09FIXED in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231218/ [09:17:02] 10Phabricator (2025-10-14), 06Release-Engineering-Team (Doing 😎): Change Maniphest AdvancedSearch default for "Group by" to "None" and for "Order by" to "Updated" - https://phabricator.wikimedia.org/T404843#11348541 (10hashar) That is wonderful, thank you @Aklapper for the quick implementation :] [09:41:32] (03open) 10zabe: Update location for dumpInterwiki script [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1037 [10:16:11] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T408271#11348783 (10IKhitron) Please pay attention to [[https://meta.wikimedia.org/wiki/Tech#A_breaking_change?|m:Tech#A breaking change]].... [11:34:08] 06Release-Engineering-Team (Radar), 06collaboration-services, 06SRE, 06Traffic, 05WMF-NDA: Change Gitiles caching config - https://phabricator.wikimedia.org/T409422 (10LSobanski) 03NEW [11:40:16] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T408271#11349124 (10mszwarc) >>! In T408271#11348783, @IKhitron wrote: > Please pay attention to [[https://meta.wikimedia.org/wiki/Tech#A_b... [11:46:21] (03CR) 10Phedenskog: "Ok cool. ItΒ¨s hard to measure if we do not try it though, but if you say it will be the same, then we do not need to try it. I'll update t" [integration/config] - 10https://gerrit.wikimedia.org/r/1201540 (https://phabricator.wikimedia.org/T408360) (owner: 10Phedenskog) [11:50:45] (03PS1) 10Phedenskog: Dockerfiles: Remove Chromedriver start/stop from browser images [integration/config] - 10https://gerrit.wikimedia.org/r/1201546 (https://phabricator.wikimedia.org/T408327) [11:54:40] 10Phabricator: Request to create a Phabricator project for Wikidata Reference Validator - https://phabricator.wikimedia.org/T409427 (10JosefAnthony) 03NEW [11:57:13] 06Project-Admins: Request to create a Phabricator project for Wikidata Reference Validator - https://phabricator.wikimedia.org/T409427#11349236 (10A_smart_kitten) [11:59:22] 10Phabricator: Create a global Herald rule to add tasks tagged with #MediaWiki-CreateAccount-page to #MediaWiki-User-login-and-signup - https://phabricator.wikimedia.org/T408253#11349257 (10A_smart_kitten) Gentle bump on this task, given it's been ~a couple of weeks (& should hopefully be relatively easy to acti... [12:45:58] 10MediaWiki-Releasing, 06translatewiki.net, 06LPL Essential (FY26 Q2), 10LPL Projects (Other), 05MW-1.45-release: Configure Translatewiki.net for REL1_45 translation backports - https://phabricator.wikimedia.org/T408469#11349426 (10abi_) 05In progressβ†’03Resolved Exports happened to 1.45 branch -... [13:03:25] (03PS1) 10Hashar: utils: zuul-mw-jobs-runner add --[no]-phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/1202699 [13:06:04] (03CR) 10Hashar: [C:03+2] utils: zuul-mw-jobs-runner add --[no]-phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/1202699 (owner: 10Hashar) [13:07:37] (03Merged) 10jenkins-bot: utils: zuul-mw-jobs-runner add --[no]-phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/1202699 (owner: 10Hashar) [13:14:19] 10Continuous-Integration-Config, 13Patch-For-Review: Create a CI job to enforce tests to pass with solely required extensions - https://phabricator.wikimedia.org/T407797#11349504 (10hashar) [13:19:30] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work, 13Patch-For-Review: Proposal for refactoring of backport dependency handling - https://phabricator.wikimedia.org/T362987#11349513 (10jnuche) > ` > 3: If `Change` belongs to the MW configuration repo and `Dep` belongs > to a MW... [13:21:40] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work, 13Patch-For-Review: Proposal for refactoring of backport dependency handling - https://phabricator.wikimedia.org/T362987#11349520 (10jnuche) >>! In T362987#11345531, @Lucas_Werkmeister_WMDE wrote: > To be honest, I would prefer a warning promp... [13:24:04] (03PS1) 10Hashar: utils: zuul-mw-jobs-runner fixes threads spawning [integration/config] - 10https://gerrit.wikimedia.org/r/1202707 [13:24:18] (03CR) 10Hashar: [C:03+2] utils: zuul-mw-jobs-runner fixes threads spawning [integration/config] - 10https://gerrit.wikimedia.org/r/1202707 (owner: 10Hashar) [13:26:09] (03Merged) 10jenkins-bot: utils: zuul-mw-jobs-runner fixes threads spawning [integration/config] - 10https://gerrit.wikimedia.org/r/1202707 (owner: 10Hashar) [13:32:45] !log Running 24 Jenkins jobs concurrently using: config/utils/zuul-mw-jobs-runner.py --start --jobs 24 --no-selenium --no-phpunit --requires-only -- config/zuul/layout.yaml config/zuul/parameter_functions.py |& tee run-requires-only-20251106.log [13:32:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:33:13] !log Running 24 Jenkins jobs concurrently using: config/utils/zuul-mw-jobs-runner.py --start --jobs 24 --no-selenium --no-phpunit --requires-only -- config/zuul/layout.yaml config/zuul/parameter_functions.py |& tee run-requires-only-20251106.log # T407797 [13:33:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:33:16] T407797: Create a CI job to enforce tests to pass with solely required extensions - https://phabricator.wikimedia.org/T407797 [13:36:18] 06Project-Admins: Request to create a Phabricator project for Wikidata Reference Validator - https://phabricator.wikimedia.org/T409427#11349576 (10JJMC89) 05Openβ†’03Stalled You should use #striker (https://toolsadmin.wikimedia.org/) to do this yourself. [13:48:14] 06Project-Admins: Request to create a Phabricator project for Wikidata Reference Validator - https://phabricator.wikimedia.org/T409427#11349607 (10JosefAnthony) 05Stalledβ†’03Resolved a:03JosefAnthony okay thanks, done! [13:53:15] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work, 13Patch-For-Review: Proposal for refactoring of backport dependency handling - https://phabricator.wikimedia.org/T362987#11349622 (10Lucas_Werkmeister_WMDE) https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1202661 is another exam... [13:55:14] 06Project-Admins: Request to create a Phabricator project for Wikidata Reference Validator - https://phabricator.wikimedia.org/T409427#11349630 (10A_smart_kitten) (for the records, #wikidata_reference_validator was created) [14:00:06] (03CR) 10Kevin Bazira: "Thanks for the review @hashar@free.fr" [integration/config] - 10https://gerrit.wikimedia.org/r/1202379 (https://phabricator.wikimedia.org/T406179) (owner: 10Kevin Bazira) [14:10:53] 10Continuous-Integration-Config, 13Patch-For-Review: Create a CI job to enforce tests to pass with solely required extensions - https://phabricator.wikimedia.org/T407797#11349714 (10hashar) Some of them are failing `ResourcesTest::testValidDependencies` which is that some Resource Loader modules are missing.... [14:18:37] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work, 13Patch-For-Review: Proposal for refactoring of backport dependency handling - https://phabricator.wikimedia.org/T362987#11349751 (10jnuche) >>! In T362987#11349622, @Lucas_Werkmeister_WMDE wrote: > https://gerrit.wikimedia.org/r/c/operations/... [15:18:59] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10ConfirmEdit (CAPTCHA extension), 06Trust and Safety Product Team, 05WE4.2 Bot detection (WE4.2 hCaptcha editing trial): ConfirmEdit hCaptcha: Get QUnit tests to run in CI - https://phabricator.wikimedia.org/T406231#11350087 (10hashar) The... [15:46:24] (03merge) 10dancy: Update location for dumpInterwiki script [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1037 (owner: 10zabe) [15:46:58] (03open) 10dancy: Release 4.225.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1038 [15:49:16] (03merge) 10dancy: Release 4.225.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1038 [16:18:06] 10MediaWiki-Releasing, 10MediaWiki-Vendor, 05MW-1.45-release, 13Patch-For-Review: Prune /vendor for REL1_45 - https://phabricator.wikimedia.org/T408470#11350419 (10Jdforrester-WMF) Does the `okvpn/clock-lts` discussion need to finish first, or shall we consider this Resolved? [17:16:31] 10Continuous-Integration-Config, 13Patch-For-Review: Create a CI job to enforce tests to pass with solely required extensions - https://phabricator.wikimedia.org/T407797#11350680 (10hashar) After running `--resolve-requires --run phpunit-unit,phpunit,qunit` against all repos known to CI: | Failure | 271 | Suc... [18:31:40] 10Continuous-Integration-Infrastructure: quibble-with-gated-extensions-vendor-mysql-php81 failure due to postbuild jobs - https://phabricator.wikimedia.org/T409479 (10thcipriani) 03NEW [18:38:17] 10Continuous-Integration-Infrastructure: quibble-with-gated-extensions-vendor-mysql-php81 failure due to postbuild jobs - https://phabricator.wikimedia.org/T409479#11351081 (10thcipriani) I see there was a job update on Nov 4th in the Jenkins UI: {F69977413} Timing-wise this seems to match up to https://gerrit... [18:40:16] 10Continuous-Integration-Infrastructure: quibble-with-gated-extensions-vendor-mysql-php81 failure due to postbuild jobs - https://phabricator.wikimedia.org/T409479#11351086 (10thcipriani) But this also matches a Jenkins upgrade https://sal.toolforge.org/log/dfalTpoB8tZ8Ohr0CB3W which might be more likely to caus... [18:50:52] 10Continuous-Integration-Infrastructure: quibble-with-gated-extensions-vendor-mysql-php81 failure due to postbuild jobs - https://phabricator.wikimedia.org/T409479#11351126 (10Jdforrester-WMF) I think this mis-behaviour (of post-build steps failing and breaking a job that otherwise worked fine) is rare but pre-e... [19:16:54] Project beta-scap-sync-world build #231277: 04FAILURE in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231277/ [19:17:54] 19:16:53 [8 hits] Error: Failed opening required '/srv/mediawiki/php-master/includes/libs/rdbms/expression/Expression.php' (include_path='/srv/mediawiki/php-master/vendor/pear/console_getopt:/srv/mediawiki/php-master/vendor/pear/mail:/srv/mediaw [19:17:54] 19:16:53 [8 hits] PHP Warning: require_once(/srv/mediawiki/php-master/includes/libs/rdbms/expression/Expression.php): Failed to open stream: No such file or directory [19:17:59] opcache-y again [19:18:56] Hmm. php-fpm should be getting restarted every deployment. [19:19:46] (03CR) 10Jforrester: "For this to work in terms of producing new images, we'll need to rev the changelog (and cascade to dependent images). I can do that bit." [integration/config] - 10https://gerrit.wikimedia.org/r/1201546 (https://phabricator.wikimedia.org/T408327) (owner: 10Phedenskog) [19:20:20] Reedy: Or are you saying that those errors were triggered while files were being updated during deployment? [19:20:26] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Nicolasmichel] - https://phabricator.wikimedia.org/T409210#11351197 (10Nicolasmichel) Request is being made for [[ https://gitlab.wikimedia.org/cloudvps-repos/videocuttool/VideoCutTool | VideoCutTool ]] bug: htt... [19:20:56] dancy: Those files were moved in a batch of commits just merged [19:21:07] It did something similar yesterday, but recovered on the next run [19:21:34] ok good. [19:27:14] Yippee, build fixed! [19:27:14] Project beta-scap-sync-world build #231278: 09FIXED in 2 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231278/ [19:29:04] like that [19:29:34] I guess when 100+ files are renamed (casing) in one pull... :D [19:31:56] 10Continuous-Integration-Infrastructure: quibble-with-gated-extensions-vendor-mysql-php81 failure due to postbuild jobs - https://phabricator.wikimedia.org/T409479#11351234 (10Reedy) It's definitely wider in terms of jobs - https://integration.wikimedia.org/ci/job/quibble-for-mediawiki-core-browser-tests-only-ve... [20:13:09] 06Project-Admins: Create project tag for Monaco - https://phabricator.wikimedia.org/T409487 (10SomeRandomDeveloper) 03NEW [20:18:15] 06Project-Admins: Create project tag for Monaco - https://phabricator.wikimedia.org/T409487#11351425 (10Peachey88) Has this been discussed with the current maintainer? There also appears to be two repos, one on gerrit and then github with issues turned on(https://github.com/Universal-Omega/Monaco/issues), and a... [20:48:02] 06Project-Admins: Create project tag for Monaco - https://phabricator.wikimedia.org/T409487#11351498 (10SomeRandomDeveloper) >>! In T409487#11351425, @Peachey88 wrote: > Has this been discussed with the current maintainer? Due to time constraints he isn't active right now; he moved the repo to gerrit so the com... [21:36:53] Project beta-scap-sync-world build #231288: 04FAILURE in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231288/ [21:40:32] 10Continuous-Integration-Config, 10Codex, 10doc.wikimedia.org: Codex PHP: publish the demo page on doc.wikimedia.org - https://phabricator.wikimedia.org/T382414#11351719 (10Catrope) [21:41:05] 10Continuous-Integration-Config, 10Codex, 10doc.wikimedia.org: Codex PHP: publish the demo page on doc.wikimedia.org - https://phabricator.wikimedia.org/T382414#11351726 (10Catrope) [21:46:53] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work, 13Patch-For-Review: Proposal for refactoring of backport dependency handling - https://phabricator.wikimedia.org/T362987#11351734 (10dancy) >>! In T362987#11349520, @jnuche wrote: > @dancy do you have a take on this? I thought about this a lo... [21:47:08] Yippee, build fixed! [21:47:09] Project beta-scap-sync-world build #231289: 09FIXED in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/231289/ [21:51:37] (03PS1) 10BryanDavis: dockerfiles(quibble-bullseye): Add PYTHONUNBUFFERED=1 envvar [integration/config] - 10https://gerrit.wikimedia.org/r/1202844 [22:01:34] 10MediaWiki-Releasing, 10MediaWiki-Vendor, 05MW-1.45-release: Prune /vendor for REL1_45 - https://phabricator.wikimedia.org/T408470#11351770 (10ABreault-WMF) 05Openβ†’03Resolved >>! In T408470#11350419, @Jdforrester-WMF wrote: > Does the `okvpn/clock-lts` discussion need to finish first, or shall we co... [22:32:34] (03open) 10dancy: backport.py: Revise backport vote rule 3b [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1039 (https://phabricator.wikimedia.org/T362987) [22:32:36] (03update) 10dancy: backport.py: Revise backport vote rule 3b [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1039 (https://phabricator.wikimedia.org/T362987) [22:41:12] (03update) 10dancy: backport.py: Revise backport vote rule 3b [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1039 (https://phabricator.wikimedia.org/T362987)