[00:00:02] (03PS1) 10Dzahn: add MOTD banner to announce the new SSH host key [software/gerrit] - 10https://gerrit.wikimedia.org/r/1312639 (https://phabricator.wikimedia.org/T240266) [05:35:03] maintenance-disconnect-full-disks build 826067 integration-agent-docker-1084 (/: 28%, /srv: 98%, /var/lib/docker: 29%): OFFLINE due to disk space [05:40:03] maintenance-disconnect-full-disks build 826068 integration-agent-docker-1084 (/: 28%, /srv: 73%, /var/lib/docker: 29%): RECOVERY disk space OK [07:08:23] !log integration: cleaned up Jenkins workspace on integration-agent-docker-1084 [07:08:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:33:21] Project mediawiki-core-doxygen build #21644: 04FAILURE in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21644/ [09:15:17] (03PS1) 10Phedenskog: castor: measure cache-load rsync [integration/config] - 10https://gerrit.wikimedia.org/r/1313076 (https://phabricator.wikimedia.org/T432685) [09:23:58] (03PS2) 10Phedenskog: castor: measure cache-load rsync [integration/config] - 10https://gerrit.wikimedia.org/r/1313076 (https://phabricator.wikimedia.org/T432685) [09:34:29] Project mediawiki-core-doxygen build #21645: 04STILL FAILING in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21645/ [10:22:47] 06Project-Admins: Create project tag for Tyap Wikimedians User Group - https://phabricator.wikimedia.org/T432706 (10Kambai_Akau) 03NEW [10:34:51] Yippee, build fixed! [10:34:51] Project mediawiki-core-doxygen build #21646: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21646/ [10:52:25] (03update) 10volker-e: releases: Bump stylelint-config-wikimedia to 0.19.3 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/134 (owner: 10jforrester) [10:52:35] (03update) 10volker-e: releases: Bump stylelint-config-wikimedia to 0.19.3 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/134 (owner: 10jforrester) [12:52:06] (03open) 10taavi: Allow running web interface as a Toolforge tool [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/117 (https://phabricator.wikimedia.org/T432728) [13:00:28] (03update) 10taavi: Allow running web interface as a Toolforge tool [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/117 (https://phabricator.wikimedia.org/T432728) [13:02:45] (03update) 10taavi: Allow running web interface as a Toolforge tool [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/117 (https://phabricator.wikimedia.org/T432728) [13:11:26] (03update) 10taavi: Allow running web interface as a Toolforge tool [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/117 (https://phabricator.wikimedia.org/T432728) [13:16:57] (03update) 10taavi: Allow running web interface as a Toolforge tool [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/117 (https://phabricator.wikimedia.org/T432728) [13:17:41] (03update) 10taavi: Allow running web interface as a Toolforge tool [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/117 (https://phabricator.wikimedia.org/T432728) [13:26:12] (03update) 10taavi: Allow running web interface as a Toolforge tool [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/117 (https://phabricator.wikimedia.org/T432728) [13:29:52] (03update) 10taavi: Allow running web interface as a Toolforge tool [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/117 (https://phabricator.wikimedia.org/T432728) [13:49:53] (03update) 10taavi: Allow running web interface as a Toolforge tool [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/117 (https://phabricator.wikimedia.org/T432728) [13:51:44] James_F: https://libraryupgrader.toolforge.org/ [13:52:23] taavi: Well, it's there. [13:52:41] I trust you that the site I'm seeing is now served differently. :-) [13:52:43] Nice. [13:54:38] yeah, that's now running in toolforge based on !117 so we can get rid of the separate web VM that would've otherwise needed an OS upgrade [14:08:24] (03update) 10taavi: Allow running web interface as a Toolforge tool [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/117 (https://phabricator.wikimedia.org/T432728) [14:15:04] (03update) 10taavi: Allow running web interface as a Toolforge tool [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/117 (https://phabricator.wikimedia.org/T432728) [14:46:40] (03merge) 10taavi: Allow running web interface as a Toolforge tool [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/117 (https://phabricator.wikimedia.org/T432728) [15:11:30] (03CR) 10Ahmon Dancy: "I think this change needs to be made on the deploy/wmf/stable-3.10 branch (which already has an existing plugins/wm-motd.js file)." [software/gerrit] - 10https://gerrit.wikimedia.org/r/1312639 (https://phabricator.wikimedia.org/T240266) (owner: 10Dzahn) [15:19:04] (03PS1) 10Ahmon Dancy: deploy: Drop buster support [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1313215 [15:19:33] (03CR) 10CI reject: [V:04-1] deploy: Drop buster support [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1313215 (owner: 10Ahmon Dancy) [15:28:33] (03PS1) 10Ahmon Dancy: jb/integration.yaml: integration-zuul-deploy: Drop buster, pin image [integration/config] - 10https://gerrit.wikimedia.org/r/1313220 (https://phabricator.wikimedia.org/T432636) [15:31:45] (03CR) 10CI reject: [V:04-1] jb/integration.yaml: integration-zuul-deploy: Drop buster, pin image [integration/config] - 10https://gerrit.wikimedia.org/r/1313220 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [15:35:27] (03PS2) 10Ahmon Dancy: jb/integration.yaml: integration-zuul-deploy: Drop buster, pin image [integration/config] - 10https://gerrit.wikimedia.org/r/1313220 (https://phabricator.wikimedia.org/T432636) [15:43:58] dancy: RE: error during scap, it happenen again. I retried it because last week I think you saw a localisation build faiure in beta that went away [15:44:09] While that may have been true, I see now that my patch is in fact flawed. [15:44:34] I'm missing a global import for `$wmgLocalServices` so that's producing an error almost certainly [15:45:17] It's a shame though that I can't get that output, ad because the l10n build happens before the php health check I think, it means the visiblility into the equivalent of whever we do `echo 1 | mwscript eval.php` nowdays, might have regressed. [15:45:50] because while that step is presumalby still there, it's probalby a no-op when placed after mergeMessageFileList.php, and that one is forced offline so that prevents errors from making it to stderr with the offline failure making it there first. [15:46:08] * Krinkle checks logstash for any verbose messags that might be hiding underneath that we can perhaps surfface instead [15:46:40] Krinkle: I'll see what I can do to improve that situation. [15:48:27] (03PS3) 10Ahmon Dancy: integration/zuul/deploy: Drop buster, pin image [integration/config] - 10https://gerrit.wikimedia.org/r/1313220 (https://phabricator.wikimedia.org/T432636) [15:49:49] Krinkle: I also wonder what could have been improved on the CI side to prevent this particular error from slipping through [15:50:11] generally speaking wmf-config has no invocation of any real code involving MediaWiki. [15:50:15] We look at the config arrays, but that's it. [15:50:40] once we get to hooks and runtime callbacks, that's well outside static analysis to safely execute [15:51:30] (03CR) 10Ahmon Dancy: [C:03+2] integration/zuul/deploy: Drop buster, pin image [integration/config] - 10https://gerrit.wikimedia.org/r/1313220 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [15:51:34] re-ordering these two steps might suffice, althoguh I guess there's a chicken-egg problem in that we're calling this whilw building the container, and so you want to build it and then run the eval.php test once you've built it. [15:51:59] We used to run it before rsync, on the localhost. That's still possible, but it's certainly prudent to (also) run on the built container for sure. I can see that. [15:52:02] I vaguely recall that eval.php won't work without l10n set up, but I'll double check that [15:52:02] 10GitLab (Project Migration), 06Traffic: Migrate DNS repository from Gerrit to Gitlab - https://phabricator.wikimedia.org/T355906#12142181 (10BCornwall) 05Open→03Declined Heh, yeah, forgot this existed. Declining for now, we're keeping dns-related items on gerrit. Thanks! [15:52:19] Yeah, that seems probable. It wouldn't work during the Docker image build step before that point indeed. [15:53:22] (03Merged) 10jenkins-bot: integration/zuul/deploy: Drop buster, pin image [integration/config] - 10https://gerrit.wikimedia.org/r/1313220 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [15:53:57] hm. maybe it's not testing the container? [15:54:01] _check_fatals calls mwscript.run [15:54:13] https://gitlab.wikimedia.org/repos/releng/scap/-/blob/ff3559ba043d5bfac12e829b32a5d0002a6ad889/scap/main.py#L168 [15:54:19] right before _build_and_push_container_images [15:56:00] OK. I'm guessing this isn't the right function I'm looking at, because it's not running that from what I can see [15:56:23] ah _before_cluster_sync does l10n [15:56:30] I figured l10n is inside the build [15:58:02] hm.. well, no, not exactly _before_cluster_sync is not doing the l10n build, it just checks a few files. [15:59:25] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1313220 [15:59:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:00:42] (03PS2) 10Ahmon Dancy: deploy: Drop buster support [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1313215 [16:01:04] (03PS1) 10Dzahn: add MOTD banner to announce the new SSH host key [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1313233 (https://phabricator.wikimedia.org/T240266) [16:01:14] (03CR) 10CI reject: [V:04-1] deploy: Drop buster support [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1313215 (owner: 10Ahmon Dancy) [16:02:27] (03CR) 10Dzahn: "I meant to amend to this but ended up with this new change. so it's here now:" [software/gerrit] - 10https://gerrit.wikimedia.org/r/1312639 (https://phabricator.wikimedia.org/T240266) (owner: 10Dzahn) [16:02:47] (03Abandoned) 10Dzahn: add MOTD banner to announce the new SSH host key [software/gerrit] - 10https://gerrit.wikimedia.org/r/1312639 (https://phabricator.wikimedia.org/T240266) (owner: 10Dzahn) [16:03:14] (03Restored) 10Dzahn: add MOTD banner to announce the new SSH host key [software/gerrit] - 10https://gerrit.wikimedia.org/r/1312639 (https://phabricator.wikimedia.org/T240266) (owner: 10Dzahn) [16:03:24] (03Abandoned) 10Dzahn: add MOTD banner to announce the new SSH host key [software/gerrit] - 10https://gerrit.wikimedia.org/r/1312639 (https://phabricator.wikimedia.org/T240266) (owner: 10Dzahn) [16:05:24] OK. I got mixed up in different implementations of _update_caches. It's not just doing git_info cache, it also does l10n cache build there, in the case of ScapWorld. [16:05:37] so yeah, the order makes sense [16:06:07] reading the verbose log at https://logstash.wikimedia.org/goto/c152d612dd06582c93089af39db52d41 [16:07:56] I wonder how this used to work then. I'm guessing when we used rsync. we didn't force-remove the l10n cache, so there was at least something there, enough to satify a basic mwscript invocation and the rebuild could happen later. [16:09:13] but I don't see a force clear of it here either. but that doesn't matter because we try it earlier now, which means whether it needs to do a rebuild or not, the script will have run and thus encountered most of the likely issues running MW with wmf-config right there. [16:09:35] so assuming there is at least some l10n cache on disk, seems we can run eval.php just fine, and is probably what we used to do would be my guess. [16:12:17] I wonder if I'm missing where the in-countainer boundary starts. I recall that at some point we considered (or did) building the l10n cache during a Dockefile step and that being part of why it's offline. But looking at it now, it seems like this runs on the host? [16:14:21] 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Add RSS extension to PatchDemo - https://phabricator.wikimedia.org/T426130#12142334 (10brennen) 05In progress→03Resolved [16:19:41] (03CR) 10Ahmon Dancy: "recheck" [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1313215 (owner: 10Ahmon Dancy) [16:22:42] (03open) 10taavi: Improve stylelint autofixing [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/118 [16:23:08] (03PS1) 10Hslater: Zuul: [skins/BlueSpiceDiscovery] Add StandardDialogs dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1313244 [16:23:18] (03CR) 10Ahmon Dancy: "recheck" [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1313215 (owner: 10Ahmon Dancy) [16:25:09] (03CR) 10Ahmon Dancy: [C:03+1] "I'll deploy this later" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1313233 (https://phabricator.wikimedia.org/T240266) (owner: 10Dzahn) [16:26:37] (03CR) 10Ahmon Dancy: [C:03+2] deploy: Drop buster support [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1313215 (owner: 10Ahmon Dancy) [16:27:21] (03Merged) 10jenkins-bot: deploy: Drop buster support [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1313215 (owner: 10Ahmon Dancy) [16:28:08] (03merge) 10taavi: Improve stylelint autofixing [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/118 [16:28:25] (03PS1) 10Ahmon Dancy: Dockerfile.bullseye.build: Pin python2-build-bullseye image [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1313245 (https://phabricator.wikimedia.org/T432636) [16:29:21] (03CR) 10Ahmon Dancy: [C:03+2] Dockerfile.bullseye.build: Pin python2-build-bullseye image [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1313245 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [16:30:45] (03Merged) 10jenkins-bot: Dockerfile.bullseye.build: Pin python2-build-bullseye image [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1313245 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [16:31:18] (03PS2) 10Hashar: Update Zuul src to catch up with our fork of Zuul [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1312557 (https://phabricator.wikimedia.org/T258630) [16:31:56] (03CR) 10Ahmon Dancy: [C:03+2] Update Zuul src to catch up with our fork of Zuul [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1312557 (https://phabricator.wikimedia.org/T258630) (owner: 10Hashar) [16:32:31] (03Merged) 10jenkins-bot: Update Zuul src to catch up with our fork of Zuul [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1312557 (https://phabricator.wikimedia.org/T258630) (owner: 10Hashar) [17:19:12] (03open) 10taavi: branches: REL1_44 is EOL [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/135 [17:19:53] (03update) 10taavi: branches: REL1_44 is EOL [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/135 [17:24:33] (03update) 10taavi: branches: REL1_44 is EOL [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/135 [18:23:16] (03CR) 10Dzahn: "thank you! do you think I should merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1282395 slightly before / after / at the same " [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1313233 (https://phabricator.wikimedia.org/T240266) (owner: 10Dzahn) [19:13:39] (03CR) 10Ahmon Dancy: [C:03+2] add MOTD banner to announce the new SSH host key [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1313233 (https://phabricator.wikimedia.org/T240266) (owner: 10Dzahn) [19:14:23] (03Merged) 10jenkins-bot: add MOTD banner to announce the new SSH host key [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1313233 (https://phabricator.wikimedia.org/T240266) (owner: 10Dzahn) [20:13:02] (03open) 10volker-e: releases: Bump Codex to 2.6.2 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/136 (https://phabricator.wikimedia.org/T431473) [20:29:01] (03approved) 10catrope: releases: Bump Codex to 2.6.2 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/136 (https://phabricator.wikimedia.org/T431473) (owner: 10volker-e) [20:29:12] (03merge) 10catrope: releases: Bump Codex to 2.6.2 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/136 (https://phabricator.wikimedia.org/T431473) (owner: 10volker-e) [20:34:28] 06Release-Engineering-Team, 10Codex: 2.6.1 not exists in NPM - https://phabricator.wikimedia.org/T432786 (10Les4353) 03NEW [20:36:37] 06Release-Engineering-Team, 10Codex: 2.6.1 not exists in NPM - https://phabricator.wikimedia.org/T432786#12143444 (10Les4353) @Volker_E [21:43:52] 10Continuous-Integration-Config, 10Codex: [Request] Add me to trusted contributor in integration/config - https://phabricator.wikimedia.org/T432791 (10Les4353) 03NEW [21:44:13] 10Continuous-Integration-Config: [Request] Add me to trusted contributor in integration/config - https://phabricator.wikimedia.org/T432791#12143586 (10Les4353) [21:45:09] (03open) 10krinkle: Restore php-xhprof and add php-excimer [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 (https://phabricator.wikimedia.org/T401115) [21:50:52] (03open) 10krinkle: fundraising: Fix broken Depends reference [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/112 [21:52:17] 06Release-Engineering-Team, 10dev-images, 10MediaWiki-Core-Profiler, 10MediaWiki-Docker, and 2 others: Replace php-tideways_xhprof with php-xhprof in MediaWiki-Docker - https://phabricator.wikimedia.org/T401115#12143620 (10Krinkle) [21:52:26] 06Release-Engineering-Team, 10dev-images, 10MediaWiki-Core-Profiler, 10MediaWiki-Docker, and 2 others: Replace php-tideways_xhprof with php-xhprof in MediaWiki-Docker - https://phabricator.wikimedia.org/T401115#12143621 (10Krinkle) p:05Triage→03Medium a:03Krinkle