[01:20:21] 10MediaWiki-Releasing, 10MediaWiki-Vendor, 05MW-1.43-release: Prune /vendor for REL1_43 - https://phabricator.wikimedia.org/T372319#10310933 (10Reedy) [01:20:21] 10MediaWiki-Releasing, 05MW-1.43-release: Release MW 1.43.0-rc.0 - https://phabricator.wikimedia.org/T372320#10310934 (10Reedy) [02:20:00] Project beta-update-databases-eqiad build #80306: 04FAILURE in 0.44 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/80306/ [03:32:14] Yippee, build fixed! [03:32:15] Project beta-update-databases-eqiad build #80307: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/80307/ [09:02:00] 10GitLab (Support), 06Tech-Docs-Team, 07Documentation: Simplify fork-and-merge-request workflow descriptions in local documentation - https://phabricator.wikimedia.org/T373122#10311322 (10KBach) 05In progress→03Resolved Thanks @bd808, this is much better. The changes are now [[ https://www.mediawiki.... [09:05:11] hashar: hi, the quibble jobs needed to cut the train are all failing with "Error: your composer.lock file is not up to date. Run "composer update --no-dev" to install newer dependencies" [09:05:15] does that sound familiar by any chance? [09:11:50] 10GitLab (Infrastructure), 06collaboration-services: GitLab OpenSSL 3 upgrade in 17.7 - https://phabricator.wikimedia.org/T379598 (10Jelto) 03NEW [09:12:20] 10GitLab (Infrastructure), 06collaboration-services: GitLab OpenSSL 3 upgrade in 17.7 - https://phabricator.wikimedia.org/T379598#10311361 (10Jelto) p:05Triage→03High [10:04:07] Krinkle, James_F: the core quibble tests keep failing with "Error: your composer.lock file is not up to date": https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1090435?tab=checks [10:04:16] that's keeping the train jobs from cutting the branch for `wmf/1.44.0-wmf.3` [10:04:27] it looks like the `wikimedia/relpath` dep version may need to be bumped: "wikimedia/relpath: 4.0.1 installed, 4.0.0 required" [10:04:33] is that something you may be able to help with? [10:06:24] actually, just noticed it's asking for an older version, weird [10:34:36] ok, so there was already a a patch for that dep, funny thing is it got merged into master shortly after the the core branch was cut: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1089908 [10:35:59] hashar: maybe the solution here is to delete the core branch for `wmf/1.44.0-wmf.3` and run everything again? [10:36:05] don't know how to do that in gerrit though [10:45:55] 06Project-Admins: Create project tag for Bangla WikiConference Hackathon 2024 - https://phabricator.wikimedia.org/T379610 (10Khattab) 03NEW [10:51:45] 06Project-Admins: Create project tag for Bangla WikiConference Hackathon 2024 - https://phabricator.wikimedia.org/T379610#10311686 (10Khattab) I apologize for raising the issue in Phabricator lately. Due to some misunderstandings among us, the task opening in Phabricator got delayed. I hope this won't be a probl... [11:27:53] jnuche: that should not happen :D [11:28:10] the composer.json in mediawiki/core is not in sync with the lock in mediawiki/vendor [11:28:30] my guess is we had some race condition while cutting the branch [11:28:50] yeah, take a look at the backscroll [11:29:06] ah you found it :) [11:30:03] so you can cherry pick it to the wmf branch [11:31:02] ok, that would also work, do you have +2 permissions for the core repo? [11:31:25] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1090456 :) [11:31:54] and iirc CR+2 is granted to member of the deployment group [11:32:13] can you +2 that one? [11:32:15] else I will do it [11:32:16] true, silly question, it's needded for backporting [11:32:38] +2'd [11:32:43] that should solve it! [11:34:21] alright, going to wait for the merge and the rerun the jenkins jobs [11:34:23] ty hashar [11:34:35] \o/ [11:34:48] and I apologize I was coding this mornign :b [11:35:12] I am out for lunch! [11:35:19] it's ok, we have plenty of time until the train window in US time [11:35:23] enjoy! [13:37:32] !log Deleted mediawiki-tarball Jenkins job, created as a WIP by https://gerrit.wikimedia.org/r/471036 as part of T208527 [13:37:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:37:37] T208527: Create jenkins job to build MediaWiki tarball releases - https://phabricator.wikimedia.org/T208527 [13:38:01] 10MediaWiki-Releasing, 10Release-Engineering-Team (Seen), 07Security: Create jenkins job to build MediaWiki tarball releases - https://phabricator.wikimedia.org/T208527#10312180 (10hashar) 05Open→03Declined [13:40:14] 10MediaWiki-Releasing, 10Release-Engineering-Team (Seen), 07Security: Streamline/automate MW tarball security release process - https://phabricator.wikimedia.org/T156445#10312188 (10hashar) 05Open→03Declined [13:41:13] 10MediaWiki-Releasing, 10Release-Engineering-Team (Seen), 07Security: Create fab task to deploy jenkins job to either ci-jenkins or releases-jenkins - https://phabricator.wikimedia.org/T208528#10312184 (10hashar) 05Open→03Declined This is no more relevant, the release Jenkins is now configured using... [13:46:24] so hmm [13:46:26] tests fail locally [13:46:36] but passes on upstream CI \o/ [13:46:46] so I guess it is all fine and my local setup is borked somehow [13:56:04] 10GitLab (Infrastructure), 06collaboration-services: GitLab OpenSSL 3 upgrade in 17.7 - https://phabricator.wikimedia.org/T379598#10312297 (10Jelto) Before checking any integrations, I verified the availability of `openssl` on the existing Bullseye GitLab hosts. They currently have `openssl` version `1.1.1`, w... [14:02:04] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T375662#10312347 (10jnuche) [14:28:21] 10Scap: scap backport fails at purgeMessageBlobStore.php with getaddrinfo failed - https://phabricator.wikimedia.org/T379589#10312521 (10Tgr) > Name or service not known (WMF_MAINTENANCE_OFFLINE_placeholder) The only place where that placeholder is used is the [[https://gerrit.wikimedia.org/g/operations/mediawi... [14:28:47] jnuche: as hashar said, this is an unlucky coincidence it seems where the branch was cut and/or the job run once just in-between those two commits. should be fine on re-try and/or when the accompanying change is backported. [14:29:08] The check for composer.lock was broken for a few months, but now fixed per T370380. [14:29:09] T370380: mediawiki/core and mediawiki/vendor both skip composer.lock checks - https://phabricator.wikimedia.org/T370380 [14:29:38] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 06Java-Scala-Standardization, 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Add CI_RELEASE_TOKEN secret for {name}-maven-release job... - https://phabricator.wikimedia.org/T379203#10312527 [14:30:37] Krinkle: yeah, the cut job is working again now, ty :) [14:31:45] it's good in a way, the alternative is to have the job cut break deps and not notice until we hit traffic in production in some subtle way. [14:31:47] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 06Java-Scala-Standardization, 10Data-Platform-SRE (2024.11.09 - 2024.11.29): Add CI_RELEASE_TOKEN secret for {name}-maven-release job... - https://phabricator.wikimedia.org/T379203#10312532 [14:31:55] but last few months this check was not running [14:41:23] 10Scap: scap backport fails at purgeMessageBlobStore.php with getaddrinfo failed - https://phabricator.wikimedia.org/T379589#10312593 (10Tgr) git log -S says the use of WMF_MAINTENANCE_OFFLINE is new (first used in the "Containerize MediaWiki script execution" patch that got reverted and unreverted multiple times). [14:50:12] ack, definitely better to catch it at the job cut stage [15:59:17] 10Scap: scap backport fails at purgeMessageBlobStore.php with getaddrinfo failed - https://phabricator.wikimedia.org/T379589#10313035 (10Krinkle) The purgeMessageBlobStore.php maintenance script effectively just runs one line of code, `WANObjectCache->touchCheckKey()` which invalidates 1 shared "check" key in me... [16:02:29] 10Scap, 06MediaWiki-Platform-Team, 10MediaWiki-ResourceLoader: scap backport fails at purgeMessageBlobStore.php with getaddrinfo failed - https://phabricator.wikimedia.org/T379589#10313048 (10Krinkle) Tagging RL since this likely needs a patch to maintenance/purgeMessageBlobStore.php which is part of Resourc... [16:08:56] 10Release-Engineering-Team (Radar), 06serviceops, 06SRE, 05Train Deployments: MW script "eval.php" failing for "testcommonswiki" during train operations - https://phabricator.wikimedia.org/T379628#10313092 (10brennen) [16:18:05] (03open) 10dancy: WIP: mwscript: Don't set WMF_MAINTENANCE_OFFLINE when --network is used [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/565 [16:18:07] (03update) 10dancy: WIP: mwscript: Don't set WMF_MAINTENANCE_OFFLINE when --network is used [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/565 [16:22:22] (03update) 10dancy: mwscript: Don't set WMF_MAINTENANCE_OFFLINE when --network is used [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/565 [16:22:26] (03update) 10dancy: mwscript: Don't set WMF_MAINTENANCE_OFFLINE when --network is used [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/565 [16:24:29] (03merge) 10dancy: mwscript: Don't set WMF_MAINTENANCE_OFFLINE when --network is used [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/565 [16:24:58] (03open) 10dancy: Release 4.123.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/566 [16:26:51] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T375662#10313156 (10brennen) [16:27:10] (03merge) 10dancy: Release 4.123.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/566 [16:37:47] 10Scap, 06MediaWiki-Platform-Team, 10MediaWiki-ResourceLoader: scap backport fails at purgeMessageBlobStore.php with getaddrinfo failed - https://phabricator.wikimedia.org/T379589#10313252 (10dancy) scap 4.123.0 has been deployed which should address the scap side of this problem. [16:41:46] 10Release-Engineering-Team (Radar), 06serviceops, 06SRE, 05Train Deployments: MW script "eval.php" failing for "testcommonswiki" during train operations - https://phabricator.wikimedia.org/T379628#10313250 (10dancy) scap 4.123.0 has been deployed which should address this problem. [17:06:12] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 06SRE Observability, 13Patch-For-Review: Scap prometheus migration: Reduce the cardinality of scap timers/statsd metrics - https://phabricator.wikimedia.org/T377883#10313477 (10dancy) @lmata @colewhite Does the existing statsd listener accept tagged... [17:13:30] (03CR) 10Ahmon Dancy: [C:03+1] "OK w/ me" [releng/phatality] - 10https://gerrit.wikimedia.org/r/1088659 (https://phabricator.wikimedia.org/T342476) (owner: 10Cwhite) [17:28:36] 10Release-Engineering-Team (Radar), 06serviceops, 06SRE, 05Train Deployments: MW script "eval.php" failing for "testcommonswiki" during train operations - https://phabricator.wikimedia.org/T379628#10313587 (10brennen) a:05brennen→03dancy [17:32:22] 10Release-Engineering-Team (Radar), 06serviceops, 06SRE, 05Train Deployments: MW script "eval.php" failing for "testcommonswiki" during train operations - https://phabricator.wikimedia.org/T379628#10313583 (10brennen) 05Open→03Resolved a:03brennen > scap 4.123.0 has been deployed which should add... [18:02:37] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 06SRE Observability, 13Patch-For-Review: Scap prometheus migration: Reduce the cardinality of scap timers/statsd metrics - https://phabricator.wikimedia.org/T377883#10313775 (10colewhite) >>! In T377883#10313476, @dancy wrote: > If it does, that woul... [19:03:26] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T375662#10314116 (10brennen) [21:19:37] 10Release-Engineering-Team (Priority Backlog 📥), 07OKR-Work: [WE6.2.6] Create design document for Group -1 deployment - https://phabricator.wikimedia.org/T379683 (10bd808) 03NEW [21:20:23] 10Release-Engineering-Team (Priority Backlog 📥), 07OKR-Work: [WE6.2.6] Create design document for Group -1 deployment - https://phabricator.wikimedia.org/T379683#10314551 (10bd808) p:05Triage→03High [21:21:13] 10Release-Engineering-Team (Doing 😎), 07OKR-Work: [WE6.2.6] Create design document for Group -1 deployment - https://phabricator.wikimedia.org/T379683#10314552 (10bd808) 05Open→03In progress [21:22:30] 10Release-Engineering-Team (Doing 😎): Draft and get approval for next hypothesis to follow WE6.2.1 - https://phabricator.wikimedia.org/T375145#10314558 (10bd808) 05In progress→03Resolved {T379683} [22:32:05] Project beta-update-databases-eqiad build #80326: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/80326/ [22:55:06] * bd808 manually triggers the database update job to see if it will fix itself [22:56:46] should do, as its vendor bumps [22:58:07] The logging output made it hard for me to guess if that was actually the failure or something else. I think I've gotten out of practice at interpreting that chicken scratch [23:00:00] scroll almost to the bottom [23:00:16] I'm guessing it only failed towards the end because of a code update in done before this had finished? [23:06:39] Yippee, build fixed! [23:06:39] Project beta-update-databases-eqiad build #80327: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/80327/