[07:24:15] (03PS1) 10Novem Linguae: zuul: Add MPGuy2824 to the CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/833703 [07:29:42] (03PS1) 10Hashar: Drop Debian Stretch from integration testing [tools/scap] - 10https://gerrit.wikimedia.org/r/833704 [07:30:25] (03PS1) 10Hashar: Remove Stretch from mediawiki/tools/scap [integration/config] - 10https://gerrit.wikimedia.org/r/833705 [07:30:32] (03CR) 10CI reject: [V: 04-1] Drop Debian Stretch from integration testing [tools/scap] - 10https://gerrit.wikimedia.org/r/833704 (owner: 10Hashar) [07:32:23] (03CR) 10Hashar: "recheck" [tools/scap] - 10https://gerrit.wikimedia.org/r/833704 (owner: 10Hashar) [07:51:45] o/ on https://gerrit.wikimedia.org/r/c/mediawiki/tools/phan/SecurityCheckPlugin/+/833436 Jenkins says: "This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository.", but this patch is already on top of master and it has no depends-on, is there something I'm missing and/or need to check? [07:55:03] dcausse: bonjour ;) [07:55:29] that is CI unable to merge one of the changes against the tip of the branches [07:55:39] or the daemon responsible for doing those merges having an issue [07:55:49] checking [07:56:08] hashar: merci! :) [08:08:17] dcausse: I have restarted Gerrit [08:08:24] ok :) [08:08:31] somehow sometime Zuul ssh session to Gerrit port 29418 get stuck [08:08:33] :-\ [08:20:39] hashar: green! thanks for having a look! :) [08:20:59] it is magic [08:21:54] it is! at least from my point of view! ;) [08:22:37] 10Continuous-Integration-Config, 10MW-1.35-release, 10MW-1.37-release, 10MW-1.38-release, and 2 others: Make PHP 8.1 voting on currently supported MW release branches - https://phabricator.wikimedia.org/T316080 (10Bawolff) [09:38:26] 10Continuous-Integration-Config: Consider replacing the heavy-weight quibble-composertest jobs with normal composer jobs, for speed - https://phabricator.wikimedia.org/T318093 (10hashar) The job only runs against `mediawiki/core`. It runs Quibble `composer-test` stage which when triggered for `mediawiki/core` ha... [09:39:52] 10Release-Engineering-Team (Blocking 🧱), 10MediaWiki-extensions-ExternalData, 10Quibble, 10Patch-For-Review, 10ci-test-error: Quibble fails with UnicodeDecodeError when test output has invalid unicode - https://phabricator.wikimedia.org/T318029 (10hashar) [09:48:34] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10SRE, and 2 others: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183 (10hashar) >>! In T191183#8192855, @kostajh wrote: > Coming back to this again... since the Gravatar issue (T263161) is unl... [10:06:16] 10Gerrit, 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Patch-For-Review: Automate Gerrit deployment steps - https://phabricator.wikimedia.org/T317412 (10hashar) [10:06:39] 10Release-Engineering-Team (Priority Backlog 📥), 10Gerrit (Gerrit 3.5), 10Patch-For-Review: Upgrade to Gerrit 3.5 - https://phabricator.wikimedia.org/T307334 (10hashar) I should probably have upgraded to 3.5 a few weeks ago but instead I went to finish the scap automatization which is T317412 As a note, Ger... [10:06:49] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10SRE, and 2 others: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183 (10kostajh) >>! In T191183#8249977, @hashar wrote: >>>! In T191183#8192855, @kostajh wrote: >> Coming back to this again...... [11:07:11] 10Phabricator, 10Trust-and-Safety: Reset Phabricator 2FA for Anoop - https://phabricator.wikimedia.org/T318172 (10Aklapper) 05Open→03Resolved a:03Aklapper Thanks everyone involved for unblocking this! <3 2FA has been stripped from @Anoop Phab account. [11:27:11] (03PS1) 10Samtar: layout.yaml: Add MPGuy2824 to trusted users [integration/config] - 10https://gerrit.wikimedia.org/r/833756 [11:33:50] (03Abandoned) 10Samtar: layout.yaml: Add MPGuy2824 to trusted users [integration/config] - 10https://gerrit.wikimedia.org/r/833756 (owner: 10Samtar) [11:34:57] (03CR) 10Samtar: [C: 03+1] "Great minds etc etc - I590c30f729a120daf84ac586dfb791e6bf2f5303" [integration/config] - 10https://gerrit.wikimedia.org/r/833703 (owner: 10Novem Linguae) [11:37:18] 10Release-Engineering-Team, 10Scap: Verify images before MW K8s deployments - https://phabricator.wikimedia.org/T318223 (10jnuche) [12:32:12] (03CR) 10Jaime Nuche: [C: 03+2] "Love removing stuff. Thanks for these changes!" [integration/config] - 10https://gerrit.wikimedia.org/r/833705 (owner: 10Hashar) [12:32:57] (03CR) 10Jaime Nuche: [C: 03+2] Drop Debian Stretch from integration testing [tools/scap] - 10https://gerrit.wikimedia.org/r/833704 (owner: 10Hashar) [12:34:56] (03Merged) 10jenkins-bot: Remove Stretch from mediawiki/tools/scap [integration/config] - 10https://gerrit.wikimedia.org/r/833705 (owner: 10Hashar) [12:35:56] jnuche: welcome back ;) [12:36:04] I am deploying the ci config change [12:36:36] (03CR) 10Hashar: "recheck" [tools/scap] - 10https://gerrit.wikimedia.org/r/833704 (owner: 10Hashar) [12:36:58] !log Reloaded Zuul for Remove Stretch from mediawiki/tools/scap - https://gerrit.wikimedia.org/r/833705 [12:36:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:37:51] hashar: thx! [12:38:20] jnuche: you would have to CR+2 again the scap change ;) [12:39:12] (03CR) 10Jaime Nuche: Drop Debian Stretch from integration testing [tools/scap] - 10https://gerrit.wikimedia.org/r/833704 (owner: 10Hashar) [12:39:15] (03CR) 10Jaime Nuche: [C: 03+2] Drop Debian Stretch from integration testing [tools/scap] - 10https://gerrit.wikimedia.org/r/833704 (owner: 10Hashar) [12:40:32] all of that cause I wrote a python test with `{}` instead of an `collections.OrderedDict()` [12:42:09] (03Merged) 10jenkins-bot: Drop Debian Stretch from integration testing [tools/scap] - 10https://gerrit.wikimedia.org/r/833704 (owner: 10Hashar) [12:42:55] (03CR) 10Hashar: "recheck after removal of Debian Stretch / python 3.5 ( https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/833704/ )" [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 (owner: 10Hashar) [13:04:04] jnuche: You need to drop the jenkins jobs too (`./jjb-delete`), as their deployment/removal isn't automated. I'll do it for that one though. :-) [13:05:00] !log Jenkins: Dropped scap-pipeline-stretch and trigger-scap-pipeline-stretch following 26c74a1 [13:05:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:05:32] Oh, hmm, it won't remove them. Odd. [13:06:28] Done manually from within Jenkins instead, then. [13:09:05] Also, hmm, current jjb output is 856 jobs, but production Jenkins has 962 jobs defined. [13:10:07] James_F: hashar mentioned he was doing the CI config change. Should have checked though, thanks for taking care of it [13:10:21] jnuche: It's a mess of a system. :-( [13:10:26] "Also, hmm, current jjb output is 856 jobs, but production Jenkins has 962 jobs defined." Hum, not great [13:11:26] yeah I often dont delete them [13:11:33] !log Jenkins: Dropped four old node10 jobs left on the server (oojs-core-node10-browser-docker, ooui-special-node10-plus-php80-composer-docker, wikipeg-special-node10-plus-php72-composer-docker, wikipeg-special-node10-plus-php80-composer-docker) [13:11:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:11:45] in most case the jobs are only triggered by zuul thus if they are no more in zuul they are no more run [13:12:24] go: cannot find GOROOT directory: go :-( [13:12:32] True, but it's a possible risk. [13:12:55] E.g. if we're relying on a job but can't regenerate it, if the contint server dies we're in a bad place. [13:18:45] !log Jenkins: Dropped 16 more old node jobs left on the server. [13:18:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:19:11] OK, now it's 941 vs. 856. Progress! [13:23:43] 10Phabricator, 10Trust-and-Safety: Reset Phabricator 2FA for Anoop - https://phabricator.wikimedia.org/T318172 (10Anoop) Thanks all for quick resolve [13:29:52] Project beta-update-databases-eqiad build #61653: 04FAILURE in 9 min 51 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/61653/ [13:35:22] beta issue being discussed over in -operations at the moment [13:53:34] 10Beta-Cluster-Infrastructure: prometheus-beta.wmcloud.org 504 timeout - https://phabricator.wikimedia.org/T315699 (10TheresNoTime) 05Stalled→03Resolved After speaking with taavi, this seems to be okay to do but should instead be reverse-proxied via apache — on having a quick look, this is already set up but... [13:54:09] (03PS1) 10Hashar: make test-docker to test Blubber with Blubber [blubber] - 10https://gerrit.wikimedia.org/r/833777 [13:54:11] (03PS1) 10Hashar: python: add no-deps flag for pip installation [blubber] - 10https://gerrit.wikimedia.org/r/833778 (https://phabricator.wikimedia.org/T316755) [13:55:25] !log modified deployment-prep "prometheus" security group - port 80, T315699 [13:55:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:55:27] T315699: prometheus-beta.wmcloud.org 504 timeout - https://phabricator.wikimedia.org/T315699 [13:56:50] (03CR) 10Hashar: "This is directly inspired from mediawiki/tools/scap which takes advantage of Blubber build kit to do all the magic." [blubber] - 10https://gerrit.wikimedia.org/r/833777 (owner: 10Hashar) [13:58:22] (03CR) 10Hashar: "See T316755 for the rationale. I confess I have mostly borrowed from UseSystemFlag. That surely should be tested against mediawiki/tools/" [blubber] - 10https://gerrit.wikimedia.org/r/833778 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [13:58:57] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 (10Lucas_Werkmeister_WMDE) db09 seems to be missing some grants; can be tested on the command line: `lang=shell-session,counterexample lucaswerkmeister-wmde@deployment-me... [13:59:44] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Patch-For-Review: Strictly lock scap requirements dependencies - https://phabricator.wikimedia.org/T316755 (10hashar) [14:05:04] Yippee, build fixed! [14:05:05] Project beta-update-databases-eqiad build #61654: 09FIXED in 9 min 41 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/61654/ [14:09:11] (03PS4) 10Jforrester: [WIP] dockerfiles: Provide PHP 8.2 CI test images [integration/config] - 10https://gerrit.wikimedia.org/r/818239 (https://phabricator.wikimedia.org/T314093) [14:10:25] hashar: Do you have a thought about what to do for PHP 8.2? It's now `8.2.0RC1` but still some critical PECL extensions haven't been packaged. :-( (ast, pcov, redis, xdebug, yaml). Is it worth having a semi-useless early image without those extensions for testing e.g. wikidiff2? [14:11:27] nop [14:11:37] we can't wait for sure [14:11:44] :-( [14:11:57] unless somehow supporting 8.2 is a top priority (which it is not for production) [14:12:12] but maybe for the next mediawiki release? [14:12:22] I understand from Perf that the next target after we land 7.4 is 8.1. [14:13:16] we are supposed to release 1.39 in November, I guess it would be nice to have it blessed with support for 8.2 [14:13:31] but worth case we can state in its release note that 8.2 is not guaranteed/untested [14:13:37] It will ship with 8.0/8.1 which none of the other releases have, so… [14:13:50] yeah it is already an improvement I guess [14:13:56] We'll hopefully get 1.38.x out with 8.0/8.1 at the same time. [14:14:15] I am guessing the various php extensions are being prepared for 8.2 as well [14:14:28] and I am guessing sury.org will eventually ship them [14:14:46] For php-yaml I guess I could bug bd.808 as it's his extension ;-) but I guess we'll just have to wait. [14:15:08] I found https://github.com/oerdnj/deb.sury.org/issues/1816 from a while ago that was just outright-rejected. [14:15:10] well if 1.38 does not really support php8, I guess that will be a good incentive for users to upgrade to the 1.39 LTS [14:15:57] something such as a statement that 1.38 is not endorsed on php 8 and people are encouraged to upgrade to 1.39 [14:16:14] We've previously said we hope to get 8.0+ working on 1.35. [14:16:15] So… [14:16:39] well [14:16:47] yeah that might make sense [14:16:53] (03CR) 10Jforrester: "(Still waiting for upstream packaging.)" [integration/config] - 10https://gerrit.wikimedia.org/r/818239 (https://phabricator.wikimedia.org/T314093) (owner: 10Jforrester) [14:16:57] or we could stop chasing the unicorns and instruct to move to 1.39 [14:17:04] True. [14:17:19] If I knew how Debian packaging worked I'd offer to do the work myself. [14:17:27] oh [14:17:35] But it's all confusing. [14:17:51] the whole debian packaging system is outdated by 20 years or so [14:18:05] coming from an area when people distributed software on CD rom (if not on floppy disk) [14:18:12] and software was released as a tarball [14:18:17] https://salsa.debian.org/php-team/pecl/php-yaml/-/tree/debian/main/debian is pretty shiny. [14:18:21] But yes. [14:18:38] and the toolchain has various antiquated tooling [14:18:45] that being said [14:19:03] Onrej (the person being sury.org) is maintaining the official Debian package [14:19:07] and nicely polished them up [14:19:09] Clearly I need to nerd-snipe l.egoktm into teaching me how to help out. [14:19:16] Yes, he's doing awesome work. [14:19:26] so it is almost straightforward to build them assuming you have a build environment locally (which is not any trivial to setup) [14:19:47] but our CI does have debian-glue job which relies on an environment setup from Puppet and which is the same SRE are using to build the packages [14:19:49] so theorically [14:20:09] fork the upstream php-pcov extension to our Gerrit, add the debian-glue job to the repo in our CI [14:20:14] Eww. [14:20:15] send a patch to whatever appropriate branhc [14:20:22] with a bump of debian/changelog [14:20:22] and [14:20:27] it should result in a .deb ;) [14:20:59] for php core, I think SRE do it directly on their build host [14:21:10] probably the same for extensions with some manual hacks [14:21:16] Yeah, I don't want to duplicate SRE's work. I was meaning more actually helping Onrej. [14:21:16] !log deployment-db09: restart mariadb # T318126 [14:21:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:21:20] T318126: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 [14:21:21] +d [14:21:34] Kunal once game the notes they are using to build the php extensions [14:21:57] Good morning [14:22:14] Oh tsk, you've summoned legoktm. There's no escaping now! ;-) [14:22:19] legoktm: Hey! [14:22:32] James_F: https://wikitech.wikimedia.org/wiki/PHP_packaging ;) [14:23:05] Yeah, but we don't currently have anyone in SRE looking to help us build WMF forks of 8.0, let alone 8.2 in pre-release state. :-) [14:23:09] Hiiii. That's the page! [14:23:10] But interesting. [14:23:14] anyway for 8.2 we gotta wait for sury, or maybe we can sponsors him to speed it up ;) [14:23:14] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 (10Zabe) I restart mariadb on deployment-db09 (after gracefully stopping replication), which fixed the issue. ` zabe@deployment-mwmaint02:~$ sql --wiki=enwiki --host=depl... [14:23:23] I am not sure whether he is doing the packaging as a hobby [14:23:31] I thought sury already had 8.2 up? [14:23:33] hashar: If only there was a team at the Foundation who could sponsor him. [14:23:44] legoktm: It's missing half the PECL extensions we need. :-( [14:24:01] legoktm: And I'd rather help out than just whine upstream, but I have no idea how I'd do that. [14:24:11] my point james [14:24:47] it happens on the good will of a few, and that surely doesn't scale [14:24:50] anyway [14:25:08] I suspect they're missing because they're broken [14:25:09] we could surely maintain forks of the repo we care about on our system and have CI to build the package which would surely ease a few things [14:25:16] He has https://www.patreon.com/oerdnj [14:25:19] or hold and wait for sury.org to do them [14:25:27] Yeah. [14:25:38] oh [14:25:40] The 8.2 transition hasn't started in Debian yet, which is usually when stuff gets fixed [14:25:53] What's the "transition"? [14:25:58] speaking of debian packaging - is the plan still to do 1.39-rc.0 this week? [14:26:30] James_F: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1014460 [14:26:32] taavi: I guess? Reedy self-assigned T313923 and set a deadline of this Friday. [14:26:32] T313923: Release 1.39.0-rc.0 - https://phabricator.wikimedia.org/T313923 [14:26:35] well theoretically I guess we could get the foundation to sponsors him via patreon or some over mean. I don't know whether we have a precedent [14:27:06] Basically make 8.2 the default, which means everything else must be compatible or dropped [14:27:06] hashar: Phacility was a direct contract, right? [14:27:19] legoktm: Ah, interesting! So is 8.1 currently the default? [14:27:39] well maybe we can hire him as a SRE ;) [14:27:41] problem solved! [14:27:59] I see https://release.debian.org/transitions/html/php8.2.html lists the packages I care about in red (and a bunch more). [14:28:07] Yes [14:28:12] It also lists wikidiff2, so I guess someone in Platform(?) will have to fix that. [14:28:31] Oh no, our packages are red for a different reason [14:28:51] Also I'm behind, I fixed LuaSandbox but didn't do a new release [14:29:00] * James_F hugs legoktm. [14:29:09] Well, maybe. Even is they were fixed they'd still be red [14:29:09] legoktm: You're awesome etc. [14:30:14] Can I see why a particular package is red? The build logs don't go to failures. [14:31:03] :), OK, but how to help, just try building the missing extensions on 8.2, (just plain make, no complicated packaging stuff), then check if someone has filed a bug upstream, if not, do so. Or ask for a new release if master is fixed, etc. [14:31:13] Ack. [14:41:20] Some issues might be straightforward to fix too? E.g. the LuaSandbox update was all PHP test cases, not C IIRC. [14:42:01] I think e.g. for php-ast it's just needing to bump to the new release of 1.1.0. [14:42:07] https://github.com/nikic/php-ast/releases/tag/v1.1.0 added PHP 8.2 support. [14:42:16] But https://salsa.debian.org/php-team/pecl/php-ast/-/branches doesn't have that yet. [14:42:20] Etc. [14:43:09] I am not sure that transition page is up to date [14:43:46] or I guess packages are marked failing cause they fail to build on some architectures (notably arc, kbsd32 and kbsd64)) [14:44:02] when [14:44:02] https://buildd.debian.org/status/package.php?p=libguestfs,php-amqp,php-apcu,php-ast,php-ds,php-excimer,php-gearman,php-geos,php-gmagick,php-gnupg,php-imagick,php-luasandbox,php-mailparse,php-memcache,php-mongodb,php-msgpack,php-oauth,php-pcov,php-ps,php-psr,php-raphf,php-rrd,php-smbclient,php-solr,php-ssh2,php-uopz,php-uploadprogress,php-uuid,php-xmlrpc,php-yac,php-yaml,php-zmq,remctl,uwsgi-plugin-php,wikidiff2,xdebug,zeroc-ice&compact=c [14:44:02] ompact [14:44:05] shows a ton of green [14:44:45] ex for wikidiff2 https://buildd.debian.org/status/package.php?p=wikidiff2 [14:49:57] Yeah, but I assumed those are for 8.1 or whatever? [14:51:03] Given that the amd64 link takes you to https://buildd.debian.org/status/logs.php?pkg=wikidiff2&ver=1.13.0-1%2Bb2&arch=amd64 which was built in January (way before 8.2 was even a beta). [14:54:21] ohhh [14:54:39] I got the links from https://release.debian.org/transitions/html/php8.2.html [14:58:43] (03CR) 10Ahmon Dancy: [C: 03+2] "Very useful" [blubber] - 10https://gerrit.wikimedia.org/r/833777 (owner: 10Hashar) [15:03:39] (03CR) 10Ahmon Dancy: [C: 03+1] python: add no-deps flag for pip installation (032 comments) [blubber] - 10https://gerrit.wikimedia.org/r/833778 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [15:03:59] (03Merged) 10jenkins-bot: make test-docker to test Blubber with Blubber [blubber] - 10https://gerrit.wikimedia.org/r/833777 (owner: 10Hashar) [15:18:33] (03CR) 10Ahmon Dancy: Support checks running before a stage (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 (owner: 10Hashar) [15:36:51] 10Beta-Cluster-Infrastructure: Set up an alerting server - https://phabricator.wikimedia.org/T318247 (10TheresNoTime) [15:38:17] (03CR) 10Ahmon Dancy: [V: 04-1 C: 03+1] "The call to self.get_serial_lock_file in ./scap/plugins/prep.py needs to be updated." [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [15:45:42] (03CR) 10Ahmon Dancy: [V: 04-1 C: 03+1] Replace Lock with TimeoutLock (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [15:59:55] (03CR) 10Dduvall: [C: 03+1] "Looks great, Antoine. Thanks for the blubber contribution! I think this can merge after the two typos that Ahmon pointed out are fixed." [blubber] - 10https://gerrit.wikimedia.org/r/833778 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [16:00:22] (03CR) 10Hashar: Support checks running before a stage (0310 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 (owner: 10Hashar) [16:00:28] (03PS5) 10Hashar: Support checks running before a stage [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 [16:01:06] (03CR) 10Dduvall: "<3" [blubber] - 10https://gerrit.wikimedia.org/r/833777 (owner: 10Hashar) [16:07:46] 10Continuous-Integration-Config, 10Release-Engineering-Team (Radar), 10MediaWiki-Vendor, 10Parsoid (Tracking), and 2 others: `mediawiki-core-php72-phan-docker` job runs `composer install` instead of using packages from mediawiki/vendor - https://phabricator.wikimedia.org/T287419 (10cscott) Dumping my own... [16:08:24] bah blubber postmerge fails https://integration.wikimedia.org/ci/blue/organizations/jenkins/blubber-pipeline-publish/detail/blubber-pipeline-publish/66/ :( [16:08:41] it tries to push back to gerrit without a change-id [16:12:04] (03PS18) 10Jeena Huneidi: Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) [16:12:50] dcausse: next Gerrit will let me retrieve the list of registered events instead of relying on reflection ( https://gerrit-review.googlesource.com/c/gerrit/+/345017 ! ) [16:13:16] hashar: \o/ [16:13:56] (03CR) 10Hashar: [C: 04-2] "https://gerrit-review.googlesource.com/c/gerrit/+/345017 got merged upstream in stable-3.4 branch but is obviously not released yet ;-]" [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/830654 (https://phabricator.wikimedia.org/T304947) (owner: 10Hashar) [16:14:33] dcausse: merci! [16:58:48] (03CR) 10Hashar: [C: 03+2] "Thanks! :)" [integration/config] - 10https://gerrit.wikimedia.org/r/833703 (owner: 10Novem Linguae) [17:00:43] (03Merged) 10jenkins-bot: zuul: Add MPGuy2824 to the CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/833703 (owner: 10Novem Linguae) [17:03:32] https://www.jenkins.io/security/advisory/2022-09-21/ got published [17:03:35] and we are not affected [17:03:44] (private task https://phabricator.wikimedia.org/T318218 ) [17:04:26] (03CR) 10Hashar: [C: 03+2] "Change deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/833703 (owner: 10Novem Linguae) [17:12:36] why do we use "gerrit2" to run Gerrit? All occurrences have been removed from the upstream doc notably by https://gerrit-review.googlesource.com/c/gerrit/+/112274 [17:12:52] and Luca has an interesting theory: Always hated the 'gerrit2' user, was it because in the early installation there was a 'gerrit' username already taken by Mr. Gerrit Rietvel ? [17:14:42] on that historical lore, I am off for the evening [17:42:44] (03CR) 10Dduvall: [C: 04-1] "Nice refactor and nice tests! I have some follow-up feedback and suggestions." [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 (owner: 10Hashar) [17:53:34] (03PS19) 10Ahmon Dancy: Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [17:54:50] !log create deployment-db10 as g3.cores8.ram16.disk20 # T318126 [17:54:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:54:52] T318126: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 [17:55:14] !log create volume db10 and attach to deployment-db10 # T318126 [17:55:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:56:39] (03CR) 10CI reject: [V: 04-1] Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [18:01:07] (03PS20) 10Ahmon Dancy: Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [18:02:26] (03CR) 10Ahmon Dancy: [V: 03+1 C: 03+1] "Tested patchset 20 in train-dev." [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [18:07:58] (03CR) 10Ahmon Dancy: Support checks running before a stage (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 (owner: 10Hashar) [18:18:44] (03CR) 10Jeena Huneidi: [C: 03+2] "LGTM" [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [18:19:52] !log install mariadb 10.6 via role::mariadb::beta on deployment-db10 # T318126 [18:19:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:19:54] T318126: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 [18:20:28] hashar: https://phabricator.wikimedia.org/P34893 might be interesting. [18:22:43] (03Merged) 10jenkins-bot: Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [18:30:22] (03PS1) 10Dduvall: Ensure git hooks are enabled for deployment-charts when committing [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833840 (https://phabricator.wikimedia.org/T318246) [18:30:44] (03PS2) 10Dduvall: Ensure git hooks are enabled for deployment-charts during promote [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833840 (https://phabricator.wikimedia.org/T318246) [18:31:58] (03CR) 10CI reject: [V: 04-1] Ensure git hooks are enabled for deployment-charts during promote [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833840 (https://phabricator.wikimedia.org/T318246) (owner: 10Dduvall) [18:32:00] (03PS3) 10Dduvall: Ensure git hooks are enabled for deployment-charts during promote [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833840 (https://phabricator.wikimedia.org/T318246) [18:33:19] (03CR) 10CI reject: [V: 04-1] Ensure git hooks are enabled for deployment-charts during promote [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833840 (https://phabricator.wikimedia.org/T318246) (owner: 10Dduvall) [18:35:00] (03PS4) 10Dduvall: Ensure git hooks are enabled for deployment-charts during promote [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833840 (https://phabricator.wikimedia.org/T318246) [18:36:21] (03CR) 10CI reject: [V: 04-1] Ensure git hooks are enabled for deployment-charts during promote [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833840 (https://phabricator.wikimedia.org/T318246) (owner: 10Dduvall) [18:36:44] (03CR) 10Dduvall: [C: 04-1] Support checks running before a stage (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 (owner: 10Hashar) [18:37:38] (03PS5) 10Dduvall: Ensure git hooks are enabled for deployment-charts during promote [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833840 (https://phabricator.wikimedia.org/T318246) [18:39:31] (03CR) 10Ahmon Dancy: [C: 03+2] "Looks legit." [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833840 (https://phabricator.wikimedia.org/T318246) (owner: 10Dduvall) [18:40:54] (03Merged) 10jenkins-bot: Ensure git hooks are enabled for deployment-charts during promote [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833840 (https://phabricator.wikimedia.org/T318246) (owner: 10Dduvall) [18:43:07] dancy: nice turnaround! :) [18:43:30] i'm going to re-run that failed blubber-pipeline-publish job [18:43:42] nice debugging and fix! [18:44:05] ty :) [18:49:28] (03PS4) 10Ahmon Dancy: cli: invoke all setup calls from a single method [tools/scap] - 10https://gerrit.wikimedia.org/r/805909 (owner: 10Hashar) [18:49:51] !log cherry-picked [[gerrit:833839]] to deployment-puppetmaster04, testing T317417 [18:49:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:49:54] T317417: Phonos links to unroutable domain/URL for the MP3 file - https://phabricator.wikimedia.org/T317417 [18:51:19] (03CR) 10Ahmon Dancy: "Hi Antoine. I didn't notice this change before (no reviewers added!)." [tools/scap] - 10https://gerrit.wikimedia.org/r/805909 (owner: 10Hashar) [18:53:17] (03PS2) 10Ahmon Dancy: Remove legacy sync.flag check [tools/scap] - 10https://gerrit.wikimedia.org/r/833464 [18:54:05] (03PS3) 10Ahmon Dancy: Remove legacy sync.flag check [tools/scap] - 10https://gerrit.wikimedia.org/r/833464 [19:08:40] (03CR) 10Jeena Huneidi: [C: 03+1] "LGTM" [tools/scap] - 10https://gerrit.wikimedia.org/r/833464 (owner: 10Ahmon Dancy) [19:18:32] (03CR) 10Hashar: "Grmblb I must have broken it when upgrading the git plugin. Sorry about that and kudos for the fix!" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833840 (https://phabricator.wikimedia.org/T318246) (owner: 10Dduvall) [19:36:19] 10Beta-Cluster-Infrastructure, 10SRE-swift-storage: deployment-ms-be05 Swift object-replicator sync error: Connection refused - https://phabricator.wikimedia.org/T318268 (10TheresNoTime) [19:41:39] 10Beta-Cluster-Infrastructure, 10SRE-swift-storage: deployment-ms-be05 Swift object-replicator sync error: Connection refused - https://phabricator.wikimedia.org/T318268 (10TheresNoTime) I added ` Ingress IPv4 TCP 6000 172.16.0.0/21 ` to the `swift-be` [[ https://horizon.wikimedia.org/project/security_grou... [19:43:43] (03CR) 10Jeena Huneidi: [C: 03+1] Make "scap backport" skip sync for labs-only changes [tools/scap] - 10https://gerrit.wikimedia.org/r/833456 (https://phabricator.wikimedia.org/T317242) (owner: 10Ahmon Dancy) [19:44:17] 10Beta-Cluster-Infrastructure, 10SRE-swift-storage: deployment-ms-be05 Swift object-replicator sync error: Connection refused - https://phabricator.wikimedia.org/T318268 (10TheresNoTime) 05Open→03Resolved a:03TheresNoTime Crossing fingers and assuming that's all it was (**//why??//**) [19:53:49] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Phonos, 10Community-Tech (CommTech-Sprint-33), 10MW-1.40-notes (1.40.0-wmf.2; 2022-09-19), 10Patch-For-Review: Phonos links to unroutable domain/URL for the MP3 file - https://phabricator.wikimedia.org/T317417 (10TheresNoTime) ^ that above is a cherry... [20:01:05] (03CR) 10Hashar: "I have to formally test it against scap code base and verify it actually fail until the requirements.txt has the exhaustive list of depend" [blubber] - 10https://gerrit.wikimedia.org/r/833778 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [20:01:16] (03CR) 10Hashar: [C: 04-1] python: add no-deps flag for pip installation [blubber] - 10https://gerrit.wikimedia.org/r/833778 (https://phabricator.wikimedia.org/T316755) (owner: 10Hashar) [20:01:46] bunch of reviews thank you dduvall & dancy! I will continue tomorrow [20:43:45] 10Release-Engineering-Team (Priority Backlog 📥), 10Release Pipeline (Blubber): Blubber: runuser permissions to node_modules/.cache - https://phabricator.wikimedia.org/T255434 (10dduvall) 05Open→03Declined >>! In T255434#8158217, @wiese wrote: > I (author of the ticket) do not have stake in this and, I gues... [20:46:02] !log clone deployment-db10 from deployment-db08 # T318126 [20:46:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:46:04] T318126: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 [21:22:33] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Quota-requests): Request increased quota for deployment-prep Cloud VPS project - https://phabricator.wikimedia.org/T318287 (10Zabe) [21:44:38] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Release Pipeline (Blubber): Sunset Blubber's CLI and microservice (blubberoid) interfaces - https://phabricator.wikimedia.org/T318289 (10dduvall) [21:44:58] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Release Pipeline (Blubber): Sunset Blubber's CLI and microservice (blubberoid) interfaces - https://phabricator.wikimedia.org/T318289 (10dduvall) p:05Triage→03Medium [21:45:17] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Release Pipeline (Blubber): Sunset Blubber's CLI and microservice (blubberoid) interfaces - https://phabricator.wikimedia.org/T318289 (10dduvall) 05Open→03In progress a:03dduvall [21:47:33] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Release Pipeline (Blubber): Refactor PipelineLib to use blubber's buildkit frontend - https://phabricator.wikimedia.org/T318290 (10dduvall) [21:47:53] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Release Pipeline (Blubber): Sunset Blubber's CLI and microservice (blubberoid) interfaces - https://phabricator.wikimedia.org/T318289 (10dduvall) [21:47:55] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Release Pipeline (Blubber): Refactor PipelineLib to use blubber's buildkit frontend - https://phabricator.wikimedia.org/T318290 (10dduvall) 05Open→03In progress p:05Triage→03Medium [21:55:13] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Release Pipeline (Blubber): Sunset Blubber's CLI and microservice (blubberoid) interfaces - https://phabricator.wikimedia.org/T318289 (10dduvall) [22:53:23] Is there an issue with Zuul/jenkins? [22:53:57] Quite a few patches are not getting feedback from Jenkins/not merging [22:58:32] I'll try restarting zuul [23:00:31] !log restarting zuul to try and fix CI issues [23:00:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:02:50] thanks jeena [23:04:45] np [23:05:28] How long does the reboot usually take? https://integration.wikimedia.org/zuul/ is still awfully quiet. [23:10:56] I saw a test running a couple of minutes ago after reboot [23:15:43] (03PS1) 10Dduvall: Use Blubber BuildKit frontend instead of blubberoid [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/833863 (https://phabricator.wikimedia.org/T318290) [23:18:14] jeena: Looks like my patches are showing up now! yay! thank you! [23:19:33] Oh good! [23:34:49] !log shutoff deployment-db08 # T318126 [23:34:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:34:51] T318126: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126