[04:08:37] 10Release-Engineering-Team (Yak Shaving 🐃🪒): Migrate from pws gpg to 1password - https://phabricator.wikimedia.org/T290337 (10hashar) [04:08:42] 10Phabricator, 10Release-Engineering-Team, 10serviceops: move "releng-secrets" git repo away from Phabricator - https://phabricator.wikimedia.org/T301170 (10hashar) [04:14:09] 10Phabricator, 10Release-Engineering-Team, 10serviceops: move "releng-secrets" git repo away from Phabricator - https://phabricator.wikimedia.org/T301170 (10hashar) We have settled on migrating out of pws/gpg/git to store our credentials in favor of 1password.com . The migration itself is not that complicate... [04:14:31] 10Release-Engineering-Team (Yak Shaving 🐃🪒): Migrate from pws gpg to 1password - https://phabricator.wikimedia.org/T290337 (10hashar) [04:14:35] 10Phabricator, 10Release-Engineering-Team (Next), 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10hashar) [05:47:59] (03CR) 10Hashar: [C: 03+2] Add new test dependencies to `BlueSpiceProDistributionConnector` [integration/config] - 10https://gerrit.wikimedia.org/r/761531 (owner: 10Robert Vogel) [05:50:21] (03Merged) 10jenkins-bot: Add new test dependencies to `BlueSpiceProDistributionConnector` [integration/config] - 10https://gerrit.wikimedia.org/r/761531 (owner: 10Robert Vogel) [05:50:32] (03CR) 10Hashar: [C: 03+2] "analytics-gobblin-wmf-maven-release-docker updated" [integration/config] - 10https://gerrit.wikimedia.org/r/761386 (https://phabricator.wikimedia.org/T297938) (owner: 10Joal) [05:51:07] (03CR) 10Hashar: [C: 03+2] Remove outdated beta feature dependency for FileExporter [integration/config] - 10https://gerrit.wikimedia.org/r/756949 (https://phabricator.wikimedia.org/T259690) (owner: 10Awight) [05:52:33] (03Merged) 10jenkins-bot: Fix analytics gobblin-wmf maven release job [integration/config] - 10https://gerrit.wikimedia.org/r/761386 (https://phabricator.wikimedia.org/T297938) (owner: 10Joal) [05:52:53] (03Merged) 10jenkins-bot: Remove outdated beta feature dependency for FileExporter [integration/config] - 10https://gerrit.wikimedia.org/r/756949 (https://phabricator.wikimedia.org/T259690) (owner: 10Awight) [06:00:42] (03PS3) 10Hashar: Enforce editorconfig settings [integration/config] - 10https://gerrit.wikimedia.org/r/756047 [06:01:31] (03CR) 10Hashar: [C: 03+1] "I have rebased the change and fixed an indent issue in one of the groovy file. I slightly amended the commit message while at it." [integration/config] - 10https://gerrit.wikimedia.org/r/756047 (owner: 10Hashar) [06:01:43] (03PS2) 10Hashar: utils/shellchecker: install shellcheck from pypi [integration/config] - 10https://gerrit.wikimedia.org/r/756087 [06:01:55] (03PS3) 10Hashar: Run shellcheck against shell files [integration/config] - 10https://gerrit.wikimedia.org/r/756088 [06:03:40] (03CR) 10jerkins-bot: [V: 04-1] Run shellcheck against shell files [integration/config] - 10https://gerrit.wikimedia.org/r/756088 (owner: 10Hashar) [06:08:27] 10Phabricator, 10DBA, 10Patch-For-Review, 10User-notice: Switchover m3 master (db1107 -> db1183) - https://phabricator.wikimedia.org/T301219 (10Marostegui) [08:12:54] (03PS1) 10Majavah: Add myself to the morning window [tools/release] - 10https://gerrit.wikimedia.org/r/762395 [08:23:31] PROBLEM - Check systemd state on doc1001 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [09:20:18] RECOVERY - Check systemd state on doc1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [13:11:07] (03CR) 10Hashar: "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/756949 (https://phabricator.wikimedia.org/T259690) (owner: 10Awight) [13:54:31] !log Jenkins contint instances are going to be restarted soon [13:54:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:26:11] !log Jenkins upgrade complete T301361 [14:26:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:49:56] ^ congratulations [14:54:42] 10Continuous-Integration-Infrastructure, 10Performance-Team, 10Patch-For-Review: Provide one or more Qemu agents in CI that use a newer version than 2.x - https://phabricator.wikimedia.org/T284774 (10hashar) I am back from vacations. The wall I have hit was that the `docker pull` was extremely slow either du... [15:41:37] !log Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration [15:41:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:46:03] poor qemu is way too slow at downloading a docker image :[ https://integration.wikimedia.org/ci/job/fresh-test/227/console [15:47:08] (03PS3) 10Hashar: jjb: adjust qemu-run.bash to use a qcow2 image [integration/config] - 10https://gerrit.wikimedia.org/r/759499 (https://phabricator.wikimedia.org/T284774) [15:47:40] hmm no it is cpu bound [15:50:50] 10Phabricator, 10Release-Engineering-Team, 10serviceops: move "releng-secrets" git repo away from Phabricator - https://phabricator.wikimedia.org/T301170 (10Dzahn) 05duplicate→03Open [15:51:51] 10Phabricator, 10Release-Engineering-Team, 10serviceops: move "releng-secrets" git repo away from Phabricator - https://phabricator.wikimedia.org/T301170 (10Dzahn) This ticket wasn't about migrating pws to another solution. It was about moving the repo out of phabricator or, alternatively, to stop using ssh... [16:09:47] (03PS4) 10Hashar: jjb: adjust qemu-run.bash to use a qcow2 image [integration/config] - 10https://gerrit.wikimedia.org/r/759499 (https://phabricator.wikimedia.org/T284774) [16:11:38] (03PS1) 10Ladsgroup: zuul: Add operations/software/schema-changes [integration/config] - 10https://gerrit.wikimedia.org/r/762471 [16:12:40] (03CR) 10Ladsgroup: [C: 03+2] zuul: Add operations/software/schema-changes [integration/config] - 10https://gerrit.wikimedia.org/r/762471 (owner: 10Ladsgroup) [16:14:28] (03Merged) 10jenkins-bot: zuul: Add operations/software/schema-changes [integration/config] - 10https://gerrit.wikimedia.org/r/762471 (owner: 10Ladsgroup) [16:15:14] (03CR) 10Ahmon Dancy: "Commit message typo, otherwise looks ok." [integration/config] - 10https://gerrit.wikimedia.org/r/756047 (owner: 10Hashar) [16:16:30] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/762471 [16:16:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:26:32] (03CR) 10Ahmon Dancy: help output: align command descriptions (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/761980 (https://phabricator.wikimedia.org/T243659) (owner: 10Jaime Nuche) [16:28:40] !log Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c [16:28:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:37:54] finding, I need more cpu for qemu [16:38:15] and pass `-snapshot` which prevent qemu from writing to the image and also turns disk cache to be unsafe (aka never write back) [16:40:32] maintenance-disconnect-full-disks build 360355 integration-agent-qemu-1003 (/: 96%, /srv: 59%, /var/lib/docker: 1%): OFFLINE due to disk space [16:41:14] hashar: regarding more cpu): Performance issues? [16:41:37] dancy: yeah I am trying to tune the perfs of a `docker pull` running inside a qemu image [16:41:42] https://integration.wikimedia.org/ci/job/fresh-test/229/console [16:41:52] it is slow, apparently due to docker-untar [16:42:00] nod.. which is single threaded. [16:42:04] it seems to be CPU bound, allocating 7 cpu helps [16:42:05] yeah [16:42:06] :-\ [16:42:26] Another issue is that you're running qemu on a host that itself is a VM [16:43:07] (so qemu can't employ the speedsup that kvm usually supplies) [16:43:30] and I imagine it is fully transcribing all cpu instructions as a result? :\ [16:44:08] I suspect the base qemu image we used already had a docker pull issued [16:44:10] (03PS5) 10Hashar: jjb: adjust qemu-run.bash to use a qcow2 image [integration/config] - 10https://gerrit.wikimedia.org/r/759499 (https://phabricator.wikimedia.org/T284774) [16:44:21] to prevent having to retrieve and uncompress thedocker image on each build [16:44:36] anyway [16:44:55] I will send some dumb refactoring patches first and continue tomorrow [16:45:41] Good luck! [17:12:38] (03PS6) 10Hashar: jjb: adjust qemu-run.bash to use a qcow2 image [integration/config] - 10https://gerrit.wikimedia.org/r/759499 (https://phabricator.wikimedia.org/T284774) [17:12:40] (03PS1) 10Hashar: qemu-run: use one line per qemu-system-x86_64 option [integration/config] - 10https://gerrit.wikimedia.org/r/762482 (https://phabricator.wikimedia.org/T284774) [17:12:42] (03PS1) 10Hashar: qemu-run: avoid copying image and faster disk IO [integration/config] - 10https://gerrit.wikimedia.org/r/762483 (https://phabricator.wikimedia.org/T284774) [17:12:44] (03PS1) 10Hashar: qemu-run: allocate more CPU to the VM [integration/config] - 10https://gerrit.wikimedia.org/r/762484 (https://phabricator.wikimedia.org/T284774) [17:15:15] (03CR) 10Hashar: "I have split the improvements I have found and made a series of tiny patch. Notably:" [integration/config] - 10https://gerrit.wikimedia.org/r/759499 (https://phabricator.wikimedia.org/T284774) (owner: 10Hashar) [17:16:45] 10Continuous-Integration-Infrastructure, 10Performance-Team, 10Patch-For-Review: Provide one or more Qemu agents in CI that use a newer version than 2.x - https://phabricator.wikimedia.org/T284774 (10hashar) The magic is to use `-snapshot` which disable writing back to the image and also set caching to unsaf... [17:59:30] dancy: regarding kvm, in an earlier iteration we experimented with some kvm options from the WMCS side but iirc those were considered too risky or incompatible with some of the operational constraints WMCS very reasonably prefers to uphold. [18:00:25] Nod. The nested VM experiment worked, but it didn't fit into the normal WMCS configs well (or something like that). [18:00:26] I can't find the exact task but https://phabricator.wikimedia.org/T276208 mentions some of it [18:00:44] yeah, live migration.. that was the issue [18:00:46] ah I see you're on the parent-parent already [18:00:50] col [18:00:52] cool* [18:01:29] Andrew does offer to prepare one-off configurations. [18:09:06] dancy: hashar: are we staying on the new qemu or should I assume we might switch back again? (i.e. is it stable enough to iterate further on the new host?) [18:09:17] if so, then I can start testing node12,14 with fresh and remove node10 [18:10:19] (03CR) 10Krinkle: [C: 03+1] qemu-run: use one line per qemu-system-x86_64 option [integration/config] - 10https://gerrit.wikimedia.org/r/762482 (https://phabricator.wikimedia.org/T284774) (owner: 10Hashar) [18:10:51] (03CR) 10Krinkle: [C: 03+1] qemu-run: avoid copying image and faster disk IO [integration/config] - 10https://gerrit.wikimedia.org/r/762483 (https://phabricator.wikimedia.org/T284774) (owner: 10Hashar) [18:11:29] ah, I misread, we already switched back again. Alright, I'll hold off a bit more then :) [18:21:53] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T300197 (10jeena) 05Open→03Resolved [18:32:44] (03PS1) 10Ahmon Dancy: Don't update wmf-config/ExtensionMessages* if unchanged [tools/scap] - 10https://gerrit.wikimedia.org/r/762506 [18:34:17] (03PS2) 10Ahmon Dancy: Don't update wmf-config/ExtensionMessages* if unchanged [tools/scap] - 10https://gerrit.wikimedia.org/r/762506 [18:36:00] (03PS3) 10Ahmon Dancy: Don't update wmf-config/ExtensionMessages* if unchanged [tools/scap] - 10https://gerrit.wikimedia.org/r/762506 [18:37:24] (03PS3) 10Ahmon Dancy: scap prep auto: Add staging fingerprint support [tools/scap] - 10https://gerrit.wikimedia.org/r/761451 (https://phabricator.wikimedia.org/T301417) [19:03:51] 10Release-Engineering-Team, 10Scap: Delete scap sync command - https://phabricator.wikimedia.org/T301716 (10dancy) [19:04:20] 10Release-Engineering-Team (Done by Feb 23🔥), 10Scap: Delete scap sync command - https://phabricator.wikimedia.org/T301716 (10dancy) [19:04:55] 10Scap, 10Documentation: scap help needs updating - https://phabricator.wikimedia.org/T301343 (10dancy) {T301716} [19:05:15] 10Scap, 10Documentation: scap help needs updating - https://phabricator.wikimedia.org/T301343 (10dancy) [19:08:24] 10Release-Engineering-Team, 10Scap: Refactor scap sync-canary - https://phabricator.wikimedia.org/T301717 (10dancy) [19:09:03] 10Release-Engineering-Team, 10Scap, 10Documentation: scap help needs updating - https://phabricator.wikimedia.org/T301343 (10dancy) [19:17:43] 10Phabricator, 10Release-Engineering-Team, 10serviceops: move "releng-secrets" git repo away from Phabricator - https://phabricator.wikimedia.org/T301170 (10Aklapper) [19:17:49] 10Release-Engineering-Team (Yak Shaving 🐃🪒): Migrate from pws gpg to 1password - https://phabricator.wikimedia.org/T290337 (10Aklapper) [19:33:27] 10Beta-Cluster-Infrastructure, 10Performance-Team: Upgrade deployment-webperf hosts to Debian Buster or Bullseye - https://phabricator.wikimedia.org/T301638 (10dpifke) a:03dpifke [19:34:00] 10Beta-Cluster-Infrastructure, 10Performance-Team: Upgrade deployment-mdb01 to Buster/Bullseye - https://phabricator.wikimedia.org/T301637 (10dpifke) a:03dpifke [20:25:12] (03CR) 10Thcipriani: [C: 03+2] "Yay! Thanks for volunteering <3" [tools/release] - 10https://gerrit.wikimedia.org/r/762395 (owner: 10Majavah) [20:25:52] (03Merged) 10jenkins-bot: Add myself to the morning window [tools/release] - 10https://gerrit.wikimedia.org/r/762395 (owner: 10Majavah) [21:42:23] Could someone with access to T291439 unsubscribe me from that task? It is in some non-public space, not sure which one. [21:45:02] trying... [21:45:31] I can't find that one. [21:52:09] its in the fundraising space [21:52:42] best to ask one of the fundraising team [21:53:10] although if you don't have access it shouldn't cause notification issues, it should just mark them as rubbish [22:00:02] (03PS1) 10Ahmon Dancy: Add utils.suppress_backtrace context manager [tools/scap] - 10https://gerrit.wikimedia.org/r/762544 [22:00:30] It has some weird side affects. It makes other notifications disappear aswell, which is quite annoying. I had this twice in the past and the first time it turned out to be caused by sba_ssett subscribing me to T285414. The second time it was like this time when I was subscribed to a task I still couldn't access. Getting myself unsubscribed helped back then. ¯\_(ツ)_/¯ [22:00:31] T285414: Write and send supplementary release announcement for extensions and skins with security patches (1.31.16/1.35.4/1.36.2) - https://phabricator.wikimedia.org/T285414 [22:00:52] But thanks I am going to ask some fundraising team folk [22:01:12] (03CR) 10Ahmon Dancy: [C: 03+2] Add utils.suppress_backtrace context manager [tools/scap] - 10https://gerrit.wikimedia.org/r/762544 (owner: 10Ahmon Dancy) [22:01:55] (03Merged) 10jenkins-bot: Add utils.suppress_backtrace context manager [tools/scap] - 10https://gerrit.wikimedia.org/r/762544 (owner: 10Ahmon Dancy) [22:02:59] if its causing other notifications to be effected, sounds like a bug that needs to be reported. [22:03:11] (03PS1) 10Ahmon Dancy: Optionally build mw container image during scap sync-* [tools/scap] - 10https://gerrit.wikimedia.org/r/762546 (https://phabricator.wikimedia.org/T297673) [22:04:29] (03PS2) 10Ahmon Dancy: Optionally build mw container image during scap sync-* [tools/scap] - 10https://gerrit.wikimedia.org/r/762546 (https://phabricator.wikimedia.org/T297673) [22:05:14] (03Abandoned) 10Ahmon Dancy: Optionally build mw container image during scap sync-* [tools/scap] - 10https://gerrit.wikimedia.org/r/762007 (https://phabricator.wikimedia.org/T297673) (owner: 10Ahmon Dancy) [22:05:46] (03PS3) 10Ahmon Dancy: Optionally build mw container image during scap sync-* [tools/scap] - 10https://gerrit.wikimedia.org/r/762546 (https://phabricator.wikimedia.org/T297673) [22:09:45] 10Release-Engineering-Team (Done by Feb 23🔥), 10MediaWiki Train Development Environment: Train-dev: Update to helm 3 - https://phabricator.wikimedia.org/T301266 (10jeena) a:03jeena [22:09:56] 10Release-Engineering-Team (Done by Feb 23🔥), 10MediaWiki Train Development Environment: Train-dev: Update to helm 3 - https://phabricator.wikimedia.org/T301266 (10jeena) 05Open→03In progress [22:15:18] (03PS1) 10Ahmon Dancy: Use utils.suppress_backtrace() everywhere [tools/scap] - 10https://gerrit.wikimedia.org/r/762548 [22:16:26] (03CR) 10Ahmon Dancy: [C: 03+2] Use utils.suppress_backtrace() everywhere [tools/scap] - 10https://gerrit.wikimedia.org/r/762548 (owner: 10Ahmon Dancy) [22:17:15] (03Merged) 10jenkins-bot: Use utils.suppress_backtrace() everywhere [tools/scap] - 10https://gerrit.wikimedia.org/r/762548 (owner: 10Ahmon Dancy) [22:19:21] (03PS1) 10BryanDavis: python: ban setuptools==60.9.0 from installing [blubber] - 10https://gerrit.wikimedia.org/r/762552 (https://phabricator.wikimedia.org/T301690) [22:21:21] (03PS2) 10Ahmon Dancy: scap prep: Add --copy-private-settings flag [tools/scap] - 10https://gerrit.wikimedia.org/r/761735 [22:22:44] (03CR) 10Ahmon Dancy: [C: 03+2] scap prep: Add --copy-private-settings flag [tools/scap] - 10https://gerrit.wikimedia.org/r/761735 (owner: 10Ahmon Dancy) [22:24:10] (03Merged) 10jenkins-bot: scap prep: Add --copy-private-settings flag [tools/scap] - 10https://gerrit.wikimedia.org/r/761735 (owner: 10Ahmon Dancy) [22:28:01] (03PS4) 10Ahmon Dancy: scap prep auto: Add staging fingerprint support [tools/scap] - 10https://gerrit.wikimedia.org/r/761451 (https://phabricator.wikimedia.org/T301417) [23:14:51] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T300197 (10Ladsgroup)