[00:01:38] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:06:38] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:06:52] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:11:53] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [03:16:04] 10Continuous-Integration-Config, 10MobileFrontend, 10Developer Productivity: Export MobileFrontend dist files as CI artifacts - https://phabricator.wikimedia.org/T352240 (10Tgr) [08:32:43] 10Release-Engineering-Team, 10LandingCheck, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, and 4 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10Joe) >>! In T352156#9365189, @hashar wrote:... [08:35:13] 10Release-Engineering-Team, 10LandingCheck, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, and 4 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10Joe) 05Resolvedβ†’03Open The task is not... [08:58:20] 10Continuous-Integration-Config, 10PHP 8.3 support: Add experimental PHP 8.3 jobs - https://phabricator.wikimedia.org/T339349 (10Michael) I tried it out (`check experimental`) on a Wikibase change and it quickly failed with the following: ` 00:00:04.612 Unable to find image 'docker-registry.wikimedia.org/rele... [09:09:47] 10Release-Engineering-Team, 10LandingCheck, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, and 4 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10Joe) 05Openβ†’03Resolved I created the su... [10:14:44] 10GitLab (Auth & Access), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10gitlab-settings, 10User-brennen: Create an approved-accounts project in GitLab for distributing account approval process to non-administrators - https://phabricator.wikimedia.org/T350832 (10Aklapper) Stumbled upon fd.o's approach... [10:21:41] (03Abandoned) 10Hashar: dockerfiles: Install pnpm in Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/813215 (https://phabricator.wikimedia.org/T305525) (owner: 10Kosta Harlan) [10:23:32] 10Phabricator: Consider disallowing creating projects with an empty description - https://phabricator.wikimedia.org/T344610 (10Aklapper) p:05Triageβ†’03Low a:03Aklapper When I had some spare time I explored a bunch of potential implementation approaches in P50619. All have some disadvantages though. :P Ther... [10:25:30] 10Phabricator, 10Patch-For-Review: Consider disallowing creating projects with an empty description - https://phabricator.wikimedia.org/T344610 (10CodeReviewBot) aklapper opened https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/24 Disallow creating projects with empty description [10:34:35] 10Phabricator, 10Patch-For-Review: Disallow creating projects with an empty description - https://phabricator.wikimedia.org/T344610 (10Aklapper) [10:38:21] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.42.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T350083 (10IKhitron) Please pay attention to T352257. [11:00:26] 10Phabricator: Phabricator should not embed videos from Wikimedia Commons when the file is deleted - https://phabricator.wikimedia.org/T293139 (10Aklapper) 05Openβ†’03Declined p:05Lowestβ†’03Low I locally set http://phorge.localhost/config/edit/remarkup.enable-embedded-youtube/ to `true` and checked Phorge's... [11:00:35] 10Phabricator: Phabricator should not embed videos from Wikimedia Commons when the file is deleted - https://phabricator.wikimedia.org/T293139 (10Aklapper) For those curious about numbers, ` > SELECT COUNT(content) FROM phabricator_maniphest.maniphest_transaction_comment WHERE content LIKE "%commons.wikimedia.or... [11:00:51] we seem to have another npm cache / _cacache corruption on integration-agent-docker1051 https://integration.wikimedia.org/ci/job/mwgate-node16-docker/90346/console [11:04:24] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Deployment pipeline (GitLab/Kokkuri/Blubber) documentation improvements - https://phabricator.wikimedia.org/T342317 (10KBach) [11:04:26] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Review deployment pipeline documentation - https://phabricator.wikimedia.org/T348855 (10KBach) 05In progressβ†’03Resolved Initial, high-level review is now done, but further review is necessary and will continue in separate tasks under... [11:08:30] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/Nuke] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/978492 (https://phabricator.wikimedia.org/T321837) [11:10:21] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/PdfHandler] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/978494 (https://phabricator.wikimedia.org/T321837) [11:10:55] 10Release-Engineering-Team, 10Tech-Docs-Team, 10Documentation: Review critical-path deployment pipeline documentation - https://phabricator.wikimedia.org/T352259 (10KBach) [11:11:28] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/Poem] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/978495 (https://phabricator.wikimedia.org/T321837) [11:11:34] 10Release-Engineering-Team, 10Tech-Docs-Team, 10Documentation: Review critical-path deployment pipeline documentation - https://phabricator.wikimedia.org/T352259 (10KBach) [11:20:38] 10Release-Engineering-Team, 10Tech-Docs-Team, 10Documentation: Review supporting deployment pipeline documentation - https://phabricator.wikimedia.org/T352262 (10KBach) [11:21:34] 10Release-Engineering-Team, 10Tech-Docs-Team, 10Documentation: Review supporting deployment pipeline documentation - https://phabricator.wikimedia.org/T352262 (10KBach) a:05KBachβ†’03None [11:22:14] 10Release-Engineering-Team, 10Tech-Docs-Team, 10Documentation: Review supporting deployment pipeline documentation - https://phabricator.wikimedia.org/T352262 (10KBach) p:05Triageβ†’03Low [11:24:36] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Improve deployment pipeline documentation structure - https://phabricator.wikimedia.org/T352264 (10KBach) [11:25:09] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Improve deployment pipeline documentation structure - https://phabricator.wikimedia.org/T352264 (10KBach) 05Openβ†’03In progress p:05Triageβ†’03Low [11:25:11] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Deployment pipeline (GitLab/Kokkuri/Blubber) documentation improvements - https://phabricator.wikimedia.org/T342317 (10KBach) [11:27:43] 10Phabricator, 10Browser-Support-Apple-Safari: Viewing raw files (on phab.wmfusercontent.org) fails with ERROR_MESSAGE_MAIN (QuickLookErrorDomain error 16) on iOS mobile - https://phabricator.wikimedia.org/T169454 (10Aklapper) As I do not have an iOS device, I used a trial account on browserstack.com (os=iOS&o... [11:30:42] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Improve deployment pipeline documentation structure - https://phabricator.wikimedia.org/T352264 (10KBach) [11:41:28] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Create repository with configuration files for common CI use cases - https://phabricator.wikimedia.org/T352273 (10KBach) [11:41:46] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Create repository with configuration files for common CI use cases - https://phabricator.wikimedia.org/T352273 (10KBach) p:05Triageβ†’03Low [11:44:47] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Write tutorials for common CI tasks - https://phabricator.wikimedia.org/T352274 (10KBach) [11:45:11] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Write tutorials for common CI tasks - https://phabricator.wikimedia.org/T352274 (10KBach) p:05Triageβ†’03Low [11:53:35] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Deployment pipeline (GitLab/Kokkuri/Blubber) documentation improvements - https://phabricator.wikimedia.org/T342317 (10KBach) [12:06:47] 10Gerrit, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap, 10Patch-For-Review: Automate Gerrit deployment steps - https://phabricator.wikimedia.org/T317412 (10hashar) The aftermath of running https://gerrit.wikimedia.org/r/c/operations/puppet/+/844998/ on gerrit2002: ` Notice: /Stage[main]/Ssh::Ser... [12:25:31] (03PS1) 10Hashar: scap: change deploy user from gerrit2 to gerrit-deploy [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/978505 (https://phabricator.wikimedia.org/T317412) [12:26:03] (03CR) 10Hashar: [C: 03+2] scap: change deploy user from gerrit2 to gerrit-deploy [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/978505 (https://phabricator.wikimedia.org/T317412) (owner: 10Hashar) [12:26:34] (03Merged) 10jenkins-bot: scap: change deploy user from gerrit2 to gerrit-deploy [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/978505 (https://phabricator.wikimedia.org/T317412) (owner: 10Hashar) [12:29:52] (03PS1) 10Hashar: Add .gitreview file [software/gerrit/tools/gervert/deploy] - 10https://gerrit.wikimedia.org/r/978526 [12:29:54] (03PS1) 10Hashar: Change deploy user from gerrit2 to gerrit-deploy [software/gerrit/tools/gervert/deploy] - 10https://gerrit.wikimedia.org/r/978527 (https://phabricator.wikimedia.org/T317412) [12:34:31] (03PS1) 10Hashar: Use gerrit dsh group from deployment server [software/gerrit/tools/gervert/deploy] - 10https://gerrit.wikimedia.org/r/978528 [12:44:32] 10Gerrit, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap, 10Patch-For-Review: Automate Gerrit deployment steps - https://phabricator.wikimedia.org/T317412 (10hashar) Manual steps I took after rolling https://gerrit.wikimedia.org/r/c/operations/puppet/+/844998/ and mentioned in the commit message:... [12:47:35] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [12:50:34] 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-FundraiserLandingPage, 10donate.wikimedia.org, 10MW-1.42-notes (1.42.0-wmf.5; 2023-11-14): donate.wikimedia.org landing pages don't have descriptions in search engine results - https://phabricator.wikimedia.org/T254808 (10Pcoombe)... [12:55:29] PROBLEM - Work requests waiting in Zuul Gearman server on contint2002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [13:04:39] 10Phabricator maintenance bot, 10API Platform, 10Content-Transform-Team, 10MediaWiki-Engineering: Stop adding Platform Engineering to new wiki creation tasks - https://phabricator.wikimedia.org/T346811 (10MSantos) 05Openβ†’03Resolved a:03MSantos [13:07:35] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [13:16:37] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/Nuke] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/978492 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:16:39] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/PdfHandler] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/978494 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:16:42] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/Poem] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/978495 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:17:34] (03Abandoned) 10Hashar: Switch to Wikimedia parent pom [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/975847 (owner: 10Hashar) [13:19:18] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Nuke] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/978492 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:19:20] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PdfHandler] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/978494 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:19:23] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Poem] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/978495 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:20:14] !log Reloaded Zuul to enable Sonar Codehealth on Nuke, PdfHandler and Poem # T321837 [13:20:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:20:18] T321837: Repositories integrated into Codehealth Pipeline (Production) - https://phabricator.wikimedia.org/T321837 [13:24:51] (03PS1) 10Ilias Sarantopoulos: ml-services: renamed directory for article_descriptions [integration/config] - 10https://gerrit.wikimedia.org/r/978535 (https://phabricator.wikimedia.org/T351940) [13:57:56] (03CR) 10Gehel: "A bunch of things could be further simplified." [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/975847 (owner: 10Hashar) [13:58:21] hashar: I just realized the CR above has already been dropped. But maybe the comments are still useful in some way... [14:07:28] gehel: ahhhh sorry [14:07:33] (03Restored) 10Hashar: Switch to Wikimedia parent pom [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/975847 (owner: 10Hashar) [14:07:42] if you reviewed it, I guess I should commit to complete it :) [14:07:52] the issue is the code base is super old (like 10 / 15 yars old) [14:07:57] and checkstyle complains all over the place [14:08:05] and I am not even sure it is worth polishing the code base :D [14:08:38] you can also have checkstyle publish reports, but not fail the build [14:08:51] ah good idea [14:09:20] and I haven't yet looked at figuring out the duplicated definitions with the parent, but since you did, I guess I can follow your lead [14:09:22] even better, if there is sonar integration, sonar will complain if you increase the number of violations and fail in that case [14:09:38] \o/ [14:10:03] example: https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/pom.xml#336 [14:13:16] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.42.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T350085 (10hashar) As a follow to T338205 we have rolled some changes to the `fix-staging-perms` script which has altered the group owner ship of `/srv/mediawi... [14:14:03] hashar: https://gerrit.wikimedia.org/r/c/wikimedia/discovery/discovery-parent-pom/+/978607 [14:14:29] 10Deployments, 10Release-Engineering-Team, 10Sustainability (Incident Followup): Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) 05Openβ†’03Resolved @claime gave the final round of review and rolled all the patches. He ran the `fix-staging-perms` s... [14:14:53] 10Deployments, 10Release-Engineering-Team, 10Sustainability (Incident Followup): Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) [14:19:09] hashar: What's the magic when we're getting npm corruption like https://integration.wikimedia.org/ci/job/mwgate-node16-docker/90451/console ? Is it to delete the cache on the agent? On castor05? [14:20:34] https://phabricator.wikimedia.org/T348243#9283049 says to just rm -rf the dir on castor05 but I tried that and it didn't solve anything. [14:21:46] gehel: another thing I don't get is my pom has the checkstyle-plugin under `` section and if I add some configuration bits,the build still fails. I guess it is for a different phase? [14:22:31] you can remove it from the reporting section and add an entry in the section with that configuration [14:23:00] James_F: that should be it yes. The dir under which the cache got saved shows up at the top of the build: `Defined: CASTOR_NAMESPACE="castor-mw-ext-and-skins/master/mwgate-node16-docker"` which is relative to /srv/castor on the castor05 instance [14:23:08] Yeah… [14:23:12] I can try again? [14:23:28] gehel: I will give it a try :) [14:23:44] James_F: maybe another corrupted build got saved after you deleted the directory [14:23:51] hashar: Possibly. [14:24:19] and I don't know why they get corrupted, I suspect it is because some build fetch from the cache while another one is being stored in it which might lead to some files being 0 bytes [14:24:27] aka a race condition [14:24:33] * James_F nods. [14:24:57] No, https://integration.wikimedia.org/ci/job/mwgate-node16-docker/90456/console is failing again. :-( [14:25:31] I guess I could just terminate all running CI by restarting zuul, and delete the cache during the restart? But that sounds unkind. [14:29:23] well https://integration.wikimedia.org/ci/job/mwgate-node16-docker/ [14:29:33] there are no jobs running right now, so no need to stop Zuul :D [14:29:47] OK, will try a fifth time. :-) [14:30:21] it is also possible something is corrupted on the npmjs side [14:30:41] Maybe. [14:36:01] Well, it failed yet again. [14:42:19] (03PS1) 10Hashar: doc: link to Discovery parent pom [integration/docroot] - 10https://gerrit.wikimedia.org/r/978612 [14:46:01] 10Release-Engineering-Team, 10LandingCheck, 10MW-on-K8s, 10MediaWiki-extensions-WikimediaEvents, and 4 others: PHP Warning: geoip_country_code_by_name(): Required database not available at /usr/share/GeoIP/GeoIP.dat. - https://phabricator.wikimedia.org/T352156 (10Jdforrester-WMF) [15:02:59] 10Phabricator (Upstream), 10Upstream: Remove "Action Has No Effect" warning dialog when removing auto assigned user - https://phabricator.wikimedia.org/T337017 (10Aklapper) a:03Aklapper [15:05:09] 10Continuous-Integration-Config, 10PHP 8.3 support: Add experimental PHP 8.3 jobs - https://phabricator.wikimedia.org/T339349 (10Jdforrester-WMF) >>! In T339349#9366041, @Michael wrote: > I tried it out (`check experimental`) on a Wikibase change and it quickly failed with the following: > > ` > 00:00:04.612... [15:23:48] (03CR) 10Hashar: [C: 04-1] "Doing a first pass, there are more things that needs to be addressed :)" [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/975847 (owner: 10Hashar) [15:23:54] (03PS2) 10Hashar: Switch to Wikimedia parent pom [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/975847 [15:25:04] hashar: A alert is firing for a high Jenkins/zuul queue: https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10. Do you have an idea what's going on there and if that's critical? [15:27:04] jelto: it is not critical, the alarm is merely to alert us there are too many patches being processed in CI [15:27:13] (03CR) 10CI reject: [V: 04-1] Switch to Wikimedia parent pom [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/975847 (owner: 10Hashar) [15:27:15] which it will eventually finishes [15:27:23] 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-FundraiserLandingPage, 10donate.wikimedia.org, 10MW-1.42-notes (1.42.0-wmf.5; 2023-11-14): donate.wikimedia.org landing pages don't have descriptions in search engine results - https://phabricator.wikimedia.org/T254808 (10EWilfong_W... [15:27:26] ack, thanks :) [15:27:28] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10SecTeam-Processed, and 2 others: Jenkins plugins security advisory - 2023-11-29 - https://phabricator.wikimedia.org/T352292 (10sbassett) p:05Triageβ†’03Low [15:28:25] yeah there is a fairly large series of patches being processed https://gerrit.wikimedia.org/r/q/topic:lsc-T352284 [15:28:26] T352284: Move user-options classes to the MediaWiki\User\Options namespace - https://phabricator.wikimedia.org/T352284 [15:29:22] 10Release-Engineering-Team, 10ci-test-error: Wikibase CI blocked again by cacache corruption error in mwgate-node16-docker - https://phabricator.wikimedia.org/T352305 (10Jakob_WMDE) [15:30:06] James_F: https://phabricator.wikimedia.org/T352305 you are not alone [15:31:28] 10Release-Engineering-Team, 10ci-test-error: Wikibase CI blocked again by cacache corruption error in mwgate-node16-docker - https://phabricator.wikimedia.org/T352305 (10Lucas_Werkmeister_WMDE) CCing @Jdforrester-WMF who was already working on this in #wikimedia-releng. [15:31:30] dammit you beat me to the CC ^^ [15:32:12] 10Continuous-Integration-Config, 10PHP 8.3 support: Add experimental PHP 8.3 jobs - https://phabricator.wikimedia.org/T339349 (10Michael) Thank you! πŸ™ [15:32:48] I guess one of hte issue is the cache for npm is shared between all repositories [15:32:56] 10Release-Engineering-Team, 10ci-test-error: Multiple extensions' CI blocked again by cache corruption error in mwgate-node16-docker - https://phabricator.wikimedia.org/T352305 (10Jdforrester-WMF) [15:33:07] when for running `npm ci && npm test` for a single repo, it should be split by repo [15:37:41] hashar: Isn't the whole point that we want to share common resources and not shard by repo? [15:38:07] A CI-local per-repo cache is better than no cache, but it's a lot less good then all-repos? [15:38:08] 10Release-Engineering-Team, 10castor, 10ci-test-error: Multiple extensions' CI blocked again by cache corruption error in mwgate-node16-docker - https://phabricator.wikimedia.org/T352305 (10hashar) That is the same issue described on T295351 which to me is most probably a race condition specially showing up... [15:38:15] But if it fixes it then… [15:39:24] well for mwgate-node16-docker , it is cloning a single repo so potentially the cahce should be split on a per repo basis [15:39:37] then most repositories more or less share the same dependencies so I guess it makes sense to share the cache [15:39:53] Yeah, grunt/eslint/etc. are common across ~2000 repos. [15:39:55] I commented on the task, but the acutal task is T295351 [15:39:56] T295351: npm cache saved by castor get corrupted for unknown reason - https://phabricator.wikimedia.org/T295351 [15:40:01] Right. [15:40:14] Also CI is not helped by `[ERROR] An error occured during post-build processing.` failures on otherwise perfectly-fine jobs. [15:40:20] though this one apparently has the slight twist that rm -rf’ing the cache hasn’t fixed it so far [15:40:26] and I think the issue we are facing right now is that for the last 3 hours we had a bunch of changes send to MediaWiki which are being +2ed [15:40:28] `Caused by: hudson.AbortException: castor-save-workspace-cache aborted.` [15:40:30] (but yeah same issue) [15:40:31] which saves the cache [15:40:35] and might cause the corruption [15:41:06] on top of that when ever a change fails in the +2 queue, it get cancelled, and I am not entirely sure what Jenkins is doing with the onfly saving [15:42:27] for `castor-save-workspace-cache aborted` I am not sure, if you have a build having that issue I am interested [15:42:39] that might be due to a fix I pushed last week for the postbuildscript plugin [15:42:47] https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php80-noselenium-docker/25222/console [15:42:58] But I've seen a few over the past few days. [15:43:02] which is that when a job is canceled while it triggering castor-save-workspace-cache, it should also cancel that sub job [15:44:54] 15:18:44 [PostBuildScript] - [ERROR] An error occured during post-build processing. [15:44:54] 15:20:18 org.jenkinsci.plugins.postbuildscript.PostBuildScriptException: hudson.AbortException: castor-save-workspace-cache aborted. [15:44:55] fun [15:45:08] 10Release-Engineering-Team (Quid Pro Crow πŸ¦ƒ), 10Security-Team, 10SecTeam-Processed: Ensure no security patches is an error condition for MediaWiki deploys - https://phabricator.wikimedia.org/T350070 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/198 Ensure no... [15:46:09] hashar: should we ask Gerrit maintenance bot’s maintainer to add throttling to this lsc feature? [15:46:20] definitely yes [15:46:24] (which I’m guessing stands for large-scale changes – Amir showed it to me once, I’m not sure how documented it is yet) [15:46:28] alright, I’ll file a task [15:46:42] LibUP does that [15:46:47] it throttles sneding the patches [15:46:59] and throttles CR+2 ing them by watching the Zuul status queue [15:47:09] also LibUp runs over night when there is less activity :) [15:47:12] AFAICT these ones don’t get auto-+2ed? [15:47:19] sergi0 just manually +2ed a lot of them :S [15:47:35] (but the initial upload and test build would still put stress on zuul, so that should be throttled too) [15:47:42] yeah hence the large queue [15:47:54] then Zuul has a limitatoin window so it is not running everything at once [15:48:38] it will resolves all of them eventually [15:51:50] hashar: created T352309 [15:51:51] T352309: Gerrit maintenance bot should throttle uploading patches - https://phabricator.wikimedia.org/T352309 [15:52:43] hashar: LibUp is amazingly throttled by virtue of not running since June, too. [15:53:11] ;-) [15:55:13] ;_; [15:57:50] 10Release-Engineering-Team (Quid Pro Crow πŸ¦ƒ), 10Security-Team, 10SecTeam-Processed: Ensure no security patches is an error condition for MediaWiki deploys - https://phabricator.wikimedia.org/T350070 (10dancy) 05Openβ†’03Resolved Scap 4.64.0 has been deployed with this change. [16:04:38] RECOVERY - Work requests waiting in Zuul Gearman server on contint2002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [16:07:17] James_F: is that cause LibUp was solely handled by Kunal and he is now busy with other adventures? [16:12:57] Yeah. :-( [16:24:38] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Quid Pro Crow πŸ¦ƒ), 10User-brennen: Define "Terms of Service and Privacy Policy" text for GitLab - https://phabricator.wikimedia.org/T285354 (10Aklapper) @brennen, @jelto: Any of you know if settings set under https://gitlab.wikimedia.or... [16:25:16] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Quid Pro Crow πŸ¦ƒ), 10User-brennen: Define "Terms of Service and Privacy Policy" text for GitLab - https://phabricator.wikimedia.org/T285354 (10Aklapper) Final text set under https://gitlab.wikimedia.org/admin/application_settings/genera... [16:27:42] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Quid Pro Crow πŸ¦ƒ), 10User-brennen: Define "Terms of Service and Privacy Policy" text for GitLab - https://phabricator.wikimedia.org/T285354 (10thcipriani) >>! In T285354#9330060, @Aklapper wrote: > @brennen, @thcipriani: Is setting the... [16:27:45] 10Gitlab-Application-Security-Pipeline, 10Patch-For-Review, 10SecTeam-Processed, 10Security, 10user-sbassett: Improve related gitlab test repository - https://phabricator.wikimedia.org/T348788 (10CodeReviewBot) mmartorana merged https://gitlab.wikimedia.org/repos/security/ci-cd-testing-gitlab-ci-security... [16:28:41] 10Scap: scap fails to bootstrap itself - https://phabricator.wikimedia.org/T352223 (10Dzahn) Should I simply copy the scap-wheels directory from production deployment server over to cloud deployment server? Or do we need to create a new deployment server from scratch and then it would appear? [16:35:09] 10Continuous-Integration-Infrastructure, 10Jenkins, 10castor: castor-save-workspace-cache aborted during postbuild - https://phabricator.wikimedia.org/T352319 (10hashar) [16:36:04] James_F: I filed the stacktrace you mentioned earlier as https://phabricator.wikimedia.org/T352319 [16:36:32] I have no idea what is going on :-] [16:38:39] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Quid Pro Crow πŸ¦ƒ), 10User-brennen: Define "Terms of Service and Privacy Policy" text for GitLab - https://phabricator.wikimedia.org/T285354 (10Jelto) We don't set `sign_in_text` in the config for application settings API (see [gitlab200... [16:42:24] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Quid Pro Crow πŸ¦ƒ), 10User-brennen: Define "Terms of Service and Privacy Policy" text for GitLab - https://phabricator.wikimedia.org/T285354 (10Aklapper) [16:42:28] 10GitLab (Administration, Settings & Policy), 10collaboration-services, 10Patch-For-Review: Wikimedia GitLab should use a Wikimedia logo - https://phabricator.wikimedia.org/T349167 (10Aklapper) [16:42:31] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Quid Pro Crow πŸ¦ƒ), 10User-brennen: Define "Terms of Service and Privacy Policy" text for GitLab - https://phabricator.wikimedia.org/T285354 (10Aklapper) a:05Aklapperβ†’03None Ah, thanks for the bigger picture I was missing! Unassignin... [16:42:53] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Quid Pro Crow πŸ¦ƒ), 10gitlab-settings, 10User-brennen: Define "Terms of Service and Privacy Policy" text for GitLab - https://phabricator.wikimedia.org/T285354 (10Aklapper) [16:45:03] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Quid Pro Crow πŸ¦ƒ), 10gitlab-settings, 10User-brennen: Define "Terms of Service and Privacy Policy" text for GitLab - https://phabricator.wikimedia.org/T285354 (10Jelto) a:03Jelto I can make sure the same text is set by [gitlab-setti... [16:54:02] @dduvall: Yo, if you by any chance remember usecases for https://phabricator.wikimedia.org/source/scap/?go-get=1 in https://phabricator.wikimedia.org/D775 and D776 , I'd love to hear if these use cases are still valid six years later or if we can scrap that custom code. Short "Still useful" vs "Scrap it" answer welcome - TIA! [16:54:03] D776: Support go-import meta tags - https://phabricator.wikimedia.org/D776 [16:56:54] andre: tl;dr: scrap it! it's needed by the go toolchain to resolve a git repo from a module path and version string. since we no longer (or are trying not to) host repos using diffusion, it shouldn't be needed [17:01:18] Thanks! [17:04:41] 10Phabricator: Remove custom go-get / go-import code in Diffusion repositories - https://phabricator.wikimedia.org/T352322 (10Aklapper) p:05Triageβ†’03Low [17:12:02] 10Phabricator: Remove custom go-get / go-import code in Diffusion repositories - https://phabricator.wikimedia.org/T352322 (10Aklapper) As the two Differential changes do not reliably link to merged commits, here's what I believe are related code changes: * rPHAB853a5e1d409a6616cc275fc83eeff041ffd79c83 * rPHAB63... [17:15:35] 10Phabricator: Remove custom go-get / go-import code in Diffusion repositories - https://phabricator.wikimedia.org/T352322 (10Aklapper) a:05Aklapperβ†’03None [17:33:25] (03CR) 10Hashar: [C: 03+2] ml-services: renamed directory for article_descriptions [integration/config] - 10https://gerrit.wikimedia.org/r/978535 (https://phabricator.wikimedia.org/T351940) (owner: 10Ilias Sarantopoulos) [17:36:56] (03Merged) 10jenkins-bot: ml-services: renamed directory for article_descriptions [integration/config] - 10https://gerrit.wikimedia.org/r/978535 (https://phabricator.wikimedia.org/T351940) (owner: 10Ilias Sarantopoulos) [17:41:40] 10Scap: scap fails to bootstrap itself - https://phabricator.wikimedia.org/T352223 (10jnuche) I hate that you don't get a notification when you get subscribed to something. I didn't get a message until @Dzahn posted his last comment. My guess is that the [[ https://gerrit.wikimedia.org/r/plugins/gitiles/operati... [17:43:04] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/978535 # T351940 [17:43:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:43:08] T351940: Enable local runs for article-descriptions model - https://phabricator.wikimedia.org/T351940 [17:43:14] (03CR) 10Hashar: [C: 03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/978535 (https://phabricator.wikimedia.org/T351940) (owner: 10Ilias Sarantopoulos) [17:50:55] andre: heya, I've noticed that I don't get notified whenever someone subscribes me to a phab ticket. I also don't see any option to change that in my setttings [17:51:01] had you noticed that before? [17:51:21] jnuche, basically https://www.mediawiki.org/wiki/Phabricator/Help/Managing_mail [17:51:36] email or web notification? [17:51:42] See https://phabricator.wikimedia.org/settings/panel/emailpreferences/ for your settings [17:52:28] I don't think there's a "Notify my once when I get subscribed" setting, if that is what you mean. Herald could do that though :-/ [17:54:05] there's a "A task's subscribers change", I wonder if that will send you something the moment you get subscribed [17:54:10] will give it a try [17:54:11] thanks! [17:55:00] (03PS3) 10Hashar: Switch to Wikimedia parent pom [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/975847 [17:58:36] jnuche, that will spam you whenever any subscribers change, I'm afraid [18:00:13] yeah, I know, but maybe it's bearable with a filter. In any case I just tried it and didn't get notified on first subscribe, bummer [18:02:15] (03CR) 10CI reject: [V: 04-1] Switch to Wikimedia parent pom [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/975847 (owner: 10Hashar) [18:41:51] I'm running into some issues with docker-registry.wikimedia.org/{bookworm,bullseye}:latest - Installing java packages is erroring out due to lack of a /usr/share/man directory. Searching seems to suggest this is a common issue on slim variants of debian. Are these images based on slim? (I can't find the registry dockerfile/containerfile) [18:53:44] 10GitLab, 10Release-Engineering-Team (Seen): buildctl reports `error: failed to solve: failed to compute cache key: failed to read expected number of bytes: unexpected EOF` - https://phabricator.wikimedia.org/T352100 (10thcipriani) [18:55:16] dancy: I also would love to know who the "reggie" registry image is. That person looks so familiar.... [18:56:04] brett: It is meant to be resemble https://en.wikipedia.org/wiki/Reggie_Jackson [18:56:20] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Seen), 10Upstream: CI Jenkins fails to restart after applying plugin updates - https://phabricator.wikimedia.org/T351865 (10thcipriani) [18:56:54] hm, not at all what I was reminded of. I can see the resemblance. But dang he looks so familiar [19:13:08] 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-FundraiserLandingPage, 10donate.wikimedia.org, and 2 others: donate.wikimedia.org landing pages don't have descriptions in search engine results - https://phabricator.wikimedia.org/T254808 (10AKanji-WMF) [19:42:04] 10Scap: scap fails to bootstrap itself - https://phabricator.wikimedia.org/T352223 (10Dzahn) Thank you @jnuche! The `role::deployment_server` is applied on the deployment server. (just like production deployment servers). This includes `profile::mediawiki::deployment::server` and that uses `class { '::scap::m... [19:46:16] 10Scap: scap fails to bootstrap itself - https://phabricator.wikimedia.org/T352223 (10Dzahn) I noticed puppet is also broken on that deployment_server: ` Function lookup() did not find a value for the name 'deployment_group' (file: /etc/puppet/modules/profile/manifests/mediawiki/deployment/server.pp, line: 7)... [19:48:13] 10Scap: scap fails to bootstrap itself - https://phabricator.wikimedia.org/T352223 (10Dzahn) The puppet breakage seems to come from https://gerrit.wikimedia.org/r/c/operations/puppet/+/927675 [19:50:53] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security: Add commit-message-validator for gitlab security ci includes merge requests - https://phabricator.wikimedia.org/T352331 (10sbassett) [19:51:06] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: [EPIC] Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10sbassett) [19:55:34] 10Scap: scap fails to bootstrap itself - https://phabricator.wikimedia.org/T352223 (10Dzahn) In `scap::master` I found the code that should run the bootstrap script: ` exec { 'bootstrap_scap_master': command => "${scap_source_path}/bin/bootstrap_scap_master.sh scap ${scap_source_path}", creates... [19:56:20] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10Patch-For-Review, 10SecTeam-Processed, 10Security: Add commit-message-validator for gitlab security ci includes merge requests - https://phabricator.wikimedia.org/T352331 (10CodeReviewBot) sbassett opened https://gitlab.wikimedia.org/repos/securi... [20:01:00] 10Scap: scap fails to bootstrap itself - https://phabricator.wikimedia.org/T352223 (10Dzahn) ` $scap_home = '/var/lib/scap' .. creates => "${scap_home}/scap/bin/scap", ` ` file /var/lib/scap/scap/bin/scap /var/lib/scap/scap/bin/scap: Python script, UTF-8 Unicode text executable ` ` file /var/lib/scap/scap... [20:01:49] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for USER[S] - https://phabricator.wikimedia.org/T352332 (10Remsense) [20:04:47] 10Scap: scap fails to bootstrap itself - https://phabricator.wikimedia.org/T352223 (10Dzahn) ` dzahn@deploy-1004:/srv/deployment/scap$ git status On branch master Your branch and 'origin/master' have diverged, and have 16 and 221 different commits each, respectively. (use "git pull" to merge the remote branch... [20:05:18] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for Remsense - https://phabricator.wikimedia.org/T352332 (10Soda) [20:05:58] 10Scap: scap fails to bootstrap itself - https://phabricator.wikimedia.org/T352223 (10Dzahn) ` git reset origin .. git pull ... sudo /srv/deployment/scap/bin/bootstrap_scap_master.sh scap /srv/deployment/scap M .pipeline/blubber.yaml M bin/install_local_version.sh M changelog M release-scripts/perform-release... [20:08:34] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for Remsense - https://phabricator.wikimedia.org/T352332 (10Soda) a:05Remsenseβ†’03None [20:09:21] 10Scap: scap fails to bootstrap itself - https://phabricator.wikimedia.org/T352223 (10Dzahn) ` git reset origin master .. git reset --hard origin/master .. git checkout master .. sudo /srv/deployment/scap/bin/bootstrap_scap_master.sh scap /srv/deployment/scap HEAD is now at 75ac24e Release 4.64.0-1 INFO: Scap di... [20:12:46] 10Phabricator, 10Release-Engineering-Team (Radar), 10collaboration-services: upgrade phab (phorge) hosts to bullseye - https://phabricator.wikimedia.org/T334519 (10Dzahn) Fixes applied on the deployment server, see T352223#9368436 ff After this, re-enabled puppet on the phab machine and: ` Nov 29 20:08:39... [20:12:58] 10Scap, 10collaboration-services: scap fails to bootstrap itself - https://phabricator.wikimedia.org/T352223 (10Dzahn) [20:14:54] 10Release-Engineering-Team (Radar), 10LDAP-Access-Requests, 10SRE: Grant Access to wmf, releng, ciadmin for sandeeps - https://phabricator.wikimedia.org/T352334 (10thcipriani) [20:16:05] 10Scap, 10collaboration-services: scap fails to bootstrap itself - https://phabricator.wikimedia.org/T352223 (10Dzahn) 05Openβ†’03Resolved a:03Dzahn [20:18:45] 10Scap, 10collaboration-services: scap fails to bootstrap itself - https://phabricator.wikimedia.org/T352223 (10Dzahn) summary: The scap deployment git repo was in a messy state. It is unknown why specifically it was in that state but: ` /srv/deployment/scap$ git status On branch master Your branch and 'origi... [20:25:59] 10Diffusion, 10Phabricator, 10Diffusion-Repository-Administrators: Rename repository names in Diffusion which have different names than in (canonical) Gerrit or Github (consistency) - https://phabricator.wikimedia.org/T135403 (10Aklapper) 05Openβ†’03Declined I'm afraid I //nowadays// don't understand which... [20:27:18] 10Phabricator: When tagging tasks from one board to another, automatically restore tasks to the same columns - https://phabricator.wikimedia.org/T253897 (10Aklapper) p:05Triageβ†’03Low [20:29:02] 10Phabricator, 10MediaWiki-extensions-CentralAuth, 10Stewards-and-global-tools, 10WMF-General-or-Unknown: CentralAuth locks should log linked users out of Phabricator - https://phabricator.wikimedia.org/T338384 (10Aklapper) p:05Triageβ†’03Low [20:37:41] !log Ran jforrester@integration-castor05:~$ sudo rm -fR /srv/castor/castor-mw-ext-and-skins/master/mw*-node16-* for T352305 [20:37:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:37:46] T352305: Multiple extensions' CI blocked again by cache corruption error in mwgate-node16-docker - https://phabricator.wikimedia.org/T352305 [20:41:39] 10Phabricator: Consider using the Badges application for a few special roles to highlight those users' comments - https://phabricator.wikimedia.org/T106924 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper The currently available badges are listed on https://phabricator.wikimedia.org/badges/ Badges can be create... [20:42:11] 10Phabricator: Decide guidelines for creation and how to distribute phabricator badges - https://phabricator.wikimedia.org/T196390 (10Aklapper) 05Openβ†’03Declined > Or do we keep the idea of distributing them on an ad hoc basis, That seems to reflect the reality. > but we still need to have a minimum numbe... [20:42:14] 10Phabricator: Consider using the Badges application for a few special roles to highlight those users' comments - https://phabricator.wikimedia.org/T106924 (10Aklapper) [20:44:04] 10Phabricator, 10Wikibase Cloud, 10Wikibase Product Platform Team WPP, 10Wikidata, and 2 others: Update Herald rule to tag Wikidata tech tasks - https://phabricator.wikimedia.org/T349104 (10Aklapper) @Itamar_WMDE: Is there more to do in this task, or can this be resolved? TIA! [21:09:39] 10Phabricator, 10Wikibugs: Case of wikibugs displaying unrelated user when Herald performed an action - https://phabricator.wikimedia.org/T116477 (10Aklapper) 05Openβ†’03Invalid Using https://phabricator.wikimedia.org/conduit/method/maniphest.gettasktransactions/ with `ids` set to `["72869"]`, or https://pha... [21:24:21] 10Phabricator, 10Release-Engineering-Team (Radar), 10collaboration-services: upgrade phab (phorge) hosts to bullseye - https://phabricator.wikimedia.org/T334519 (10Dzahn) Brennen made an attempt at deployment which showed us what other things we need to fix. for example: E: Package 'python-mysqldb' has no i... [21:26:51] 10Phabricator, 10Patch-For-Review, 10Voice & Tone: When a user clicks on Create Project but doesn't have permission to create a project, provide a link to learn how to create a project - https://phabricator.wikimedia.org/T165192 (10CodeReviewBot) aklapper opened https://gitlab.wikimedia.org/repos/phabricator... [21:43:51] 10Phabricator, 10Patch-For-Review, 10Voice & Tone: When a user clicks on Create Project but doesn't have permission to create a project, provide a link to learn how to create a project - https://phabricator.wikimedia.org/T165192 (10Aklapper) p:05Lowestβ†’03Low a:03Aklapper [21:45:06] 10Phabricator (Upstream), 10Upstream: Enhance tokens by adding rainbow-related ones - https://phabricator.wikimedia.org/T337986 (10Aklapper) [21:49:12] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Priority Backlog πŸ“₯): Reports: "Older" bar link in Histogram has wrong task query URL - https://phabricator.wikimedia.org/T336175 (10Aklapper) [21:49:16] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Priority Backlog πŸ“₯): Filter default Phabricator calendar view to show items tagged with #events only - https://phabricator.wikimedia.org/T343258 (10Aklapper) [21:49:33] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Priority Backlog πŸ“₯): Update custom allowlist in src/applications/phurl/storage/PhabricatorPhurlURL.php - https://phabricator.wikimedia.org/T343267 (10Aklapper) [21:50:10] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Quid Pro Crow πŸ¦ƒ): Remove broken custom code in DifferentialRevisionSearchEngine.php - https://phabricator.wikimedia.org/T351957 (10Aklapper) [21:50:31] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Quid Pro Crow πŸ¦ƒ): Remove unused custom "Add more repository details to harbormaster build steps" - https://phabricator.wikimedia.org/T351958 (10Aklapper) [21:50:37] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Technical-Debt: Revert unused downstream Phab code to suppress triggering mail (predates silent batch-editing) - https://phabricator.wikimedia.org/T343264 (10Aklapper) [22:00:35] 10Scap: Unclear scap warning message about "unexpected commits pulled from origin" - https://phabricator.wikimedia.org/T352339 (10Tgr) [22:01:29] 10Scap: Unclear scap warning message about "unexpected commits pulled from origin" - https://phabricator.wikimedia.org/T352339 (10Tgr) [23:19:14] 10Release-Engineering-Team, 10castor, 10ci-test-error: Multiple extensions' CI blocked again by cache corruption error in mwgate-node16-docker - https://phabricator.wikimedia.org/T352305 (10Jdforrester-WMF) It was working briefly, but now it's corrupted again it seems. :-(