[00:49:43] 10Phabricator, 07Wikimedia-production-error: In some environments, Phabricator or its resources returns 403 - https://phabricator.wikimedia.org/T431644 (10Les4353) 03NEW p:05Triage→03Medium [00:54:37] 10Phabricator, 06collaboration-services: In some environments, Phabricator or its resources returns 403 - https://phabricator.wikimedia.org/T431644#12102752 (10JJMC89) p:05Medium→03Triage You have not reported the issue using the link in the message you quoted. Therefore, it is missing relevant details. [00:57:03] 10Phabricator, 06collaboration-services: In some environments, Phabricator or its resources returns 403 - https://phabricator.wikimedia.org/T431644#12102756 (10Les4353) >>! In T431644#12102753, @JJMC89 wrote: > You have not reported the issue using the link in the message you quoted. Therefore, it is missing r... [06:06:36] 10Phabricator, 06collaboration-services: In some environments, Phabricator or its resources returns 403 - https://phabricator.wikimedia.org/T431644#12103003 (10Aklapper) We need the following information: * Resource/path: * Error message: * Browser/client: * IP address: I'm going to make this ticket non-publi... [08:27:36] 10Continuous-Integration-Infrastructure: integration.wikimedia.org should have a FOSS license - https://phabricator.wikimedia.org/T431663#12103342 (10Shashwatojha369) a:03Shashwatojha369 [08:27:43] 10Continuous-Integration-Infrastructure: Add dark mode to integration.wikimedia.org - https://phabricator.wikimedia.org/T431661#12103343 (10Shashwatojha369) a:03Shashwatojha369 [08:35:53] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T430829#12103418 (10Jdforrester-WMF) [08:44:01] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Krrish Sehgal - https://phabricator.wikimedia.org/T413600#12103462 (10Aklapper) 05Stalled→03Declined [09:03:56] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Remove remaining diffusion.ssh-host entries from Phab config and Puppet - https://phabricator.wikimedia.org/T429367#12103546 (10Aklapper) [09:03:58] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Remove remaining diffusion.ssh-host entries from Phab config and Puppet - https://phabricator.wikimedia.org/T429367#12103547 (10Aklapper) 05Open→03Resolved [09:06:12] (03open) 10kbach: Update DEVELOPERS.md and the OAD example [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/16 (https://phabricator.wikimedia.org/T422927) [09:17:18] (03merge) 10addshore: Fix composer question in CI (no interaction) [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/659 [09:17:58] (03update) 10addshore: Update from redis:6.2 to redis:8.6.4 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/658 [09:18:15] (03update) 10addshore: Update to eventgate-wikimedia:v1.31.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/657 [09:18:22] (03update) 10addshore: Update to releng/node20-test-browser:20.20.2 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/656 [09:18:31] (03update) 10addshore: Update shellbox to shellbox:2026-07-03-212105 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/655 [09:18:46] (03update) 10addshore: Update various go packages [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/654 [09:23:51] 10Deployments, 06Release-Engineering-Team: OpenSearch mediawiki new errors filters out some errors - https://phabricator.wikimedia.org/T431675 (10hashar) 03NEW [09:25:53] 10Deployments, 06Release-Engineering-Team: OpenSearch mediawiki new errors filters out some errors - https://phabricator.wikimedia.org/T431675#12103660 (10hashar) [09:38:46] 10Deployments, 06Release-Engineering-Team (Doing 😎): OpenSearch mediawiki new errors filters out some errors - https://phabricator.wikimedia.org/T431675#12103756 (10Aklapper) a:03Aklapper [09:42:38] 10Deployments, 06Release-Engineering-Team (Doing 😎): OpenSearch mediawiki new errors filters out some errors - https://phabricator.wikimedia.org/T431675#12103774 (10Aklapper) 05Open→03Resolved I deleted the filter for `T394850` on the MW New Errors ECS dashboard which, for reasons I don't know, ignored... [09:46:58] 10Deployments, 06Release-Engineering-Team (Doing 😎): OpenSearch mediawiki new errors filters out some errors - https://phabricator.wikimedia.org/T431675#12103804 (10hashar) Good catch, and for the record the filter had the custom label {nav NOT T394850 - JsonConfig: Undefined array key} with the OpenSearch... [09:47:42] 10Deployments, 06Release-Engineering-Team (Doing 😎): OpenSearch mediawiki new errors filters out some errors - https://phabricator.wikimedia.org/T431675#12103817 (10AlexisJazz) In T431668 one request gave me `Call to a member function getDifficulty() on null` ([8ee88844-e7c0-402f-bbf6-660c3ef43043]) and an... [09:53:35] 10Deployments, 06Release-Engineering-Team (Doing 😎): OpenSearch mediawiki new errors filters out some errors - https://phabricator.wikimedia.org/T431675#12103848 (10Aklapper) @AlexisJazz: This ticket is about fixing what is displayed on the Logstash dashboard. This ticket has nothing to do with any errors... [09:54:17] andre: I have found the reason the filter were excluding stuff :] [09:54:40] hashar, you XORed an AND I guess? :P [09:54:56] I've wondered for a while if some of the filters are too broad [09:59:18] 10Deployments, 06Release-Engineering-Team (Doing 😎): OpenSearch mediawiki new errors filters out some errors - https://phabricator.wikimedia.org/T431675#12103889 (10hashar) https://docs.opensearch.org/latest/query-dsl/compound/bool/ > `must_not` > > Logical `not` operator. All matches are excluded from th... [10:00:20] andre: yeah I pasted some explanation [10:00:29] which is not easily readable, there could be a nicer summary with some examples [10:00:39] and I don't know whether we have a documentation to setting up filters [10:00:53] fiddling with that QueryDSL has always been... "fun" [10:00:55] eg to cover a case like: I want to filter out a generic message for a single specific file [10:01:04] https://wikitech.wikimedia.org/wiki/OpenSearch_Dashboards [10:01:11] I am reopening that task if you don't mind? [10:01:22] I'd like others in devx to be aware of it and shime in [10:01:24] oh sure do it [10:01:32] s/shime// [10:02:30] 10Continuous-Integration-Infrastructure, 07Software-Licensing: integration.wikimedia.org should have a FOSS license - https://phabricator.wikimedia.org/T431663#12103914 (10Bugreporter) [10:03:00] 10Deployments, 06Release-Engineering-Team (Doing 😎): OpenSearch mediawiki new errors filters out some errors - https://phabricator.wikimedia.org/T431675#12103919 (10hashar) 05Resolved→03Open **TLDR** must_not without {nav Exclude results} ends up being an `OR` filter. After chatting with Andrew I am reop... [10:03:12] andre: ^ did some summary :] [10:04:08] would you mind raising awareness of it in #developer-experience on Slack for me? [10:04:14] s/for me/for us/ [10:04:27] 10Continuous-Integration-Config, 10Diffusion, 10Phabricator: integration-agent-docker machines excessively pull some Wikibase related Git repos in Diffusion - https://phabricator.wikimedia.org/T349921#12103923 (10Aklapper) Looking at only internal pulls, I believe that we've reached the baseline: ` mysql:phs... [10:04:38] (I don't mind doing it, but I don't mind not doing it in interest of licking less cookies :b ) [10:08:44] I'm not really on Slack either [10:09:08] I prefer doing work. :P [10:13:03] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: contint1003 is missing docker-buildx, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12103961 (10kevinbazira) >>! In T431582#1210... [10:15:41] o/ looks like image publishing in CI is broken: https://phabricator.wikimedia.org/T431582#12103961 [10:15:41] audit_audldhmks_1783592141 [10:16:11] link_audldhmks https://google.com [10:16:30] andre: fair!! I'll do it :-] thank you! [10:16:55] if we suspect other filters to be catch-alls, I guess we can go and audit all the filters using `must_not` we shall see [10:16:57] fun times [10:17:20] I am quite happy you found the root cause [10:50:51] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: contint1003 is missing docker-buildx, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12104166 (10hashar) Thanks @kevinbazira! The... [10:55:16] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: contint1003 is missing docker-buildx, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12104208 (10hashar) The docker daemon has: `... [11:10:58] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: contint1003 is missing docker-buildx, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12104312 (10hashar) I have confirmed the cre... [11:21:09] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: contint1003 is missing docker-buildx, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12104350 (10hashar) If I try to push an exis... [11:55:04] !log retriggering postmerge change for T431582: zuul enqueue --trigger gerrit --pipeline postmerge --project machinelearning/liftwing/inference-services --change 1308631,3 [11:55:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:55:07] T431582: contint1003 is missing docker-buildx / auth to registry fails, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582 [11:57:33] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: contint1003 is missing docker-buildx / auth to registry fails, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12104625 (10hashar) [12:05:15] (03merge) 10addshore: Update to eventgate-wikimedia:v1.31.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/657 [12:05:33] (03merge) 10addshore: Update to releng/node20-test-browser:20.20.2 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/656 [12:05:53] (03merge) 10addshore: Update shellbox to shellbox:2026-07-03-212105 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/655 [12:13:56] 10Beta-Cluster-Infrastructure, 10Page Content Service, 06Content-Transform-Team (Work In Progress), 07Essential-Work: PCS/summary returns errors for beta requests - https://phabricator.wikimedia.org/T415877#12104772 (10MSantos) [12:24:52] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: contint1003 is missing docker-buildx / auth to registry fails, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12104838 (10kevinbaz... [12:34:41] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: contint1003 is missing docker-buildx / auth to registry fails, causing blubber-based CI pipelines to fail - https://phabricator.wikimedia.org/T431582#12104934 (10hashar)... [13:01:05] 10Phabricator Translations Extension, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Interruptions), 10LPL sprints (S1 2026 July): Phabricator translations have been failing i18n-check for almost a month - https://phabricator.wikimedia.org/T431487#12105091 (10Nikerabbit) https://gerrit.wikimedia.... [13:01:38] 10Phabricator Translations Extension, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Interruptions), 10LPL sprints (S1 2026 July): Phabricator translations have been failing i18n-check for almost a month - https://phabricator.wikimedia.org/T431487#12105093 (10Nikerabbit) [13:02:09] 10Phabricator Translations Extension, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Interruptions), 10LPL sprints (S1 2026 July): Phabricator translations have been failing i18n-check for almost a month - https://phabricator.wikimedia.org/T431487#12105096 (10Nikerabbit) 05Open→03Resolved [13:20:53] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12105202 (10Arnoldokoth) I've setup `phabricato-trixie` and the lats merge... [13:27:13] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): PostBuild changing the status of successful builds to failure for no apparent reason - https://phabricator.wikimedia.org/T419488#12105225 (10hashar) I am pretty sure that is the... [13:27:30] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Upstream: castor-save-workspace-cache aborted during postbuild - https://phabricator.wikimedia.org/T352319#12105227 (10hashar) [13:27:31] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): PostBuild changing the status of successful builds to failure for no apparent reason - https://phabricator.wikimedia.org/T419488#12105230 (10hashar) →14Duplicate dup:03T3... [13:29:19] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Upstream: castor-save-workspace-cache aborted during postbuild - https://phabricator.wikimedia.org/T352319#12105234 (10hashar) Self note review the two cases in T352319#10288944 > receiving a CancellationException after four minutes. The opti... [13:40:57] (03merge) 10addshore: Update various go packages [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/654 [13:41:41] (03merge) 10addshore: Update from redis:6.2 to redis:8.6.4 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/658 [14:05:38] 10Phabricator Translations Extension: Suspicious plurals in Phabricator messages - https://phabricator.wikimedia.org/T431715 (10Nikerabbit) 03NEW [14:07:43] 10Continuous-Integration-Infrastructure, 07Software-Licensing: integration.wikimedia.org should have a FOSS license - https://phabricator.wikimedia.org/T431663#12105424 (10Tacsipacsi) @Shashwatojha369, thanks for your willingness to help, but I think this task should be solved by the original authors of the co... [14:17:15] 10Continuous-Integration-Infrastructure, 07Software-Licensing: integration.wikimedia.org should have a FOSS license - https://phabricator.wikimedia.org/T431663#12105453 (10Dzahn) At least 3 different sources are involved to create that page. [[ https://gerrit.googlesource.com/gitiles/ | gitiles ]], [[ https:/... [14:26:17] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T430829#12105529 (10hashar) 05Open→03Resolved I am claiming 1.47.0-wmf.10 to have successfully rolled out. [14:34:25] 10Phabricator: Request for mpopov to be added to acl*phabricator - https://phabricator.wikimedia.org/T431720 (10mpopov) 03NEW [14:34:40] 10Phabricator: Request for mpopov to be added to acl*phabricator - https://phabricator.wikimedia.org/T431720#12105612 (10mpopov) [14:36:59] 10Continuous-Integration-Infrastructure, 10doc.wikimedia.org, 07Software-Licensing: integration.wikimedia.org should have a FOSS license - https://phabricator.wikimedia.org/T431663#12105637 (10hashar) + #doc.wikimedia.org since the repo is shared with https://doc.wikimedia.org/ Tacsipacsi wrote: > Relatedly... [14:48:25] 10Gerrit: Decide if Gerrit's indices should get backed up - https://phabricator.wikimedia.org/T257744#12105722 (10hashar) 05Open→03Resolved a:03hashar We have historically not backed up the search indices. We had them backed up by mistake as part of some path change and they take a sizeable amount of... [14:54:04] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Compact Gerrit H2 databases to reclaim disk - https://phabricator.wikimedia.org/T430774#12105786 (10hashar) 05Open→03Resolved a:03hashar This happens automatically when Gerrit is being stopped (more exactly when the H2 DB Driver is abo... [15:00:17] 10Gerrit, 06collaboration-services: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12105835 (10hashar) a:05ABran-WMF→03hashar From my list of action on T411583#12074321 **Compact the H2 DB** The Gerrit primary has been restart a few time which does trigger a compaction. Right... [15:07:11] 10Gerrit, 06collaboration-services: Validate Gerrit backup recoverability after the fileset exclusions - https://phabricator.wikimedia.org/T430775#12105890 (10hashar) The backup is being conducted while the service is actively writing to LFS object files (rarely happens), git packfiles and objects. We also h... [15:15:24] 10Phabricator Translations Extension: Suspicious plurals in Phabricator messages - https://phabricator.wikimedia.org/T431715#12105927 (10Pppery) a:03Pppery [15:27:23] 10Beta-Cluster-Infrastructure, 07Developer Productivity: Set up a process for regularly updating the Beta Cluster's copies of common.js, gadgets, and similar files - https://phabricator.wikimedia.org/T212244#12105986 (10bd808) This seems like something that someone could create a tool in Toolforge to do with p... [15:35:19] (03PS1) 10Kosta Harlan: Provide a disposable secondary database for multi-DB tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/1309235 (https://phabricator.wikimedia.org/T342763) [15:39:08] (03CR) 10CI reject: [V:04-1] Provide a disposable secondary database for multi-DB tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/1309235 (https://phabricator.wikimedia.org/T342763) (owner: 10Kosta Harlan) [15:49:15] 10Phabricator: Potentially disable H489 - https://phabricator.wikimedia.org/T431735 (10Aklapper) 03NEW p:05Triage→03Low [15:51:01] (03PS2) 10Kosta Harlan: Provide a disposable secondary database for multi-DB tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/1309235 (https://phabricator.wikimedia.org/T342763) [15:51:21] 10Phabricator: Potentially disable H488 - https://phabricator.wikimedia.org/T431737 (10Aklapper) 03NEW [15:54:25] 10Phabricator: Disable H485 - https://phabricator.wikimedia.org/T431738 (10Aklapper) 03NEW [15:55:05] 10Phabricator: Disable H484 and H485 - https://phabricator.wikimedia.org/T431738#12106114 (10Aklapper) [15:56:57] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Request for mpopov to be added to acl*phabricator - https://phabricator.wikimedia.org/T431720#12106129 (10Aklapper) 05Open→03Resolved a:03Aklapper Added. [16:03:50] (03PS3) 10Kosta Harlan: Provide a disposable secondary database for multi-DB tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/1309235 (https://phabricator.wikimedia.org/T342763) [16:25:18] (03PS1) 10Kosta Harlan: DNM: verify Special:UserRights interwiki test via a 2nd DB [integration/quibble] - 10https://gerrit.wikimedia.org/r/1309252 (https://phabricator.wikimedia.org/T342763) [16:30:13] (03open) 10pppery: Add newline at end of file [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/67 [16:31:32] (03update) 10pppery: Add newline at end of file [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/67 [16:33:45] (03close) 10pppery: Add newline at end of file [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/67 [16:34:26] (03open) 10pppery: Add newline at end of file [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/68 [16:36:58] (03update) 10aklapper: Hide Source Branch column in listed GitLab patches [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/61 (https://phabricator.wikimedia.org/T420059) [16:44:30] (03merge) 10aklapper: Add newline at end of file [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/68 (owner: 10pppery) [16:51:23] (03CR) 10CI reject: [V:04-1] DNM: verify Special:UserRights interwiki test via a 2nd DB [integration/quibble] - 10https://gerrit.wikimedia.org/r/1309252 (https://phabricator.wikimedia.org/T342763) (owner: 10Kosta Harlan) [16:54:11] 10Phabricator Translations Extension: Make WMF translations extension pass `bin/i18n validate` - https://phabricator.wikimedia.org/T412651#12106305 (10Pppery) As of https://gerrit.wikimedia.org/r/c/phabricator/translations/+/1309254: {P94782} That's a lot better. I see three classes of errors here: * Upstream... [17:16:54] dzahn: Re T240266, is there a "test instance" of Gerrit somewhere that we can use to experiment? [17:17:16] mutante: ^ [17:18:22] 06Release-Engineering-Team, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12106350 (10cmooney) [17:19:35] dancy: i think there was one in devtools at one point, lemme check [17:20:35] thx [17:21:20] dancy: it exists but it is shut down [17:21:20] there's a gerrit-bullseye VM in that project at least [17:21:26] cant boot it :p [17:21:48] gotcha. [17:21:49] nice timing that this comes up now. just the other day we were saying there is probably no point if nobody uses it :P [17:22:13] the reason it was shut down: because it cant be connected to cloud auth [17:22:18] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Request for mpopov to be added to acl*phabricator - https://phabricator.wikimedia.org/T431720#12106351 (10mpopov) Thank you very much! [17:22:35] so to make that work there would have to be some workaround for how to let users login [17:23:24] For the ticket in question, it would be good enough to be able to confirm the ssh host key on connect, and confirm Gerrit startup behavior depending on what keys are present. [17:23:38] no need to successfully log in. [17:24:42] ah, for that. hmmm [17:24:45] https://phabricator.wikimedia.org/T363196 [17:26:18] dancy: let me just say... you are allowed to use gerrit2002 for that test. this is "gerrit-spare.wikimedia.org" [17:26:35] please test in prod :) but it's just the _spare_ prod :) [17:26:46] haha, okay that'll do! [17:29:01] Alright. I'm looking through the Gerrit code now. [17:29:58] I was planning to write a patch to support multiple keys at once. [17:30:28] I am interested if hashar's claim that "once we add an openssh key other (serialized) keys will be ignored" is true. [17:31:20] well maybe it doesnt matter anymore since at this point we already have the unserialized version of the old key [17:31:44] (03PS2) 10Kosta Harlan: DNM: verify Special:UserRights interwiki test via a 2nd DB [integration/quibble] - 10https://gerrit.wikimedia.org/r/1309252 (https://phabricator.wikimedia.org/T342763) [17:31:50] I believe it will refuse to start if the java-style key exists alongside openssh style keys. https://github.com/GerritCodeReview/gerrit/blob/v3.10.6/java/com/google/gerrit/sshd/HostKeyProvider.java#L41-L87 [17:32:15] dancy: should I make that patch to support more than one key in puppet code? [17:32:33] Yes, ultimately that will be needed. And it will be important to absent the old style file [17:32:43] alright! [17:32:57] thanks for working on this very old ticket [17:33:04] Also noting that the filenames that it expects are fixed: https://github.com/GerritCodeReview/gerrit/blob/v3.10.6/java/com/google/gerrit/server/config/SitePaths.java#L108-L113 [17:33:17] oh, important point [17:33:19] You're welcome! hash.ar asks for assistance moving it along. [17:33:24] *asked [17:33:56] :) it's because I asked about it but we did not have the time to tackle both this week [17:34:22] the names being fixed means we should do some cleanup in the private repo [17:34:31] there are a bunch of keys there [17:34:46] from previous attempts to solve this [17:35:14] for example there is already a key with that exact name ssh_host_rsa_key but it's 2048 bit instead of 1024 [17:42:12] PROBLEM - Host contint2003 is DOWN: PING CRITICAL - Packet loss = 100% [17:43:32] RECOVERY - Host contint2003 is UP: PING OK - Packet loss = 0%, RTA = 31.65 ms [17:50:07] ^ reboots [19:41:11] mutante: it is not claim, that is how the Gerrit code behave :] [19:42:06] if there is a file with the java serialized keys, it does not bother looking at other files that holds the class base64 encoded keys. At least that was the case when I have looked at how Gerrit handled it at the time [19:42:50] thank you dancy for stepping in! [19:43:18] You got it! [20:00:50] and I think my theory is once the RSA key is in an open ssh format (ssh_rsa_key file name?) [20:01:08] we could then add the other more modern algos [20:01:41] and I imagine Gerrit SSH would offer them during the key negotiation and might be added on the client side [20:01:46] or not. I don't know the behavior [20:02:02] I imagine the client would ask for verification of the fingerprint [20:02:24] so tldr, I don't quite know how to migrate the myriad of clients in a way that is smooth enough [20:03:00] Gerrit will advertise host keys in a specific order: https://github.com/GerritCodeReview/gerrit/blob/v3.10.6/java/com/google/gerrit/sshd/SshDaemon.java#L446-L454 [20:03:14] (or which algo Zuul obsolete Paramiko supporrts) [20:03:38] it might be possible to test the behavior live using the gerrit-spare [20:03:54] Nod. That was suggested earlier. [20:04:09] neat! [20:05:28] and the whole reason for my push back is avoiding having to deal with a ton of support requests and fighting fires all over the place :b [20:05:35] Agreed [20:05:41] It's the right patch [20:05:43] *path [20:05:50] cool that reassure me [20:26:39] dancy: there is also a javascript message of the day plugin to make some announce via the web UI. https://gerrit.wikimedia.org/g/operations/software/gerrit/+/refs/heads/deploy/wmf/stable-3.10/plugins/wm-motd.js [20:26:42] deployable via scap [20:27:06] mutante: ^ in case you need add a message banner in Gerrit web UI, similar to Gitlab [20:27:47] configurable via the 3 constants at the top, merge, scap deploy, done (no need to restart Gerrit, it serves js files directly) [20:27:56] that was Antoine's brain dump [20:39:49] 10Beta-Cluster-Infrastructure, 07Kubernetes: Set up new Zotero on beta cluster - https://phabricator.wikimedia.org/T212549#12106966 (10bd808) 05Open→03Resolved Happened sometime before {T306049}. The versions of citoid and zotero running on deployment-docker-citoid02 are ancient, but {T380165} exists t... [20:41:31] 10Beta-Cluster-Infrastructure, 10Citoid: Update citoid service on beta - https://phabricator.wikimedia.org/T380165#12106975 (10bd808) This has lingered for a very long time. It should be possible to do something as part of {T401839} where the docker hosts need to be replaced too. [20:45:57] 10Beta-Cluster-Infrastructure: cswiki beta has incorrect wgGENewcomerTasksRemoteApiUrl - https://phabricator.wikimedia.org/T238321#12106984 (10bd808) `lang=shell-session bd808@deployment-deploy04.deployment-prep.eqiad1:~$ mwscript shell.php cswiki Psy Shell v0.12.24 (PHP 8.3.31 — cli) by Justin Hileman > $wgGENe... [21:29:18] 10Continuous-Integration-Infrastructure: Add dark mode to integration.wikimedia.org - https://phabricator.wikimedia.org/T431661#12107064 (10bd808) > I find it a bit odd that while we put a lot of effort into making wikis dark mode-compatible, this simple website (which already uses Codex, so should be easy to ha... [23:07:04] (03merge) 10bd808: Support Assisted-by as a git trailer [repos/ci-tools/commit-message-validator] - 10https://gitlab.wikimedia.org/repos/ci-tools/commit-message-validator/-/merge_requests/22 (https://phabricator.wikimedia.org/T427387) (owner: 10kharlan) [23:18:05] (03open) 10bd808: HISTORY: Prep for 2.3.0 release [repos/ci-tools/commit-message-validator] - 10https://gitlab.wikimedia.org/repos/ci-tools/commit-message-validator/-/merge_requests/23 [23:20:11] (03merge) 10bd808: HISTORY: Prep for 2.3.0 release [repos/ci-tools/commit-message-validator] - 10https://gitlab.wikimedia.org/repos/ci-tools/commit-message-validator/-/merge_requests/23 [23:48:01] (03open) 10bd808: ci: use v2.3.0 in CI/CD template [repos/ci-tools/commit-message-validator] - 10https://gitlab.wikimedia.org/repos/ci-tools/commit-message-validator/-/merge_requests/24 (https://phabricator.wikimedia.org/T427387) [23:49:56] (03merge) 10bd808: ci: use v2.3.0 in CI/CD template [repos/ci-tools/commit-message-validator] - 10https://gitlab.wikimedia.org/repos/ci-tools/commit-message-validator/-/merge_requests/24 (https://phabricator.wikimedia.org/T427387)