[06:17:49] 10WikimediaDebug, 06MW-Interfaces-Team, 06serviceops-deprecated, 10Wikidata Change Dispatching & Watchlists, and 3 others: Add support for xhgui for jobs - https://phabricator.wikimedia.org/T292382#12122417 (10aaron) p:05Triageβ†’03Low [08:27:42] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10QS-Test-Automation, 13Patch-For-Review, 06Quality-and-Test-Engineering-Team (Quality Engineering): Better prevent changes in MW core from breaking WMF production extensions - https://phabricator.wikimedia.org/T404403#12122703 (10Peter... [09:10:40] (03PS1) 10Jforrester: Docker: [php83] Update PHP to 8.3.32 [integration/config] - 10https://gerrit.wikimedia.org/r/1310981 (https://phabricator.wikimedia.org/T431099) [09:10:41] (03PS1) 10Jforrester: jjb: Update php83 jobs to PHP 8.3.32 [integration/config] - 10https://gerrit.wikimedia.org/r/1310982 (https://phabricator.wikimedia.org/T431099) [09:10:51] (03CR) 10Jforrester: [C:03+2] Docker: [php83] Update PHP to 8.3.32 [integration/config] - 10https://gerrit.wikimedia.org/r/1310981 (https://phabricator.wikimedia.org/T431099) (owner: 10Jforrester) [09:12:43] (03Merged) 10jenkins-bot: Docker: [php83] Update PHP to 8.3.32 [integration/config] - 10https://gerrit.wikimedia.org/r/1310981 (https://phabricator.wikimedia.org/T431099) (owner: 10Jforrester) [09:13:12] !log Docker: [php83] Update PHP to 8.3.32 for T431099 [09:13:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:14:44] 10Continuous-Integration-Config, 10CirrusSearch, 10Wikidata, 10Wikidata Lexicographical data, and 4 others: LexemeFullTextQueryBuilderTest::testSearchElastic CI test errors - https://phabricator.wikimedia.org/T428975#12122898 (10karapayneWMDE) [09:25:35] (03CR) 10Jforrester: [C:03+2] jjb: Update php83 jobs to PHP 8.3.32 [integration/config] - 10https://gerrit.wikimedia.org/r/1310982 (https://phabricator.wikimedia.org/T431099) (owner: 10Jforrester) [09:27:52] (03Merged) 10jenkins-bot: jjb: Update php83 jobs to PHP 8.3.32 [integration/config] - 10https://gerrit.wikimedia.org/r/1310982 (https://phabricator.wikimedia.org/T431099) (owner: 10Jforrester) [09:45:09] (03update) 10jforrester: Provide initial trixie images [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/81 [09:48:52] (03merge) 10jforrester: Provide initial trixie images [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/81 [10:04:26] (03open) 10jforrester: bookworm-php-sury: Re-build PHP images for latest point releases [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/110 (https://phabricator.wikimedia.org/T431099) [10:09:37] (03open) 10jforrester: releases: Bump stylelint-config-wikimedia to 0.19.3 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/134 [10:10:29] (03merge) 10jforrester: releases: Bump stylelint-config-wikimedia to 0.19.3 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/134 [10:20:43] (03PS3) 10Abijeet Patro: Add rake-ruby3.3 image and switch translatewiki to it [integration/config] - 10https://gerrit.wikimedia.org/r/1310997 (https://phabricator.wikimedia.org/T424966) [11:45:28] (03CR) 10Nikerabbit: "I'm not familiar enough to review, but I can vouch we want this change." [integration/config] - 10https://gerrit.wikimedia.org/r/1310997 (https://phabricator.wikimedia.org/T424966) (owner: 10Abijeet Patro) [12:33:21] 10GitLab (Pipeline Services Migration🐀), 10Wikispeech-Jobrunner, 10Wikispeech-Text-to-Speech: Migrate Wikispeech services to GitLab - https://phabricator.wikimedia.org/T344741#12123911 (10Jelto) All projections have been archived! ` mediawiki/services/wikispeech/mary-tts mediawiki/services/wikispeech/ahotts... [12:42:25] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10BlueSpice: BlueSpice/HalloWelt force merging to branches and messing up mediawiki/extensions.git - https://phabricator.wikimedia.org/T431845#12123973 (10Reedy) ` fatal: remote error: want b3f1bae1ecb096b1726ce955d25f199f43dc2192 not valid fatal:... [12:44:44] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316#12123998 (10Jelto) [12:50:48] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316#12124028 (10Jelto) 05Openβ†’03Resolved All projects have been either updated or ar... [13:00:34] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10BlueSpice: BlueSpice/HalloWelt force merging to branches and messing up mediawiki/extensions.git - https://phabricator.wikimedia.org/T431845#12124067 (10Hslater) 05Openβ†’03Resolved a:03Hslater Interesting; thank you. [13:02:08] 06Release-Engineering-Team, 10Cloud-VPS, 06collaboration-services, 06Data-Persistence, and 8 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12124086 (10cmooney) 05Openβ†’03Resolved [13:33:21] Project mediawiki-core-doxygen build #21508: 04FAILURE in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21508/ [14:05:58] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10BlueSpice: BlueSpice/HalloWelt force merging to branches and messing up mediawiki/extensions.git - https://phabricator.wikimedia.org/T431845#12124433 (10Reedy) 05Resolvedβ†’03Open Sorry, no, you can't just close it like that. What does "Intere... [14:13:24] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10BlueSpice: BlueSpice/HalloWelt force merging to branches and messing up mediawiki/extensions.git - https://phabricator.wikimedia.org/T431845#12124454 (10Hslater) It meant I understood the report and appreciated you bringing it to my attention. No... [14:33:04] Yippee, build fixed! [14:33:04] Project mediawiki-core-doxygen build #21509: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21509/ [14:52:43] 10Beta-Cluster-Infrastructure, 10Observability-Logging: scap on deployment-deploy04 failing to reach logging-logstash-04.logging.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T432254 (10bd808) 03NEW [14:57:39] 10Beta-Cluster-Infrastructure, 10Observability-Logging: scap on deployment-deploy04 failing to reach logging-logstash-04.logging.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T432254#12124787 (10bd808) Horizon console log for logging-logstash-04.logging.eqiad1.wikimedia.cloud shows only an OOM cra... [15:03:54] 10Beta-Cluster-Infrastructure, 10Observability-Logging: scap on deployment-deploy04 failing to reach logging-logstash-04.logging.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T432254#12124818 (10bd808) The reboot brought https://beta-logs.wmcloud.org/ back online. Ssh is working. And a manual curl... [15:07:07] 10Beta-Cluster-Infrastructure, 10Observability-Logging: scap on deployment-deploy04 failing to reach logging-logstash-04.logging.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T432254#12124832 (10bd808) @colewhite There were some blips with this server earlier too (maybe a full 24 hours?) that seem... [15:10:45] 10Beta-Cluster-Infrastructure, 10Observability-Logging: scap on deployment-deploy04 failing to reach logging-logstash-04.logging.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T432254#12124845 (10bd808) p:05Triageβ†’03Low Marking as low because it is just a post-incident investigation task now. [15:26:35] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12124881 (10Arnoldokoth) There is one issue though. Avatars are not loadin... [15:27:06] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Anirudh_K - https://phabricator.wikimedia.org/T432257 (10Anirudh_KK) 03NEW [15:32:35] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for Anirudh_K - https://phabricator.wikimedia.org/T432257#12124900 (10brennen) 05Openβ†’03Resolved a:03brennen [15:57:34] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12124983 (10Arnoldokoth) Actually, the dump might have caused file integr... [16:14:35] 10Phabricator, 06Release-Engineering-Team (Doing 😎), 06ServiceOps new: Custom task form for Production Readiness Checklist process (SCROLL) - https://phabricator.wikimedia.org/T429692#12125116 (10jijiki) @Aklapper many thanks! [16:34:50] 10Phabricator Translations Extension: Make WMF translations extension pass `bin/i18n validate` - https://phabricator.wikimedia.org/T412651#12125215 (10Pppery) ` The following validation errors were found: The locale `ar` defines a translation for the string "Norwegian", however that string does not appear to be... [17:44:23] 10Gerrit, 06collaboration-services: Rename of a repository is not replicated - https://phabricator.wikimedia.org/T398401#12125512 (10Dzahn) based on the question on T423253#12002143 if this was caused by our mismatching system user (gerrit vs gerrit2) I made another test and create a project just to rename it... [17:48:28] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10BlueSpice: BlueSpice/HalloWelt force merging to branches and messing up mediawiki/extensions.git - https://phabricator.wikimedia.org/T431845#12125544 (10bd808) Is there a reasonable use case for these repos to have force merge enabled in gerrit?... [18:02:50] 10Gerrit, 06collaboration-services: Rename of a repository is not replicated - https://phabricator.wikimedia.org/T398401#12125606 (10Dzahn) This is because we renamed the shell user from gerrit2 to gerrit (In hindsight I wish we hadn't) but did not also rename the Gerrit-internal user in Notesdb. That is still... [18:06:41] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10BlueSpice: BlueSpice/HalloWelt force merging to branches and messing up mediawiki/extensions.git - https://phabricator.wikimedia.org/T431845#12125616 (10Dzahn) I suggest since the issue has been acknowledged we just disable it in Gerrit config an... [18:07:07] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10BlueSpice: BlueSpice/HalloWelt force merging to branches and messing up mediawiki/extensions.git - https://phabricator.wikimedia.org/T431845#12125619 (10Hslater) Thanks for the constructive feedback. I've revoked the explicit force-push permissi... [18:13:43] 10Gerrit, 06collaboration-services: Rename of a repository is not replicated - https://phabricator.wikimedia.org/T398401#12125666 (10Dzahn) ` [Primary Server] └─ Linux Shell User (gerrit) └─ Reads key from /home/gerrit/.ssh/id_rsa β”‚ β–Ό (Network via Port 29418) [Target Serv... [18:15:43] 10GerritBot, 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Gerritbot is not commenting on tasks with merged patches - https://phabricator.wikimedia.org/T432168#12125681 (10matmarex) Thanks! [18:29:04] 10GitLab (Project and group requests), 06Release-Engineering-Team: GitLab Private Repository Request for: product-safety-and-integrity/wikimedia-anti-abuse-private - https://phabricator.wikimedia.org/T432281 (10kostajh) 03NEW [18:30:10] 10GitLab (Project and group requests), 06Release-Engineering-Team, 05DE4.1 Platform capacity, 06Product Safety and Integrity (Sprint Venus Flytrap (Jun 29 - Jul 17)): GitLab Private Repository Request for: product-safety-and-integrity/wikimedia-anti-abuse-... - https://phabricator.wikimedia.org/T432281#12125751 [18:38:27] 10GitLab (Project and group requests), 06Release-Engineering-Team, 05DE4.1 Platform capacity, 07Essential-Work, 06Product Safety and Integrity (Sprint Venus Flytrap (Jun 29 - Jul 17)): GitLab Private Repository Request for: product-safety-and-integrity/... - https://phabricator.wikimedia.org/T432281#12125804 [18:40:23] 10GitLab (Project and group requests), 06Release-Engineering-Team, 05DE4.1 Platform capacity, 07Essential-Work, 06Product Safety and Integrity (Sprint Venus Flytrap (Jun 29 - Jul 17)): GitLab Private Repository Request for: product-safety-and-integrity/... - https://phabricator.wikimedia.org/T432281#12125814 [18:55:54] (03merge) 10jforrester: bookworm-php-sury: Re-build PHP images for latest point releases [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/110 (https://phabricator.wikimedia.org/T431099) [18:58:02] !log dev-images: Re-build PHP images for latest point releases, for T431099 [18:58:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:34:29] could someone here add me as an invitee to the train log triage recurring event? it's gone from my calendar for some reason. thanks! [19:34:56] I mean, gone as something I can ack to and get a reminder about. [19:47:45] apergos: Done. Hope that works? [19:48:39] I don't see it (or an invite in my mail either). hrm [19:49:17] maybe it takes a few minutes for the change to propagate? [19:50:34] it's there! thank you. [20:11:28] 10Phabricator: Potentially disable H488 - https://phabricator.wikimedia.org/T431737#12126015 (10Pskyechology) 05Openβ†’03Resolved a:03Pskyechology Wasn't aware of this limitation; disabled it. [20:21:46] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12126025 (10Dzahn) At this point; are we just waiting for upstream? Or was there anything else I could do here? [20:31:32] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [20:31:42] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T432292 (10wmcs-alerts) 03NEW [21:19:13] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T432292#12126157 (10bd808) `lang=shell-session gitpuppet@deployment-puppetserver-1:/srv/git/operations/puppet$ /usr/loc... [21:19:27] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T432292#12126158 (10bd808) p:05Triageβ†’03High [21:23:53] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T432292#12126178 (10bd808) `lang=git-conflict-markers <<<<<<< HEAD:modules/role/lib/puppet/functions/kafka_config.rb... [21:28:40] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T432292#12126213 (10bd808) The conflict comes from the merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/13... [23:23:06] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T432292#12126441 (10bd808) 05Openβ†’03In progress a:03bd808 I updated the patch in gerrit (https://gerrit.wikimedia... [23:26:32] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [23:46:48] 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Add RSS extension to PatchDemo - https://phabricator.wikimedia.org/T426130#12126506 (10jeena) @Danielyepezgarces is there a specific feed we should add to the allowlist settings for this? [23:54:23] 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Add RSS extension to PatchDemo - https://phabricator.wikimedia.org/T426130#12126523 (10Danielyepezgarces) I don't have a specific one in mind right now, but it would be good if it were easy to configure, so...