[00:36:19] 10Scap, 10MediaWiki-Configuration: Scap reliance on extension-list in turn requires extensions to be deployed to WMF production before being deployed to the Beta Cluster - https://phabricator.wikimedia.org/T125678#11634890 (10bd808) https://www.mediawiki.org/w/index.php?title=Writing_an_extension_for_deploymen... [00:55:09] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T413808#11634913 (10SecurityPatchBot) [01:06:32] 10Scap, 10MediaWiki-Configuration: Consider using `--extension-dir` instead of `--list-file` with mergeMessageFileList.php - https://phabricator.wikimedia.org/T417954 (10bd808) 03NEW [01:15:39] 10Scap, 10MediaWiki-Configuration: Scap reliance on extension-list in turn requires extensions to be deployed to WMF production before being deployed to the Beta Cluster - https://phabricator.wikimedia.org/T125678#11634935 (10bd808) 05In progress→03Resolved We have at this point determined that having... [07:01:20] 06Release-Engineering-Team, 10Scap, 06collaboration-services: scap error on Gerrit first setup - https://phabricator.wikimedia.org/T417767#11635174 (10ABran-WMF) >>! In T417767#11632455, @Dzahn wrote: >> the last blocker seems to come from the puppet5 CA > > This sounds related to the recent work by inf... [07:38:42] FIRING: [2x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#gerrit2003:29418 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:38:55] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T417968 (10phaultfinder) 03NEW [07:41:56] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit1003 - https://phabricator.wikimedia.org/T417246#11635234 (10ABran-WMF) 05Stalled→03In progress Once gerrit1003 has been repooled, replication_log raised the following issue: ` [2026-02-20 07:38:34,958] Cannot replicate to gerrit@gerri... [07:43:42] RESOLVED: [2x] ProbeDown: Service gerrit2003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#gerrit2003:29418 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:44:24] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T417968#11635240 (10ABran-WMF) →14Duplicate dup:03T417967 [07:44:29] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T417968#11635244 (10ABran-WMF) →14Duplicate dup:03T417966 [07:44:39] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T417968#11635245 (10ABran-WMF) [07:44:45] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit1003 - https://phabricator.wikimedia.org/T417246#11635246 (10ABran-WMF) [07:45:34] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit1003 - https://phabricator.wikimedia.org/T417246#11635248 (10ABran-WMF) [07:56:59] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit1003 - https://phabricator.wikimedia.org/T417246#11635287 (10ABran-WMF) after a fix ad-hoc: ` gerrit@gerrit2003:~$ diff ~/.ssh/known_hosts ~/.ssh/known_hosts.bak 3a4 > gerrit1003.wikimedia.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDbcAskAy... [08:11:21] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit1003 - https://phabricator.wikimedia.org/T417246#11635298 (10ABran-WMF) 05In progress→03Resolved This can be considered resolved, replication is running [08:58:25] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Zuul, 06collaboration-services, and 2 others: Gerrit events not received by Zuul due to TCP Proxy timeout (CI is not triggered for some patches) - https://phabricator.wikimedia.org/T417497#11635380 (10hashar) Turns out Zuul is still being disconnected af... [09:32:36] 10Continuous-Integration-Config, 10MediaWiki-Codesniffer, 10MediaWiki-Core-Tests: Enforce usage of MediaWikiUnitTestCase and MediaWikiIntegrationTestCase in MW PHPUnit Test - https://phabricator.wikimedia.org/T319536#11635469 (10Tacsipacsi) If the method would be a no-op, wouldn’t an [[https://www.php.net/ma... [10:21:38] 10Continuous-Integration-Infrastructure, 06Infrastructure-Foundations: Upgrade releng/operations-puppet CI image from Bullseye to Bookworm - https://phabricator.wikimedia.org/T417976 (10hashar) 03NEW [10:38:52] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11635561 (10ABran-WMF) 05Stalled→03Open [10:53:45] (03open) 10taavi: monitoring: Link to PAWS Pywikibot update docs [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/121 [10:53:50] (03update) 10taavi: monitoring: Link to PAWS Pywikibot update docs [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/121 [10:54:59] (03approved) 10filippo: monitoring: Link to PAWS Pywikibot update docs [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/121 (owner: 10taavi) [10:55:10] (03merge) 10taavi: monitoring: Link to PAWS Pywikibot update docs [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/121 [11:30:00] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T413808#11635700 (10Dreamy_Jazz) [12:30:20] 10Continuous-Integration-Config, 06Release-Engineering-Team (Priority Backlog 📥), 10Browser Test Platform, 10Wikibase (3rd party installations), and 3 others: Move some Wikibase selenium tests to a standalone job - https://phabricator.wikimedia.org/T287582#11635819 (10hashar) @WMDE-leszek sorry I had a ver... [12:35:58] 10Continuous-Integration-Config, 06Release-Engineering-Team (Priority Backlog 📥), 10Browser Test Platform, 10Wikibase (3rd party installations), and 3 others: Move some Wikibase selenium tests to a standalone job - https://phabricator.wikimedia.org/T287582#11635824 (10hashar) For the later actions, we'd wa... [12:40:45] (03PS24) 10Hashar: Use standalone jobs for Wikibase Selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/676107 (https://phabricator.wikimedia.org/T287582) [12:42:30] (03CR) 10Hashar: "Rebased to resolve conflicts with:" [integration/config] - 10https://gerrit.wikimedia.org/r/676107 (https://phabricator.wikimedia.org/T287582) (owner: 10Hashar) [13:43:27] I am fed up by OpenSearch search system really [13:43:38] that is such an antipattern of doom [14:25:46] apologies if this is a better question for service ops, but I have a gitlab problem: I changed my SSH key yesterday and I can't seem to pull or push. Fingerprint`ssh-keygen -E md5 -l -v -f ~/.ssh/bking_other_ed25519.pub [14:25:47] 256 MD5:6e:62:cf:c6:22:e2:cf:43:f3:3c:15:e9:2c:e9:f4:7c` seems to match`6e:62:cf:c6:22:e2:cf:43:f3:3c:15:e9:2c:e9:f4:7c` from the gitlab WebUI, any ideas? [14:26:28] inflatador: does `ssh -v git@gitlab.wikimedia.org` show the right key being presented? [14:28:29] (03PS1) 10Jforrester: Zuul: Clean up a few to-be-deployed comments, now done [integration/config] - 10https://gerrit.wikimedia.org/r/1240986 [14:28:29] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/Monstranto] Move out of Wikimedia prod section [integration/config] - 10https://gerrit.wikimedia.org/r/1240987 [14:28:29] (03PS1) 10Jforrester: Zuul: Split out a new 'headed to prod' section [integration/config] - 10https://gerrit.wikimedia.org/r/1240988 [14:28:29] (03PS1) 10Jforrester: [DNM] Zuul: Enforce PHP 8.4 in MW extensions and skins for development branch [integration/config] - 10https://gerrit.wikimedia.org/r/1240989 (https://phabricator.wikimedia.org/T386108) [14:28:34] cdanis is it NOT offering the right key. Thanks, I think I can fix that in SSH config [14:28:47] 🫡 [14:30:53] confirming that did the trick! thanks again [14:31:27] (03CR) 10Jforrester: [C:03+2] Zuul: Clean up a few to-be-deployed comments, now done [integration/config] - 10https://gerrit.wikimedia.org/r/1240986 (owner: 10Jforrester) [14:31:30] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/Monstranto] Move out of Wikimedia prod section [integration/config] - 10https://gerrit.wikimedia.org/r/1240987 (owner: 10Jforrester) [14:31:35] (03CR) 10Jforrester: [C:03+2] Zuul: Split out a new 'headed to prod' section [integration/config] - 10https://gerrit.wikimedia.org/r/1240988 (owner: 10Jforrester) [14:33:00] (03Merged) 10jenkins-bot: Zuul: Clean up a few to-be-deployed comments, now done [integration/config] - 10https://gerrit.wikimedia.org/r/1240986 (owner: 10Jforrester) [14:33:11] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Monstranto] Move out of Wikimedia prod section [integration/config] - 10https://gerrit.wikimedia.org/r/1240987 (owner: 10Jforrester) [14:33:18] (03Merged) 10jenkins-bot: Zuul: Split out a new 'headed to prod' section [integration/config] - 10https://gerrit.wikimedia.org/r/1240988 (owner: 10Jforrester) [14:35:39] !log Zuul: [mediawiki/extensions/Monstranto] Move out of Wikimedia prod section [14:35:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:53:03] (03PS1) 10Jforrester: Zuul: [3d2png] Add basic Node CI at version 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1240998 [15:23:09] 06Gerrit-Privilege-Requests, 06Security-Team: Request membership in operations group for Rsilvola - https://phabricator.wikimedia.org/T418004 (10Rsilvola) 03NEW [15:23:43] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE: Request membership in operations group for Rsilvola - https://phabricator.wikimedia.org/T418004#11636593 (10Rsilvola) [15:33:09] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for [kuzsh] - https://phabricator.wikimedia.org/T418000#11636623 (10Aklapper) 05Open→03Resolved a:03Aklapper Welcome! I've approved your account. Please also [link your LDAP account to your Phabric... [15:33:56] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Wyslijp16 Toolforge - https://phabricator.wikimedia.org/T417394#11636627 (10Aklapper) 05Open→03Stalled [15:34:44] 10Continuous-Integration-Config, 10Shellbox, 07PHP 8.5 support: Test Shellbox on PHP 8.5 - https://phabricator.wikimedia.org/T416064#11636629 (10Jdforrester-WMF) 05Open→03Resolved a:03Reedy [15:35:35] 10Continuous-Integration-Config, 07Epic, 07PHP 8.5 support: Make PHP 8.5 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T411814#11636631 (10Jdforrester-WMF) [15:38:21] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11636635 (10taavi) [15:39:54] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11636638 (10taavi) +2 in deployment-charts is linked to the ability to actually deploy those changes, and as far as I can tell... [15:40:57] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T413807#11636639 (10dancy) 05Open→03Resolved [15:48:34] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11636661 (10Rsilvola) Ah, well this clarifies the situation. Thanks @taavi! [15:57:58] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: NDA ticket access for user trueg - https://phabricator.wikimedia.org/T417991#11636674 (10Aklapper) 05Open→03Resolved a:03Aklapper Done. [16:10:04] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: "Copy" button on Jenkins console output page copies "null" in clipboard - https://phabricator.wikimedia.org/T410633#11636735 (10thcipriani) 05Open→03Resolved a:03thcipriani Fix was backported to https://www.jenkins.io/changelog/2.541.1... [16:31:42] 10Gerrit, 06collaboration-services: Gerrit switchover between secondary instances - https://phabricator.wikimedia.org/T406334#11636796 (10ABran-WMF) [16:31:45] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Reimage gerrit2002 - https://phabricator.wikimedia.org/T417247#11636795 (10ABran-WMF) [17:13:23] 06Gerrit-Privilege-Requests: Request membership in wmf-deployment group for alex.sanford - https://phabricator.wikimedia.org/T418015 (10ASanford-WMF) 03NEW [17:13:34] (03CR) 10Dmaza: Zuul: [3d2png] Add basic Node CI at version 20 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1240998 (owner: 10Jforrester) [17:13:47] !log Unblock 122.187.64.0/18 (T417964) [17:13:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:24:03] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11637038 (10sbassett) (we just did this for Alex [T418015], if you'd like an example request to work from) [17:28:40] 06Gerrit-Privilege-Requests, 10SRE-Access-Requests: Request membership in wmf-deployment group for alex.sanford - https://phabricator.wikimedia.org/T418015#11637047 (10Dzahn) [17:28:56] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11637048 (10Dzahn) [17:29:44] 06Gerrit-Privilege-Requests, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for alex.sanford - https://phabricator.wikimedia.org/T418015#11637063 (10Dzahn) Being able to +2 in the repo should be combined with being able to actually deploy changes. Which then turns it into a request... [17:29:54] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11637065 (10Dzahn) Being able to +2 in the repo should be combined with being able to actually deploy... [17:40:53] 06Gerrit-Privilege-Requests, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for alex.sanford - https://phabricator.wikimedia.org/T418015#11637097 (10ASanford-WMF) I am already in the `deployment` shell group - https://phabricator.wikimedia.org/source/operations-puppet/browse/product... [17:49:04] 06Gerrit-Privilege-Requests, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for alex.sanford - https://phabricator.wikimedia.org/T418015#11637118 (10Dzahn) @ASanford-WMF Oh! I see, yes. Sorry about that. I just added you to the Gerrit group as requested. [17:50:50] 06Gerrit-Privilege-Requests, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for alex.sanford - https://phabricator.wikimedia.org/T418015#11637119 (10ASanford-WMF) 05Open→03Resolved a:03ASanford-WMF Great, thanks! 🙌 [18:03:00] (03CR) 10Dmaza: [C:03+1] "I don't have +2 on this repo" [integration/config] - 10https://gerrit.wikimedia.org/r/1240998 (owner: 10Jforrester) [18:48:38] (03open) 10matmarex: Remove leading newline from the "what" parameters in the calendar [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/235 [19:06:01] (03CR) 10Hashar: "@jforrester@wikimedia.org we have dropped 3d2png from CI when we stopped supporting NodeJS 6." [integration/config] - 10https://gerrit.wikimedia.org/r/1240998 (owner: 10Jforrester) [19:10:52] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Create an OpenSearch dashboard for Gerrit sshd logs - https://phabricator.wikimedia.org/T417753#11637276 (10hashar) self note: I get the disconection for a given user with Lucene search query: ` message:gerrit.stream-events AND user.name:jenkins... [19:55:31] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11637409 (10Dzahn) a:03thcipriani Hi Tyler, they would need both shell "deployment" and gerrit "wmf... [19:59:26] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11637416 (10Dzahn) Well.. wait. I am saying that but it's actually just about access to deployment... [20:06:41] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11637421 (10thcipriani) k8s deployment does require `deployment` for read access to the configs to us... [20:08:01] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11637423 (10Dzahn) a:05thcipriani→03None Thanks for that confirmation and approval:) [20:14:26] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11637434 (10Dzahn) @Rsilvola So.. there are 2 things you need. The shell access to the deployment ser... [20:17:06] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11637440 (10Dzahn) [20:20:27] 06Gerrit-Privilege-Requests, 06Release-Engineering-Team, 06Security-Team, 06SRE, 10SRE-Access-Requests: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11637446 (10Dzahn) @Rsilvola I copied the template for shell access requests into the ticket descript... [21:47:49] !log Unblock 168.184.84.0/24 (T418020) [21:47:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:17:08] 10Phabricator, 06collaboration-services, 13Patch-For-Review: Replace deprecated Phabricator Conduit API call by @ProdPasteBot with its stable equivalent - https://phabricator.wikimedia.org/T410572#11637613 (10Aklapper) a:03Aklapper [22:29:57] 10Phabricator, 10ReleaseTaggerBot: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402453#11637624 (10Aklapper) [23:00:03] maintenance-disconnect-full-disks build 782868 integration-agent-docker-1042 (/: 36%, /srv: 99%, /var/lib/docker: 36%): OFFLINE due to disk space [23:05:03] maintenance-disconnect-full-disks build 782869 integration-agent-docker-1042 (/: 36%, /srv: 73%, /var/lib/docker: 37%): RECOVERY disk space OK [23:23:26] 10Gerrit, 06Release-Engineering-Team, 14Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repo... - https://phabricator.wikimedia.org/T246763#11637703 [23:26:01] 10Phabricator, 10ReleaseTaggerBot: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402453#11637719 (10Aklapper) [23:29:38] 10Phabricator (2026-02-17), 06Release-Engineering-Team (Doing 😎): Change Maniphest AdvancedSearch default for "Group by" to "None" and for "Order by" to "Updated" - https://phabricator.wikimedia.org/T404843#11637725 (10Aklapper) 05Open→03Resolved and deployed [23:37:06] 10Phabricator: Uninstall Countdown (Phabricator application) - https://phabricator.wikimedia.org/T418033 (10Aklapper) 03NEW p:05Triage→03Low [23:37:19] 10Phabricator: Uninstall Countdown (Phabricator application) - https://phabricator.wikimedia.org/T418033#11637740 (10Aklapper) [23:38:30] 10Phabricator: Uninstall Countdown (Phabricator application) - https://phabricator.wikimedia.org/T418033#11637741 (10Aklapper) [23:54:15] 10Continuous-Integration-Config, 10MediaWiki-Codesniffer, 10MediaWiki-Core-Tests: Enforce usage of MediaWikiUnitTestCase and MediaWikiIntegrationTestCase in MW PHPUnit Test - https://phabricator.wikimedia.org/T319536#11637745 (10Daimona) Yes, I think attributes could work too! I'm also not sure how much we r...