[01:50:18] 06Release-Engineering-Team, 10Catalyst, 10Cloud-VPS, 06tools-infrastructure-team: Restore ingress to patchdemo / catalyst - https://phabricator.wikimedia.org/T442664 (10brennen) 03NEW [04:44:10] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for shwetamishra - https://phabricator.wikimedia.org/T440798#12482795 (10Shwetamish12) Thank you for reviewing and approving my GitLab account activation request. I appreciate your time and assist... [06:02:48] (03CR) 10Hashar: [C:03+2] Zuul: Add Sanskardubey to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1359312 (owner: 10Urbanecm) [06:04:02] (03Merged) 10jenkins-bot: Zuul: Add Sanskardubey to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1359312 (owner: 10Urbanecm) [06:05:00] (03CR) 10Hashar: [C:03+2] "@tausheef.hassan@gmail.com that is the nicest commit message have seen in ages, and I see a lot of them every day 👍" [integration/config] - 10https://gerrit.wikimedia.org/r/1358329 (owner: 10Tausheef Hassan) [06:06:20] (03CR) 10CI reject: [V:04-1] Zuul: add Tausheef Hassan to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1358329 (owner: 10Tausheef Hassan) [06:07:56] (03PS2) 10Hashar: Zuul: add Tausheef Hassan to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1358329 (owner: 10Tausheef Hassan) [06:12:18] (03CR) 10Hashar: [C:03+2] Zuul: add Tausheef Hassan to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1358329 (owner: 10Tausheef Hassan) [06:13:24] (03Merged) 10jenkins-bot: Zuul: add Tausheef Hassan to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1358329 (owner: 10Tausheef Hassan) [06:15:00] (03CR) 10Hashar: [C:03+2] zuul: [mediawiki/extensions/Sanctions] Add DiscussionTools dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1358325 (owner: 10Lens0021) [06:16:17] (03Merged) 10jenkins-bot: zuul: [mediawiki/extensions/Sanctions] Add DiscussionTools dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1358325 (owner: 10Lens0021) [06:31:38] (03CR) 10Hashar: [C:03+2] "Why is VisualEditor coupled with AbuseFilter and SpamBlacklist escapes me 😊 But I guess that can be decoupled in the future or maybe ther" [integration/config] - 10https://gerrit.wikimedia.org/r/1357377 (owner: 10Fomafix) [06:32:54] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/VisualEditor] Add Phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1357377 (owner: 10Fomafix) [07:11:09] (03PS1) 10Hashar: Update Zuul plugin to align CSS styling [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1359738 (https://phabricator.wikimedia.org/T441684) [07:19:56] 06Release-Engineering-Team, 06Collaboration-Services, 06NFDI-Germany, 10Wikidata, and 3 others: Github actions 403 errors cloning from Gerrit - https://phabricator.wikimedia.org/T441387#12482937 (10hashar) 05Open→03Resolved a:05ABran-WMF→03hashar Some changes were made by @Jelto on Monday and a... [07:21:14] (03CR) 10Hashar: [C:03+2] Update Zuul plugin to align CSS styling [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1359738 (https://phabricator.wikimedia.org/T441684) (owner: 10Hashar) [07:21:51] (03Merged) 10jenkins-bot: Update Zuul plugin to align CSS styling [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1359738 (https://phabricator.wikimedia.org/T441684) (owner: 10Hashar) [07:37:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:37:44] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - https://phabricator.wikimedia.org/T442677 (10phaultfinder) 03NEW [07:39:05] 10Gerrit, 07Upstream: Gerrit Zuul plugin style mismatches related changes styling - https://phabricator.wikimedia.org/T441684#12482966 (10hashar) [07:42:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:52:43] 10Gerrit, 07Upstream: Gerrit Zuul plugin style mismatches related changes styling - https://phabricator.wikimedia.org/T441684#12482981 (10hashar) I havedeployed the zuul.jar update, Gerrit reloaded the plugin but the CSS changes did not reflect in the UI :-\ The deployed plugin has: - md5sum 55e1d105359500d5d... [08:02:41] 10Gerrit, 07Upstream: Gerrit Zuul plugin style mismatches related changes styling - https://phabricator.wikimedia.org/T441684#12482989 (10hashar) 05Open→03Resolved a:03hashar Looking at GET https://gerrit.wikimedia.org/r/plugins/zuul/static/zuul.js, my browser indicates it is cached :/ The response h... [08:17:25] 06Release-Engineering-Team, 10Catalyst, 10Cloud-VPS, 06tools-infrastructure-team: Restore ingress to patchdemo / catalyst - https://phabricator.wikimedia.org/T442664#12483035 (10taavi) I cannot look further at the moment why, but our audit script is flagging the presence of an outdated PHP 8.2 version on C... [08:34:32] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - https://phabricator.wikimedia.org/T442677#12483060 (10phaultfinder) [08:51:49] 10Continuous-Integration-Infrastructure (Zuul upgrade), 07Upstream: Zuul web API calls are cached - https://phabricator.wikimedia.org/T430462#12483109 (10hashar) p:05Triage→03Low To serve as a reference for later https://gerrit.wikimedia.org/r/c/operations/puppet/+/1307459/1/modules/zuul/files/zuul.wikimed... [08:52:03] 10Continuous-Integration-Infrastructure (Zuul upgrade), 07Upstream: Zuul web should emit cache headers (was: API calls are cached) - https://phabricator.wikimedia.org/T430462#12483111 (10hashar) [09:30:05] 10Gerrit, 13Patch-For-Review: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465#12483235 (10hashar) a:03hashar [09:42:19] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown (vrts1003) - https://phabricator.wikimedia.org/T442677#12483263 (10Jelto) [11:12:47] 10Gerrit, 06Collaboration-Services, 06SRE Observability, 13Patch-For-Review: Prometheus scrapping denied by Gerrit - https://phabricator.wikimedia.org/T441498#12483709 (10hnowlan) 05Open→03Resolved a:03hnowlan Disabling gerrit scraping in pontoon will hopefully have addressed this. [11:47:37] phabricator seems to be down? [11:49:14] ok, correction per -operation, a lot of things seem to be down [11:50:06] FIRING: ProbeDown: Service gerrit1003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#gerrit1003:29418 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:50:47] FIRING: [2x] ProbeDown: Service gerrit1003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:59:03] looks like phabricator appserver is back but db came up readonly [12:03:03] RESOLVED: [2x] ProbeDown: Service gerrit1003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:05:39] (phabricator is back up) [12:14:11] 06Release-Engineering-Team: Add chlod as deployer to the UTC morning backport window - https://phabricator.wikimedia.org/T442702 (10Chlod) 03NEW [12:17:30] (03update) 10chlod: deployments-calendar: add chlod to UTC morning [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/281 (https://phabricator.wikimedia.org/T442702) [12:17:58] (03update) 10chlod: deployments-calendar: add chlod to UTC morning [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/281 (https://phabricator.wikimedia.org/T442702) [12:19:11] (03approved) 10jforrester: deployments-calendar: add chlod to UTC morning [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/281 (https://phabricator.wikimedia.org/T442702) (owner: 10chlod) [12:19:13] (03update) 10jforrester: deployments-calendar: add chlod to UTC morning [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/281 (https://phabricator.wikimedia.org/T442702) (owner: 10chlod) [12:21:25] (03merge) 10jforrester: deployments-calendar: add chlod to UTC morning [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/281 (https://phabricator.wikimedia.org/T442702) (owner: 10chlod) [12:22:51] 06Release-Engineering-Team: Add chlod as deployer to the UTC morning backport window - https://phabricator.wikimedia.org/T442702#12483984 (10Chlod) 05Open→03Resolved a:03Jdforrester-WMF [12:23:03] 06Release-Engineering-Team: Add chlod as deployer to the UTC morning backport window - https://phabricator.wikimedia.org/T442702#12483986 (10Jdforrester-WMF) a:05Jdforrester-WMF→03Chlod This is landed, so will appear in the calendar starting next week if it's not broken. Welcome! [12:23:20] 06Release-Engineering-Team: Add chlod as deployer to the UTC morning backport window - https://phabricator.wikimedia.org/T442702#12483989 (10Jdforrester-WMF) Ha, you did the hard bit, I just pressed a button. :-) [12:24:02] 06Release-Engineering-Team: Add chlod as deployer to the UTC morning backport window - https://phabricator.wikimedia.org/T442702#12483991 (10Chlod) That's true, thanks! :D [13:36:21] 10GitLab (Project and group requests), 06Release-Engineering-Team, 06App Growth, 07Essential-Work: Finding right group for microsite project in GitLab - https://phabricator.wikimedia.org/T439904#12484336 (10Cklimas) @Dzahn thanks! It looks like I have permissions to do everything I'll need to on that repo.... [13:36:30] 10GitLab (Project and group requests), 06Release-Engineering-Team, 06App Growth, 07Essential-Work: Finding right group for microsite project in GitLab - https://phabricator.wikimedia.org/T439904#12484337 (10Cklimas) 05Stalled→03Resolved [14:45:24] gcifdivrket [15:01:48] !log `SET GLOBAL read_only = 0;` on deployment-db15.deployment-prep.eqiad1.wikimedia.cloud [15:01:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:04:01] (03CR) 10Hashar: "I have poked my team (release engineering) about it with the intent of having more persons able to review changes made to Fresh" [fresh] - 10https://gerrit.wikimedia.org/r/1347594 (https://phabricator.wikimedia.org/T439828) (owner: 10Hashar) [15:13:55] (03CR) 10Ahmon Dancy: "Love the idea!" [fresh] - 10https://gerrit.wikimedia.org/r/1347594 (https://phabricator.wikimedia.org/T439828) (owner: 10Hashar) [15:19:21] !log wmf-beta-update-all operation restored. [15:19:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:40:20] (03open) 10dancy: jobrunner: Handle mid-line io messages [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1385 [15:41:13] (03update) 10dancy: jobrunner: Handle mid-line io messages [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1385 [15:43:25] (03merge) 10dancy: jobrunner: Handle mid-line io messages [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1385 [15:44:39] (03open) 10dancy: spiderpig: Use one database engine for all apiserver sessions [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1386 [15:44:39] (03update) 10dancy: spiderpig: Use one database engine for all apiserver sessions [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1386 [15:44:58] (03update) 10dancy: spiderpig: Use one database engine for all apiserver sessions [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1386 [15:46:47] (03merge) 10dancy: spiderpig: Use one database engine for all apiserver sessions [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1386 [15:52:39] (03open) 10dancy: spiderpig: Add tests of the interaction and job log routes [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1387 [15:53:08] (03update) 10dancy: spiderpig: Add tests of the interaction and job log routes [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1387 [15:54:42] (03merge) 10dancy: spiderpig: Add tests of the interaction and job log routes [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1387 [15:55:54] (03open) 10dancy: spiderpig: Add tests of Interaction, JobLog and JobViewerPage [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1388 [15:56:33] (03update) 10dancy: spiderpig: Add tests of Interaction, JobLog and JobViewerPage [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1388 [15:58:57] (03merge) 10dancy: spiderpig: Add tests of Interaction, JobLog and JobViewerPage [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1388 [15:59:28] (03open) 10dancy: Release 4.307.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1389 [16:00:02] (03PS1) 10Hashar: Refactor tests framework [fresh] - 10https://gerrit.wikimedia.org/r/1347688 [16:01:39] (03merge) 10dancy: Release 4.307.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1389 [16:08:20] 06Release-Engineering-Team (Radar), 07Essential-Work, 10Test Kitchen (Experiment Platform Sprint 31): Determine if we can expand TK experiment deployment windows - https://phabricator.wikimedia.org/T440612#12485256 (10JVanderhoop-WMF) > Would it make more sense to do your experiment-status changes *immediate... [16:12:40] (03CR) 10Hashar: Consolidate fresh-node* scripts to a single file (032 comments) [fresh] - 10https://gerrit.wikimedia.org/r/1347594 (https://phabricator.wikimedia.org/T439828) (owner: 10Hashar) [16:12:45] (03PS4) 10Hashar: Consolidate fresh-node* scripts to a single file [fresh] - 10https://gerrit.wikimedia.org/r/1347594 (https://phabricator.wikimedia.org/T439828) [16:13:50] (03CR) 10Ahmon Dancy: [C:03+1] Consolidate fresh-node* scripts to a single file [fresh] - 10https://gerrit.wikimedia.org/r/1347594 (https://phabricator.wikimedia.org/T439828) (owner: 10Hashar) [16:18:00] 06Release-Engineering-Team, 10Scap, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485354 (10Milimetric) a:03tchin [16:18:23] 06Release-Engineering-Team, 10Scap, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485356 (10Milimetric) [16:48:29] 06Project-Admins: Create Phabricator projects for two Toolforge tools: Tool-whichtool and Tool-script-publisher - https://phabricator.wikimedia.org/T442737 (10Dev) 03NEW [16:56:07] 06Project-Admins: Create Phabricator projects for two Toolforge tools: Tool-whichtool and Tool-script-publisher - https://phabricator.wikimedia.org/T442737#12485648 (10Nemoralis) There is a "Create project" button on toolsadmin [16:59:58] 06Release-Engineering-Team, 10Scap, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485671 (10Jdforrester-WMF) This is probably not all being done by @tchin for all ~22 outstanding services across 10 teams. Maybe you w... [17:01:35] 06Release-Engineering-Team, 10Scap, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485675 (10taavi) [17:31:59] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12485815 (10Volans) If it helps I've run the audit for the whole project just now and those are the results: ` # With PHP unable to check the vers... [17:39:06] FIRING: WidespreadInstanceDown: Widespread instances down in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadInstanceDown [17:46:24] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12485893 (10taavi) >>! In T442482#12485815, @Volans wrote: > If it helps I've run the audit for the whole project just now and those are the result... [17:47:21] 06Release-Engineering-Team, 10Catalyst, 10Cloud-VPS, 06tools-infrastructure-team: Restore ingress to patchdemo / catalyst - https://phabricator.wikimedia.org/T442664#12485897 (10hashar) [17:51:39] (03PS2) 10Hashar: jjb: reenable timed and scm polling jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1357483 [17:54:10] (03PS1) 10Hashar: jjb: reenable timed and scm polling jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1359927 (https://phabricator.wikimedia.org/T442664) [17:55:41] (03CR) 10Hashar: [C:03+2] "@vwalters@wikimedia.org I have moved the restoration of the Catalyst jobs to a new change Iff4070e0f469f76eaab0bdf10f03df5495bad975 which " [integration/config] - 10https://gerrit.wikimedia.org/r/1357483 (owner: 10Hashar) [17:56:28] (03PS2) 10Hashar: jjb: reenable Catalys based timed jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1359927 (https://phabricator.wikimedia.org/T442664) [17:56:44] (03CR) 10Hashar: [C:04-1] "-1 pending T442664" [integration/config] - 10https://gerrit.wikimedia.org/r/1359927 (https://phabricator.wikimedia.org/T442664) (owner: 10Hashar) [17:56:58] (03Merged) 10jenkins-bot: jjb: reenable timed and scm polling jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1357483 (owner: 10Hashar) [17:57:44] 06Release-Engineering-Team, 10Scap, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12485957 (10karapayneWMDE) [17:57:46] !log integration: reenabled in Jenkins the jobs relaying on scm polling or on a timer # https://gerrit.wikimedia.org/r/c/integration/config/+/1357483/ [17:57:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:19:43] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Lexicographical data, 06Wikidata-Omega (The Board): OutOfBoundsException: Attempt to get non-existing setting "tmpEnableMobileEditingUIBetaFeature" (Beta Commons down) - https://phabricator.wikimedia.org/T441718#12486063 (10dduvall) >>! In T441718#12453... [18:22:07] 10GitLab (Account Approval), 10Tool-wikinewsie: Requesting GitLab account activation for Wikinewsie Columbia students - https://phabricator.wikimedia.org/T442748 (10Pharos) 03NEW [18:23:29] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for SnowFire - https://phabricator.wikimedia.org/T442255#12486101 (10Pharos) We have some Columbia University students working on the Wikinewsie project too, that have faced the same issue. Manual approval for th... [18:23:52] 10GitLab (Account Approval), 10Tool-wikinewsie: Requesting GitLab account activation for Wikinewsie Columbia students - https://phabricator.wikimedia.org/T442748#12486103 (10Pharos) [18:37:54] 06Project-Admins: Create Phabricator projects for two Toolforge tools: Tool-whichtool and Tool-script-publisher - https://phabricator.wikimedia.org/T442737#12486158 (10Dev) 05Open→03Resolved a:03Dev Thanks @Nemoralis, I've created both projects through toolsadmin. Closing this as resolved. [18:42:44] 06Release-Engineering-Team (Priority Backlog 📥), 10Scap: find a way to systematically update the deployment server name across all repos - https://phabricator.wikimedia.org/T197470#12486176 (10dancy) @Dzahn, please run this on deploy1003 to close out this ticket. ` # Remove obsolete settings from DEPLOY_HEA... [18:51:45] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12486204 (10Volans) @taavi I trust your knowledge of those services more than the audit script 😉 if you run it without the `-s` it will tell you wh... [18:57:48] 06Release-Engineering-Team, 10Catalyst, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Restore ingress to patchdemo / catalyst - https://phabricator.wikimedia.org/T442664#12486233 (10brennen) We deleted a lingering failed job and some old images from the registry. We think the script shoul... [19:27:46] hi! It would seem the fingerprint of my work machine is getting blocked from phab.wmfusercontent.org . In incognito, it errors out entirely and I can't load Phab at all. On my phone all works fine. What is the suggested recourse? [19:31:52] (03CR) 10Ahmon Dancy: "I don't know how often fresh developers do this but this change makes it so you can no longer run:" [fresh] - 10https://gerrit.wikimedia.org/r/1347594 (https://phabricator.wikimedia.org/T439828) (owner: 10Hashar) [19:33:40] (03CR) 10Ahmon Dancy: [C:04-1] "Failed when testing locally:" [fresh] - 10https://gerrit.wikimedia.org/r/1347594 (https://phabricator.wikimedia.org/T439828) (owner: 10Hashar) [19:53:57] (03open) 10dancy: install_local_version.sh: Restore the previous venv after a failed install [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1390 (https://phabricator.wikimedia.org/T382271) [20:01:56] 10Scap, 13Patch-For-Review: Weird scap install-world failure for aphlict2001.codfw.wmnet - https://phabricator.wikimedia.org/T382271#12486501 (10dancy) Most probable: Puppet starts running on the target while `scap install-world` is deploying to it. ` exec { 'bootstrap-scap-target': command... [20:02:58] (03update) 10dancy: install_local_version.sh: Restore the previous venv after a failed install [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1390 (https://phabricator.wikimedia.org/T382271) [20:03:19] (03update) 10dancy: install_local_version.sh: Restore the previous venv after a failed install [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1390 (https://phabricator.wikimedia.org/T382271) [20:06:01] (03merge) 10dancy: install_local_version.sh: Restore the previous venv after a failed install [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1390 (https://phabricator.wikimedia.org/T382271) [20:06:53] (03open) 10dancy: install-world: Lock the install on targets [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1391 (https://phabricator.wikimedia.org/T382271) [20:08:16] (03update) 10dancy: install-world: Lock the install on targets [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1391 (https://phabricator.wikimedia.org/T382271) [20:09:52] (03merge) 10dancy: install-world: Lock the install on targets [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1391 (https://phabricator.wikimedia.org/T382271) [20:12:11] (03open) 10dancy: Release 4.308.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1392 [20:12:15] (03update) 10dancy: Release 4.308.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1392 [20:15:12] (03merge) 10dancy: Release 4.308.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1392 [21:04:21] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T438222#12486787 (10dduvall) [21:36:14] 06Release-Engineering-Team (Priority Backlog 📥), 10Scap: find a way to systematically update the deployment server name across all repos - https://phabricator.wikimedia.org/T197470#12486950 (10Dzahn) done! ` root@deploy1003:~# sudo find . -path '*/.git/DEPLOY_HEAD' -exec sed -i -E '/^git_(server|scheme|deplo... [21:45:02] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12486969 (10Southparkfan) @Volans @taavi Thanks for your help. >>! In T442482#12485893, @taavi wrote: > [...] > The shellboxes are showing up as... [22:03:46] 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12487051 (10komla) Could you please confirm whether you still intend to complete the mig... [22:57:32] 10GitLab (Infrastructure), 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: Trusted runners failing to connect to docker registry - https://phabricator.wikimedia.org/T439046#12487223 (10Dzahn) >>! In T439046#12481885, @thcipriani wrote: > Better fix: update puppet to allow... [22:58:32] (03open) 10dancy: spiderpig: Improve deploy-service --confirm-diffs UX [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1393 (https://phabricator.wikimedia.org/T412941) [23:00:22] (03update) 10dancy: spiderpig: Improve deploy-service --confirm-diffs UX [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1393 (https://phabricator.wikimedia.org/T412941) [23:12:55] (03update) 10dancy: spiderpig: Improve deploy-service --confirm-diffs UX [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1393 (https://phabricator.wikimedia.org/T412941) [23:13:03] (03update) 10dancy: spiderpig: Improve deploy-service --confirm-diffs UX [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1393 (https://phabricator.wikimedia.org/T412941) [23:13:17] (03update) 10dancy: spiderpig: Improve deploy-service --confirm-diffs UX [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1393 (https://phabricator.wikimedia.org/T412941) [23:15:58] (03merge) 10dancy: spiderpig: Improve deploy-service --confirm-diffs UX [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1393 (https://phabricator.wikimedia.org/T412941) [23:51:46] 10Scap, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 14 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12487406 (10Jdforrester-WMF)