[06:22:09] (03CR) 10Deni: "Hi Antoine," [integration/config] - 10https://gerrit.wikimedia.org/r/853078 (owner: 10Deni) [06:46:29] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, 10Thanks: Exception caught: Could not enqueue jobs when attempting to 'thank' on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10Tgr) Seems to happen for [[https://beta-logs.wmcloud.org/goto/b25ee... [07:21:39] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, 10Thanks: JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10TheresNoTime) [07:21:55] 10Beta-Cluster-Infrastructure: Account creation broken on beta - https://phabricator.wikimedia.org/T322442 (10TheresNoTime) [07:22:20] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, 10Thanks: JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10TheresNoTime) [07:23:33] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, 10Thanks: JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10TheresNoTime) [07:33:46] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, 10Thanks: JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10TheresNoTime) On attempting to create an account, I logged [[ https://beta-logs.wmcloud.org/app... [07:43:54] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, 10Thanks: JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10TheresNoTime) More digging, found [[ https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Echo... [07:57:01] TheresNoTime: I think that's a train blocker for next week [07:57:03] And UBN [07:57:07] Or not next week [07:57:09] Week after [07:57:15] No train I believe next week [08:04:09] Project beta-code-update-eqiad build #416412: 15ABORTED in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/416412/ [08:06:34] 10GitLab, 10Release-Engineering-Team, 10Data-Engineering, 10serviceops-collab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10hashar) a:05hashar→03None I have done the first pass investigation back in June to free up disk space but I am otherwise n... [08:33:43] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.40.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T320515 (10TheresNoTime) [08:33:45] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, 10Thanks: JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10TheresNoTime) [08:34:06] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, 10Thanks: JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10TheresNoTime) p:05Triage→03Unbreak! On advisement of @RhinosF1, I agree this may warrant ho... [08:34:07] RhinosF1: ^ (thank you) [08:35:15] TheresNoTime: thanks [08:36:34] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, 10Thanks: JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10RhinosF1) >>! In T322342#8372147, @TheresNoTime wrote: > More digging, found [[ https://gerrit.... [08:36:47] TheresNoTime: it's our favourites to blame too, Amir1 and Reedy [08:39:55] 10Beta-Cluster-Infrastructure: DBError: Access denied for user 'wikiadmin'@'172.16.%' to database 'mainstash' on Beta Cluster - https://phabricator.wikimedia.org/T322469 (10TheresNoTime) [10:13:12] Thanks zabe [10:14:23] TheresNoTime: patch for review [10:17:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: gerrit-bot holding open SSH sessions - https://phabricator.wikimedia.org/T309376 (10hashar) [10:21:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10User-DannyS712: Gerrit: all patches are being reported as merge conflicts - https://phabricator.wikimedia.org/T309371 (10hashar) This was for the immediate response in July. The tracking canonical task is T309376 [11:48:20] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, and 2 others: JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10RhinosF1) Can this be resolved? [11:50:28] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.40.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T320515 (10Ladsgroup) [11:50:58] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, and 2 others: JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10Ladsgroup) 05Open→03Resolved a:03Zabe [11:52:29] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, and 2 others: JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10TheresNoTime) Still getting `Y2ZOP1rG9juoZpSmy0kEugAAAAQ] /w/index.php?returnto=Main+Page&tit... [15:25:31] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, and 2 others: JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10Zabe) 05Resolved→03Open [15:25:37] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.40.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T320515 (10Zabe) [16:59:40] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.40.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T320515 (10Zabe) [16:59:42] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform Value Stream, 10Growth-Team, and 3 others: JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322342 (10Zabe) 05Open→03Resolved [18:15:24] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10serviceops-collab: Self-build and publish buildkit helper images - https://phabricator.wikimedia.org/T321316 (10dduvall) >>! In T321316#8369939, @Jelto wrote: > In the last IC sync meeting we discussed that it makes more sense to add the self-hosted...