[00:34:34] (03update) 10thcipriani: JobCard: Move job actions to their own column [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1091 (owner: 10dancy) [00:45:10] (03merge) 10thcipriani: JobCard: Move job actions to their own column [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1091 (owner: 10dancy) [01:14:22] (03open) 10thcipriani: CSS: Space between retry/cancel stop/cancel buttons [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1094 [01:14:37] (03update) 10thcipriani: CSS: Space between retry/cancel stop/cancel buttons [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1094 [01:16:36] 10Continuous-Integration-Config, 10Shellbox, 07PHP 8.5 support: Test Shellbox on PHP 8.5 - https://phabricator.wikimedia.org/T416064 (10Reedy) 03NEW [01:17:02] 10Continuous-Integration-Config, 10Shellbox, 07PHP 8.5 support: Test Shellbox on PHP 8.5 - https://phabricator.wikimedia.org/T416064#11571170 (10Reedy) p:05Triage→03Low [19:00:55] (03CR) 10Reedy: [C:03+2] Zuul: [mediawiki/extensions/Echo] Add MobileFrontend phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1235380 (owner: 10Fomafix) [19:02:37] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Echo] Add MobileFrontend phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1235380 (owner: 10Fomafix) [19:03:12] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1235380 [19:03:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:09:21] (03CR) 10Reedy: [C:03+2] build: Updating lodash to 4.17.23 [performance/fresnel] - 10https://gerrit.wikimedia.org/r/1232776 (owner: 10Libraryupgrader) [20:28:28] 10Continuous-Integration-Infrastructure: CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078 (10Reedy) 03NEW [20:28:47] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078#11571543 (10Reedy) p:05Triage→03High [20:29:38] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078#11571545 (10Reedy) [20:30:37] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078#11571547 (10Reedy) [21:05:28] Hello. Could you tell me, please, how can I push the Gerrit tests if they stack on the start point for almost an hour? Thank you. [21:06:12] * stuck [21:06:35] lol [21:06:43] * stuck [21:06:59] IKhitron: it's not stuck [21:07:08] RhinosF1: Yeah it is [21:07:17] T416078 [21:07:19] T416078: CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078 [21:08:11] Ah, it's not just me. Great. Thank you for your help. [21:08:28] I've poked someone for help, but obviously it's an evening on a weekend... [21:09:54] Reedy: I was about to link to https://integration.wikimedia.org/zuul/ and say there's a lot in the backlog [21:09:58] Oh they left again [21:10:05] I didn't notice they all said an hour [21:10:11] I thought it was just busy at first [21:11:22] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078#11571567 (10IKhitron) {F71638083} The UI looks like this for the last 56 minutes. [21:12:17] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07ci-test-error (WMF-deployed Build Failure): CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078#11571568 (10A_smart_kitten) [21:18:05] it might not necessarily make a practical difference (given the weekend evening), but personally I'd be tempted to mark T416078 as UBN... it seems like it might be blocking merges in a *lot* of repos (maybe all repos that have CI, but I'm not confident enough on that to say for certain) [21:18:10] T416078: CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078 [21:19:58] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078#11571569 (10A_smart_kitten) [21:20:04] There's no obvious escalation path, hence I've poked someone OOB [21:21:29] yeah, fair enough :) [21:23:08] unfortunately I don't know enough about Castor to even know where to start with it [21:35:44] Hey hey. [21:36:26] hey :P [21:42:26] 10Gerrit, 06collaboration-services: Opt-in testing of Gerrit-via-CDN - https://phabricator.wikimedia.org/T414719#11571573 (10Dzahn) I edited my /etc/hosts directly to opt-in and works fine for me so far. [21:43:14] OK, I took integration-castor05 offline, killed threads, and re-connected; no dice. [21:43:19] I'll try to restart Jenkins itself. [21:43:46] Hmm, I should !log this. [21:44:42] !log Fighting T416078, took integration-castor-5 offline, disconnected, sshed in to kill threads, then reconnected; no change in aspect. [21:44:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:44:49] T416078: CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078 [21:45:20] !log Running `sudo systemctl restart jenkins` on contint for T416078 [21:45:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:48:03] Bah, no, something (Zuul itself?) is still tracking job 6193776. [21:49:37] !log Deleted Jenkins's job entry for castor-save-workspace-cache 6193776 and this seems to have unstuck things for T416078? [21:49:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:51:19] maintenance-disconnect-full-disks build 777094 integration-agent-docker-1043 (/: 36%, /srv: 99%, /var/lib/docker: 39%): OFFLINE due to disk space [21:54:14] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078#11571578 (10IKhitron) It started to run, but extremel... [21:55:04] maintenance-disconnect-full-disks build 777095 integration-agent-docker-1043 (/: 36%, /srv: 70%, /var/lib/docker: 39%): RECOVERY disk space OK [21:55:50] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078#11571580 (10Jdforrester-WMF) >>! In T416078#11571578,... [22:02:30] Thanks James_F [22:02:36] Happy to help. [22:02:46] I didn't realise you were this side of the pond [22:03:01] Yeah, on my way to Anatolia. [22:03:10] Should we Resolve? Or leave open in case it recurs? [22:03:25] I'm guessing it's not going to reoccur... [22:03:35] * James_F crosses fingers. [22:04:33] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078#11571595 (10Jdforrester-WMF) 05Open→03Resolved... [22:04:53] A quick look at Zuul just shows the usual catching up on the backlog, with obvious prioritisation of certain queues [22:05:10] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078#11571599 (10IKhitron) >>! In T416078#11571580, @J... [22:09:23] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078#11571600 (10Jdforrester-WMF) >>! In T416078#11571... [22:10:03] maintenance-disconnect-full-disks build 777098 integration-agent-docker-1043 (/: 36%, /srv: 96%, /var/lib/docker: 39%): OFFLINE due to disk space [22:11:44] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI frozen waiting for "Waiting for next available executor on ‘integration-castor05" - https://phabricator.wikimedia.org/T416078#11571603 (10IKhitron) I do not know enough to see... [22:15:03] maintenance-disconnect-full-disks build 777099 integration-agent-docker-1043 (/: 36%, /srv: 82%, /var/lib/docker: 39%): RECOVERY disk space OK [23:20:01] Project beta-update-databases-eqiad build #91007: 04FAILURE in 0.72 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/91007/ [23:27:31] Yippee, build fixed! [23:27:31] Project beta-update-databases-eqiad build #91008: 09FIXED in 3 min 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/91008/