[02:37:16] <wikibugs>	 10Phabricator, 10Wikibugs, 13Patch-For-Review: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#12012937 (10HakanIST) Thanks both for the context. No rush on the review from my side, I can rebase whenever needed. Also, I can h...
[03:47:42] <wikibugs>	 10Beta-Cluster-Infrastructure: Beta cluster access via open proxies (due to internet censorship) - https://phabricator.wikimedia.org/T428989 (10Dringsim) 03NEW
[05:28:43] <wikibugs>	 10Beta-Cluster-Infrastructure: Beta cluster access via open proxies (due to internet censorship) - https://phabricator.wikimedia.org/T428989#12013110 (10Dringsim) or perhaps I should use a more reliable proxy?
[06:11:48] <wikibugs>	 (03open) 10phedenskog: Add dashboard that show agent load and how it affects performance [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/7 (https://phabricator.wikimedia.org/T428994)
[06:36:49] <wikibugs>	 10Gerrit, 06collaboration-services, 07ci-test-error (WMF-deployed Build Failure): Gerrit keeps returning 503 errors - https://phabricator.wikimedia.org/T428981#12013183 (10Jelto) 05Open→03Resolved a:03Jelto Hi, thanks for reporting the issue!  Traffic against Gerrit was increased for the last hours...
[08:34:27] <wikibugs>	 (03PS1) 10Hslater: Zuul: [mediawiki/extensions/DrawioEditor] Disable selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/1301302
[09:17:13] <wikibugs>	 (03open) 10fceratto: Add https://gitlab.wikimedia.org/repos/sre/wmfdb [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/176 (https://phabricator.wikimedia.org/T427900)
[09:32:51] <wikibugs>	 (03update) 10aklapper: Remove custom "Expert Mode" [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/64 (https://phabricator.wikimedia.org/T351289)
[09:52:46] <wikibugs>	 (03open) 10phedenskog: Add last jobs that finish per patch [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/8
[10:09:28] <jelto>	 GitLab needs a short maintenance break in one hour
[10:21:42] <Krinkle>	 !log `krinkle@{doc1004,doc2003}:/srv/doc/mediawiki-core$ sudo -u doc-uploader rm -rf list/` - remove doc build for git-tag test.
[10:21:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[11:02:33] <wikibugs>	 (03close) 10phedenskog: Add last jobs that finish per patch [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/8
[11:04:28] <wikibugs>	 (03open) 10phedenskog: Add last job that finish per patch [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/9
[11:05:22] <wikibugs>	 (03update) 10phedenskog: Add last job that finish per patch [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/9
[11:13:48] <wikibugs>	 (03merge) 10aklapper: Remove custom "Expert Mode" [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/64 (https://phabricator.wikimedia.org/T351289)
[11:14:39] <wikibugs>	 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎), 10Wikimedia-Phabricator-Extensions: Remove custom "Expert Mode" - https://phabricator.wikimedia.org/T351289#12013918 (10Aklapper)
[11:27:12] <jelto>	 GitLab upgrade finished
[11:39:14] <wikibugs>	 (03open) 10jelto: gitlab-runner: bump image version to alpine-v18.11.3 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/614 (https://phabricator.wikimedia.org/T428842)
[11:49:32] <wikibugs>	 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upgrade GitLab to major version 19 - https://phabricator.wikimedia.org/T426164#12014047 (10Jelto)
[11:51:21] <wikibugs>	 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upgrade GitLab to major version 19 - https://phabricator.wikimedia.org/T426164#12014065 (10Jelto) In  T428842 postgresql was updated to version 17. So all blockers for the GitLab major version upgrade are resolved.  I'll update...
[11:54:40] <wikibugs>	 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316#12014077 (10Jelto)
[11:55:33] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] jjb: [catalyst-daily-Popups] Add Popups job [integration/config] - 10https://gerrit.wikimedia.org/r/1300926 (https://phabricator.wikimedia.org/T427011) (owner: 10Vaughn Walters)
[11:56:38] <wikibugs>	 10Diffusion, 10Phabricator, 06Release-Engineering-Team, 07Performance Issue: Diffusion code view: Per-file change info is slow to load despite caching (diffusion.lastmodifiedquery) - https://phabricator.wikimedia.org/T403215#12014088 (10Aklapper) Ofc the output I'm interested in got truncated in the debug...
[11:57:20] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: [catalyst-daily-Popups] Add Popups job [integration/config] - 10https://gerrit.wikimedia.org/r/1300926 (https://phabricator.wikimedia.org/T427011) (owner: 10Vaughn Walters)
[12:04:26] <wikibugs>	 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316#12014101 (10Jelto)
[12:11:36] <wikibugs>	 10Phabricator, 10Catalyst (PatchDemo): Replace deprecated Phabricator Conduit API calls with their stable equivalents (PatchDemoBot) - https://phabricator.wikimedia.org/T428850#12014110 (10Aklapper) Updated the profile in Phab; unCC'ing you as the hot stewardship potato has been passed in the meantime :)
[12:26:45] <wikibugs>	 10Phabricator, 06collaboration-services: Add cache policy to static resources in phab.wmfusercontent.org - https://phabricator.wikimedia.org/T429019 (10Jelto) 03NEW
[13:22:53] <James_F>	 jnuche: I'm going to add a docker-pull-and-report-image builder to the jjb to explicitly show what image is being used, rather than just :latest.
[13:23:29] <jnuche>	 James_F: ack 👍
[13:26:08] <James_F>	 `Using image: docker-registry.wikimedia.org/repos/test-platform/catalyst/catalyst-ci-client@sha256:4ab20552e510097fb553d6e4541f6e83bc03a8c49f723fee307507b5a41fb8d9 (created 2026-06-12T09:31:17.394197249Z)`
[13:26:09] <James_F>	 Yay.
[13:28:45] <jnuche>	 that is indeed the right one, was that pulled by the docker runner just now?
[13:29:15] <jnuche>	 ah, I see it, neat: https://integration.wikimedia.org/ci/view/All/job/wikilambda-catalyst-end-to-end/3811/console
[13:29:21] <jnuche>	 yeah, that's the new format alright
[13:29:22] <jnuche>	 thanks!
[13:29:54] <James_F>	 Aye.
[13:29:58] <wikibugs>	 (03PS1) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end*] Report resolved client image digest [integration/config] - 10https://gerrit.wikimedia.org/r/1301363
[13:31:34] <wikibugs>	 (03CR) 10CI reject: [V:04-1] jjb: [wikilambda-catalyst-end-to-end*] Report resolved client image digest [integration/config] - 10https://gerrit.wikimedia.org/r/1301363 (owner: 10Jforrester)
[13:31:55] <wikibugs>	 (03PS2) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end*] Fetch & report Catalyst image [integration/config] - 10https://gerrit.wikimedia.org/r/1301363
[13:32:05] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] jjb: [wikilambda-catalyst-end-to-end*] Fetch & report Catalyst image [integration/config] - 10https://gerrit.wikimedia.org/r/1301363 (owner: 10Jforrester)
[13:33:43] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: [wikilambda-catalyst-end-to-end*] Fetch & report Catalyst image [integration/config] - 10https://gerrit.wikimedia.org/r/1301363 (owner: 10Jforrester)
[13:43:37] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Write lightweight OCI-image-based Puppet plans for beta cluster - https://phabricator.wikimedia.org/T425585#12014385 (10bking)
[14:04:23] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Migrate beta cluster to OpenSearch 2.x - https://phabricator.wikimedia.org/T421763#12014441 (10bking) →14Duplicate dup:03T425585
[14:04:24] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Write lightweight OCI-image-based Puppet plans for beta cluster - https://phabricator.wikimedia.org/T425585#12014443 (10bking)
[14:14:30] <wikibugs>	 (03PS6) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end*] Add catalyst api-tests step [integration/config] - 10https://gerrit.wikimedia.org/r/1289415 (https://phabricator.wikimedia.org/T343378)
[14:14:41] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] jjb: [wikilambda-catalyst-end-to-end*] Add catalyst api-tests step [integration/config] - 10https://gerrit.wikimedia.org/r/1289415 (https://phabricator.wikimedia.org/T343378) (owner: 10Jforrester)
[14:16:22] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: [wikilambda-catalyst-end-to-end*] Add catalyst api-tests step [integration/config] - 10https://gerrit.wikimedia.org/r/1289415 (https://phabricator.wikimedia.org/T343378) (owner: 10Jforrester)
[14:50:37] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.47.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T423916#12014581 (10Lucas_Werkmeister_WMDE) Note: T428620, a task causing heavy logspam (including “MediaWikiHighErrorRate: Elevated rate of MediaWiki errors -...
[15:07:18] <dancy>	 !log deployment-db15 configured as a replica of deployment-db11 (T428930)
[15:07:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:07:21] <stashbot>	 T428930: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930
[15:11:02] <wikibugs>	 (03PS2) 10Jforrester: Docker: Drop quibble-bullseye, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1300216 (https://phabricator.wikimedia.org/T362705)
[15:11:03] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Docker: Drop quibble-bullseye, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1300216 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester)
[15:13:52] <wikibugs>	 (03Merged) 10jenkins-bot: Docker: Drop quibble-bullseye, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1300216 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester)
[15:17:00] <wikibugs>	 (03PS1) 10Jforrester: Docker: [php83] Re-platform to Debian Bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1301387 (https://phabricator.wikimedia.org/T383337)
[15:17:07] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Docker: [php83] Re-platform to Debian Bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1301387 (https://phabricator.wikimedia.org/T383337) (owner: 10Jforrester)
[15:18:56] <wikibugs>	 (03Merged) 10jenkins-bot: Docker: [php83] Re-platform to Debian Bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1301387 (https://phabricator.wikimedia.org/T383337) (owner: 10Jforrester)
[15:19:10] <wikibugs>	 (03PS1) 10Jforrester: jjb: Switch non-Quibble PHP 8.3 jobs over to 8.3.31 on Bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1301388 (https://phabricator.wikimedia.org/T383337)
[15:19:27] <James_F>	 !log Docker: [php83] Re-platform to Debian Bookworm, for T383337
[15:19:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:19:30] <stashbot>	 T383337: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337
[15:23:34] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] jjb: Switch non-Quibble PHP 8.3 jobs over to 8.3.31 on Bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1301388 (https://phabricator.wikimedia.org/T383337) (owner: 10Jforrester)
[15:24:16] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337#12014660 (10Jdforrester-WMF)
[15:25:14] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Switch non-Quibble PHP 8.3 jobs over to 8.3.31 on Bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1301388 (https://phabricator.wikimedia.org/T383337) (owner: 10Jforrester)
[15:30:52] <wikibugs>	 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎), 06Fundraising-Backlog: Phabricator Bot Request: Fundraising Tech Bug Reporter - https://phabricator.wikimedia.org/T426459#12014704 (10AKanji-WMF) Thank you!
[16:22:32] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch13 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[16:22:48] <wikibugs>	 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cirrussearch13 on project deployment-prep - https://phabricator.wikimedia.org/T429035 (10wmcs-alerts) 03NEW
[16:27:32] <wmcs-alerts>	 FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch13 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[16:32:32] <wmcs-alerts>	 FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[16:37:32] <wmcs-alerts>	 FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[16:48:04] <wikibugs>	 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12014912 (10ssingh) > Any capacity or isolation concern with git-over-HTTPS traffic on text-lb?  No direc...
[16:57:46] <wikibugs>	 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12014938 (10BBlack) Well, for this and other similar use-cases, I'd clarify that while technically it is...
[17:04:01] <inflatador>	 Anything going on with Zuul ATM? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1301396 has been testing for over an hour
[17:04:43] <wikibugs>	 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12014961 (10Dzahn) >>! In T428903#12014938, @BBlack wrote: > what's the biggest repo in gitlab?  I think...
[17:05:04] <inflatador>	 Maybe it's just busy? Looks like a lot of stuff in the queue https://integration.wikimedia.org/zuul/
[17:05:42] <wikibugs>	 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12014964 (10ssingh) >>! In T428903#12014938, @BBlack wrote: > Well, for this and other similar use-cases,...
[17:07:00] <wikibugs>	 (03merge) 10dancy: Add https://gitlab.wikimedia.org/repos/sre/wmfdb [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/176 (https://phabricator.wikimedia.org/T427900) (owner: 10fceratto)
[17:13:15] <wikibugs>	 (03merge) 10jhuneidi: gitlab-runner: bump image version to alpine-v18.11.3 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/614 (https://phabricator.wikimedia.org/T428842) (owner: 10jelto)
[17:23:35] <SomeRandomDev>	 Hi, I hope this is the right channel for this; https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/40460/console (for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/1300224 ) has been stuck for 1h43mins and seems to be blocking gate-and-submit
[17:30:38] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Castor: CI not running - integration-castor06 is overloaded and castor-save-workspace-cache is freezed - https://phabricator.wikimedia.org/T429042 (10Umherirrender) 03NEW
[17:30:52] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Castor: CI not running - integration-castor06 is overloaded and castor-save-workspace-cache is freezed - https://phabricator.wikimedia.org/T429042#12015052 (10Umherirrender) p:05Triage→03Unbreak!
[17:33:00] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI not running - integration-castor06 is overloaded and castor-save-workspace-cache is freezed - https://phabricator.wikimedia.org/T429042#12015058 (10Umherirrender)
[17:35:22] <taavi>	 trying to bounce the worker process via the jenkins interface
[17:37:33] <wmcs-alerts>	 FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[17:41:10] <taavi>	 tried to kick a bunch of things but jenkins still thinks that one complete job is taking the castor06 slot
[17:41:17] <wmcs-alerts>	 FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[17:41:20] <taavi>	 i don't see much other options than restarting jenkins at this point
[17:45:17] <taavi>	 why is the jenkins service masked on all contint1002 where it's actually running???
[17:46:32] <wmcs-alerts>	 FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[17:47:03] <wikibugs>	 (03update) 10thcipriani: gitlab-runner: set nofile ulimit of 4096 for runners [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/610 (https://phabricator.wikimedia.org/T426827) (owner: 10dancy)
[17:48:08] <thcipriani>	 taavi: lemme take a look, maybe I can just kill that one
[17:48:23] <taavi>	 thcipriani: so I'm very confused about the state of things
[17:48:33] <taavi>	 AIUI there's an attempt to move jenkins from 1002 to 1003?
[17:48:51] <taavi>	 1003 should be running jenkins according to puppet, but has puppet disabled with a message saying it's to prevent jenkins from starting there
[17:48:52] <thcipriani>	 that's right, unsure about the current state of that attempt though
[17:49:19] <taavi>	 1002 has jenkins running, but has `profile::ci::jenkins::service_enabled: false` in puppet so the unit is trying to mask itself (but somehow it's still running)
[17:49:30] <thcipriani>	 !log attempting to cancel castor-save-workspace-cache 6710545
[17:49:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:49:51] <taavi>	 and because it's masked, it can't be restarted for example, even though it's running
[17:51:32] <wmcs-alerts>	 RESOLVED: [4x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[17:52:27] <thcipriani>	 well. The journalctl for jenkins is angry about this agent.
[17:52:40] <thcipriani>	 lemme make sure this agent is even up
[17:55:00] <thcipriani>	 looks fine/idle, java remoting thing from jenkins on the agent is running. I support restarting jenkins. mutante how do I restart jenkins on contint1002? Should I just unmask, restart, remask? Are you in the middle of something?
[17:56:24] <thcipriani>	 also: should it be masked?
[17:59:54] <taavi>	 I'm just going to update Puppet to match reality so the restart doesn't need to fight puppet trying to mask it
[18:00:30] <thcipriani>	 !log unmasking jenkins on contint1002 and restarting
[18:00:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[18:23:22] <dduvall>	 thcipriani, taavi: looks like castor jobs are running again. are we just waiting for that queue to drain at this point?
[18:24:01] <dduvall>	 down to 18 from 41 a few minutes ago, so that's good
[18:24:28] <taavi>	 dduvall: yeah, it'll be a while for the backlog to fully recover, but I'm not sure if we have any better options?
[18:25:09] <dduvall>	 can't think of any. canceling jobs would just cause a series of terrible re-queues in zuul/gearman
[18:25:43] <dduvall>	 i really hate that castor bottleneck :/
[19:04:45] <thcipriani>	 this is something that's on test platform's radar at the moment. Peter H has some interest in this, he has some data on the thing we all feel: it seems to be getting worse over time: https://releng-data.wmcloud.org/-/dashboards/ci-by-repo-and-job/castor-cache-monthly (Peter made this table showing times climbing month-by-month, part of another dashboard he made:
[19:04:47] <thcipriani>	 https://releng-data.wmcloud.org/-/dashboards/ci-by-repo-and-job )
[19:06:22] <thcipriani>	 see also: percentage of time there's a backlog on castor: https://releng-data.wmcloud.org/-/dashboards/ci-by-repo-and-job/castor-queue-monthly
[19:39:57] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI not running - integration-castor06 is overloaded and castor-save-workspace-cache is freezed - https://phabricator.wikimedia.org/T429042#12015271 (10Umherirrender) 05Open...
[20:42:52] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI not running - integration-castor06 is overloaded and castor-save-workspace-cache is freezed - https://phabricator.wikimedia.org/T429042#12015411 (10thcipriani) Thanks @Um...
[21:19:05] <wikibugs>	 (03merge) 10thcipriani: gitlab-runner: set nofile ulimit of 4096 for runners [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/610 (https://phabricator.wikimedia.org/T426827) (owner: 10dancy)
[21:39:09] <wikibugs>	 (03PS1) 10Vaughn Walters: jjb: [wikilambda-catalyst-end-to-end-daily] Add ZUUL_PATCHSET [integration/config] - 10https://gerrit.wikimedia.org/r/1301447
[21:40:27] <wikibugs>	 (03CR) 10Vaughn Walters: "This should fix the unbound variable issue in the daily tests." [integration/config] - 10https://gerrit.wikimedia.org/r/1301447 (owner: 10Vaughn Walters)
[21:45:44] <dancy>	 !log Unstuck wmf-beta-update-all service on deployment-deploy04.deployment-prep (sudo systemctl stop wmf-beta-update-all)
[21:45:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:19:31] <jinxer-wm>	 FIRING: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[22:19:40] <wikibugs>	 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T429055 (10phaultfinder) 03NEW
[22:24:31] <jinxer-wm>	 RESOLVED: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[23:09:25] <wikibugs>	 10Gerrit, 06collaboration-services: Investigate Gerrit root disk usage and logging - https://phabricator.wikimedia.org/T425667#12015603 (10thcipriani) Noticed that the current apache log keeps on growing, seems like it's no longer logrotated (previously, under `/var/log/apache` they seem to rotate ... nigh...