[00:17:11] <wikibugs>	 (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1297676 (owner: 10L10n-bot)
[00:46:57] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/26
[00:46:58] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/26
[00:46:59] <wikibugs>	 (03open) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/26
[00:47:27] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/17
[00:48:18] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/10
[00:49:09] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/30
[00:49:13] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/30
[00:49:44] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/20
[00:49:50] <wikibugs>	 (03open) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/10
[00:49:53] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/20
[00:50:20] <wikibugs>	 (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/20
[00:50:53] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/10
[02:23:55] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Release-Engineering-Team (Radar): Magnum cluster stuck in DELETE_FAILED status - https://phabricator.wikimedia.org/T428312#11992004 (10Andrew) Keystone says:   ` raise exception.Forbidden(_('Trustee has no delegated roles.')) `
[02:35:18] <wikibugs>	 10Tools, 07Performance Issue: Stalktoy is often slow or not responding - https://phabricator.wikimedia.org/T427941#11992005 (10Pathoschild) I suspect the performance issues are due to Toolforge resource limits. Stalktoy is very quick right after a restart, but it slows significantly when many connections a...
[02:52:17] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job openstack in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[02:53:22] <jinxer-wm>	 FIRING: [2x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1011.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[02:57:17] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[02:58:22] <jinxer-wm>	 RESOLVED: [2x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1011.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[03:38:28] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Release-Engineering-Team (Radar): Magnum cluster stuck in DELETE_FAILED status - https://phabricator.wikimedia.org/T428312#11992007 (10Andrew) trust_role_id: e2dfeddb1c514120b467835266f4bc06 (k8s_admin) current_effective_trustor_roles: ['38676f30eaeb44518bf7e144a73c8da6'...
[04:39:55] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [V:03+2 C:03+2] Add new private info stub for hiddenparma [labs/private] - 10https://gerrit.wikimedia.org/r/1297493 (https://phabricator.wikimedia.org/T428119) (owner: 10Giuseppe Lavagetto)
[04:56:21] <wikibugs>	 (03close) 10countcount: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/20 (owner: 10renovatebot)
[04:56:24] <wikibugs>	 (03close) 10countcount: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/10 (owner: 10renovatebot)
[04:57:58] <wikibugs>	 (03close) 10countcount: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/20 (owner: 10renovatebot)
[04:58:06] <wikibugs>	 (03close) 10countcount: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/10 (owner: 10renovatebot)
[04:59:01] <wikibugs>	 (03close) 10countcount: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/26 (owner: 10renovatebot)
[04:59:45] <wikibugs>	 (03close) 10countcount: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/30 (owner: 10renovatebot)
[06:21:33] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki urwikisource - https://phabricator.wikimedia.org/T415977#11992097 (10taavi) Hmh: ` ===== NODE GROUP ===== (1) clouddb1020.eqiad.wmnet ----- OUTPUT for command #2: '/usr/bin/maintai...ses urwikisource' ----- 2026-06-08 06:15:22,737...
[07:11:05] <wikibugs>	 10Tools: [RFC] Toolscope — a fully custom observability platform for Toolforge tools, built from scratch (SDK + backend + dashboard) - https://phabricator.wikimedia.org/T428379#11992220 (10Aklapper)
[07:50:18] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki urwikisource - https://phabricator.wikimedia.org/T415977#11992323 (10ops-monitoring-bot) Section s5: Wikis urwikisource redacted - fceratto@cumin1003
[07:53:21] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki urwikisource - https://phabricator.wikimedia.org/T415977#11992332 (10ops-monitoring-bot) Section s5: Wikis urwikisource redacted - fceratto@cumin1003
[08:03:38] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki urwikisource - https://phabricator.wikimedia.org/T415977#11992348 (10ops-monitoring-bot) Section s5: Wikis urwikisource set up on clouddb - fceratto@cumin1003
[08:15:54] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki urwikisource - https://phabricator.wikimedia.org/T415977#11992385 (10taavi) 05Open→03Resolved a:03taavi
[08:37:06] <wikibugs>	 06cloud-services-team, 10Toolforge: Buildservice failing - https://phabricator.wikimedia.org/T428396 (10magnusmanske) 03NEW
[09:05:57] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11992730 (10fnegri) Transaction History Length is back in check, though it has never reached zero:  {F87390614}
[09:13:17] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.32.13 (T427918)
[09:13:22] <stashbot>	 T427918: Upgrade toolsbeta cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427918
[09:13:26] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade to 1.32.13 (T427918)
[09:13:59] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests
[09:14:04] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99)
[09:18:58] <wikibugs>	 10Toolforge, 06tools-platform-team: Toolforge test suite runs hide gitlab 502 errors - https://phabricator.wikimedia.org/T428409 (10taavi) 03NEW
[09:24:56] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.32.13 (T427918)
[09:25:01] <stashbot>	 T427918: Upgrade toolsbeta cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427918
[09:26:21] <wikibugs>	 06cloud-services-team, 10Toolforge: Buildservice failing - https://phabricator.wikimedia.org/T428396#11992828 (10fnegri) GitLab was having some issues and throttling requests:  ` [step-inject-buildpacks] 2026-06-08T08:34:47.400269246Z Connecting to gitlab.wikimedia.org (208.80.154.145:443) [step-inject-buildpa...
[09:26:32] <wmcs-alerts>	 FIRING: InstanceDown: Project metricsinfra instance metricsinfra-controller-2 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[09:26:32] <wmcs-alerts>	 FIRING: TargetDown: Job manager is unreachable in project metricsinfra instance metricsinfra-controller-2   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[09:31:32] <wmcs-alerts>	 RESOLVED: InstanceDown: Project metricsinfra instance metricsinfra-controller-2 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[09:31:32] <wmcs-alerts>	 RESOLVED: TargetDown: Job manager is unreachable in project metricsinfra instance metricsinfra-controller-2   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[09:39:30] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade to 1.32.13 (T427918)
[09:39:36] <stashbot>	 T427918: Upgrade toolsbeta cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427918
[09:40:32] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.31.14 to 1.32.13 (T427918)
[09:48:07] <wmcs-alerts>	 FIRING: ToolsDBReplicationLagIsTooHigh: ToolsDB replication is lagging - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationLagIsTooHigh
[09:49:39] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.31.14 to 1.32.13 (T427918)
[09:49:44] <stashbot>	 T427918: Upgrade toolsbeta cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427918
[09:49:46] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.31.14 to 1.32.13 (T427918)
[09:55:23] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.31.14 to 1.32.13 (T427918)
[09:55:29] <stashbot>	 T427918: Upgrade toolsbeta cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427918
[09:55:30] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.31.14 to 1.32.13 (T427918)
[09:57:54] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.31.14 to 1.32.13 (T427918)
[09:58:47] <wikibugs>	 06cloud-services-team, 10Toolforge: Update Toolforge pause image - https://phabricator.wikimedia.org/T428418 (10taavi) 03NEW
[09:59:49] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T427918)
[10:03:10] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 (T427918)
[10:03:16] <stashbot>	 T427918: Upgrade toolsbeta cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427918
[10:04:15] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T427918)
[10:04:52] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T427918)
[10:09:44] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T427918)
[10:09:49] <stashbot>	 T427918: Upgrade toolsbeta cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427918
[10:14:27] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 (T427918)
[10:16:34] <wmcs-alerts>	 FIRING: ToolsNfsAlmostFull: Toolforge NFS is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull
[10:17:46] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-1 from 1.31.14 to 1.32.13 (T427918)
[10:17:51] <stashbot>	 T427918: Upgrade toolsbeta cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427918
[10:18:42] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-1 from 1.31.14 to 1.32.13 (T427918)
[10:18:44] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-2 from 1.31.14 to 1.32.13 (T427918)
[10:19:30] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-2 from 1.31.14 to 1.32.13 (T427918)
[10:19:33] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-3 from 1.31.14 to 1.32.13 (T427918)
[10:20:27] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-3 from 1.31.14 to 1.32.13 (T427918)
[10:20:45] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud (T427918)
[10:21:00] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud (T427918)
[10:21:40] <wikibugs>	 10Toolforge, 06tools-infrastructure-team: Upgrade toolsbeta cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427918#11993121 (10taavi) 05Open→03Resolved
[10:22:19] <wikibugs>	 (03PS1) 10Atsuko: kubernetes: dummy secret for opensearch CFSSL profile [labs/private] - 10https://gerrit.wikimedia.org/r/1298722 (https://phabricator.wikimedia.org/T427517)
[10:23:25] <wikibugs>	 (03CR) 10Atsuko: "check experimental" [labs/private] - 10https://gerrit.wikimedia.org/r/1298722 (https://phabricator.wikimedia.org/T427517) (owner: 10Atsuko)
[10:23:42] <wikibugs>	 06cloud-services-team, 10Toolforge: 2026-06-08 Toolforge NFS cleanup - https://phabricator.wikimedia.org/T428422 (10taavi) 03NEW
[10:25:18] <wikibugs>	 (03PS1) 10Giuseppe Lavagetto: Remove api token definitions [labs/private] - 10https://gerrit.wikimedia.org/r/1298723 (https://phabricator.wikimedia.org/T428119)
[10:25:31] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [V:03+2 C:03+2] Remove api token definitions [labs/private] - 10https://gerrit.wikimedia.org/r/1298723 (https://phabricator.wikimedia.org/T428119) (owner: 10Giuseppe Lavagetto)
[10:26:27] <wikibugs>	 06cloud-services-team, 10Toolforge: Buildservice failing - https://phabricator.wikimedia.org/T428396#11993167 (10magnusmanske) 05Open→03Resolved a:03magnusmanske Yes, seems to work, thanks!
[10:28:57] <wikibugs>	 10Toolforge, 06tools-platform-team: [jobs-api] Disable file logging by default - https://phabricator.wikimedia.org/T400915#11993185 (10taavi) Is there anything left to do before this can move forward?
[10:34:40] <wikibugs>	 10Tools: capx tool has unusually large cache folders - https://phabricator.wikimedia.org/T428425 (10taavi) 03NEW
[10:38:07] <wmcs-alerts>	 RESOLVED: ToolsDBReplicationLagIsTooHigh: ToolsDB replication is lagging - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationLagIsTooHigh
[10:38:56] <wikibugs>	 10Tools: cluebotng-review tool stores 180G of backups - https://phabricator.wikimedia.org/T428427 (10taavi) 03NEW
[10:51:18] <wikibugs>	 10Tools: cluebotng-review tool stores 180G of backups - https://phabricator.wikimedia.org/T428427#11993317 (10DamianZaremba) Once T428214 is resolved I can adjust the retention, this has over the last months with ~6.5k edits added last week.
[10:51:47] <wikibugs>	 06cloud-services-team, 10Toolforge: 'digero' tool uses an unreasonable amount of disk space - https://phabricator.wikimedia.org/T428430 (10taavi) 03NEW
[10:56:02] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Horizon project access - https://phabricator.wikimedia.org/T428432 (10DamianZaremba) 03NEW
[11:01:09] <wikibugs>	 (03CR) 10Atsuko: [V:03+2 C:03+2] kubernetes: dummy secret for opensearch CFSSL profile [labs/private] - 10https://gerrit.wikimedia.org/r/1298722 (https://phabricator.wikimedia.org/T427517) (owner: 10Atsuko)
[11:13:21] <wikibugs>	 10Tools: capx tool has unusually large cache folders - https://phabricator.wikimedia.org/T428425#11993428 (10Albertoleoncio) Done! `bash 1.1M    .yarn 232M    .npm 5.8M    .cache `
[11:15:34] <wikibugs>	 10Tools: capx tool has unusually large cache folders - https://phabricator.wikimedia.org/T428425#11993450 (10taavi) 05Open→03Resolved a:03Albertoleoncio Thanks.
[11:17:54] <wikibugs>	 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix description and example issues for transform paths in the MediaWiki REST API OAD (part 1) - https://phabricator.wikimedia.org/T428151#11993457 (10KBach)
[11:33:52] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-platform-team: Audit tools memory requests vs actual usage - https://phabricator.wikimedia.org/T420565#11993566 (10fgiunchedi) 05In progress→03Resolved I'll call this done and good enough™: I tackled the low-hanging fruits, namely webservice defaults have b...
[11:35:05] <wikibugs>	 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix description and example issues for transform paths in the MediaWiki REST API OAD (part 1) - https://phabricator.wikimedia.org/T428151#11993570 (10KBach)
[11:35:50] <wikibugs>	 10Tools, 07Performance Issue: Stalktoy is often slow or not responding - https://phabricator.wikimedia.org/T427941#11993576 (10Jules78120) 05Invalid→03Open
[11:52:18] <wikibugs>	 06cloud-services-team, 10Striker: Report account age, number of edits, etc in Striker membership requests - https://phabricator.wikimedia.org/T428440 (10fgiunchedi) 03NEW
[11:54:45] <wikibugs>	 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge: [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones - https://phabricator.wikimedia.org/T313030#11993674 (10fgiunchedi) Following up from team meeting, only the k8s etcd checks are relevant nowadays to port to AM, the rest can be ditched
[12:06:11] <wikibugs>	 (03open) 10filippo: kubernetes: add k8s etcd alerts from toolschecker [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/63 (https://phabricator.wikimedia.org/T313030)
[12:06:19] <wikibugs>	 (03update) 10filippo: kubernetes: add k8s etcd alerts from toolschecker [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/63 (https://phabricator.wikimedia.org/T313030)
[12:13:46] <wikibugs>	 (03open) 10aklapper: README: Replace repo mirror URI with canonical URI [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/7
[12:16:21] <wikibugs>	 (03open) 10aklapper: README: Replace repo mirror URI with canonical URI [toolforge-repos/my-first-flask-oauth-tool] - 10https://gitlab.wikimedia.org/toolforge-repos/my-first-flask-oauth-tool/-/merge_requests/1
[12:25:20] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/72
[12:25:21] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/43
[12:38:40] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/72 (owner: 10l10n-bot)
[12:38:43] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/72 (owner: 10l10n-bot)
[12:42:45] <wm-bot2>	 !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156826 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)
[12:42:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL
[12:43:05] <wm-bot2>	 !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156804 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)
[12:43:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL
[12:44:00] <wm-bot2>	 !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156886 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)
[12:44:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL
[12:46:36] <wikibugs>	 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix type, media type, and array issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428149#11993871 (10KBach)
[12:54:52] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Release-Engineering-Team (Radar): Magnum cluster stuck in DELETE_FAILED status - https://phabricator.wikimedia.org/T428312#11993922 (10Andrew) ...and after all that, I'm curious to hear if either of you (@dduvall or @bd808) can still create clusters.  Or delete them, for...
[13:09:41] <wikibugs>	 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Add examples to the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428153#11994028 (10KBach)
[13:10:03] <wikibugs>	 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Add examples to the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428153#11994029 (10KBach)
[13:10:32] <wikibugs>	 06cloud-services-team, 10Toolforge: [builds-api] images are not persisted / [jobs-api] is incorrectly caching tool container images - https://phabricator.wikimedia.org/T428449 (10DamianZaremba) 03NEW
[13:12:42] <wikibugs>	 06cloud-services-team, 10Toolforge: [builds-api] images are not persisted / [jobs-api] is incorrectly caching tool container images - https://phabricator.wikimedia.org/T428449#11994051 (10DamianZaremba)
[13:19:15] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11994098 (10fnegri) InnoDB reads have also returned to their previous level:  {F87427947}  It looks like all or most of the increase in reads was driven...
[13:46:34] <wmcs-alerts>	 RESOLVED: ToolsNfsAlmostFull: Toolforge NFS is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull
[13:48:23] <wikibugs>	 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix type, media type, and array issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428149#11994193 (10KBach)
[13:50:20] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11994212 (10fnegri) p:05High→03Low I have restarted the tool `dimastbkbot`, as I think it was not the main cause of the History Length growth, but m...
[13:57:20] <wikibugs>	 06cloud-services-team, 10Toolforge: 'digero' tool uses an unreasonable amount of disk space - https://phabricator.wikimedia.org/T428430#11994270 (10jberkel) Recently, the tool has been used to produce community HTML snapshots for enwiktionary (see T305407). These are quite storage intensive and there is no cle...
[15:37:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[15:42:44] <jinxer-wm>	 RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[15:51:29] <wikibugs>	 (03CR) 10Jayprakash12345: [C:03+2] messages: Remove obsolete lang-* translation keys [labs/tools/bookreader] - 10https://gerrit.wikimedia.org/r/1297661 (owner: 10Siebrand)
[15:51:51] <wikibugs>	 (03CR) 10Jayprakash12345: [C:03+2] gitignore: Add frontend/dist/ build output [labs/tools/bookreader] - 10https://gerrit.wikimedia.org/r/1297659 (owner: 10Siebrand)
[15:52:01] <wikibugs>	 (03CR) 10Jayprakash12345: [C:03+2] Use @wikimedia/language-data for language autonyms [labs/tools/bookreader] - 10https://gerrit.wikimedia.org/r/1297660 (owner: 10Siebrand)
[15:53:25] <wikibugs>	 (03Merged) 10jenkins-bot: gitignore: Add frontend/dist/ build output [labs/tools/bookreader] - 10https://gerrit.wikimedia.org/r/1297659 (owner: 10Siebrand)
[15:53:32] <wikibugs>	 (03Merged) 10jenkins-bot: Use @wikimedia/language-data for language autonyms [labs/tools/bookreader] - 10https://gerrit.wikimedia.org/r/1297660 (owner: 10Siebrand)
[15:53:47] <wikibugs>	 (03Merged) 10jenkins-bot: messages: Remove obsolete lang-* translation keys [labs/tools/bookreader] - 10https://gerrit.wikimedia.org/r/1297661 (owner: 10Siebrand)
[16:25:15] <wikibugs>	 (03CR) 10MusikAnimal: "Just two tiny things :)" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444)
[16:43:24] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Release-Engineering-Team (Radar): Magnum cluster stuck in DELETE_FAILED status - https://phabricator.wikimedia.org/T428312#11995247 (10dduvall) Thanks, @Andrew! I was able to delete the cluster template via `openstack coe cluster template delete` just now FWIW. I'll atte...
[17:19:56] <wikibugs>	 06cloud-services-team, 10Data-Services, 10BetaFeatures, 06Data-Engineering, and 4 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11995400 (10SD0001) >>! In https://gerrit.wikimedia.org/r/c/operations/puppet/+/1298329/comments/9648a202_f27...
[17:28:28] <wikibugs>	 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix summary issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428150#11995446 (10TBurmeister) 05Open→03In progress
[18:05:23] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure: horizon: hiera config reseted to an empty state for deployment-prep instances - https://phabricator.wikimedia.org/T262284#11995699 (10bd808) 05Open→03Declined Too old to do anything about
[18:06:55] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: [tofu-cloudvps] Add support for importing legacy cloudvps_puppet_prefix objects - https://phabricator.wikimedia.org/T423970#11995707 (10bd808)
[18:33:20] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 06Release-Engineering-Team (Radar): Magnum cluster stuck in DELETE_FAILED status - https://phabricator.wikimedia.org/T428312#11995793 (10dduvall) 05Open→03Resolved a:03dduvall I tested a full create/destroy cycle via `tofu` and it worked. Thanks, @Andrew
[19:45:47] <wikibugs>	 10Tools, 06Commons, 07Community-Wishlist: Image Editor for Commons - https://phabricator.wikimedia.org/T426106#11996028 (10Soda) @mikez-WMF Who exactly is working on this?
[19:57:42] <wikibugs>	 (03close) 10countcount: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/17 (owner: 10renovatebot)
[19:59:31] <wikibugs>	 (03PS14) 10Alien4444: Add proofreadpage quality support [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012)
[19:59:44] <wikibugs>	 (03CR) 10Alien4444: Add proofreadpage quality support (032 comments) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444)
[20:19:57] <wikibugs>	 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix summary issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428150#11996204 (10TBurmeister) Today I worked on updates for the core endpoints that had `wikimedia-paths-GET-summary` errors.  For  `/v1/revision/{from}/compare...
[20:48:52] <wikibugs>	 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix summary issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428150#11996331 (10apaskulin) Thanks for working on this, @TBurmeister!   >>! In T428150#11996204, @TBurmeister wrote: > I think the verb capturing the meaning of...
[21:44:45] <wikibugs>	 06cloud-services-team, 10Toolforge: Account disabled - https://phabricator.wikimedia.org/T426544#11996511 (10Hawkeye7) The bot logs on at the start of the run, and stays logged on for the duration of the run. I do not know why it would become logged out. I could check that the session is still current before s...
[22:03:54] <wikibugs>	 10Cloud-VPS (Quota-requests), 06Release-Engineering-Team (Radar): Quota increase request for zuul - https://phabricator.wikimedia.org/T428515 (10dduvall) 03NEW
[22:18:30] <wikibugs>	 (03CR) 10MusikAnimal: [C:03+2] "Apologies for the very, *very* long delay in finally getting this merged. Fantastic work, as always! Thank you 😊" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444)
[22:20:57] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add proofreadpage quality support [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444)
[22:22:07] <wikibugs>	 (03CR) 10MusikAnimal: [C:03+2] "…" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444)
[22:23:07] <wikibugs>	 (03Merged) 10jenkins-bot: Add proofreadpage quality support [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444)
[23:04:19] <wikibugs>	 10VPS-project-Wikistats: wikistats broken due to MediaWiki API dropping format=php - https://phabricator.wikimedia.org/T428521 (10Dzahn) 03NEW
[23:08:10] <wikibugs>	 10VPS-project-Wikistats: wikistats broken due to MediaWiki API dropping format=php - https://phabricator.wikimedia.org/T428521#11996825 (10Dzahn) Such a sad day when the API drops what you are using since years.
[23:19:43] <wikibugs>	 10Cloud-VPS (Quota-requests), 06Release-Engineering-Team (Radar): Quota increase request for zuul - https://phabricator.wikimedia.org/T428515#11996847 (10bd808) @dduvall The `tls-server-name: 127.0.0.1` trick in the kubeconfig does not work for the executor access?
[23:47:28] <wikibugs>	 10VPS-project-Wikistats: wikistats broken due to MediaWiki API dropping format=php - https://phabricator.wikimedia.org/T428521#11996941 (10-jem-) Thanks, @Dzahn. I was the one been warned about my list of Wikipedias in eswiki not been updated anymore. The task about this was T118538, executed by @Ladsgroup, and...