[02:56:03] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:04:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [03:09:17] FIRING: [2x] PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [03:23:05] 10Tool-quickcategories, 10MediaWiki-Action-API, 10Notifications (Echo), 06Traffic: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11787201 (10DavidBrooks) Can someone explain the rationale for this breaking change? Some user... [03:34:17] RESOLVED: [2x] PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [05:05:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [05:35:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [05:51:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [06:56:04] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:05:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [09:35:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [10:16:18] (03update) 10danyya: Create language evolution tab [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/2 [10:16:36] (03merge) 10danyya: Create language evolution tab [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/2 [10:56:04] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [11:05:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [11:20:38] 06cloud-services-team, 10Toolforge: Toolforge Prometheus instance is unstable - https://phabricator.wikimedia.org/T422287 (10taavi) 03NEW [11:20:44] 06cloud-services-team, 10Toolforge: [infra,o11y] Alert on Prometheus instability / unexpected restarts - https://phabricator.wikimedia.org/T421416#11787369 (10taavi) 05Open→03Resolved Tracking follow-ups on {T422287}. [11:35:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [11:48:44] FIRING: [3x] IstioGatewayPodMisplaced: istio-gateway pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIstioGatewayPodMisplaced [11:53:44] RESOLVED: [6x] IstioGatewayPodMisplaced: istio-gateway pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIstioGatewayPodMisplaced [11:56:44] FIRING: [3x] IstioGatewayPodMisplaced: istio-gateway pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIstioGatewayPodMisplaced [12:01:44] RESOLVED: [9x] IstioGatewayPodMisplaced: istio-gateway pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIstioGatewayPodMisplaced [12:09:11] (03open) 10taavi: istio-gateway: Reduce Istio metric cardinality [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1206 (https://phabricator.wikimedia.org/T421386) [12:09:12] (03update) 10taavi: istio-gateway: Reduce Istio metric cardinality [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1206 (https://phabricator.wikimedia.org/T421386) [12:09:41] (03update) 10taavi: istio-gateway: Reduce Istio metric cardinality [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1206 (https://phabricator.wikimedia.org/T421386) [12:21:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [12:22:25] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [12:27:10] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [12:54:19] 10Tool-delintbot: Add ability to fix night-mode-unaware-background-colors errors in table styles - https://phabricator.wikimedia.org/T422013#11787493 (10Redmin) 05Open→03Invalid I can't remember what prompted the creation of this task or when I fixed this (or was this never accurate to begin with?) but t... [14:08:52] 10Tool-delintbot: Add ability to fix night-mode-unaware-background-colors errors in table styles - https://phabricator.wikimedia.org/T422013#11787553 (10Redmin) 05Invalid→03Open This is the kind of edit that prompted this: https://hr.wiktionary.org/w/index.php?title=Predložak:crveni_karton&curid=5233&diff=27... [14:56:04] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:08:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [15:33:20] 10Tool-quickcategories, 10MediaWiki-Action-API, 10Notifications (Echo), 06Traffic: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11787598 (10DavidBrooks) The AutoWikiBrowser community has been on a wild goose chase since th... [15:38:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [16:00:24] (03open) 10danyya: Draft: Migrate to SQLite [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/3 [16:21:05] 10Tool-wiktlexbot: Properly retrieve lemmas for sign languages - https://phabricator.wikimedia.org/T418890#11787639 (10Redmin) 05Open→03Resolved [[ https://gitlab.wikimedia.org/toolforge-repos/wiktlexbot/-/commit/33e24d32f114a3a36ae66785ab40c85cf245c105 | Done ]]. [16:40:27] 10Tool-wikimonitor: 30-Minute SseEmitter Timeout with Event Resumption - https://phabricator.wikimedia.org/T422194#11787644 (10Gerges) @Praffq, What will you do about that? > Architectural Note: Since the server currently maintains a single global EventSource connection to Wikimedia and broadcasts to all SseEmit... [17:07:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [17:37:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [18:13:22] 10Tool-quickcategories, 10MediaWiki-Action-API, 10Notifications (Echo), 06Traffic: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11787690 (10daniel) >>! In T421991#11787201, @DavidBrooks wrote: > Can someone explain the rat... [18:40:49] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [19:33:29] (03CR) 10Alien4444: "recheck" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1267007 (owner: 10L10n-bot) [19:41:18] (03CR) 10Alien4444: [C:03+2] Localisation updates from https://translatewiki.net. [labs/xtools] - 10https://gerrit.wikimedia.org/r/1267007 (owner: 10L10n-bot) [21:05:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [21:20:27] (03CR) 10Alien4444: [C:03+2] eslint: autofix several rules [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260616 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [21:21:05] (03Merged) 10jenkins-bot: eslint: autofix several rules [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260616 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [21:35:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [22:30:22] 06cloud-services-team, 10Toolforge, 06tools-platform-team: `toolforge jobs logs` misplaces my logs - https://phabricator.wikimedia.org/T421929#11787825 (10Soda) >>! In T421929#11780368, @Raymond_Ndibe wrote: > @Soda You can now see all your logs using `--since` to adjust how far in the past the logs should b... [23:07:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [23:37:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [23:48:53] 06cloud-services-team, 10Toolforge: Building/Running dotnet job fails on Toolforge - https://phabricator.wikimedia.org/T422224#11787861 (10Hawkeye7) `tools.milhistbot@tools-bastion-15:~$ buildit-autoreport Waiting for the logs... if the build just started this might take a minute [prepare] 2026-04-03T00:02:49.... [23:49:40] 06cloud-services-team, 10Toolforge: Building/Running dotnet job fails on Toolforge - https://phabricator.wikimedia.org/T422224#11787862 (10Hawkeye7) Permission problem?