[00:00:52] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers [00:03:14] RESOLVED: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [02:01:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:11:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:10:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:20:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:26:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:31:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:23:33] 10Tool-upload-workbench: no custom wikitext-template columns (!59) - https://phabricator.wikimedia.org/T426449#11927733 (10Daanvr) please also remove the custom wikitext template column button form the add new column dropdown at the end of the table [06:28:16] 10Tool-upload-workbench: Always-visible Feedback button (encouraging beta-feedback modal) (!56) - https://phabricator.wikimedia.org/T426454#11927747 (10Daanvr) no changes visible in the frontend. of !56 this feedback still aplies: nice! lets move it to the top center of the page and keep it sticky always in sigh... [06:30:59] 10Tool-upload-workbench: advanced title and caption generation (!54) - https://phabricator.wikimedia.org/T425887#11927749 (10Daanvr) explaine the technical considerations in more generic termis insted of the specific functions ore code peaces since i am not nose deep in the codebut want to understand and make a... [06:42:02] 10Tool-upload-workbench: advanced title and caption generation (!54) - https://phabricator.wikimedia.org/T425887#11927754 (10Daanvr) @Daanvr Re your 06:30 ask — here is the same situation explained without code-specific references. The previous comment was right that a maintainer-level design call is needed; thi... [06:42:32] 10Tool-upload-workbench: no custom wikitext-template columns (!59) - https://phabricator.wikimedia.org/T426449#11927756 (10Daanvr) Addressed feedback: the `Custom wikitext-template column` entry in the "+ Add column" popover was still in the committed code on this branch (the prior agent had it as unstaged work... [06:47:26] 10Tool-upload-workbench: Always-visible Feedback button (encouraging beta-feedback modal) (!56) - https://phabricator.wikimedia.org/T426454#11927772 (10Daanvr) Address-feedback round 2 on !56: * Pill repositioned to **top-centre** (was mid-right). `.feedback-fab` now uses `position: fixed; left: 50%; top: 8px;... [06:50:16] 10Tool-upload-workbench: Add per-language Caption columns (multi-lang captions) (!52) - https://phabricator.wikimedia.org/T426422#11927773 (10Daanvr) 05Open→03Resolved Merged as v0.37.0 via [!52](https://gitlab.wikimedia.org/daanvr/upload-workbench/-/merge_requests/52). Live at https://upload-workbench.t... [07:47:41] 10Tool-upload-workbench: advanced title and caption generation (!54) - https://phabricator.wikimedia.org/T425887#11927794 (10Daanvr) the answer is: the auto-sequence concept become "just another chip kind" inside the new editor. So the use of the hashtag uh as the last character in um the string will still trigg... [07:59:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:24:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:25:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:30:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:40:02] 10Tool-upload-workbench: advanced title and caption generation (!54) - https://phabricator.wikimedia.org/T425887#11927868 (10Daanvr) Picked up after maintainer's resume signal in [comment 11927794](https://phabricator.wikimedia.org/T425887#11927794) ("auto-sequence concept becomes 'just another chip kind' inside... [09:10:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:15:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:18:47] 10Tool-curator: Curator: Select only not-uploaded images of a sequence - https://phabricator.wikimedia.org/T426494 (10PantheraLeo1359531) 03NEW [09:23:25] 10Tool-upload-workbench: New logo - https://phabricator.wikimedia.org/T426495 (10Daanvr) 03NEW [09:23:34] 10Tool-curator: Curator: Queuing aborts when internet connection is unstable - https://phabricator.wikimedia.org/T426496 (10PantheraLeo1359531) 03NEW [09:35:05] 10Tool-upload-workbench: advanced title and caption generation (!54) - https://phabricator.wikimedia.org/T425887#11927929 (10Daanvr) Uh some new new feedback is that the um file name suggestions are still overlapping the Suggestions for chips. So whenever the chips menu is activated, the file name suggestions sh... [09:41:02] 10Tool-upload-workbench: New logo - https://phabricator.wikimedia.org/T426495#11927956 (10Daanvr) a:03Daanvr [09:42:05] 10Tool-upload-workbench: Always-visible Feedback button (encouraging beta-feedback modal) (!56) - https://phabricator.wikimedia.org/T426454#11927957 (10Daanvr) 05Open→03Resolved == Lessons learned == * **Shared-helper DRY opportunity exists for follow-up.** The feedback modal reuses the v0.28.0 "Report... [09:47:53] 10Wikibugs: Wikibugs is no longer in some IRC channels - https://phabricator.wikimedia.org/T426501 (10A_smart_kitten) 03NEW [09:48:48] 10Tool-upload-workbench: no custom wikitext-template columns (!59) - https://phabricator.wikimedia.org/T426449#11927990 (10Daanvr) 05Open→03Resolved == Lessons learned == * **Pre-deciding ambiguous design questions before implementation accelerated this MR significantly.** The maintainer's Q-1/Q-2/Q-3 a... [09:49:59] 10Tool-upload-workbench: New logo (!60) - https://phabricator.wikimedia.org/T426495#11927994 (10Daanvr) MR: https://gitlab.wikimedia.org/daanvr/upload-workbench/-/merge_requests/60 Preview: https://upload-workbench.toolforge.org/mr-60/ [09:57:39] 10Wikibugs: Wikibugs is no longer in some IRC channels (2026-05-16) - https://phabricator.wikimedia.org/T426501#11927997 (10A_smart_kitten) [10:04:54] 10Tool-upload-workbench: advanced title and caption generation (!54) - https://phabricator.wikimedia.org/T425887#11928000 (10Daanvr) Addressed all six points from your 09:34 comment in two new commits on !54 (72299f2 + 124229e — base 7a1d8ad preserved as you asked). 1. Filename suggestions are hidden whenever t... [10:10:59] 10Tool-Pageviews, 06Data-Engineering, 10Pageviews-API: Mediaviews Analysis returns API not found error - https://phabricator.wikimedia.org/T426373#11928003 (10AlexisJazz) >>! In T426373#11924484, @Aklapper wrote: > I don't see how this is related to RestBase? Why were all those folks added as task subscr... [10:37:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:55:10] 10Tool-upload-workbench: New logo (!60) - https://phabricator.wikimedia.org/T426495#11928102 (10Shiksha28) I would like to work on this task. [10:57:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:53:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:58:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:06:04] 10Wikibugs: Wikibugs is no longer in some IRC channels (2026-05-16) - https://phabricator.wikimedia.org/T426501#11928149 (10A_smart_kitten) [13:16:56] FIRING: SystemdUnitDown: The service unit backup_glance_images.service is in failed status on host cloudbackup1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:34:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:39:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:11:56] FIRING: SystemdUnitDown: The systemd unit backup_glance_images.service on node cloudbackup1003 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:38:01] 10Tool-upload-workbench: persistent login - https://phabricator.wikimedia.org/T426515 (10Daanvr) 03NEW [16:01:56] RESOLVED: SystemdUnitDown: The service unit backup_glance_images.service is in failed status on host cloudbackup1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:01:56] RESOLVED: SystemdUnitDown: The systemd unit backup_glance_images.service on node cloudbackup1003 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:04:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:09:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:17:31] 10Wikibugs: Wikibugs is no longer in some IRC channels (2026-05-16) - https://phabricator.wikimedia.org/T426501#11928335 (10bd808) 05Open→03Resolved a:03bd808 Probably was a {T410540} recurrence. `lang=shell-session tools.wikibugs@tools-bastion-14:~$ kubectl get po NAME READY S... [17:51:41] 10Wikibugs: Wikibugs is no longer in some IRC channels (2026-05-16) - https://phabricator.wikimedia.org/T426501#11928375 (10A_smart_kitten) Thank you @bd808 :) [18:56:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:06:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:47:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:52:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:56:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:01:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:17:24] (03update) 10danyya: Draft: Resolve T422343 "Integrate Denelezh and merge stats tables" [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/56 [21:48:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:49:27] (03update) 10danyya: Draft: Resolve T422343 "Integrate Denelezh and merge stats tables" [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/56 [21:56:23] (03update) 10danyya: Draft: Resolve T422343 "Integrate Denelezh and merge stats tables" [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/56 [21:58:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:59:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:22:34] (03PS5) 10Wandji collins: Bug:Remove wrong import call [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1286279 [22:44:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:57:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:07:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown