[00:04:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:16:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:21:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:35:45] (03PS6) 10Wandji collins: Bug:Remove wrong import call [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1286279 [00:40:06] (03CR) 10Wandji collins: [C:03+2] Bug:Remove wrong import call [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1286279 (owner: 10Wandji collins) [00:40:59] (03Merged) 10jenkins-bot: Bug:Remove wrong import call [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1286279 (owner: 10Wandji collins) [02:01:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:06:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:36:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:01:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:40:54] (03update) 10countcount: Draft: Minimize Cargo dependency features to those actually required [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/8 [04:48:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:53:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:54:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:59:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:02:37] (03update) 10countcount: Update Rust crate reqwest to 0.13 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/5 (owner: 10renovatebot) [05:07:02] (03update) 10countcount: Update Rust crate reqwest to 0.13 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/5 (owner: 10renovatebot) [05:10:10] (03update) 10countcount: Update Rust crate reqwest to 0.13 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/5 (owner: 10renovatebot) [05:10:22] 10Tool-Pageviews, 06Community-Tech: PageView tool not updated since 23 May 2026, possibly due to disbandment of CommTech team - https://phabricator.wikimedia.org/T427171 (10Robertsky) 03NEW [05:10:29] (03merge) 10countcount: Update Rust crate reqwest to 0.13 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/5 (owner: 10renovatebot) [05:10:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:14:10] 10Tool-Pageviews, 06Community-Tech: PageView tool not updated since 23 May 2026, possibly due to disbandment of CommTech team - https://phabricator.wikimedia.org/T427171#11951529 (10Robertsky) [05:14:55] (03update) 10renovatebot: Update Rust crate sqlx to 0.9.0 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/6 [05:15:00] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/7 [05:20:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:24:58] 10Tool-Pageviews, 06Community-Tech: PageView tool not updated since 23 May 2026, possibly due to disbandment of CommTech team - https://phabricator.wikimedia.org/T427171#11951534 (10HakanIST) This seems related to T423818. As of now, the API returns 404 for May 22 onward and the last dump file is `pageviews-20... [05:26:01] (03update) 10countcount: Draft: Update Rust crate sqlx to 0.9.0 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/6 (owner: 10renovatebot) [06:11:50] 10Tool-Pageviews, 06Community-Tech: PageView tool not updated since 23 May 2026, possibly due to disbandment of CommTech team - https://phabricator.wikimedia.org/T427171#11951565 (10Pppery) While I agree the timing is suspicious, if T423818 truly is the cause then this isn't related to the disbandment of the C... [08:07:15] (03open) 10countcount: Use shared cargo-audit CI template [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/9 [08:21:22] (03merge) 10countcount: Use shared cargo-audit CI template [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/9 [08:25:21] (03update) 10renovatebot: Draft: Update Rust crate sqlx to 0.9.0 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/6 [08:25:27] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/7 [08:27:42] (03update) 10countcount: Draft: Update Rust crate sqlx to 0.9.0 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/6 (owner: 10renovatebot) [09:14:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/7 [09:48:47] 10Cloud-VPS, 06tools-infrastructure-team: Consider allowing cumin access to all Cloud VPS VMs - https://phabricator.wikimedia.org/T422801#11951938 (10Volans) So we just need to use sudo. For another task related to cumin I had Moritz check if there was any side effect of using sudo when connected from the root... [09:52:22] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:53:10] FIRING: GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [09:57:22] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:58:10] RESOLVED: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [10:00:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:00:54] 06cloud-services-team, 10Toolforge: toolsdb disk space growing too fast - https://phabricator.wikimedia.org/T427187 (10fnegri) 03NEW [10:00:58] 06cloud-services-team, 10Toolforge: toolsdb disk space growing too fast - https://phabricator.wikimedia.org/T427187#11951956 (10fnegri) 05Open→03In progress p:05Triage→03High [10:02:01] 10Toolforge, 06tools-platform-team: toolsdb disk space growing too fast - https://phabricator.wikimedia.org/T427187#11951958 (10fnegri) [10:02:22] FIRING: [30x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [10:03:25] FIRING: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [10:04:40] FIRING: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [10:05:17] FIRING: [2x] JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:07:22] FIRING: [30x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [10:07:37] 10Toolforge, 06tools-platform-team: ToolsDB disk space growing too fast - https://phabricator.wikimedia.org/T427187#11951964 (10fnegri) [10:08:25] RESOLVED: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [10:10:17] RESOLVED: [2x] JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:11:07] FIRING: ToolsDBAlmostFull: ToolsDB is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBAlmostFull [10:12:22] FIRING: [45x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [10:12:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:13:25] FIRING: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [10:14:40] RESOLVED: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [10:17:22] RESOLVED: [31x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [10:17:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:19:15] 10Toolforge, 06tools-platform-team: ToolsDB disk space growing too fast - https://phabricator.wikimedia.org/T427187#11951983 (10fnegri) Looking at the [ToolsDB debugging](https://grafana.wmcloud.org/d/5968f014-0a22-4f48-b6dc-6d1c46636f80/toolsdb-debugging) dashboard, it's not a repeat of {T409716}, because in... [10:25:04] 06cloud-services-team, 10Cloud-VPS: cleanup mcrouter from cloudcontrol hosts - https://phabricator.wikimedia.org/T427189 (10fgiunchedi) 03NEW [10:27:08] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: memcache is a SPOF for designate/tooz coordination - https://phabricator.wikimedia.org/T422646#11952026 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is deployed, designate now uses zk for coordination. A rolling-res... [10:28:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:33:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:58:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:23:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:23:53] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/37 [12:23:54] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/71 [12:23:54] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/41 [12:37:03] 10Tool-Pageviews, 06Community-Tech: PageView tool not updated since 21 May 2026 - https://phabricator.wikimedia.org/T427171#11952297 (10Johannnes89) [13:21:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:26:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:48:14] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/71 (owner: 10l10n-bot) [13:48:17] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/71 (owner: 10l10n-bot) [13:48:50] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/37 (owner: 10l10n-bot) [13:48:53] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/37 (owner: 10l10n-bot) [13:49:39] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/41 (owner: 10l10n-bot) [13:49:41] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/41 (owner: 10l10n-bot) [13:51:17] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:55:39] (03approved) 10fnegri: status: add StatusShort enum and basic job status detection [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/299 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [13:56:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:56:33] (03update) 10fnegri: status: add StatusShort enum and basic job status detection [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/299 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [13:58:23] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: experiment with moving rabbitmq behind haproxy - https://phabricator.wikimedia.org/T420937#11952611 (10fgiunchedi) 05Open→03Declined Yes I'm happy with pointing rabbit clients to the whole cluster and let client-side logic handle failures/retrie... [14:06:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:06:56] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops: bird bfd session with 172.20.1.1 down - Bad packet from 172.20.1.1 - unknown session id - https://phabricator.wikimedia.org/T427202 (10fgiunchedi) 03NEW [14:11:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:17:48] 06cloud-services-team, 10Toolforge: Investigate toolforge failure to schedule pods due to insufficient cpu - https://phabricator.wikimedia.org/T427204 (10fgiunchedi) 03NEW [14:21:54] 10Tool-scout, 10Wikidata: Scout – Wikidata task discovery and contribution support tool - https://phabricator.wikimedia.org/T426892#11952723 (10JosefAnthony) [14:24:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:44:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:52:14] 10Tool-humaniki-2: Fix project/language selectors not disabled correctly - https://phabricator.wikimedia.org/T427209 (10Danya) 03NEW [14:52:30] 10Tool-humaniki-2: Fix project/language selectors not disabled correctly - https://phabricator.wikimedia.org/T427209#11952792 (10Danya) a:03Danya [14:52:38] 10Tool-humaniki-2: Fix project/language selectors not disabled correctly - https://phabricator.wikimedia.org/T427209#11952795 (10Danya) p:05Triage→03High [14:56:32] (03open) 10filippo: Add note on policy for clinic duty [toolforge-repos/cloudvps-quota] - 10https://gitlab.wikimedia.org/toolforge-repos/cloudvps-quota/-/merge_requests/1 [14:57:06] (03merge) 10filippo: Add note on policy for clinic duty [toolforge-repos/cloudvps-quota] - 10https://gitlab.wikimedia.org/toolforge-repos/cloudvps-quota/-/merge_requests/1 [15:03:27] 10Tool-humaniki-2: Remove unused per_language_chart i18n entry in LanguagesPage.vue - https://phabricator.wikimedia.org/T427211 (10Danya) 03NEW [15:03:30] 10Tool-humaniki-2: Remove unused per_language_chart i18n entry in LanguagesPage.vue - https://phabricator.wikimedia.org/T427211#11952833 (10Danya) [15:03:35] 10Tool-humaniki-2: Remove unused per_language_chart i18n entry in LanguagesPage.vue - https://phabricator.wikimedia.org/T427211#11952835 (10Danya) p:05Triage→03Low [15:03:46] 10Tool-humaniki-2: Setup push to deploy - https://phabricator.wikimedia.org/T422347#11952836 (10Danya) [15:03:54] 10Tool-humaniki-2: Retro-active statistics - https://phabricator.wikimedia.org/T422342#11952837 (10Danya) [15:31:29] 10Tool-humaniki-2: Vertically align view links on home page - https://phabricator.wikimedia.org/T427213 (10Danya) 03NEW [15:31:36] 10Tool-humaniki-2: Vertically align view links on home page - https://phabricator.wikimedia.org/T427213#11952880 (10Danya) [15:31:40] 10Tool-humaniki-2: Vertically align view links on home page - https://phabricator.wikimedia.org/T427213#11952882 (10Danya) p:05Triage→03Low [15:31:51] 10Tool-humaniki-2: Update contrib guide to forbid AI-generated code - https://phabricator.wikimedia.org/T426758#11952883 (10Danya) [15:31:58] 10Tool-humaniki-2: Update contrib guide to forbid AI-generated code - https://phabricator.wikimedia.org/T426758#11952884 (10Danya) [15:32:04] 10Tool-humaniki-2: Rename contribution guide - https://phabricator.wikimedia.org/T427124#11952885 (10Danya) [15:32:42] 10Tool-humaniki-2: Fix contribution guide - https://phabricator.wikimedia.org/T427149#11952886 (10Danya) [15:48:06] 10Tool-humaniki-2: Fix translations - https://phabricator.wikimedia.org/T427211#11952890 (10Danya) [15:48:45] 10Tool-humaniki-2: Fix translations - https://phabricator.wikimedia.org/T427211#11952893 (10Danya) [15:49:06] 10Tool-humaniki-2: Fix translations - https://phabricator.wikimedia.org/T427211#11952894 (10Danya) [15:50:17] 10Tool-humaniki-2: Fix translations - https://phabricator.wikimedia.org/T427211#11952895 (10Danya) [15:50:30] 10Tool-humaniki-2: Fix translations - https://phabricator.wikimedia.org/T427211#11952896 (10Danya) [16:01:52] 10Tool-humaniki-2: Harmonize API routes - https://phabricator.wikimedia.org/T427215 (10Danya) 03NEW [16:02:08] 10Tool-humaniki-2: Translate language names - https://phabricator.wikimedia.org/T426057#11952940 (10Danya) [16:02:41] 10Tool-humaniki-2: Harmonize API routes - https://phabricator.wikimedia.org/T427215#11952943 (10Danya) [16:02:52] 10Tool-humaniki-2: Harmonize API routes - https://phabricator.wikimedia.org/T427215#11952945 (10Danya) p:05Triage→03High [16:13:39] (03update) 10fnegri: status: add StatusShort enum and basic job status detection [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/299 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [16:25:57] (03update) 10danyya: Draft: Resolve T422343 "Integrate Denelezh and merge stats tables" [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/56 [16:29:48] 10Toolforge, 06tools-platform-team: ToolsDB disk space growing too fast - https://phabricator.wikimedia.org/T427187#11952964 (10fnegri) Binlogs are stored in 100M chunks. It looks like we are still generating more binlogs than usual even if the spike in "deletes per day" has ended. There might be other operati... [16:40:36] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate Puppet CA: metricsinfra-puppetmaster-1.metricsinfra.eqiad1.wikimedia.cloud is about to expire in 19d 23h 58m 10s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [17:10:25] (03update) 10danyya: Draft: Resolve T422343 "Integrate Denelezh and merge stats tables" [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/56 [17:24:25] (03update) 10vriaa: Add basic undo functionality [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/61 (https://phabricator.wikimedia.org/T421061) (owner: 10ejegg) [17:28:46] (03approved) 10vriaa: Add basic undo functionality [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/61 (https://phabricator.wikimedia.org/T421061) (owner: 10ejegg) [17:29:01] (03update) 10vriaa: Add basic undo functionality [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/61 (https://phabricator.wikimedia.org/T421061) (owner: 10ejegg) [17:29:15] (03merge) 10vriaa: Add basic undo functionality [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/61 (https://phabricator.wikimedia.org/T421061) (owner: 10ejegg) [17:41:34] 10Tool-humaniki-2: Remove unused bar chart properties in LanguagePages.vue - https://phabricator.wikimedia.org/T427218 (10Danya) 03NEW [17:41:38] 10Tool-humaniki-2: Remove unused bar chart properties in LanguagePages.vue - https://phabricator.wikimedia.org/T427218#11953044 (10Danya) [17:41:42] 10Tool-humaniki-2: Remove unused bar chart properties in LanguagePages.vue - https://phabricator.wikimedia.org/T427218#11953046 (10Danya) p:05Triage→03Medium [17:41:48] 10Tool-humaniki-2: Fix translations - https://phabricator.wikimedia.org/T427211#11953048 (10Danya) p:05Low→03Medium [17:44:05] 10Tool-Pageviews, 06Community-Tech, 06Data-Engineering: PageView tool not updated since 21 May 2026 - https://phabricator.wikimedia.org/T427171#11953050 (10Robertsky) [17:44:10] 10Tool-humaniki-2: Add a filter bar to chart labels toggles - https://phabricator.wikimedia.org/T427219 (10Danya) 03NEW [17:44:18] 10Tool-humaniki-2: Add a filter bar to chart labels toggles - https://phabricator.wikimedia.org/T427219#11953062 (10Danya) [17:44:24] 10Tool-humaniki-2: Add a filter bar to chart labels toggles - https://phabricator.wikimedia.org/T427219#11953064 (10Danya) p:05Triage→03High [17:44:24] (03open) 10vriaa: feat: use Codex icons for undo/redo buttons [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/69 [17:45:26] (03merge) 10vriaa: feat: use Codex icons for undo/redo buttons [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/69 [17:47:14] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-34 (2026-05-19 to 2026-06-02)), 07OKR-Work: Improve linting - enum descriptions - https://phabricator.wikimedia.org/T424210#11953065 (10AGhirelli-WMF) a:03AGhirelli-WMF [18:00:12] (03update) 10vriaa: feat: add one input for all sides for editing side-based properties [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/68 (https://phabricator.wikimedia.org/T421063) [18:04:27] (03update) 10vriaa: feat: add one input for all sides for editing side-based properties [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/68 (https://phabricator.wikimedia.org/T421063) [19:02:48] (03update) 10vriaa: feat: add one input for all sides for editing side-based properties [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/68 (https://phabricator.wikimedia.org/T421063) [19:04:32] (03merge) 10vriaa: feat: add one input for all sides for editing side-based properties [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/68 (https://phabricator.wikimedia.org/T421063) [19:08:15] 10Tool-humaniki-2: Keep filters states when updating datasets - https://phabricator.wikimedia.org/T422335#11953118 (10Danya) [19:09:09] 10Tool-humaniki-2: Link to Wikidata elements in country and profession stats tables - https://phabricator.wikimedia.org/T425089#11953119 (10Danya) [19:12:25] (03update) 10vriaa: feat: add appearance settings with light/dark/auto mode [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/66 (https://phabricator.wikimedia.org/T421746) [19:13:57] (03merge) 10vriaa: feat: add appearance settings with light/dark/auto mode [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/66 (https://phabricator.wikimedia.org/T421746) [19:52:02] (03update) 10danyya: Draft: Resolve T422343 "Integrate Denelezh and merge stats tables" [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/56 [19:52:55] 10Tool-humaniki-2: Remove unused bar chart properties in LanguagePages.vue - https://phabricator.wikimedia.org/T427218#11953210 (10Danya) 05Open→03Resolved Resolved in T422343. [19:55:35] 10Tool-humaniki-2: Use dumps - https://phabricator.wikimedia.org/T423012#11953220 (10Danya) In the end I used the JSON dumps as their structure is stable and easier to parse. [20:05:50] (03update) 10danyya: Draft: Resolve T422343 "Integrate Denelezh and merge stats tables" [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/56 [20:22:42] (03update) 10danyya: Draft: Resolve T422343 "Integrate Denelezh and merge stats tables" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/56 [20:24:56] (03update) 10danyya: Resolve T422343 "Integrate Denelezh and merge stats tables" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/56 [20:25:02] (03merge) 10danyya: Resolve T422343 "Integrate Denelezh and merge stats tables" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/56 [20:25:29] 10Tool-humaniki-2, 13Patch-For-Review: Integrate Denelezh and merge stats tables - https://phabricator.wikimedia.org/T422343#11953319 (10Danya) 05In progress→03Resolved [20:25:50] 10Tool-humaniki-2: Aggregate gender data in the database instead of client - https://phabricator.wikimedia.org/T425219#11953322 (10Danya) Resolved in T422343. [20:25:59] 10Tool-humaniki-2: Aggregate gender data in the database instead of client - https://phabricator.wikimedia.org/T425219#11953325 (10Danya) 05In progress→03Resolved [20:26:21] 10Tool-humaniki-2: Use transactions for snapshots - https://phabricator.wikimedia.org/T424903#11953326 (10Danya) 05In progress→03Resolved Resolved in T422343. [20:27:52] 10Tool-humaniki-2: Use dumps - https://phabricator.wikimedia.org/T423012#11953345 (10Danya) Implemented in T422343 but needs to be tested in prod. [20:29:03] 10Tool-humaniki-2: Fix project/language selectors not disabled correctly - https://phabricator.wikimedia.org/T427209#11953348 (10Danya) 05Open→03In progress [20:31:17] (03open) 10danyya: Draft: Resolve T427209 "Fix project/language selectors not disabled correctly" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/57 [20:38:11] (03update) 10danyya: Draft: Resolve T427209 "Fix project/language selectors not disabled correctly" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/57 [20:38:31] (03update) 10danyya: Resolve T427209 "Fix project/language selectors not disabled correctly" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/57 [20:38:44] (03merge) 10danyya: Resolve T427209 "Fix project/language selectors not disabled correctly" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/57 [20:38:49] (03update) 10vriaa: fix: replace hardcoded colors and -fixed tokens with mode-aware Codex tokens [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/67 (https://phabricator.wikimedia.org/T421746) [20:39:03] 10Tool-humaniki-2, 13Patch-For-Review: Fix project/language selectors not disabled correctly - https://phabricator.wikimedia.org/T427209#11953388 (10Danya) 05In progress→03Resolved [20:40:02] (03merge) 10vriaa: fix: replace hardcoded colors and -fixed tokens with mode-aware Codex tokens [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/67 (https://phabricator.wikimedia.org/T421746) [20:41:47] 10Tool-humaniki-2: Update CONTRIBUTING.md to reflect workflow changes - https://phabricator.wikimedia.org/T427233 (10Danya) 03NEW [20:41:48] 10Tool-humaniki-2: Update CONTRIBUTING.md to reflect workflow changes - https://phabricator.wikimedia.org/T427233#11953406 (10Danya) [20:41:56] 10Tool-humaniki-2: Update CONTRIBUTING.md to reflect workflow changes - https://phabricator.wikimedia.org/T427233#11953408 (10Danya) p:05Triage→03High [20:42:17] 10Tool-humaniki-2: Rename contribution guide - https://phabricator.wikimedia.org/T427124#11953410 (10Danya) 05Open→03In progress [21:12:20] 10Tool-humaniki-2: Rename and update contribution guide - https://phabricator.wikimedia.org/T427124#11953472 (10Danya) [21:13:07] 10Tool-humaniki-2: Update CONTRIBUTING.md to reflect workflow changes - https://phabricator.wikimedia.org/T427233#11953474 (10Danya) 05Open→03Invalid Merged into T427124. [22:48:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:58:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:59:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:04:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:20:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:25:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown