[01:14:00] (03update) 10renovatebot: Update pnpm to v11.13.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/7 [01:14:25] (03update) 10renovatebot: Update pnpm to v11.13.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/7 [01:15:19] (03update) 10renovatebot: Update pnpm to v11.13.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/42 [01:15:20] (03update) 10renovatebot: Update pnpm to v11.13.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/42 [01:16:12] (03update) 10renovatebot: Update pnpm to v11.13.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [01:16:13] (03update) 10renovatebot: Update pnpm to v11.13.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [02:14:53] (03update) 10renovatebot: Update pnpm to v11.13.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/42 [02:15:22] (03update) 10renovatebot: Update pnpm to v11.13.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [03:44:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:59:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:12:20] 10PAWS, 06tools-platform-team: Create a proper way to ban a user from PAWS - https://phabricator.wikimedia.org/T431915#12126863 (10Bugreporter) Note to we need to delete files made by banned users automatically (otherwise user can bloat NFS usage via creating multiple accounts) - and this should retroactively... [04:39:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:40:21] (03open) 10countcount: Update yanked spin 0.9.8 to 0.9.9 in lock file [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/25 [04:44:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:46:49] (03merge) 10countcount: Update yanked spin 0.9.8 to 0.9.9 in lock file [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/25 [04:47:18] (03update) 10countcount: Update pnpm to v11.13.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 (owner: 10renovatebot) [04:47:35] (03merge) 10countcount: Update pnpm to v11.13.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/42 (owner: 10renovatebot) [04:47:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:47:40] (03merge) 10countcount: Update pnpm to v11.13.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/7 (owner: 10renovatebot) [04:52:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:53:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:57:14] (03update) 10countcount: Mirror local block history in ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [05:04:35] (03update) 10countcount: Mirror local block history in ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [05:08:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:14:50] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/43 [05:17:24] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [05:18:46] (03update) 10countcount: Mirror local block history in ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [05:18:55] (03update) 10countcount: Mirror local block history in ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [05:34:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:49:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:32:43] (03update) 10countcount: Mirror local block history in ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [06:33:33] (03update) 10countcount: Mirror local block history in ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [06:38:36] (03update) 10countcount: Mirror local block history in ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [06:47:13] 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops, 06Traffic: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12126934 (10fgiunchedi) [06:47:16] 10Cloud-VPS, 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team, and 5 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12126935 (10fgiunchedi) [07:54:42] 10PAWS, 06tools-platform-team: Create a proper way to ban a user from PAWS - https://phabricator.wikimedia.org/T431915#12127132 (10aputhin) >>! In T431915#12126863, @Bugreporter wrote: > Note to we need to delete files made by banned users automatically (otherwise user can bloat NFS usage via creating multiple... [07:56:10] (03update) 10dcaro: runtime: split restart functions per job type [repos/cloud/toolforge/jobs-api] (split_delete_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/354 [07:57:20] (03update) 10dcaro: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] (split_restart_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 [07:58:59] (03update) 10dcaro: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] (split_restart_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 [08:20:17] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29482938340 (https://github.com/cluebotng/component-configs/commits/6ddd234f9ddace81ab9e1e7e45b5f974c5938a11) [08:20:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [08:38:16] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [o11y] Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12127246 (10DamianZaremba) >>! In T423549#12125257, @komla wrote: > We want to know if an average of 300 SSH logins are seen a... [08:45:10] (03update) 10dcaro: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] (split_restart_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 [08:50:12] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29484727917 (https://github.com/cluebotng/component-configs/commits/328ca21897e26730c95c4dad77c34d5e1bbf9c31) [08:50:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [08:50:15] (03update) 10dcaro: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] (split_restart_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 [09:13:20] (03update) 10dcaro: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] (split_restart_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 [09:22:28] 06tools-platform-team: [lima-kilo] put sysctl config under /etc/sysctl.d/, as /etc/sysctl.conf is not being used anymore - https://phabricator.wikimedia.org/T432323 (10dcaro) 03NEW [09:22:45] 10Toolforge, 06tools-platform-team: [lima-kilo] put sysctl config under /etc/sysctl.d/, as /etc/sysctl.conf is not being used anymore - https://phabricator.wikimedia.org/T432323#12127510 (10dcaro) [09:24:59] 10Toolforge, 06tools-platform-team: [lima-kilo] put sysctl config under /etc/sysctl.d/, as /etc/sysctl.conf is not being used anymore - https://phabricator.wikimedia.org/T432323#12127517 (10dcaro) [09:26:15] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Finalize other OAD resources - https://phabricator.wikimedia.org/T422924#12127520 (10KBach) 05Open→03Resolved [09:29:22] (03open) 10dcaro: global: a couple leftover renamings of vars/strings [repos/cloud/toolforge/jobs-api] (split_update_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/355 [09:33:54] 10Cloud-VPS, 06tools-infrastructure-team, 06tools-platform-team: Make sure dumps-nfs mount/umount is propagated inside containers - https://phabricator.wikimedia.org/T432325 (10fgiunchedi) 03NEW [09:37:40] (03update) 10countcount: Mirror local block history in ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [09:38:50] (03update) 10countcount: Mirror local and global block history in ToolsDB, prune suppressed events [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [09:47:20] (03open) 10tlepage: freezegun TestStreamToolLogs to avoid flakes [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/27 [09:48:10] (03update) 10tlepage: freezegun TestStreamToolLogs to avoid flakes [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/27 [09:49:17] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:54:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:56:50] (03update) 10tlepage: freezegun TestStreamToolLogs to avoid flakes [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/27 [09:59:20] (03update) 10tlepage: freezegun TestStreamToolLogs to avoid flakes [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/27 [10:02:49] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 95.95% full for project toolsbeta-logging - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [10:20:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:26:10] 10Toolforge: Request for Toolforge account access for puspaaamm-bot - https://phabricator.wikimedia.org/T432268#12127687 (10aputhin) [10:26:51] 10Toolforge, 06tools-infrastructure-team: Replace ingress-nginx before upstream EOL date - https://phabricator.wikimedia.org/T392356#12127691 (10aputhin) [10:28:35] (03update) 10raymond-ndibe: runtime: split restart functions per job type [repos/cloud/toolforge/jobs-api] (split_delete_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/354 (owner: 10dcaro) [10:28:36] (03approved) 10raymond-ndibe: runtime: split restart functions per job type [repos/cloud/toolforge/jobs-api] (split_delete_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/354 (owner: 10dcaro) [10:29:42] (03approved) 10raymond-ndibe: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 (owner: 10dcaro) [10:29:45] (03update) 10raymond-ndibe: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 (owner: 10dcaro) [10:30:27] (03update) 10raymond-ndibe: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] (split_restart_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 (owner: 10dcaro) [10:30:28] (03approved) 10raymond-ndibe: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] (split_restart_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 (owner: 10dcaro) [10:34:00] (03CR) 10Ladsgroup: "is it making any meaningful change to the UI? IIRC, we only use colors and it hasn't changed there. Right?" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1310563 (owner: 10Neriah) [10:35:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:36:48] 10Toolforge, 06tools-infrastructure-team: Update Toolforge Cumin nodes off of Bullseye - https://phabricator.wikimedia.org/T401817#12127750 (10aputhin) [10:37:10] 10Toolforge, 06tools-infrastructure-team: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.32 - https://phabricator.wikimedia.org/T379047#12127752 (10aputhin) [10:38:09] 10Toolforge, 06tools-infrastructure-team: [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones - https://phabricator.wikimedia.org/T313030#12127753 (10aputhin) [10:38:12] (03CR) 10Neriah: "yeah, I just wanted it to stay up to date. but if you think it's unnecessary, we'll leave it at that." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1310563 (owner: 10Neriah) [10:39:10] 10Toolforge, 06tools-infrastructure-team: Upgrade Toolforge Redis cluster to Bookworm or newer - https://phabricator.wikimedia.org/T360704#12127759 (10aputhin) [10:45:49] (03open) 10tlepage: Draft: Prototype: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] (fix-midnight-test-flake) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [10:47:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:52:01] (03update) 10tlepage: Draft: Prototype: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] (fix-midnight-test-flake) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 [11:07:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:10:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:44:20] (03update) 10countcount: Mirror local and global block history in ToolsDB, prune suppressed events [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [11:44:51] (03update) 10countcount: Mirror local and global block history in ToolsDB, prune suppressed events [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [11:55:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:01:25] (03update) 10dcaro: global: a couple leftover renamings of vars/strings [repos/cloud/toolforge/jobs-api] (split_update_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/355 [12:02:05] (03merge) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [12:02:13] (03update) 10dcaro: runtime: split restart functions per job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/354 [12:05:45] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.544-20260716120237-b69caa0d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1338 (https://phabricator.wikimedia.org/T430071) [12:05:54] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.544-20260716120237-b69caa0d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1338 (https://phabricator.wikimedia.org/T430071) [12:14:34] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:15:23] 10Toolforge, 06tools-platform-team: [toolsbeta,infra,logging] We are reaching 86% of quota by objects - https://phabricator.wikimedia.org/T431248#12128034 (10dcaro) p:05Triage→03Medium [12:25:14] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:25:20] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12128065 (10cmooney) [12:26:34] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:27:43] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12128073 (10cmooney) [12:31:37] 10Cloud-VPS, 06tools-infrastructure-team, 06tools-platform-team: Make sure dumps-nfs mount/umount is propagated inside containers - https://phabricator.wikimedia.org/T432325#12128084 (10dcaro) > My first instinct was to bind-mount /mnt/nfs though I believe that's a non-starter because it also contains home/p... [12:37:32] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: [toolsbeta,infra,admin] toolsbeta-test-k8s-haproxy7 is flaky on ip6 when replying to admin - https://phabricator.wikimedia.org/T432338 (10dcaro) 03NEW [12:37:45] 10Toolforge, 06tools-infrastructure-team: [toolsbeta,infra,admin] toolsbeta-test-k8s-haproxy7 is flaky on ip6 when replying to admin - https://phabricator.wikimedia.org/T432338#12128121 (10dcaro) [12:39:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:41:55] (03approved) 10dcaro: jobs-api: bump to 0.0.544-20260716120237-b69caa0d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1338 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:42:01] (03merge) 10dcaro: jobs-api: bump to 0.0.544-20260716120237-b69caa0d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1338 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:42:10] (03merge) 10dcaro: runtime: split restart functions per job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/354 [12:42:14] (03update) 10dcaro: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 [12:42:20] (03update) 10raymond-ndibe: Draft: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T431421) [12:43:16] 10Toolforge, 06tools-infrastructure-team: [toolsbeta,infra,admin] toolsbeta-test-k8s-haproxy7 is flaky on ip6 when replying to admin - https://phabricator.wikimedia.org/T432338#12128125 (10dcaro) [12:45:57] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.545-20260716124224-b19edd2d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1339 (https://phabricator.wikimedia.org/T430071) [12:45:59] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.545-20260716124224-b19edd2d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1339 (https://phabricator.wikimedia.org/T430071) [12:46:10] (03approved) 10dcaro: freezegun TestStreamToolLogs to avoid flakes [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/27 (owner: 10tlepage) [12:46:24] (03update) 10raymond-ndibe: Draft: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T431421) [12:49:18] 10Toolforge, 06tools-platform-team: [toolsbeta,infra,logging] We are reaching 96% of quota by objects - https://phabricator.wikimedia.org/T431248#12128144 (10dcaro) [12:49:35] (03approved) 10raymond-ndibe: global: a couple leftover renamings of vars/strings [repos/cloud/toolforge/jobs-api] (split_update_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/355 (owner: 10dcaro) [12:49:36] (03update) 10raymond-ndibe: global: a couple leftover renamings of vars/strings [repos/cloud/toolforge/jobs-api] (split_update_functions) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/355 (owner: 10dcaro) [12:49:48] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:52:51] (03update) 10countcount: Mirror local and global block history in ToolsDB, prune suppressed events [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [12:53:14] (03update) 10countcount: Mirror local and global block history in ToolsDB, prune suppressed events [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [13:01:15] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:01:40] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:13:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:16:22] (03approved) 10dcaro: jobs-api: bump to 0.0.545-20260716124224-b19edd2d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1339 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:16:28] (03merge) 10dcaro: jobs-api: bump to 0.0.545-20260716124224-b19edd2d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1339 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:20:39] (03update) 10countcount: Mirror local and global block history in ToolsDB, prune suppressed events [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [13:23:19] (03merge) 10dcaro: runtime: split update_* function per job-type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/350 [13:23:28] (03update) 10dcaro: global: a couple leftover renamings of vars/strings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/355 [13:26:27] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.546-20260716132338-92a05b3e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1340 (https://phabricator.wikimedia.org/T430071) [13:26:36] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.546-20260716132338-92a05b3e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1340 (https://phabricator.wikimedia.org/T430071) [13:30:52] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:42:14] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:42:40] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:54:15] (03update) 10countcount: Mirror local and global block history in ToolsDB, prune suppressed events [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [13:55:14] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:55:58] 10Toolforge, 06tools-platform-team: [toolsbeta,infra,logging] We are reaching 96% of quota by objects - https://phabricator.wikimedia.org/T431248#12128388 (10dcaro) a:03Raymond_Ndibe [13:57:30] (03merge) 10countcount: Mirror local and global block history in ToolsDB, prune suppressed events [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/45 [14:02:50] (03approved) 10dcaro: jobs-api: bump to 0.0.546-20260716132338-92a05b3e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1340 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:03:00] (03merge) 10dcaro: jobs-api: bump to 0.0.546-20260716132338-92a05b3e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1340 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:03:03] 10Toolforge, 06tools-platform-team: [toolsbeta,infra,logging] We are reaching 96% of quota by objects - https://phabricator.wikimedia.org/T431248#12128422 (10Raymond_Ndibe) So we investigated this, and as far as we know, this issue seems to be coming from the tool-sample-complex-app tool in toolsbeta, because... [14:03:57] 10Toolforge, 06tools-platform-team: [toolsbeta,infra,logging] We are reaching 96% of quota by objects - https://phabricator.wikimedia.org/T431248#12128426 (10Raymond_Ndibe) 05Open→03Resolved [14:11:54] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Rip out barbican puppet code - https://phabricator.wikimedia.org/T432349 (10Andrew) 03NEW [14:14:19] RESOLVED: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 95.84% full for project toolsbeta-logging - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [14:15:24] (03update) 10renovatebot: Update Rust crate sha2 to 0.11 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/48 [14:15:28] (03open) 10renovatebot: Update Rust crate sha2 to 0.11 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/48 [14:15:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [14:16:40] (03merge) 10taavi: Update security groups for metricsinfra VXLAN/Dualstack IPs [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/332 (https://phabricator.wikimedia.org/T401813) [14:17:11] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [14:17:44] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [14:22:09] 10Toolforge, 06tools-platform-team: Regression: Images built with the build service images do no longer contain/configure locales specified in .locales - https://phabricator.wikimedia.org/T428230#12128497 (10dcaro) a:05dcaro→03Raymond_Ndibe [14:22:13] (03approved) 10dcaro: global: a couple leftover renamings of vars/strings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/355 [14:22:15] 10Toolforge, 06tools-platform-team: Regression: Images built with the build service images do no longer contain/configure locales specified in .locales - https://phabricator.wikimedia.org/T428230#12128512 (10dcaro) 05Open→03In progress [14:22:20] (03merge) 10dcaro: global: a couple leftover renamings of vars/strings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/355 [14:26:19] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.547-20260716142237-a8374318 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1341 (https://phabricator.wikimedia.org/T430071) [14:26:24] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.547-20260716142237-a8374318 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1341 (https://phabricator.wikimedia.org/T430071) [14:36:52] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:39:56] (03open) 10andrew: In codfw1dev, add shared magnum templates in the new 'public-resources' dir [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [14:43:34] 10Toolforge, 06tools-platform-team: [toolforge-deploy,functional-tests] use the tool `automated-toolforge-tests` on toolsbeta instead of `test` - https://phabricator.wikimedia.org/T432354 (10dcaro) 03NEW [14:46:41] (03update) 10andrew: In codfw1dev, add shared magnum templates in the new 'public-resources' dir [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [14:48:14] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:48:39] (03update) 10andrew: Add shared magnum templates in the new 'public-resources' dir [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [14:51:16] (03update) 10andrew: Add shared magnum templates in the new 'public-resources' dir [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [15:02:16] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:12:06] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12128720 (10jsn.sherman) [15:13:29] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 - https://phabricator.wikimedia.org/T431682#12128739 (10VRiley-WMF) Hey @fgiunchedi thanks for your patience. As it turns out there is a lot going on here at eqiad. I know you'... [15:14:13] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:15:20] (03approved) 10dcaro: jobs-api: bump to 0.0.547-20260716142237-a8374318 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1341 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:15:26] (03merge) 10dcaro: jobs-api: bump to 0.0.547-20260716142237-a8374318 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1341 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:16:59] 06cloud-services-team, 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] Refactor before webservice support - https://phabricator.wikimedia.org/T359804#12128755 (10dcaro) 05In progress→03Resolved [15:18:54] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12128771 (10VRiley-WMF) [15:18:55] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: New thermal paste for cloudvirt1071.eqiad.wmnet - https://phabricator.wikimedia.org/T431429#12128768 (10VRiley-WMF) 05Open→03Resolved Hey @Andrew I'm going to go ahead and close this ticket for now. Please feel... [15:20:07] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12128772 (10jsn.sherman) note that I tested email sending on the new prod and it worked as exp... [15:20:58] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 07Essential-Work, 06Moderator-Tools-Team (Kanban): twl: Replace deprecated Bullseye VMs in Cloud VPS - https://phabricator.wikimedia.org/T402054#12128776 (10jsn.sherman) leaving open until next week to discover any latent issues. [15:24:39] (03open) 10tlepage: Draft: Tentative update for glibc 2.36+ [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/2 [15:27:30] (03update) 10tlepage: Draft: Tentative update for glibc 2.36+ [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/2 [15:48:29] (03open) 10countcount: Serve IP block history via prefix stabbing instead of /16-floor window scans [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/49 [15:48:53] (03update) 10countcount: Serve IP block history via prefix stabbing instead of /16-floor window scans [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/49 [15:49:11] (03approved) 10dcaro: Revert "bandaid high RLIMIT_NOFILE in gitlab runners" [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/90 (https://phabricator.wikimedia.org/T426827) (owner: 10filippo) [15:56:22] (03update) 10andrew: Add shared magnum templates in the new 'public-resources' dir [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [15:57:46] (03update) 10andrew: Add shared magnum templates in the new 'public-resources' dir [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [16:01:41] 10Toolforge, 06tools-platform-team: [ci] enable coverage on all the repos - https://phabricator.wikimedia.org/T432365 (10dcaro) 03NEW [16:02:51] 06cloud-services-team, 10VPS-project-Phabricator, 06collaboration-services, 06Infrastructure-Foundations, 10Mail: @wikimedia.org email addresses don't seem to be receiving emails sent by the test Phabricator instance - https://phabricator.wikimedia.org/T422559#12129019 (10Dzahn) Just ran into this again... [16:03:48] (03update) 10andrew: Add shared magnum templates in the new 'public-resources' dir [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [16:08:04] (03update) 10andrew: Add shared magnum templates in the new 'public-resources' dir [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [16:08:27] (03close) 10andrew: Add a couple of default global magnum templates. [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/331 (https://phabricator.wikimedia.org/T429164) [16:08:51] 10VPS-project-Phabricator, 06collaboration-services, 06Infrastructure-Foundations, 10Mail: @wikimedia.org email addresses don't seem to be receiving emails sent by the test Phabricator instance - https://phabricator.wikimedia.org/T422559#12129092 (10taavi) [16:12:52] 10VPS-project-Phabricator, 06collaboration-services, 06Infrastructure-Foundations, 10Mail: @wikimedia.org email addresses don't seem to be receiving emails sent by the test Phabricator instance - https://phabricator.wikimedia.org/T422559#12129123 (10taavi) >>! In T422559#11945766, @jhathaway wrote: > 1. mx... [16:14:14] (03update) 10andrew: Add shared magnum templates in the new 'public-resources' dir [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [16:15:58] 10Tool-wsindex, 10Wikisource Reader App (Android): WS Index thumbnail URLs are broken if the file is redirected - https://phabricator.wikimedia.org/T432367 (10Saiphani02) 03NEW [16:16:17] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:18:17] (03merge) 10countcount: Serve IP block history via prefix stabbing instead of /16-floor window scans [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/49 [16:21:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:27:17] 10Tool-wikiloves: Add Wiki Loves Ramadan to Wikiloves Statistics Tool - https://phabricator.wikimedia.org/T427971#12129188 (10Jony) @Danilo and @Lokal_Profil could you please look into this? [16:28:35] (03update) 10andrew: Add shared magnum templates in the new 'public-resources' dir [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [16:29:26] (03update) 10andrew: Add shared magnum templates in the new 'public-resources' dir [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [16:32:13] (03update) 10andrew: Add shared magnum templates in the new 'public-resources' dir [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [16:32:56] (03update) 10andrew: Add shared magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333 (https://phabricator.wikimedia.org/T429164) [16:33:18] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12129249 (10cmooney) [16:42:11] (03update) 10dcaro: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) (owner: 10raymond-ndibe) [17:05:23] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: Created the backend for leximap - https://phabricator.wikimedia.org/T432248#12129417 (10Kengkong1) a:03Kengkong1 [17:12:24] !log andrew@cloudcumin1001 cvn START - Cookbook wmcs.nfs.add_server [17:13:32] (03update) 10dcaro: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) (owner: 10raymond-ndibe) [17:15:25] (03update) 10renovatebot: Update Rust crate sha2 to 0.11 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/48 [17:15:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/46 [17:19:28] 10Toolforge, 06tools-infrastructure-team, 13Patch-For-Review: toolforge VMs should use nftables - https://phabricator.wikimedia.org/T427799#12129467 (10cmooney) Just a note, I'm not sure if this is still the case but at one point the cloudnet hosts were wrapping their VRRP keepalives in multicast UDP packets... [17:22:16] !log andrew@cloudcumin1001 cvn END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) [17:24:39] (03open) 10countcount: Drop range_end from the IP range dimensions, key on (range_start, prefix_len) [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/50 [17:24:44] !log andrew@cloudcumin1001 cvn START - Cookbook wmcs.nfs.migrate_service (T430589) [17:24:48] T430589: CVN: Replace Bullseye nfs server with a new Trixie server - https://phabricator.wikimedia.org/T430589 [17:24:54] !log andrew@cloudcumin1001 cvn END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T430589) [17:26:11] !log andrew@cloudcumin1001 cvn START - Cookbook wmcs.nfs.migrate_service (T430589) [17:26:16] !log andrew@cloudcumin1001 cvn END (ERROR) - Cookbook wmcs.nfs.migrate_service (exit_code=97) (T430589) [17:26:30] 10Cloud-Services, 06SRE: Wikimania 2026: Request to whitelist venue IP for all Wikimedia projects, Toolforge, and Cloud VPS etc... - https://phabricator.wikimedia.org/T432377 (10Michael_Barbereau_WMFr) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://p... [17:26:44] !log andrew@cloudcumin1001 cvn START - Cookbook wmcs.nfs.migrate_service (T430589) [17:30:25] !log andrew@cloudcumin1001 cvn END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T430589) [17:30:28] 10wikitech.wikimedia.org: Wikitech static is down with a 502 HTTP error (Bad Gateway) - https://phabricator.wikimedia.org/T427081#12129510 (10taavi) What is left to do here? [17:30:29] T430589: CVN: Replace Bullseye nfs server with a new Trixie server - https://phabricator.wikimedia.org/T430589 [17:30:38] !log andrew@cloudcumin1001 cvn START - Cookbook wmcs.nfs.migrate_service (T430589) [17:30:38] !log andrew@cloudcumin1001 cvn END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T430589) [17:41:32] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 06SRE: Wikimania 2026: Request to whitelist venue IP for all Wikimedia projects, Toolforge, and Cloud VPS etc... - https://phabricator.wikimedia.org/T432377#12129582 (10Michael_Barbereau_WMFr) [17:42:44] 06tools-infrastructure-team, 10Countervandalism-Network: CVN: Replace Bullseye nfs server with a new Trixie server - https://phabricator.wikimedia.org/T430589#12129585 (10Andrew) This is now done. I did wind up doing a staggered reboot of all servers because NFS clients are not great at handling change. The o... [17:43:14] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Update all cloud-vps Bullseye NFS servers - https://phabricator.wikimedia.org/T429793#12129587 (10Andrew) 05Open→03Resolved [17:44:00] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 06SRE: Wikimania 2026: Request to whitelist venue IP for all Wikimedia projects, Toolforge, and Cloud VPS etc... - https://phabricator.wikimedia.org/T432377#12129589 (10Michael_Barbereau_WMFr) [17:44:46] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 06SRE: Wikimania 2026: Request to whitelist venue IP for all Wikimedia projects, Toolforge, and Cloud VPS etc... - https://phabricator.wikimedia.org/T432377#12129592 (10Michael_Barbereau_WMFr) [18:00:36] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 06SRE: Wikimania 2026: Request to exempt venue IP for all Wikimedia projects, Toolforge, and Cloud VPS etc... - https://phabricator.wikimedia.org/T432377#12129642 (10taavi) [18:00:57] (03CR) 10Alien4444: Readd third-party suppport (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1277281 (https://phabricator.wikimedia.org/T344090) (owner: 10Alien4444) [18:01:26] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 06SRE: Wikimania 2026: Request to exempt venue IP for all Wikimedia projects, Toolforge, and Cloud VPS etc... - https://phabricator.wikimedia.org/T432377#12129647 (10taavi) Former Wikimania events have not needed exemptions fro... [18:10:43] (03open) 10taavi: Remove mention of SRE meeting delegate [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/5 [18:15:33] 10Tool-wiki-gender-stats: Typos in Esperanto translations - https://phabricator.wikimedia.org/T432223#12129717 (10Danya) 05Open→03In progress [18:15:57] (03open) 10danyya: Draft: Resolve T432223 "Fix eo typos" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/82 [18:16:04] (03update) 10danyya: Draft: Resolve T432223 "Typos in Esperanto translations" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/82 [18:20:52] 10Tool-wikinewsie: "About" link for Wikinewsie - https://phabricator.wikimedia.org/T431955#12129735 (10ZhaoFJx) 05In progress→03Resolved [18:22:54] (03update) 10danyya: Draft: Resolve T432223 "Typos in Esperanto translations" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/82 [18:23:02] (03update) 10danyya: Resolve T432223 "Typos in Esperanto translations" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/82 [18:24:26] (03merge) 10danyya: Resolve T432223 "Typos in Esperanto translations" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/82 [18:24:43] 10Tool-wiki-gender-stats, 13Patch-For-Review: Typos in Esperanto translations - https://phabricator.wikimedia.org/T432223#12129746 (10Danya) 05In progress→03Resolved [18:25:06] 10Tool-wiki-gender-stats: Show percentages in aggregated countries dialog - https://phabricator.wikimedia.org/T432297#12129749 (10Danya) 05Open→03In progress [18:33:46] (03open) 10danyya: Draft: Resolve T432297 "Show percentages in aggregated countries dialog" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/83 [19:00:15] 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12129847 (10bking) [19:17:06] 06tools-infrastructure-team, 10wikitech.wikimedia.org: wikitech-static: get better at cleaning up old containers and/or establish a cadence for rebuilding the host - https://phabricator.wikimedia.org/T432391 (10Andrew) 03NEW [19:33:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:50:45] 10Tool-wiki-gender-stats: Show Wikidata’s icon next to links to this wiki - https://phabricator.wikimedia.org/T432395 (10Danya) 03NEW [19:51:04] 10Tool-wiki-gender-stats: Show Wikidata’s icon next to links to this wiki - https://phabricator.wikimedia.org/T432395#12129991 (10Danya) [19:51:07] 10Tool-wiki-gender-stats: Show Wikidata’s icon next to links to this wiki - https://phabricator.wikimedia.org/T432395#12129995 (10Danya) p:05Triage→03High [19:51:29] 10Tool-wiki-gender-stats: [VERSION] 0.3.0 - https://phabricator.wikimedia.org/T427979#12130000 (10Danya) [19:51:30] 10Tool-wiki-gender-stats: Show Wikidata’s icon next to links to this wiki - https://phabricator.wikimedia.org/T432395#12129999 (10Danya) [19:53:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:56:08] (03update) 10danyya: Draft: Resolve T432297 "Show percentages in aggregated countries dialog" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/83 [19:57:23] (03update) 10danyya: Resolve T432297 "Show percentages in aggregated countries dialog" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/83 [19:57:27] (03merge) 10danyya: Resolve T432297 "Show percentages in aggregated countries dialog" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/83 [19:58:36] (03open) 10danyya: Draft: Resolve T432395 "Show Wikidata’s icon next to links to this wiki" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/84 [20:05:02] 06tools-infrastructure-team, 10Countervandalism-Network: CVN: Replace Bullseye nfs server with a new Trixie server - https://phabricator.wikimedia.org/T430589#12130096 (10Krinkle) 05Open→03Resolved @Andrew Thanks, the sync bridge reflects changes from <2 minutes ago. LGTM! [20:15:00] 10Tools, 07Performance Issue: Stalktoy is often slow or not responding - https://phabricator.wikimedia.org/T427941#12130129 (10Pathoschild) Stalktoy runs by itself on the `meta3` account. When the issue happens, requests begin taking a very long time to complete (even trivial requests like loading an empty fo... [20:34:07] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Rip out barbican puppet code - https://phabricator.wikimedia.org/T432349#12130202 (10BLiviero-WMF) to add a little commentary on what exists out there that references it... on cloudcontrol2005-dev, 2006-dev, 2010-dev there appears to be a nigh... [20:36:31] (03update) 10danyya: Draft: Resolve T432395 "Show Wikidata’s icon next to links to this wiki" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/84 [20:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:02:18] (03update) 10danyya: Draft: Resolve T432395 "Show Wikidata’s icon next to links to this wiki" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/84 [21:11:27] (03update) 10danyya: Draft: Resolve T432395 "Show Wikidata’s icon next to links to this wiki" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/84 [21:30:44] (03update) 10danyya: Draft: Resolve T432395 "Show Wikidata’s icon next to links to this wiki" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/84 [21:30:50] (03update) 10danyya: Resolve T432395 "Show Wikidata’s icon next to links to this wiki" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/84 [21:31:33] (03merge) 10danyya: Resolve T432395 "Show Wikidata’s icon next to links to this wiki" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/84 [21:31:59] 10Tool-wiki-gender-stats, 13Patch-For-Review: Show Wikidata’s icon next to links to this wiki - https://phabricator.wikimedia.org/T432395#12130482 (10Danya) 05In progress→03Resolved [21:33:16] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 10Wikilink-Tool, 07Essential-Work, 06Moderator-Tools-Team (Kanban): wikilink: Replace deprecated Bullseye VM in Cloud VPS - https://phabricator.wikimedia.org/T402055#12130486 (10jsn.sherman) [22:44:10] 10Tool-wikinewsie: Create new Wikidata items from Wikinewsie - https://phabricator.wikimedia.org/T432411 (10Pharos) 03NEW [23:08:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:13:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:15:49] (03update) 10renovatebot: Update pnpm to v11.13.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [23:27:05] 10Cloud-VPS (Debian Bullseye Deprecation), 06The-Wikipedia-Library, 10Wikilink-Tool, 07Essential-Work, 06Moderator-Tools-Team (Kanban): wikilink: Replace deprecated Bullseye VM in Cloud VPS - https://phabricator.wikimedia.org/T402055#12130873 (10jsn.sherman) 05Open→03In progress [23:28:39] 10Cloud-VPS, 06tools-infrastructure-team: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12130881 (10BLiviero-WMF) a:03Andrew