[00:05:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:10:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:18:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:23:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:30:09] (03PS2) 10Jsn.sherman: EditCounterRepository: unit test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312259 (https://phabricator.wikimedia.org/T384711) [01:30:09] (03PS1) 10Jsn.sherman: Repository: unbound params in untested branches [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312267 (https://phabricator.wikimedia.org/T384711) [01:30:11] (03PS1) 10Jsn.sherman: Repository: unit test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312268 (https://phabricator.wikimedia.org/T384711) [01:53:51] (03PS2) 10Jsn.sherman: Repository: unbound params in untested branches [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312267 (https://phabricator.wikimedia.org/T384711) [01:53:52] (03PS2) 10Jsn.sherman: Repository: unit test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312268 (https://phabricator.wikimedia.org/T384711) [02:00:08] (03PS4) 10Jsn.sherman: Repository: config-driven dblist assembly [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309801 (https://phabricator.wikimedia.org/T344090) [02:00:08] (03PS3) 10Jsn.sherman: EditCounterRepository: unit test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312259 (https://phabricator.wikimedia.org/T384711) [02:00:08] (03PS3) 10Jsn.sherman: Repository: unbound params in untested branches [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312267 (https://phabricator.wikimedia.org/T384711) [02:00:09] (03PS3) 10Jsn.sherman: Repository: unit test coverage [labs/xtools] - 10https://gerrit.wikimedia.org/r/1312268 (https://phabricator.wikimedia.org/T384711) [02:32:11] FIRING: SystemdUnitDown: The systemd unit security_group_ssh-from-restricted-bastion_to_project_trove.service on node cloudcontrol1006 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:59:28] (03open) 10priyankar22: Set Secure flag on session cookie [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/13 (https://phabricator.wikimedia.org/T431897) [05:01:30] 10Tool-sentinel, 06RoadToWiki, 13Patch-For-Review: Set Secure flag on session cookie - https://phabricator.wikimedia.org/T431897#12134928 (10Priyankar22) [05:01:39] 10Tool-sentinel, 06RoadToWiki, 13Patch-For-Review: Set Secure flag on session cookie - https://phabricator.wikimedia.org/T431897#12134930 (10Priyankar22) I've submitted a fix: https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/13 [05:16:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:21:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:33:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:38:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:32:11] FIRING: SystemdUnitDown: The systemd unit security_group_ssh-from-restricted-bastion_to_project_trove.service on node cloudcontrol1006 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:32:37] 10Tool-documentation, 10Adiutor, 06Wiki-Mentor-Africa, 07good first task: [Documentation] Update and improve docs for Adiutor tool - https://phabricator.wikimedia.org/T390392#12134974 (10Priyankar22) Hi @TBurmeister, I posted on Talk:Adiutor with the draft link and have gone ahead and published it to the m... [06:52:59] (03open) 10countcount: Show a fallback message when JavaScript is unavailable [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/8 [06:55:45] (03open) 10countcount: Show a fallback message when JavaScript is unavailable [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/9 [06:56:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:07:41] (03merge) 10countcount: Show a fallback message when JavaScript is unavailable [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/9 [07:08:04] (03close) 10countcount: Show a fallback message when JavaScript is unavailable [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/8 [07:12:31] (03open) 10countcount: Link AGENTS.md to CLAUDE.md [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/10 [07:12:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [07:15:16] (03merge) 10countcount: Link AGENTS.md to CLAUDE.md [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/10 [07:16:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:17:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [08:14:09] (03update) 10renovatebot: Update pnpm to v11.15.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [08:14:28] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/8 [08:14:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/8 [09:19:04] 10Tool-wiki-gender-stats: Add button to download charts as PNGs - https://phabricator.wikimedia.org/T432545 (10Danya) 03NEW [09:19:26] 10Tool-wiki-gender-stats: Add button to download charts as PNGs - https://phabricator.wikimedia.org/T432545#12135037 (10Danya) p:05Triage→03Medium [09:21:04] 10Tool-wiki-gender-stats: Add button to download charts as PNGs - https://phabricator.wikimedia.org/T432545#12135038 (10Danya) [09:24:28] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12135039 (10Kengkong1) @Bodhisattwa - Awesome thanks [09:47:57] 10Tool-wiki-gender-stats: Add page to query data - https://phabricator.wikimedia.org/T425023#12135061 (10Danya) [09:47:58] 10Tool-wiki-gender-stats: Migrate to ToolsDB - https://phabricator.wikimedia.org/T432053#12135062 (10Danya) [09:49:11] 10Tool-wiki-gender-stats: Add page to query data - https://phabricator.wikimedia.org/T425023#12135063 (10Danya) 05Open→03Stalled Stalled until T432053 is resolved, because the current SQLite database is way too slow for making SQL queries on non-aggregated tables. [10:32:11] FIRING: SystemdUnitDown: The systemd unit security_group_ssh-from-restricted-bastion_to_project_trove.service on node cloudcontrol1006 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:06:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:10:49] 10Tool-wiki-gender-stats: Website unable to access database while dumps ingester is running - https://phabricator.wikimedia.org/T432052#12135112 (10Danya) **Analysis:** Whenever a process is writing data into the database, SQLite locks the file. This prevents any other process from writing to the file and even r... [11:11:27] 10Tool-wiki-gender-stats: Website unable to access database while dumps ingester is running - https://phabricator.wikimedia.org/T432052#12135114 (10Danya) [11:11:33] 10Tool-wiki-gender-stats: Non-Wikipedia wikis categorized as Wikipedia language codes - https://phabricator.wikimedia.org/T432474#12135115 (10Danya) [11:11:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:11:55] 10Tool-wiki-gender-stats: Chart filter states reset when dataset changes - https://phabricator.wikimedia.org/T432240#12135116 (10Danya) [11:12:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:12:41] 10Tool-wiki-gender-stats: Content-Security-Policy when loading OSM map tiles - https://phabricator.wikimedia.org/T432531#12135117 (10Danya) [11:13:11] 10Tool-wiki-gender-stats: Chart filter states reset when dataset changes - https://phabricator.wikimedia.org/T432240#12135118 (10Danya) [11:13:22] 10Tool-wiki-gender-stats: Website unable to access database while dumps ingester is running - https://phabricator.wikimedia.org/T432052#12135119 (10Danya) [11:13:29] 10Tool-wiki-gender-stats: Non-Wikipedia wikis categorized as Wikipedia language codes - https://phabricator.wikimedia.org/T432474#12135123 (10Danya) [11:22:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:57:20] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12135141 (10Kengkong1) Hello Team. I hope everyone is having a good day. As I... [12:12:12] (03PS7) 10Jsn.sherman: Repository: cover isWMF branches [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309830 (https://phabricator.wikimedia.org/T344090) [12:16:53] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12135144 (10BabyJat) Good afternoon team , Kengkong1 has identified an import... [12:29:04] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12135146 (10Bodhisattwa) >>! In T432095#12135141, @Kengkong1 wrote: > Hello Te... [12:37:36] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12135149 (10Kengkong1) Hi @Dagmawi-M. I hope your day is going well. Is it pos... [12:58:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [13:02:13] (03open) 10countcount: Add AGENTS.md instruction entrypoint [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/10 [13:02:44] (03update) 10countcount: Add AGENTS.md instruction entrypoint [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/10 [13:03:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [13:04:13] (03merge) 10countcount: Add AGENTS.md instruction entrypoint [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/10 [13:13:57] (03update) 10renovatebot: Update pnpm to v11.15.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [13:23:36] (03open) 10countcount: Build the Name History web app [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/1 [13:23:43] (03update) 10countcount: Build the Name History web app [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/1 [13:36:20] (03update) 10countcount: Build the Name History web app [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/1 [13:41:21] (03merge) 10countcount: Build the Name History web app [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/1 [13:54:33] (03open) 10countcount: Add wiki-aware contribution links and block status [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/11 [13:56:20] (03update) 10countcount: Add wiki-aware contribution links and block status [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/11 [14:09:09] (03merge) 10countcount: Add wiki-aware contribution links and block status [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/11 [14:14:05] (03update) 10renovatebot: Update pnpm to v11.15.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [14:32:12] FIRING: SystemdUnitDown: The systemd unit security_group_ssh-from-restricted-bastion_to_project_trove.service on node cloudcontrol1006 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:36:08] (03open) 10countcount: fix deployment script [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/12 [14:37:28] (03merge) 10countcount: fix deployment script [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/12 [15:14:12] (03update) 10renovatebot: Update pnpm to v11.15.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [15:31:57] RESOLVED: SystemdUnitDown: The systemd unit security_group_ssh-from-restricted-bastion_to_project_trove.service on node cloudcontrol1006 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:42:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [15:47:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [15:50:38] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12135265 (10Kengkong1) But yeah the protoype looks pretty cool! [16:14:25] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/8 [16:28:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services [16:31:22] FIRING: [6x] HAProxyBackendUnavailable: HAProxy service glance-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [16:34:56] FIRING: SystemdUnitDown: The service unit designate-producer.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:36:22] RESOLVED: [13x] HAProxyBackendUnavailable: HAProxy service glance-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [16:39:56] FIRING: [3x] SystemdUnitDown: The service unit designate-producer.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:44:53] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services [17:09:56] RESOLVED: [2x] SystemdUnitDown: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:46:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:51:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:52:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:00:15] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12135393 (10Dagmawi-M) Hi @Kengkong1, hope yours is going well too yes... I... [19:02:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:05:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:10:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:54:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:04:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:34:13] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12135437 (10Bugreporter2) Is this not a duplicate of {T426106} now? [23:14:12] (03update) 10renovatebot: Update pnpm to v11.15.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [23:14:31] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [23:15:31] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [23:15:32] (03update) 10renovatebot: Update pnpm to v11.15.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [23:16:32] (03update) 10renovatebot: Update pnpm to v11.15.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 [23:17:20] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12135448 (10Kengkong1) {F94714612} Hi team the backend functionality is ready.... [23:17:59] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 03Wikimania-Hackathon-2026: [Team Challenges #07E] Wikidata–Wikisource Lexicographical Data Visualization Tool - https://phabricator.wikimedia.org/T432095#12135449 (10Kengkong1) Hi @Dagmawi-M - no worries and safe travels! [23:31:49] (03merge) 10samwilson: Handle missing target language in translation requests [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/17 (https://phabricator.wikimedia.org/T431646) (owner: 10zaidusyy) [23:39:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:44:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:44:56] (03open) 10samwilson: Update invalid-request error message [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/18 (https://phabricator.wikimedia.org/T431646) [23:49:11] (03merge) 10samwilson: Update invalid-request error message [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/18 (https://phabricator.wikimedia.org/T431646)