[00:58:02] 10Tool-quickcategories, 10MediaWiki-Action-API, 10Notifications (Echo), 06Traffic: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11787908 (10matmarex) At least the title of the security issue has been made public: "(T420154... [01:46:46] (03PS1) 10Novem Linguae: eslint: more fixes [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268085 (https://phabricator.wikimedia.org/T392531) [01:47:15] (03PS2) 10Novem Linguae: eslint: more fixes [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268085 (https://phabricator.wikimedia.org/T392531) [01:47:16] (03CR) 10CI reject: [V:04-1] eslint: more fixes [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268085 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [01:47:45] (03CR) 10CI reject: [V:04-1] eslint: more fixes [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268085 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [01:48:50] (03PS3) 10Novem Linguae: eslint: more fixes [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268085 (https://phabricator.wikimedia.org/T392531) [02:21:36] (03CR) 10Novem Linguae: build: Enable eslint in `npm test` and make pass (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1267406 (https://phabricator.wikimedia.org/T422228) (owner: 10Krinkle) [02:27:28] (03PS1) 10Novem Linguae: eslint: manual fixes [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268086 (https://phabricator.wikimedia.org/T392531) [02:36:44] (03PS2) 10Novem Linguae: eslint: manual fixes [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268086 (https://phabricator.wikimedia.org/T392531) [02:37:22] (03PS3) 10Novem Linguae: eslint: manual fixes [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268086 (https://phabricator.wikimedia.org/T392531) [05:05:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [05:10:17] FIRING: [2x] PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [05:34:03] 10Tool-refill, 07good first task: reFill UI still shows links to former maintainer’s user pages and GitHub account - https://phabricator.wikimedia.org/T421478#11788010 (10Alachuckthebuck) I hope you get well soon! In the meantime, you can add me as maintainer on toolsadmin, and I can go ahead and fix the chang... [05:35:17] RESOLVED: [2x] PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [06:17:50] (03CR) 10Krinkle: [C:03+2] eslint: more fixes [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268085 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [06:18:50] (03Merged) 10jenkins-bot: eslint: more fixes [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268085 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [08:56:33] 10Tool-echo-chamber: Echo Chamber API error: You don't have permission - https://phabricator.wikimedia.org/T422318 (10Toadspike) 03NEW [09:07:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [09:37:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [10:01:46] 10Tool-echo-chamber: Echo Chamber API error: You don't have permission - https://phabricator.wikimedia.org/T422318#11788174 (10Johannnes89) Likely caused by {T420154}, see public task {T421991} describing a similar issue. [10:14:11] (03update) 10booksmurf: Draft: dev- machine translation [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/6 [10:39:30] (03update) 10booksmurf: Draft: dev- machine translation [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/6 [11:08:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [11:38:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [13:44:41] (03CR) 10Alien4444: eslint: manual fixes (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268086 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [13:47:10] 10Tool-wikimonitor: 30-Minute SseEmitter Timeout with Event Resumption - https://phabricator.wikimedia.org/T422194#11788303 (10Praffq) @Gerges I'm adding caching for this, a dequeue , through which our relevant event passes, and then sent to the client for processing. Now when the client goes to timeout , the ev... [17:05:17] FIRING: [2x] PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [17:35:17] RESOLVED: [2x] PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [18:29:05] 10Tool-wikimonitor: 30-Minute SseEmitter Timeout with Event Resumption - https://phabricator.wikimedia.org/T422194#11788417 (10Gerges) @Praffq, You can use via Toolforge: wikitech:Help:Toolforge/Redis It’s a better fit because it provides a shared, persistent queue, supports list operations, and works well if we... [19:38:32] (03PS4) 10Novem Linguae: eslint: manual fixes [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268086 (https://phabricator.wikimedia.org/T392531) [19:38:41] (03CR) 10Novem Linguae: eslint: manual fixes (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268086 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [19:50:31] (03CR) 10Alien4444: [C:03+2] eslint: manual fixes [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268086 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [19:51:29] (03Merged) 10jenkins-bot: eslint: manual fixes [labs/xtools] - 10https://gerrit.wikimedia.org/r/1268086 (https://phabricator.wikimedia.org/T392531) (owner: 10Novem Linguae) [20:34:31] 10Tool-curator: Curator causes critical errors - https://phabricator.wikimedia.org/T422326 (10PantheraLeo1359531) 03NEW [20:34:53] 10Tool-curator: Curator causes critical errors - https://phabricator.wikimedia.org/T422326#11788501 (10PantheraLeo1359531) [20:46:28] 10Tool-quickcategories, 10MediaWiki-Action-API, 10Notifications (Echo), 06Traffic: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11788504 (10Novem_Linguae) My bot is broken as well. Can we consider a backport for https://ge... [20:47:40] 10Tool-quickcategories, 10MediaWiki-Action-API, 10Notifications (Echo), 06Traffic: Notifications API is returning a permissions error since 2026-04-01 for a bot account - https://phabricator.wikimedia.org/T421991#11788505 (10Novem_Linguae) Also, `You don't have permission to $1.` The $1 seems like a minor... [21:05:19] 10Tool-curator: Curator causes critical errors - https://phabricator.wikimedia.org/T422326#11788507 (10DaxServer) 05Open→03Resolved p:05Triage→03Unbreak! a:03DaxServer I ran a migration and updated the worker config, but forgot to restart the worker, thus the worker which was still running remained... [21:07:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [21:20:32] (03PS1) 10Akoopal: Added progress parameter to update_database.py [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1268098 [21:37:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [21:45:08] 10Tool-wikimonitor: 30-Minute SseEmitter Timeout with Event Resumption - https://phabricator.wikimedia.org/T422194#11788536 (10Praffq) @Gerges thanks for suggestion, I'll go through the toolforge redis docs and will try to implement using it [22:10:18] 13Tool-humaniki-2: Migrate to SQL - https://phabricator.wikimedia.org/T422329 (10Danya) 03NEW [22:10:47] 13Tool-humaniki-2: Migrate to SQL - https://phabricator.wikimedia.org/T422329#11788575 (10Danya) 05Open→03In progress p:05Triage→03High [22:13:21] 13Tool-humaniki-2: Migrate to SQL - https://phabricator.wikimedia.org/T422329#11788577 (10Danya) [22:15:55] 13Tool-humaniki-2: Add button to save page state as GET params - https://phabricator.wikimedia.org/T422330 (10Danya) 03NEW [22:17:53] 13Tool-humaniki-2: List data sources - https://phabricator.wikimedia.org/T422332 (10Danya) 03NEW [22:19:20] 13Tool-humaniki-2: Fix data loading error message flicker - https://phabricator.wikimedia.org/T422333 (10Danya) 03NEW [22:19:50] 13Tool-humaniki-2: Fix data loading error message flicker - https://phabricator.wikimedia.org/T422333#11788626 (10Danya) a:03Danya [22:22:51] 13Tool-humaniki-2: [Language view] Show stats for all wiki languages - https://phabricator.wikimedia.org/T422334 (10Danya) 03NEW [22:24:44] 13Tool-humaniki-2: Keep filters states when updating datasets - https://phabricator.wikimedia.org/T422335 (10Danya) 03NEW [22:25:22] 13Tool-humaniki-2: Choose licence for charts - https://phabricator.wikimedia.org/T422336 (10Danya) 03NEW [22:26:33] 13Tool-humaniki-2: Add field to filter gender tables - https://phabricator.wikimedia.org/T422337 (10Danya) 03NEW [22:27:55] 13Tool-humaniki-2: Fix illegible colors in per-language scatter chart - https://phabricator.wikimedia.org/T422338 (10Danya) 03NEW [22:28:58] 13Tool-humaniki-2: [Birth-years view] Add a form to filter by date range - https://phabricator.wikimedia.org/T422339 (10Danya) 03NEW [22:30:42] 13Tool-humaniki-2: [Birth-years view] Stats for persons with at least one Wikipedia page - https://phabricator.wikimedia.org/T422340 (10Danya) 03NEW [22:32:09] 13Tool-humaniki-2: [Countries view] Stats based on nationality - https://phabricator.wikimedia.org/T422341 (10Danya) 03NEW [22:32:48] 13Tool-humaniki-2: Flickering data loading error message - https://phabricator.wikimedia.org/T422333#11788729 (10Danya) [22:36:48] 13Tool-humaniki-2: Retro-active statistics - https://phabricator.wikimedia.org/T422342 (10Danya) 03NEW [22:37:44] 13Tool-humaniki-2: Integrate Denelezeh - https://phabricator.wikimedia.org/T422343 (10Danya) 03NEW [22:38:30] 13Tool-humaniki-2: [Countries view] Stats based on nationality - https://phabricator.wikimedia.org/T422341#11788755 (10Danya) [22:40:06] 13Tool-humaniki-2: Header items overflow on mobile screens - https://phabricator.wikimedia.org/T422344 (10Danya) 03NEW [22:41:09] 13Tool-humaniki-2: Header items overflow on mobile screens - https://phabricator.wikimedia.org/T422344#11788768 (10Danya) a:03Danya [22:42:12] 13Tool-humaniki-2: Make visualisation menu not selectable - https://phabricator.wikimedia.org/T422345 (10Danya) 03NEW [22:43:03] 13Tool-humaniki-2: Make app title clickable - https://phabricator.wikimedia.org/T422346 (10Danya) 03NEW [22:43:19] 13Tool-humaniki-2: Make app title clickable - https://phabricator.wikimedia.org/T422346#11788796 (10Danya) p:05Triage→03Low [22:43:25] 13Tool-humaniki-2: Make visualisation menu not selectable - https://phabricator.wikimedia.org/T422345#11788797 (10Danya) p:05Triage→03Low [22:43:32] 13Tool-humaniki-2: Header items overflow on mobile screens - https://phabricator.wikimedia.org/T422344#11788798 (10Danya) p:05Triage→03High [22:43:49] 13Tool-humaniki-2: Add button to save page state as GET params - https://phabricator.wikimedia.org/T422330#11788799 (10Danya) p:05Triage→03Low [22:43:56] 13Tool-humaniki-2: List data sources - https://phabricator.wikimedia.org/T422332#11788800 (10Danya) p:05Triage→03High [22:44:18] 13Tool-humaniki-2: [Language view] Show stats for all wiki languages - https://phabricator.wikimedia.org/T422334#11788801 (10Danya) p:05Triage→03High [22:44:24] 13Tool-humaniki-2: Integrate Denelezeh - https://phabricator.wikimedia.org/T422343#11788802 (10Danya) p:05Triage→03High [22:44:35] 13Tool-humaniki-2: [Countries view] Stats based on nationality - https://phabricator.wikimedia.org/T422341#11788803 (10Danya) p:05Triage→03High [22:44:48] 13Tool-humaniki-2: [Birth-years view] Stats for persons with at least one Wikipedia page - https://phabricator.wikimedia.org/T422340#11788804 (10Danya) p:05Triage→03High [22:45:01] 13Tool-humaniki-2: Fix illegible colors in per-language scatter chart - https://phabricator.wikimedia.org/T422338#11788805 (10Danya) p:05Triage→03High [22:46:20] 13Tool-humaniki-2: Retro-active statistics - https://phabricator.wikimedia.org/T422342#11788819 (10Danya) p:05Triage→03Low [22:47:23] 13Tool-humaniki-2: Flickering data loading error message - https://phabricator.wikimedia.org/T422333#11788824 (10Danya) p:05Triage→03High [22:48:09] 13Tool-humaniki-2: Migrate to SQLite - https://phabricator.wikimedia.org/T422329#11788826 (10Danya) [22:48:39] 13Tool-humaniki-2: [Birth-years view] Add a form to filter by date range - https://phabricator.wikimedia.org/T422339#11788827 (10Danya) p:05Triage→03Medium [22:48:45] 13Tool-humaniki-2: Add field to filter gender tables - https://phabricator.wikimedia.org/T422337#11788828 (10Danya) p:05Triage→03Medium [22:48:54] 13Tool-humaniki-2: Choose licence for charts - https://phabricator.wikimedia.org/T422336#11788829 (10Danya) p:05Triage→03Medium [22:49:00] 13Tool-humaniki-2: Keep filters states when updating datasets - https://phabricator.wikimedia.org/T422335#11788830 (10Danya) p:05Triage→03Medium [22:53:21] 13Tool-humaniki-2: Setup push to deploy - https://phabricator.wikimedia.org/T422347 (10Danya) 03NEW [22:53:25] 13Tool-humaniki-2: Setup push to deploy - https://phabricator.wikimedia.org/T422347#11788847 (10Danya) p:05Triage→03High [22:54:12] 13Tool-humaniki-2: Setup unit tests - https://phabricator.wikimedia.org/T422348 (10Danya) 03NEW [22:54:25] 13Tool-humaniki-2: Setup unit tests - https://phabricator.wikimedia.org/T422348#11788860 (10Danya) p:05Triage→03High [22:54:46] 13Tool-humaniki-2: List data sources - https://phabricator.wikimedia.org/T422332#11788862 (10Danya) p:05High→03Medium [22:55:08] 13Tool-humaniki-2: Setup unit tests - https://phabricator.wikimedia.org/T422348#11788866 (10Danya) 05Open→03In progress [22:55:38] 13Tool-humaniki-2: Flickering data loading error message - https://phabricator.wikimedia.org/T422333#11788867 (10Danya) p:05High→03Medium [22:55:40] 13Tool-humaniki-2: Header items overflow on mobile screens - https://phabricator.wikimedia.org/T422344#11788868 (10Danya) p:05High→03Medium [22:58:15] (03update) 10danyya: Draft: Migrate to SQLite [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/3 [23:05:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [23:06:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [23:35:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted