[00:06:21] FIRING: PrometheusK8sCertExpirySoon: Prometheus k8s certificate is about to expire - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusK8sCertExpirySoon [00:50:38] FIRING: [4x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:55:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [00:56:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [01:00:32] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [01:05:32] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [01:06:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [01:50:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [01:55:32] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [01:55:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [02:00:32] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [02:00:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [02:33:33] RESOLVED: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [02:43:33] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [03:11:35] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11877662 (10Hasish_Infant) Hi, I'm Hasish Infant, a React/TypeScript developer from Bengaluru, India. I submitted a GSoC 2026 proposal for this project (CampWiz NxT migration to pure... [06:25:37] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11877748 (10Gopavasanth) a:03Osuji_pius [06:34:36] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/22 [06:34:44] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/22 [06:34:54] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/167 [06:35:05] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/167 [06:35:22] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/48 [06:35:25] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/48 [06:35:35] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/295 [06:35:37] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/295 [06:48:21] 10Tool-bullseye: Update maxmind GeoLite2 mmdb files - https://phabricator.wikimedia.org/T425049 (10revi) 03NEW [07:10:20] (03PS1) 10MusikAnimal: Authorship: enable dsb, fa, hi, ru, sv, uk, vi Wikipedias [labs/xtools] - 10https://gerrit.wikimedia.org/r/1281309 (https://phabricator.wikimedia.org/T372340) [07:18:33] RESOLVED: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [07:25:33] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [07:36:48] (03update) 10bd808: Draw the rest of the owl [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/1 (https://phabricator.wikimedia.org/T422958 https://phabricator.wikimedia.org/T423014) [07:52:50] (03update) 10bd808: Draw the rest of the owl [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/1 (https://phabricator.wikimedia.org/T422958 https://phabricator.wikimedia.org/T423014) [07:54:03] 10Tool-etherpad-backup, 03Wikimedia-Hackathon-2026: Archive etherpads mentioned in hackathon/techconf tasks - https://phabricator.wikimedia.org/T417207#11877985 (10bd808) [07:54:30] 10Tool-etherpad-backup, 13Patch-For-Review, 03Wikimedia-Hackathon-2026: Setup proof of concept storage and retrieval from WMCS object storage - https://phabricator.wikimedia.org/T422958#11877987 (10bd808) [07:58:22] 10Tool-curator: Resolve Mapillary sequences from URL - https://phabricator.wikimedia.org/T425054 (10DaxServer) 03NEW [07:58:40] 10Tool-curator: Resolve Mapillary sequences from URL - https://phabricator.wikimedia.org/T425054#11878005 (10DaxServer) p:05Triage→03High [07:59:35] 10Tool-curator, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: Add advanced filtering, better queuing and improvements for Curator - https://phabricator.wikimedia.org/T424994#11878024 (10DaxServer) [08:02:38] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11878064 (10Nokib_Sarkar) @Hasish_Infant You are absolutely welcome to our community and we would love to have you among us. But, the project is not only migration, but also redesigni... [08:11:40] (03update) 10bd808: Draw the rest of the owl [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/1 (https://phabricator.wikimedia.org/T422958 https://phabricator.wikimedia.org/T423014) [08:15:46] (03update) 10bd808: Draw the rest of the owl [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/1 (https://phabricator.wikimedia.org/T422958 https://phabricator.wikimedia.org/T423014) [08:29:54] (03merge) 10bd808: Draw the rest of the owl [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/1 (https://phabricator.wikimedia.org/T422958 https://phabricator.wikimedia.org/T423014) [09:04:32] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: linter: Refactor test suite to use helper spec builders instead of inline YAML strings - https://phabricator.wikimedia.org/T425068 (10KineticPelagic) 03NEW [09:06:48] 10Tool-glam-tool-hospital, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: GLAM Tool Hospital - https://phabricator.wikimedia.org/T424464#11878332 (10Quiddity) Hi, you probably saw it already, but just in case: You might find this page and associated tool to be useful (for ideas, if not more!): * https:/... [09:08:17] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: linter: Refactor test suite to use helper spec builders instead of inline YAML strings - https://phabricator.wikimedia.org/T425068#11878334 (10KineticPelagic) [09:09:21] (03PS5) 10Volans: wm_gitlab: fix getting project [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1280283 [09:10:04] (03CR) 10Volans: wm_gitlab: fix getting project (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1280283 (owner: 10Volans) [09:24:02] (03update) 10kineticpelagic: Add rules for operations and paths (T422504) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/17 [09:34:11] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work, 13Patch-For-Review: Add linting rules for operations and paths - https://phabricator.wikimedia.org/T422504#11878428 (10KineticPelagic) @KBach , @AGhirelli-WMF perceptively made [[... [09:37:23] 10Tool-glam-tool-hospital, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: GLAM Tool Hospital - https://phabricator.wikimedia.org/T424464#11878443 (10AKanji-WMF) I'd love to help with this, I've created part of this flow for my internal team use at WMF. I'll come find you! [09:46:34] (03CR) 10Alien4444: [C:03+2] Authorship: enable dsb, fa, hi, ru, sv, uk, vi Wikipedias [labs/xtools] - 10https://gerrit.wikimedia.org/r/1281309 (https://phabricator.wikimedia.org/T372340) (owner: 10MusikAnimal) [09:48:08] (03Merged) 10jenkins-bot: Authorship: enable dsb, fa, hi, ru, sv, uk, vi Wikipedias [labs/xtools] - 10https://gerrit.wikimedia.org/r/1281309 (https://phabricator.wikimedia.org/T372340) (owner: 10MusikAnimal) [09:55:02] (03open) 10bd808: web: Add X-Forwarded-Proto support for https [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/2 [10:06:10] (03merge) 10bd808: web: Add X-Forwarded-Proto support for https [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/2 [10:08:35] 10Tool-ghanasupremecases1, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: SCC TOOL - https://phabricator.wikimedia.org/T424977#11878547 (10Aklapper) [10:08:58] 10Tool-glam-tool-hospital: [TEST] GLAMorgan - Bug - Test - https://phabricator.wikimedia.org/T425078 (10Dactylantha) 03NEW [10:11:47] 10Tool-ghanasupremecases1: ghanasupremecases.toolforge.org violates Privacy Policy - https://phabricator.wikimedia.org/T425079 (10Aklapper) 03NEW [10:12:19] 10Tool-glam-tool-hospital: [TEST] Mix'n'match - Bug - test - https://phabricator.wikimedia.org/T425080 (10Dactylantha) 03NEW [10:40:02] 10Tool-humaniki-2: Remove assets/project_languages.json - https://phabricator.wikimedia.org/T425085 (10Danya) 03NEW [10:40:07] 10Tool-humaniki-2: Remove assets/project_languages.json - https://phabricator.wikimedia.org/T425085#11878689 (10Danya) p:05Triage→03High [10:40:59] 10Tool-humaniki-2: Reduce snapshots frequency - https://phabricator.wikimedia.org/T424277#11878691 (10Danya) [10:41:08] 10Tool-humaniki-2: List data sources - https://phabricator.wikimedia.org/T422332#11878693 (10Danya) [10:41:15] 10Tool-humaniki-2: Choose licence for website’s content - https://phabricator.wikimedia.org/T422336#11878694 (10Danya) [10:41:35] 10Tool-humaniki-2: Button to download the whole database? - https://phabricator.wikimedia.org/T424282#11878695 (10Danya) [10:41:46] 10Tool-humaniki-2: Add button to save page state as GET params - https://phabricator.wikimedia.org/T422330#11878696 (10Danya) [10:42:17] 10Tool-humaniki-2: Add "About" page - https://phabricator.wikimedia.org/T425022#11878697 (10Danya) [10:42:22] 10Tool-humaniki-2: Add page to query data - https://phabricator.wikimedia.org/T425023#11878698 (10Danya) [10:42:32] 10Tool-humaniki-2: Use transactions for snapshots - https://phabricator.wikimedia.org/T424903#11878699 (10Danya) [10:42:41] 10Tool-humaniki-2: Prevent page freeze when loading data - https://phabricator.wikimedia.org/T424578#11878704 (10Danya) [10:42:49] 10Tool-humaniki-2: Header items overflow on mobile screens - https://phabricator.wikimedia.org/T422344#11878705 (10Danya) [10:42:59] 10Tool-humaniki-2: Notify tool’s devs when snapshot capture fails - https://phabricator.wikimedia.org/T424279#11878706 (10Danya) [10:43:08] 10Tool-humaniki-2: Add field to filter gender tables - https://phabricator.wikimedia.org/T422337#11878707 (10Danya) [10:43:24] 10Tool-humaniki-2: Keep filters states when updating datasets - https://phabricator.wikimedia.org/T422335#11878708 (10Danya) [10:43:30] 10Tool-humaniki-2: Keep filters states when updating datasets - https://phabricator.wikimedia.org/T422335#11878709 (10Danya) [10:43:36] 10Tool-humaniki-2: Remove assets/project_languages.json - https://phabricator.wikimedia.org/T425085#11878710 (10Danya) [10:43:47] 10Tool-humaniki-2: Move views to separate URLs - https://phabricator.wikimedia.org/T425021#11878713 (10Danya) [10:43:53] 10Tool-humaniki-2: Integrate Denelezh - https://phabricator.wikimedia.org/T422343#11878714 (10Danya) [10:43:57] 10Tool-humaniki-2: Finer-grained data - https://phabricator.wikimedia.org/T425024#11878715 (10Danya) [10:44:07] 10Tool-humaniki-2: Scheduler does not take birth year snapshots - https://phabricator.wikimedia.org/T423025#11878716 (10Danya) [10:44:14] 10Tool-humaniki-2: Scheduler does not take birth year snapshots - https://phabricator.wikimedia.org/T423025#11878717 (10Danya) [10:44:21] 10Tool-humaniki-2: Localize table texts - https://phabricator.wikimedia.org/T423787#11878721 (10Danya) [10:44:24] 10Tool-campwiz-nxt: CampWiz Nxt Redesign: Root Path - https://phabricator.wikimedia.org/T415408#11878718 (10Nokib_Sarkar) 05Open→03Resolved a:03Nokib_Sarkar [10:44:33] 10Tool-humaniki-2: Favicon - https://phabricator.wikimedia.org/T423004#11878722 (10Danya) [10:45:11] 10Tool-humaniki-2: Setup push to deploy - https://phabricator.wikimedia.org/T422347#11878723 (10Danya) [10:45:13] 10Tool-humaniki-2: Setup push to deploy - https://phabricator.wikimedia.org/T422347#11878724 (10Danya) [10:45:30] 10Tool-humaniki-2: Retro-active statistics - https://phabricator.wikimedia.org/T422342#11878725 (10Danya) [10:45:46] 10Tool-humaniki-2: Use dumps? - https://phabricator.wikimedia.org/T423012#11878726 (10Danya) [10:46:39] 10Tool-humaniki-2: Scheduler does not take birth year snapshots - https://phabricator.wikimedia.org/T423025#11878732 (10Danya) [10:53:40] 10Cloud-Services, 06Data-Platform-SRE: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087 (10BTullis) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and re... [10:54:45] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11878762 (10BTullis) a:05KCVelaga_WMF→03BTullis [11:16:58] 10Tool-humaniki-2: Link to Wikidata elements in country stats table - https://phabricator.wikimedia.org/T425089 (10Danya) 03NEW [11:17:05] 10Tool-humaniki-2: Link to Wikidata elements in country stats table - https://phabricator.wikimedia.org/T425089#11878801 (10Danya) [11:17:17] 10Tool-humaniki-2: Link to Wikidata elements in country stats table - https://phabricator.wikimedia.org/T425089#11878803 (10Danya) p:05Triage→03Medium [11:23:27] (03update) 10danyya: Draft: Restructure DB tables [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/22 [11:25:19] 10Cloud-VPS (Project-requests): Request creation of wise VPS project - https://phabricator.wikimedia.org/T424068#11878810 (10Nemoralis) [11:26:39] 10Tool-humaniki-2: Add title to countries map - https://phabricator.wikimedia.org/T425090 (10Danya) 03NEW [11:26:40] 10Tool-humaniki-2: Add title to countries map - https://phabricator.wikimedia.org/T425090#11878825 (10Danya) [11:26:45] 10Tool-humaniki-2: Add title to countries map - https://phabricator.wikimedia.org/T425090#11878827 (10Danya) p:05Triage→03Medium [11:26:54] 10Tool-humaniki-2: Add title to countries map - https://phabricator.wikimedia.org/T425090#11878829 (10Danya) p:05Medium→03High [11:29:53] 10Tool-humaniki-2: Use Pytest’s PYTEST_CURRENT_TEST envvar - https://phabricator.wikimedia.org/T425091 (10Danya) 03NEW [11:29:59] 10Tool-humaniki-2: Use Pytest’s PYTEST_CURRENT_TEST envvar - https://phabricator.wikimedia.org/T425091#11878845 (10Danya) [11:30:15] 10Tool-humaniki-2: Use Pytest’s PYTEST_CURRENT_TEST envvar - https://phabricator.wikimedia.org/T425091#11878847 (10Danya) p:05Triage→03Low [11:34:33] 10Tool-humaniki-2: Fix case in chart titles of languages view - https://phabricator.wikimedia.org/T425092 (10Danya) 03NEW [11:34:37] 10Tool-humaniki-2: Fix case in chart titles of languages view - https://phabricator.wikimedia.org/T425092#11878862 (10Danya) [11:34:42] 10Tool-humaniki-2: Fix case in chart titles of languages view - https://phabricator.wikimedia.org/T425092#11878864 (10Danya) p:05Triage→03Low [11:42:18] 10Tool-humaniki-2: Fix chart titles - https://phabricator.wikimedia.org/T425092#11878869 (10Danya) [11:43:17] 10Tool-humaniki-2: Translate wiki names - https://phabricator.wikimedia.org/T425093 (10Danya) 03NEW [11:43:25] 10Tool-humaniki-2: Translate wiki names - https://phabricator.wikimedia.org/T425093#11878886 (10Danya) [11:43:30] 10Tool-humaniki-2: Translate wiki names - https://phabricator.wikimedia.org/T425093#11878888 (10Danya) p:05Triage→03Low [11:44:51] 10Tool-humaniki-2: Show warning for percentages in tables - https://phabricator.wikimedia.org/T425094 (10Danya) 03NEW [11:44:53] 10Tool-humaniki-2: Show warning for percentages in tables - https://phabricator.wikimedia.org/T425094#11878902 (10Danya) [11:44:58] 10Tool-humaniki-2: Show warning for percentages in tables - https://phabricator.wikimedia.org/T425094#11878904 (10Danya) p:05Triage→03Low [11:45:24] (03update) 10danyya: Restructure DB tables [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/22 [11:47:10] (03merge) 10danyya: Restructure DB tables [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/22 [11:51:03] 10Tool-humaniki-2: Finer-grained data - https://phabricator.wikimedia.org/T425024#11878914 (10Danya) 05In progress→03Resolved a:03Danya [11:54:20] 10Tool-humaniki-2: Make loggers always print to stdout - https://phabricator.wikimedia.org/T425095 (10Danya) 03NEW [11:54:21] 10Tool-humaniki-2: Make loggers always print to stdout - https://phabricator.wikimedia.org/T425095#11878929 (10Danya) p:05Triage→03High [11:54:25] 10Tool-humaniki-2: Make loggers always print to stdout - https://phabricator.wikimedia.org/T425095#11878930 (10Danya) [11:55:58] 10Tool-etherpad-backup, 10Wikimedia-Etherpad: Set a default wikipage target for saving the final state of an Etherpad - https://phabricator.wikimedia.org/T425039#11878937 (10Quiddity) @bd808 This might be relevant to your ideas. [11:56:01] 10Tool-humaniki-2: Make loggers always print to stdout - https://phabricator.wikimedia.org/T425095#11878939 (10Danya) 05Open→03In progress [11:56:02] (03open) 10danyya: Draft: Loggers always stdout [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/23 [11:56:15] 10Tool-humaniki-2: Make loggers always print to stdout - https://phabricator.wikimedia.org/T425095#11878945 (10Danya) a:03Danya [11:59:30] (03update) 10danyya: Draft: Loggers always stdout [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/23 [12:00:13] (03update) 10danyya: Loggers always stdout [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/23 [12:01:11] (03merge) 10danyya: Loggers always stdout [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/23 [12:07:11] 10Tool-humaniki-2: Make loggers always print to stdout - https://phabricator.wikimedia.org/T425095#11878983 (10Danya) 05In progress→03Resolved [12:16:25] (03update) 10pepepiton: Feat: Add pagination to search results and author works list [toolforge-repos/paulina] (pagination-for-search-results-and-works-list) - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/30 (owner: 10system625) [12:20:30] 10Tool-humaniki-2: Save datetime strings instead of timestamps - https://phabricator.wikimedia.org/T425101 (10Danya) 03NEW [12:20:39] 10Tool-humaniki-2: Save datetime strings instead of timestamps - https://phabricator.wikimedia.org/T425101#11879049 (10Danya) [12:20:47] 10Tool-humaniki-2: Save datetime strings instead of timestamps - https://phabricator.wikimedia.org/T425101#11879051 (10Danya) p:05Triage→03High [12:21:20] 10Tool-humaniki-2: Move views to separate URLs - https://phabricator.wikimedia.org/T425021#11879054 (10Danya) [12:22:16] 10Tool-humaniki-2: Favicon - https://phabricator.wikimedia.org/T423004#11879069 (10Danya) [12:24:15] (03update) 10pepepiton: Feat: Add pagination to search results and author works list [toolforge-repos/paulina] (pagination-for-search-results-and-works-list) - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/30 (owner: 10system625) [12:24:41] (03merge) 10pepepiton: Feat: Add pagination to search results and author works list [toolforge-repos/paulina] (pagination-for-search-results-and-works-list) - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/30 (owner: 10system625) [12:28:33] 10Tool-etherpad-backup, 03Wikimedia-Hackathon-2026: Archive etherpads mentioned in hackathon/techconf tasks - https://phabricator.wikimedia.org/T417207#11879119 (10Addshore) Phabricator etherpad links that i believe should be currently complete. Looks like some of them have bad formatting and could be cleaned... [12:30:49] 10Tool-curator: Resolve Mapillary sequences from URL - https://phabricator.wikimedia.org/T425054#11879137 (10Mroik) a:03Mroik I'm working on it! [12:30:53] 10Tool-curator: Allow admins to reset user logins - https://phabricator.wikimedia.org/T424981#11879139 (10DaxServer) 05Open→03In progress a:03DaxServer [12:31:47] (03open) 10bd808: web: add footer links to phabricator and docs [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/3 [12:34:37] 10Tool-curator: Resolve Mapillary sequences from URL - https://phabricator.wikimedia.org/T425054#11879155 (10DaxServer) 05Open→03In progress [12:47:33] 10Tool-etherpad-backup, 03Wikimedia-Hackathon-2026: Archive etherpads mentioned in hackathon/techconf tasks - https://phabricator.wikimedia.org/T417207#11879169 (10Addshore) And some more from some other sources https://phabricator.wikimedia.org/P92135 [12:56:31] (03merge) 10bd808: web: add footer links to phabricator and docs [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/3 [13:07:25] 10Tool-wikidata-geo-dumps, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: Geographic Wikidata dumps - https://phabricator.wikimedia.org/T424869#11879295 (10Danysan1) [13:11:33] 10Tool-curator, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: Add advanced filtering, better queuing and improvements for Curator - https://phabricator.wikimedia.org/T424994#11879304 (10DaxServer) a:03DaxServer [13:25:32] 10Tool-etherpad-backup: Cannot put unicode usernames into "Archived-by" metadata - https://phabricator.wikimedia.org/T425111 (10bd808) 03NEW [13:28:44] 10Tool-curator: Allow admins to reset user logins - https://phabricator.wikimedia.org/T424981#11879329 (10DaxServer) 05In progress→03Resolved [13:30:13] 10Tool-curator: Curator: Uploads by one specific user fail all the time - https://phabricator.wikimedia.org/T424757#11879331 (10DaxServer) 05In progress→03Resolved a:03DaxServer All sessions are invalidated, all users now need to login again. The T424981 made it easy to invalidate the sessions when nee... [13:32:57] 10Tool-curator: Allow users to filter by time elapsed - https://phabricator.wikimedia.org/T424993#11879343 (10DaxServer) 05Open→03In progress a:03DaxServer [13:35:39] 10Tool-wikidata-geo-dumps, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: Geographic Wikidata dumps - https://phabricator.wikimedia.org/T424869#11879348 (10Danysan1) As source for the dumps I will use `/public/dumps/public/wikidatawiki/entities/` which contains the same files listed in https://dumps.wik... [14:21:54] 10Tool-wikidata-geo-dumps, 03Wikimedia-Hackathon-2026: Wikimedia Hackathon 2026: Geographic Wikidata dumps - https://phabricator.wikimedia.org/T424869#11879452 (10Danysan1) I'm generating a dump sample for quicker development with these commands: ` cat /public/dumps/public/wikidatawiki/entities/latest-all.jso... [14:22:17] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#11879453 (10Jclark-ctr) [14:36:56] FIRING: SystemdUnitDown: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudservices1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:40:16] 10Tool-curator: Allow users to filter by time elapsed - https://phabricator.wikimedia.org/T424993#11879534 (10DaxServer) 05In progress→03Resolved [14:43:33] 10Tool-curator: Curator: Select every Xth file of Mapillary sequences - https://phabricator.wikimedia.org/T423095#11879558 (10DaxServer) 05In progress→03Resolved [14:44:28] 10Tool-glam-tool-hospital, 10Phabricator-Bot-Requests: Phabricator Bot Request: GLAM Tool Hospital - https://phabricator.wikimedia.org/T425118 (10Dactylantha) 03NEW [14:57:58] 10Tool-curator: Resolve Mapillary sequences from URL - https://phabricator.wikimedia.org/T425054#11879611 (10DaxServer) [14:58:36] (03open) 10bd808: s3: encode username with xmlcharrefreplace [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/4 (https://phabricator.wikimedia.org/T425111) [14:59:32] 10Tool-humaniki-2: Scheduler does not take birth year snapshots - https://phabricator.wikimedia.org/T423025#11879619 (10Danya) The issue seems to stem from an OOM error. [15:00:39] (03merge) 10bd808: s3: encode username with xmlcharrefreplace [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/4 (https://phabricator.wikimedia.org/T425111) [15:00:56] (03open) 10danyya: Draft: Optimize data capture [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/24 [15:00:57] 10Tool-curator: Allow users to filter by distance separation - https://phabricator.wikimedia.org/T424986#11879623 (10DaxServer) 05Open→03In progress a:03DaxServer [15:02:13] 10Cloud-VPS, 06tools-infrastructure-team: Consider allowing cumin access to all Cloud VPS VMs - https://phabricator.wikimedia.org/T422801#11879629 (10Andrew) After doing some CVE mitigation yesterday, I'm am more convinced that this is something we should do. If we don't care about key rotation then this becom... [15:04:50] 10Tool-etherpad-backup, 13Patch-For-Review, 03Wikimedia-Hackathon-2026: Cannot put unicode usernames into "Archived-by" metadata - https://phabricator.wikimedia.org/T425111#11879631 (10bd808) [15:06:15] 10Tool-etherpad-backup, 13Patch-For-Review, 03Wikimedia-Hackathon-2026: Cannot put unicode usernames into "Archived-by" metadata - https://phabricator.wikimedia.org/T425111#11879635 (10bd808) 05Open→03Resolved a:03bd808 [15:16:02] 10Tool-etherpad-backup, 10Wikimedia-Etherpad: Set a default wikipage target for saving the final state of an Etherpad - https://phabricator.wikimedia.org/T425039#11879654 (10bd808) >>! In T425039#11878936, @Quiddity wrote: > @bd808 This might be relevant to your ideas. As of today, there is a tool at https:/... [15:16:25] 10wikitech.wikimedia.org, 07HTTPS: Firefox: Be careful. Something doesn’t look right (Strict Transport Security) - https://phabricator.wikimedia.org/T425120 (10Ponor) 03NEW [15:17:34] 10Tool-curator: Filtering distance as the crow flies - https://phabricator.wikimedia.org/T424989#11879684 (10DaxServer) 05Open→03In progress a:03DaxServer [15:27:28] 10Tool-etherpad-backup: Special characters are not handled consistently in the current backups, should be improved when migrating to S3 - https://phabricator.wikimedia.org/T423905#11879744 (10bd808) @Tkarcher If you have some examples that you can test at the new https://etherpad-backup.toolforge.org/ frontend t... [15:31:56] 10Tool-glam-tool-hospital: PetScan - Bug - Test - https://phabricator.wikimedia.org/T425124 (10Dactylantha) 03NEW [15:31:56] RESOLVED: SystemdUnitDown: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudservices1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:32:48] 10Tool-glam-tool-hospital: PetScan - Bug - Test - https://phabricator.wikimedia.org/T425124#11879782 (10Dactylantha) 05Open→03Resolved a:03Dactylantha [15:35:15] 10Tool-curator: Filtering distance as the crow flies - https://phabricator.wikimedia.org/T424989#11879788 (10DaxServer) [15:35:32] 10Tool-curator: Filtering distance as the crow flies - https://phabricator.wikimedia.org/T424989#11879789 (10DaxServer) [15:44:30] 10Tool-glam-tool-hospital: Mix'n'match - Support request - Test - https://phabricator.wikimedia.org/T425125 (10Dactylantha) 03NEW [15:49:10] 10Tool-curator: Filtering distance as the crow flies - https://phabricator.wikimedia.org/T424989#11879828 (10DaxServer) 05In progress→03Resolved [15:49:59] 10Tool-curator: Filtering distance while traversing the sequence - https://phabricator.wikimedia.org/T424991#11879831 (10DaxServer) 05Open→03In progress a:03DaxServer [15:52:25] 06cloud-services-team, 10Cloud-VPS, 10Tool-Pageviews, 06Community-Tech (Fox Squad), 07Essential-Work: [Cloud VPS alert][commtech] Puppet failure on pageviews02.commtech.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T424813#11879842 (10TheresNoTime) 05Open→03Resolved [15:52:51] 10Tool-etherpad-backup: Special characters are not handled consistently in the current backups, should be improved when migrating to S3 - https://phabricator.wikimedia.org/T423905#11879847 (10bd808) I used `test-*,?,\,/,%,>,<,🤖💬` as a title string in the new pad creation form. That made the pad at https://etherp... [16:00:23] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11879870 (10Ottomata) > When they are on kafka-jumbo, we will be able to build a pipeline that pulls these logs and creates... [16:12:28] 10Tool-curator: Filtering distance while traversing the sequence - https://phabricator.wikimedia.org/T424991#11879897 (10DaxServer) 05In progress→03Resolved [16:12:45] 10Tool-curator: Allow users to filter by distance separation - https://phabricator.wikimedia.org/T424986#11879899 (10DaxServer) 05In progress→03Resolved [16:13:18] 10Tool-glam-tool-hospital: Pattypan / PattyPan - Support request - Tes t test tses - https://phabricator.wikimedia.org/T425127 (10Dactylantha) 03NEW [16:14:04] 10Tool-glam-tool-hospital: Pattypan / PattyPan - Support request - Tes t test tses - https://phabricator.wikimedia.org/T425127#11879908 (10Dactylantha) 05Open→03Resolved a:03Dactylantha [16:18:46] (03CR) 10FNegri: [C:03+1] wm_gitlab: fix getting project (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1280283 (owner: 10Volans) [16:27:19] 10Tool-glam-tool-hospital: QuickStatements - Bug - Test - https://phabricator.wikimedia.org/T425129 (10Dactylantha) 03NEW [16:28:20] 10Tool-glam-tool-hospital: QuickStatements - Bug - Test - https://phabricator.wikimedia.org/T425129#11879975 (10Dactylantha) 05Open→03Resolved a:03Dactylantha [16:59:16] 10Tool-glam-tool-hospital, 10Phabricator-Bot-Requests: Phabricator Bot Request: GLAM Tool Hospital - https://phabricator.wikimedia.org/T425118#11880056 (10Addshore) a:03Aklapper [17:11:39] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11880086 (10BTullis) >>! In T425087#11879870, @Ottomata wrote: >> When they are on kafka-jumbo, we will be able to build a... [17:56:17] 10Tool-etherpad-backup: Special characters are not handled consistently in the current backups, should be improved when migrating to S3 - https://phabricator.wikimedia.org/T423905#11880134 (10Tkarcher) Cool! I like the new form. Apart from the encoding part: How am I supposed to back up https://etherpad.wikimed... [18:25:22] 10Tool-glam-tool-hospital, 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎): Phabricator Bot Request: GLAM Tool Hospital - https://phabricator.wikimedia.org/T425118#11880190 (10Aklapper) 05Open→03Resolved Created @GLAM-Tool-Hospital-Bot. You can find the API Token in non-public P92137. [18:30:34] RESOLVED: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [18:32:08] 10Tool-inteGraality: Re-evaluate the grouping-link behaviour - https://phabricator.wikimedia.org/T237276#11880215 (10JeanFred) 05Open→03Resolved a:03JeanFred This is done. We now have a lightweight "templating" language. It’s now possible to have links with: * labels in any language (using eg `Lfr`, `L... [18:32:25] 10Tool-inteGraality, 03Wikimedia-Hackathon-2026: Re-evaluate the grouping-link behaviour - https://phabricator.wikimedia.org/T237276#11880220 (10JeanFred) [18:39:10] 10Tool-glam-tool-hospital: OpenRefine - Bug - test - https://phabricator.wikimedia.org/T425140 (10GLAM-Tool-Hospital-Bot) 03NEW [18:40:33] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [18:47:08] 10Tool-inteGraality: Create pages linked using `grouping_link` - https://phabricator.wikimedia.org/T368001#11880299 (10JeanFred) [18:49:50] 06cloud-services-team, 10Toolforge: Copyvios: Impending Google Custom Search API deprecation on 2027-01-01 - https://phabricator.wikimedia.org/T425144 (10Earwig) 03NEW [18:50:39] 06cloud-services-team, 10Toolforge, 06Community-Tech, 10Google-api-proxy: Copyvios: Impending Google Custom Search API deprecation on 2027-01-01 - https://phabricator.wikimedia.org/T425144#11880329 (10MusikAnimal) [18:55:33] RESOLVED: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [18:59:01] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11880348 (10Ottomata) > i can have a crack at the ECS event schema Nice! yea lets figure it out. Since you have direct con... [19:02:30] 10Tool-humaniki-2: Use Codex colors for links - https://phabricator.wikimedia.org/T425147 (10Danya) 03NEW [19:02:39] 10Tool-humaniki-2: Use Codex colors for links - https://phabricator.wikimedia.org/T425147#11880378 (10Danya) p:05Triage→03Low [19:02:56] 10Tool-humaniki-2: Use Codex colors for links - https://phabricator.wikimedia.org/T425147#11880379 (10Danya) [19:04:33] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [19:07:30] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#11880400 (10Andrew) I started to map all this out in a spreadsheet: https://docs.google.com/spreadsheets/d/15z4PMNhZ1l3oCbUeuEfbDnXRYerHibe7VUDjzF... [19:17:22] 10Tool-glam-tool-hospital: Other / I am not sure - Documentation problem - I am trying to make MediaWiki run on an Apollo 11 guidance computer - https://phabricator.wikimedia.org/T425150 (10GLAM-Tool-Hospital-Bot) 03NEW [19:18:41] 10Tool-glam-tool-hospital: Other / I am not sure - Documentation problem - I am trying to make MediaWiki run on an Apollo 11 guidance computer - https://phabricator.wikimedia.org/T425150#11880435 (10Dactylantha) 05Open→03Resolved a:03Dactylantha [19:18:57] 10Tool-glam-tool-hospital: Other / I am not sure - Documentation problem - I am trying to make MediaWiki run on an Apollo 11 guidance computer - https://phabricator.wikimedia.org/T425150#11880437 (10Dactylantha) 05Resolved→03Declined [19:33:47] 10Tool-glam-tool-hospital: Other / I am not sure - Feature request - Adding an interactive monthly challenge map - https://phabricator.wikimedia.org/T425152 (10GLAM-Tool-Hospital-Bot) 03NEW [19:35:11] 10Tool-glam-tool-hospital: Other / I am not sure - Feature request - Adding an interactive monthly challenge map - https://phabricator.wikimedia.org/T425152#11880490 (10Dactylantha) 05Open→03Declined