[00:45:50] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Add examples to the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428153#11987319 (10KineticPelagic) As part of my Clinic Duty rotation, I add a project tag and move this task to "To Estimate" because the task is clear enough to be si... [00:46:38] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix description issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428151#11987322 (10KineticPelagic) As part of my Clinic Duty rotation, I add a project tag and move this task to "To Estimate" because the task is clear enoug... [00:47:58] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix summary issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428150#11987326 (10KineticPelagic) As part of my Clinic Duty rotation, I add a project tag and move this task to "To Estimate" because the task is clear enough to... [00:48:20] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix type, media type, and array issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428149#11987330 (10KineticPelagic) As part of my Clinic Duty rotation, I add a project tag and move this task to "To Estimate" because the tas... [00:48:51] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix top-level issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428147#11987334 (10KineticPelagic) As part of my Clinic Duty rotation, I add a project tag and move this task to "To Estimate" because the task is clear enough... [03:57:23] (03open) 10komla: cleanup: remove debug printlining [repos/cloud/wmcs/cloud-survey-ops] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/cloud-survey-ops/-/merge_requests/7 [03:57:24] (03approved) 10komla: cleanup: remove debug printlining [repos/cloud/wmcs/cloud-survey-ops] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/cloud-survey-ops/-/merge_requests/7 [03:57:27] (03merge) 10komla: cleanup: remove debug printlining [repos/cloud/wmcs/cloud-survey-ops] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/cloud-survey-ops/-/merge_requests/7 [05:02:55] (03open) 10jelto: update blubber buildkit image to v1.8.0 [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/7 (https://phabricator.wikimedia.org/T321316) [05:05:16] (03update) 10countcount: Draft: Add report-only test coverage to CI [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/9 [05:05:19] (03update) 10countcount: Draft: Add report-only test coverage to CI [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/16 [05:05:27] (03update) 10countcount: Draft: ci: add report-only coverage for wasm crate and front-end [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/8 [05:05:39] (03update) 10countcount: Draft: ci: add report-only test coverage (shared python-ci templates) [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/29 [05:05:46] (03update) 10countcount: Draft: Add report-only test coverage to CI [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/18 [05:05:53] (03update) 10countcount: Draft: ci: add report-only test coverage (shared node-ci templates) [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/24 [05:06:19] (03update) 10countcount: Add report-only test coverage to CI [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/9 [05:06:20] (03update) 10countcount: Add report-only test coverage to CI [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/16 [05:06:23] (03update) 10countcount: ci: add report-only coverage for wasm crate and front-end [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/8 [05:06:25] (03update) 10countcount: ci: add report-only test coverage (shared python-ci templates) [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/29 [05:06:28] (03update) 10countcount: Add report-only test coverage to CI [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/18 [05:06:33] (03update) 10countcount: ci: add report-only test coverage (shared node-ci templates) [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/24 [05:16:31] (03open) 10jelto: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/97 (https://phabricator.wikimedia.org/T321316) [05:16:32] (03merge) 10countcount: ci: add report-only test coverage (shared node-ci templates) [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/24 [05:17:15] (03merge) 10countcount: Add report-only test coverage to CI [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/18 [05:17:29] (03merge) 10countcount: ci: add report-only test coverage (shared python-ci templates) [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/29 [05:18:59] (03merge) 10countcount: ci: add report-only coverage for wasm crate and front-end [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/8 [05:19:57] (03merge) 10countcount: Add report-only test coverage to CI [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/16 [05:20:08] (03merge) 10countcount: Add report-only test coverage to CI [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/9 [05:30:34] (03merge) 10countcount: Add landing page with username lookup form [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/15 [05:34:35] (03open) 10jelto: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/161 (https://phabricator.wikimedia.org/T321316) [05:41:19] (03merge) 10samwilson: T413556 - fix incorrect image offset on rotate [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/11 (owner: 10okerekechinweotito) [05:41:35] (03update) 10samwilson: T413556 - fix incorrect image offset on rotate [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/11 (https://phabricator.wikimedia.org/T413556) (owner: 10okerekechinweotito) [05:45:38] (03open) 10jelto: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/75 (https://phabricator.wikimedia.org/T321316) [05:48:53] (03open) 10jelto: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/41 (https://phabricator.wikimedia.org/T321316) [05:49:33] (03open) 10countcount: Add .python-version file for pyenv [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/17 [05:49:51] (03update) 10countcount: Add .python-version file for pyenv [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/17 [05:50:52] (03update) 10countcount: Add .python-version for Toolforge build services [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/17 [05:51:11] (03update) 10countcount: Add .python-version for Toolforge build services [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/17 [05:51:43] (03merge) 10countcount: Add .python-version for Toolforge build services [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/17 [05:53:19] (03open) 10jelto: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/67 (https://phabricator.wikimedia.org/T321316) [06:02:49] (03open) 10jelto: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/90 (https://phabricator.wikimedia.org/T321316) [06:03:22] (03update) 10jelto: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/90 (https://phabricator.wikimedia.org/T321316) [06:06:19] (03open) 10jelto: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/41 (https://phabricator.wikimedia.org/T321316) [06:12:07] (03open) 10jelto: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/48 (https://phabricator.wikimedia.org/T321316) [06:15:53] (03update) 10renovatebot: Update python Docker tag to v3.14.5 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [06:15:57] (03open) 10renovatebot: Update python Docker tag to v3.14.5 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [06:42:10] (03open) 10anafibnshahibul: Revert "Auto-update: Thu Jun 4 06:06:56 PM +06 2026" [toolforge-repos/wikiform] - 10https://gitlab.wikimedia.org/toolforge-repos/wikiform/-/merge_requests/1 [06:42:22] (03merge) 10anafibnshahibul: Revert "Auto-update: Thu Jun 4 06:06:56 PM +06 2026" [toolforge-repos/wikiform] - 10https://gitlab.wikimedia.org/toolforge-repos/wikiform/-/merge_requests/1 [06:42:42] 06cloud-services-team, 10Toolforge: Build service images do no longer contain locales specified in .locales - https://phabricator.wikimedia.org/T428230 (10Count_Count) 03NEW [06:42:59] 06cloud-services-team, 10Toolforge: Regression: build service images do no longer contain locales specified in .locales - https://phabricator.wikimedia.org/T428230#11987596 (10Count_Count) [07:15:06] (03update) 10renovatebot: Update python Docker tag to v3.14.5 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [07:31:44] (03update) 10countcount: Update python Docker tag to v3.14.5 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 (owner: 10renovatebot) [08:15:13] (03update) 10renovatebot: Update python Docker tag to v3.14.5 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [08:18:32] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix summary issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428150#11987728 (10KBach) a:03TBurmeister [08:56:36] 06cloud-services-team, 10Toolforge, 06tools-platform-team: components-api failing to connect to internal api - https://phabricator.wikimedia.org/T428214#11987829 (10DamianZaremba) This is still happening, so doesn't appear to be a transient issue. Unfortunately it makes deploying new release quite painful a... [08:56:41] (03approved) 10fnegri: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/97 (https://phabricator.wikimedia.org/T321316) (owner: 10jelto) [08:57:02] (03approved) 10fnegri: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/48 (https://phabricator.wikimedia.org/T321316) (owner: 10jelto) [08:57:07] (03approved) 10fnegri: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/41 (https://phabricator.wikimedia.org/T321316) (owner: 10jelto) [08:57:13] (03approved) 10fnegri: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/90 (https://phabricator.wikimedia.org/T321316) (owner: 10jelto) [08:57:15] (03approved) 10fnegri: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/67 (https://phabricator.wikimedia.org/T321316) (owner: 10jelto) [08:57:16] (03approved) 10fnegri: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/41 (https://phabricator.wikimedia.org/T321316) (owner: 10jelto) [08:57:17] (03approved) 10fnegri: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/75 (https://phabricator.wikimedia.org/T321316) (owner: 10jelto) [08:57:17] (03approved) 10fnegri: update blubber buildkit image to v1.8.0 [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/161 (https://phabricator.wikimedia.org/T321316) (owner: 10jelto) [08:57:23] (03approved) 10fnegri: update blubber buildkit image to v1.8.0 [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/7 (https://phabricator.wikimedia.org/T321316) (owner: 10jelto) [09:00:24] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11987832 (10fnegri) The situation after a few hours has improved: {F86722566} I'm investigating the reason for the increase in InnoDB I/O Data Reads. [09:17:18] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11987887 (10BTullis) Great! How much more testing to we have to do before promoting the schema from t... [09:18:10] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11987891 (10BTullis) [09:19:52] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11987893 (10BTullis) I believe we said that we wanted to enable canary events for the stream. Is that... [09:40:44] (03PS1) 10Tiziano Fogli: slothslos/report2drive: move secrets under profile [labs/private] - 10https://gerrit.wikimedia.org/r/1298117 (https://phabricator.wikimedia.org/T425795) [09:40:59] (03CR) 10Tiziano Fogli: [C:03+2] slothslos/report2drive: move secrets under profile [labs/private] - 10https://gerrit.wikimedia.org/r/1298117 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [09:41:10] (03CR) 10Tiziano Fogli: [V:03+2 C:03+2] slothslos/report2drive: move secrets under profile [labs/private] - 10https://gerrit.wikimedia.org/r/1298117 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [09:47:28] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11987928 (10KCVelaga_WMF) >>! In T425087#11987887, @BTullis wrote: > Great! How much more testing to... [09:56:05] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11987965 (10AndrewTavis_WMDE) From my end as well, really thanks all for the efforts to get this set... [10:07:22] (03PS1) 10Tiziano Fogli: slothslos/report2drive: replace array with hash [labs/private] - 10https://gerrit.wikimedia.org/r/1298136 (https://phabricator.wikimedia.org/T425795) [10:07:36] (03CR) 10Tiziano Fogli: [C:03+2] slothslos/report2drive: replace array with hash [labs/private] - 10https://gerrit.wikimedia.org/r/1298136 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [10:07:38] (03CR) 10Tiziano Fogli: [V:03+2 C:03+2] slothslos/report2drive: replace array with hash [labs/private] - 10https://gerrit.wikimedia.org/r/1298136 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [10:31:50] 10Cloud-VPS (Project-requests): Request creation of humaniki-2 VPS project - https://phabricator.wikimedia.org/T428090#11988074 (10Danya) For the persistent db connection, it stems from my own habits. That said, I have just discovered that there is a way with the framework I use (Flask) to have non-persistent da... [10:32:03] 06cloud-services-team, 10Toolforge: Regression: Images built with the build service images do no longer contain/configure locales specified in .locales - https://phabricator.wikimedia.org/T428230#11988075 (10Count_Count) [11:04:02] (03open) 10countcount: Remove dependency on system locale for German month names [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/19 [11:04:48] (03merge) 10countcount: Remove dependency on system locale for German month names [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/19 [11:15:15] (03update) 10renovatebot: Update python Docker tag to v3.14.5 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [11:17:32] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27011683531 (https://github.com/cluebotng/component-configs/commits/7b3ba4bae58717e47510c281fdb60f00be6f7c92) [11:17:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [11:23:37] (03open) 10okerekechinweotito:  T413556- fix: add canvas data handling for image cropping and rotation [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/12 [11:25:22] (03update) 10okerekechinweotito:  T413556- fix: add canvas data handling for image cropping and rotation [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/12 (https://phabricator.wikimedia.org/T413556) [11:35:01] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11988194 (10fnegri) > The history length started to increase around 2026-05-28, which coincides with this increase in InnoDB I/O: @magnusmanske I wonde... [11:44:25] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27012814543 (https://github.com/cluebotng/component-configs/commits/48e7143fc26338766785004996ec66dea5876b0e) [11:44:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [11:45:56] 06cloud-services-team, 10Toolforge, 06tools-platform-team: components-api failing to connect to internal api - https://phabricator.wikimedia.org/T428214#11988230 (10DamianZaremba) Also getting failures when interacting with the build api directly; ` tools.cluebotng@tools-bastion-15:~$ toolforge build start -... [11:47:17] 10Tool-wikimedia-attribution: Technical implementation: Add "User-Agent policy and rate limits" section - https://phabricator.wikimedia.org/T425976#11988243 (10Sarai-WMF) Status update: A [[ https://wikimedia-attribution.toolforge.org/technical-implementation.html#user-agent-policy-and-rate-limits | User-Agent p... [11:51:10] 10Tool-wikimedia-attribution: Add detailed recommendations for external reusers to improve accuracy of web traffic provenance - https://phabricator.wikimedia.org/T424905#11988267 (10Sarai-WMF) Status update: A [[ https://wikimedia-attribution.toolforge.org/technical-implementation.html#traffic-referral | Traffic... [12:04:08] 10Tool-humaniki-2: Try to split the compiled JS asset file - https://phabricator.wikimedia.org/T428248 (10Danya) 03NEW [12:06:09] 10Tool-translatetagger, 06Toolforge-standards-committee, 07Privacy: translatetagger tool loads third-party resources - https://phabricator.wikimedia.org/T413376#11988345 (10Aklapper) [12:06:34] 10Tool-wikicontest, 07Privacy: WikiEval Tool / WikiContest violates Toolforge Privacy Policy - https://phabricator.wikimedia.org/T427744#11988349 (10Aklapper) [12:06:40] 06Toolforge-standards-committee, 10Tools, 06Privacy Engineering, 07Privacy: Hunt for Toolforge tools that load resources from third party sites - https://phabricator.wikimedia.org/T172065#11988350 (10Aklapper) [12:07:05] 06Toolforge-standards-committee, 10Tools, 06Privacy Engineering, 07Privacy: Toolforge tools that load resources from third party sites - https://phabricator.wikimedia.org/T172065#11988360 (10Aklapper) [12:07:10] 10Tool-humaniki-2: Use Flask’s appcontext feature for database connections - https://phabricator.wikimedia.org/T428250 (10Danya) 03NEW [12:08:15] 10Tool-humaniki-2: Use Flask’s appcontext feature for database connections - https://phabricator.wikimedia.org/T428250#11988374 (10Danya) a:03Danya [12:08:19] 10Tool-humaniki-2: Use Flask’s appcontext feature for database connections - https://phabricator.wikimedia.org/T428250#11988378 (10Danya) p:05Triage→03High [12:09:16] 10Tool-humaniki-2: Investigate why server is launched 4 times - https://phabricator.wikimedia.org/T427563#11988392 (10Danya) I’m dumb, this is because of the `--workers=4` option in the Procfile. [12:09:39] 06cloud-services-team, 10Toolforge: [logs-cli] Specifying a `since` time in the future results in unclear error message - https://phabricator.wikimedia.org/T428252 (10DamianZaremba) 03NEW [12:09:42] 10Tool-humaniki-2: Investigate why server is launched 4 times - https://phabricator.wikimedia.org/T427563#11988407 (10Danya) 05Open→03Resolved [12:09:57] 10Tool-humaniki-2: Try to split the compiled JS asset file - https://phabricator.wikimedia.org/T428248#11988409 (10Danya) [12:09:59] 10Tool-humaniki-2: Try to split the compiled JS asset file - https://phabricator.wikimedia.org/T428248#11988411 (10Danya) p:05Triage→03High [12:10:04] 10Tool-humaniki-2: Try to split the compiled JS asset file - https://phabricator.wikimedia.org/T428248#11988413 (10Danya) p:05High→03Medium [12:11:39] 10Tool-humaniki-2: Investigate best solution between ToolsDB and self-hosted database - https://phabricator.wikimedia.org/T425991#11988422 (10Danya) Cloud VPS request can be found here: T428090 [12:15:04] 06cloud-services-team, 10Toolforge: [logs-cli] `since` + `until` limited by `last` - https://phabricator.wikimedia.org/T428253 (10DamianZaremba) 03NEW [12:29:30] 10Tool-humaniki-2: Convert dump ingester into a faster language - https://phabricator.wikimedia.org/T428256 (10Danya) 03NEW [12:29:35] 10Tool-humaniki-2: Convert dump ingester into a faster language - https://phabricator.wikimedia.org/T428256#11988523 (10Danya) [12:29:37] 10Tool-humaniki-2: Convert dump ingester into a faster language - https://phabricator.wikimedia.org/T428256#11988525 (10Danya) p:05Triage→03High [12:31:47] 10Tool-humaniki-2: Convert dump ingester into a faster language - https://phabricator.wikimedia.org/T428256#11988534 (10Danya) [12:34:26] 10Tools, 07Performance Issue: Stalktoy is often slow or not responding - https://phabricator.wikimedia.org/T427941#11988543 (10Aklapper) 05Open→03Invalid Hi, https://toolhub.wikimedia.org/tools/meta-stalktoy points to https://github.com/Pathoschild/Wikimedia-contrib/issues as the issue tracker for this... [12:39:50] 10Tool-humaniki-2: Convert dump ingester into a faster language and detatch it from the web server - https://phabricator.wikimedia.org/T428256#11988572 (10Danya) [12:40:21] 10Tool-humaniki-2: Convert dump ingester into a faster language and detach it from the web server - https://phabricator.wikimedia.org/T428256#11988576 (10Danya) [12:42:42] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11988580 (10fnegri) I tried two approaches to understand which user/db is driving the high rate of "Data Reads". 1. Checking innodb_buffer_page: `lang... [12:54:44] 10Tool-humaniki-2: Fix link to contrib guide in readme and update version workflow - https://phabricator.wikimedia.org/T428258 (10Danya) 03NEW [12:54:48] 10Tool-humaniki-2: Fix link to contrib guide in readme and update version workflow - https://phabricator.wikimedia.org/T428258#11988630 (10Danya) p:05Triage→03High [12:54:53] 10Tool-humaniki-2: Fix link to contrib guide in readme and update version workflow - https://phabricator.wikimedia.org/T428258#11988631 (10Danya) [12:59:12] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27016160066 (https://github.com/cluebotng/component-configs/commits/3c617707fe44f00c5ac81aa315bfc989a7bd4d00) [12:59:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [13:27:08] 06cloud-services-team, 10Data-Services, 10BetaFeatures, 06Data-Engineering, and 2 others: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11988863 (10Gehel) [13:33:55] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11989000 (10Gehel) [13:34:19] 10Cloud Services Proposals, 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence, and 2 others: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#11989010 (10Gehel) [13:34:33] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11989014 (10Gehel) [13:34:49] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, and 3 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#11989016 (10Gehel) [13:43:05] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11989075 (10magnusmanske) OK, `s51434__mixnmatch_p` and `s52421__commonsdelinquent_p` are mine. I have just stopped the mixnmatch "cache warmer", I thi... [13:44:30] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11989078 (10magnusmanske) The `anon_1` thing is not mine, AFAICT [14:04:37] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27019408986 (https://github.com/cluebotng/component-configs/commits/4de86ac4b524f394ec49dc167d0c75457e981af1) [14:04:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [14:40:06] 10Tool-humaniki-2: Convert dump ingester into a faster language and detach it from the web server - https://phabricator.wikimedia.org/T428256#11989259 (10Danya) [14:43:28] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11989284 (10fnegri) > I have just stopped the mixnmatch "cache warmer", I think it's sufficient now, see if that makes a difference. I think that helpe... [14:45:12] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021602036 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000) [14:45:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [14:45:25] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021601200 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000) [14:45:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [14:45:45] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021601458 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000) [14:45:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [15:03:38] 06cloud-services-team, 10Toolforge: MySQL toolsdb backups? - https://phabricator.wikimedia.org/T428272 (10magnusmanske) 03NEW [15:20:17] 06cloud-services-team, 10Toolforge: MySQL toolsdb backups? - https://phabricator.wikimedia.org/T428272#11989497 (10taavi) 05Open→03Invalid A documented workflow for periodic backups already exists: https://wikitech.wikimedia.org/wiki/Help:Toolforge/ToolsDB#Backups [15:23:27] 10Tool-humaniki-2: Convert dump ingester into a faster language and detach it from the web server - https://phabricator.wikimedia.org/T428256#11989504 (10Danya) 05Open→03In progress [15:24:15] (03open) 10danyya: Draft: Resolve T428256 "Convert dump ingester into a faster language and detach it from the web server" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/67 [16:19:36] 10VPS-project-Wikistats: Add magwiki to wikistats - https://phabricator.wikimedia.org/T428284#11989764 (10Dzahn) once wiki is created in T428266 [16:26:32] FIRING: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:36:32] RESOLVED: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:21:03] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11990062 (10Ottomata) > wanted to enable canary events for the stream. Is that right? Ya lets do it.... [18:21:25] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11990063 (10Ottomata) Anyway, I'm out next week, don't wait for approval from me to move forward with... [19:04:52] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11990165 (10Dzahn) [19:53:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:54:05] 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#11990195 (10komla) [19:55:42] 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#11990198 (10komla) We settled on December 1st, 2026 as the deadline for the migration/upgrade and January 1st, 2027 as the end of the grace period. [20:58:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [21:45:12] 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#11990404 (10BLiviero-WMF) Hi Komla, I thought the deadline we discussed was end of September 2026. was there a subsequent conversation on this? thank you [22:01:42] 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#11990424 (10komla) @BLiviero-WMF sorry about the confusion. I didn't understand it this way. The current date was suggested on the draft message. However, the message... [22:04:37] 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki urwikisource - https://phabricator.wikimedia.org/T415977#11990432 (10Ragesoss) I'd love to see this get taken care of soon. There's a Programs & Events Dashboard program that's trying to track uwikisource, and it won't be able to... [22:11:15] 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#11990439 (10BLiviero-WMF) Let me double check with a couple of other folks and get back to you! I may have misunderstood myself [22:53:45] 06cloud-services-team, 10Cloud-VPS, 06Release-Engineering-Team (Radar): Magnum cluster stuck in DELETE_FAILED status - https://phabricator.wikimedia.org/T428312 (10dduvall) 03NEW