[00:16:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:21:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:53:06] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/1293110 (owner: 10L10n-bot) [01:40:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:45:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:21:26] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-34 (2026-05-19 to 2026-06-02)), 07OKR-Work: [SPIKE] Add security scheme information to MediaWiki REST API description - https://phabricator.wikimedia.org/T423552#11957842 (10KineticPelagic) 05In progress→03Resolved [03:05:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:20:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:11:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:16:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:17:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:22:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:11:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:16:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:50:33] (03merge) 10filippo: Default to 64mb memory requests [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/110 (https://phabricator.wikimedia.org/T420565) [07:51:52] (03open) 10filippo: d/changelog: bump to 0.103.25 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/111 (https://phabricator.wikimedia.org/T420565) [07:52:55] !log filippo@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [07:54:11] !log filippo@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [08:22:20] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Remove obsolete maintain-kubeusers limitranges - https://phabricator.wikimedia.org/T427352 (10fgiunchedi) 03NEW [08:27:09] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Remove obsolete maintain-kubeusers limitranges - https://phabricator.wikimedia.org/T427352#11958215 (10fgiunchedi) I just did it™ in toolsbeta: ` filippo@toolsbeta-bastion-7:~$ kubectl sudo get limitrange -A | awk '$1 ~ /^tool/ && $2 !~ /^tool/ {pri... [08:28:11] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Remove obsolete maintain-kubeusers limitranges - https://phabricator.wikimedia.org/T427352#11958216 (10fgiunchedi) Now roll-restart works as expected ` filippo@toolsbeta-bastion-7:~$ ./restart-webservice.sh --cutoff-timestamp 1779869159 automated-to... [08:50:12] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support requestBody.description in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427355#11958416 (10AGhirelli-WMF) [08:50:29] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support root-level externalDocs in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427356#11958418 (10AGhirelli-WMF) [08:50:45] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Generate a valid semver info.version for the default MediaWiki REST module - https://phabricator.wikimedia.org/T427359#11958421 (10AGhirelli-WMF) [08:50:57] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support requestBody content examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427360#11958423 (10AGhirelli-WMF) [08:51:07] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support response body examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427361#11958425 (10AGhirelli-WMF) [08:51:18] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support path-level summary and description in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427362#11958427 (10AGhirelli-WMF) [09:03:53] (03CR) 10Filippo Giunchedi: [C:03+1] openstack: Add cookbook to reboot a clouservices node [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293740 (https://phabricator.wikimedia.org/T348841) (owner: 10Majavah) [09:09:24] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Remove obsolete maintain-kubeusers limitranges - https://phabricator.wikimedia.org/T427352#11958510 (10fnegri) > Happened following this commit https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/commit/9b51acbff6ecb7db733f289d996... [09:22:02] 06cloud-services-team, 10Toolforge, 06SRE Observability, 13Patch-For-Review: Support deploying paging alerts to non-production environments (was: make toolsbeta paging alerts less confusing) - https://phabricator.wikimedia.org/T424814#11958530 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This... [09:29:22] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Remove obsolete maintain-kubeusers limitranges - https://phabricator.wikimedia.org/T427352#11958562 (10fgiunchedi) 05Open→03Resolved This is done [10:04:15] (03update) 10fnegri: Catch SQL errors [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/12 (https://phabricator.wikimedia.org/T351637) [10:04:17] (03update) 10fnegri: Catch SQL errors [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/12 (https://phabricator.wikimedia.org/T351637) [10:09:30] (03approved) 10volans: Catch SQL errors [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/12 (https://phabricator.wikimedia.org/T351637) (owner: 10fnegri) [11:10:55] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-34 (2026-05-19 to 2026-06-02)), 07OKR-Work: Fix linter rule duplicates - https://phabricator.wikimedia.org/T426825#11958988 (10KBach) 05In progress→03Resolved [11:13:18] (03CR) 10Majavah: [C:03+2] openstack: Add cookbook to reboot a clouservices node [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293740 (https://phabricator.wikimedia.org/T348841) (owner: 10Majavah) [11:17:30] (03Merged) 10jenkins-bot: openstack: Add cookbook to reboot a clouservices node [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293740 (https://phabricator.wikimedia.org/T348841) (owner: 10Majavah) [11:19:25] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Alert, 07Cloud-Services-Worktype-Unplanned, 13Patch-For-Review: [wmcs-cookbooks] add a cookbook to reboot a cloudservices/cloudlb host - https://phabricator.wikimedia.org/T348841#11959035 (10taavi) 05Open→03Resolved [11:20:31] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-34 (2026-05-19 to 2026-06-02)), 07OKR-Work, 13Patch-For-Review: Improve and standardize linter messages - https://phabricator.wikimedia.org/T426821#11959041 (10KBach) [11:35:00] 06cloud-services-team, 10Cloud-VPS: cleanup mcrouter from cloudcontrol hosts - https://phabricator.wikimedia.org/T427189#11959125 (10taavi) a:03taavi [11:46:17] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support requestBody.description in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427355#11959212 (10KBach) [11:47:04] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support root-level externalDocs in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427356#11959214 (10KBach) [11:47:44] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Generate a valid semver info.version for the default MediaWiki REST module - https://phabricator.wikimedia.org/T427359#11959216 (10KBach) [11:48:15] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support path-level summary and description in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427362#11959218 (10KBach) [11:48:21] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support path-level summary and description in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427362#11959220 (10KBach) p:05Triage→03Medium [11:48:51] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support response body examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427361#11959221 (10KBach) [11:49:02] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support response body examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427361#11959223 (10KBach) p:05Triage→03High [11:49:31] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support requestBody content examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427360#11959228 (10KBach) [11:49:40] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support requestBody content examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427360#11959230 (10KBach) p:05Triage→03High [11:50:04] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Generate a valid semver info.version for the default MediaWiki REST module - https://phabricator.wikimedia.org/T427359#11959231 (10KBach) p:05Triage→03High [11:50:17] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support root-level externalDocs in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427356#11959233 (10KBach) p:05Triage→03High [11:50:30] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support requestBody.description in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427355#11959234 (10KBach) p:05Triage→03High [11:50:42] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support path-level summary and description in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427362#11959235 (10KBach) p:05Medium→03Low [11:50:55] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support response body examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427361#11959236 (10KBach) p:05High→03Medium [11:51:06] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support requestBody content examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427360#11959237 (10KBach) p:05High→03Medium [12:02:02] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [12:03:29] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [12:25:53] 10Tool-wiktlexbot: Use Wiktionary page title dump to retrieve the list of existing entries - https://phabricator.wikimedia.org/T419317#11959341 (10Redmin) 05Open→03Stalled p:05Low→03Triage This is stalled on T427195. [13:01:16] (03open) 10danyya: Draft: Resolve T427124 "Rename and update contribution guide" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/58 [13:02:20] (03update) 10danyya: Draft: Resolve T427124 "Rename and update contribution guide" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/58 [13:03:03] 10Tool-humaniki-2: Setup a version scheme - https://phabricator.wikimedia.org/T427391 (10Danya) 03NEW [13:03:33] 10Tool-humaniki-2: Setup a version scheme - https://phabricator.wikimedia.org/T427391#11959469 (10Danya) p:05Triage→03High [13:17:45] (03update) 10danyya: Draft: Resolve T427124 "Rename and update contribution guide" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/58 [13:17:57] (03update) 10danyya: Resolve T427124 "Rename and update contribution guide" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/58 [13:18:29] (03merge) 10danyya: Resolve T427124 "Rename and update contribution guide" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/58 [13:19:09] 10Tool-humaniki-2: Fix contribution guide - https://phabricator.wikimedia.org/T427149#11959525 (10Danya) 05Open→03Resolved Fixed in T427124. [13:19:10] 10Tool-humaniki-2, 13Patch-For-Review: Rename and update contribution guide - https://phabricator.wikimedia.org/T427124#11959524 (10Danya) 05In progress→03Resolved [13:20:18] 10Tool-humaniki-2: Setup a version scheme - https://phabricator.wikimedia.org/T427391#11959537 (10Danya) 05Open→03In progress [13:21:25] (03approved) 10fnegri: tests: fix wrong test data [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/290 (https://phabricator.wikimedia.org/T423880) (owner: 10raymond-ndibe) [13:21:53] (03open) 10danyya: Draft: Resolve T427391 "Setup a version scheme" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/59 [13:27:52] (03PS1) 10Brouberol: idp_test: remove deprecated growthbook client_secret [labs/private] - 10https://gerrit.wikimedia.org/r/1294289 [13:28:02] (03CR) 10Brouberol: [C:03+2] idp_test: remove deprecated growthbook client_secret [labs/private] - 10https://gerrit.wikimedia.org/r/1294289 (owner: 10Brouberol) [13:28:06] (03CR) 10Brouberol: [V:03+2 C:03+2] idp_test: remove deprecated growthbook client_secret [labs/private] - 10https://gerrit.wikimedia.org/r/1294289 (owner: 10Brouberol) [13:31:41] (03update) 10fnegri: core.py: skip one-off jobs when updating storage [repos/cloud/toolforge/jobs-api] (fix_data_handling_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/288 (https://phabricator.wikimedia.org/T423544) (owner: 10raymond-ndibe) [13:31:44] (03update) 10fnegri: core.py: skip one-off jobs when updating storage [repos/cloud/toolforge/jobs-api] (fix_data_handling_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/288 (https://phabricator.wikimedia.org/T423544) (owner: 10raymond-ndibe) [13:31:48] (03update) 10fnegri: core.py: skip one-off jobs when updating storage [repos/cloud/toolforge/jobs-api] (fix_data_handling_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/288 (https://phabricator.wikimedia.org/T423544) (owner: 10raymond-ndibe) [13:31:49] (03approved) 10fnegri: core.py: skip one-off jobs when updating storage [repos/cloud/toolforge/jobs-api] (fix_data_handling_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/288 (https://phabricator.wikimedia.org/T423544) (owner: 10raymond-ndibe) [13:31:51] (03update) 10fnegri: tests: fix wrong test data [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/290 (https://phabricator.wikimedia.org/T423880) (owner: 10raymond-ndibe) [13:31:55] (03update) 10fnegri: tests: fix wrong test data [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/290 (https://phabricator.wikimedia.org/T423880) (owner: 10raymond-ndibe) [13:41:40] (03merge) 10filippo: d/changelog: bump to 0.103.25 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/111 (https://phabricator.wikimedia.org/T420565) [13:46:56] (03update) 10danyya: Draft: Resolve T427391 "Setup a version scheme" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/59 [13:47:49] 10Tool-humaniki-2: Fix link to contribution guide - https://phabricator.wikimedia.org/T427394 (10Danya) 03NEW [13:47:50] 10Tool-humaniki-2: Fix link to contribution guide - https://phabricator.wikimedia.org/T427394#11959642 (10Danya) p:05Triage→03Medium [13:47:53] 10Tool-humaniki-2: Fix link to contribution guide - https://phabricator.wikimedia.org/T427394#11959643 (10Danya) [13:50:24] 10Tool-humaniki-2, 13Patch-For-Review: Setup a version scheme - https://phabricator.wikimedia.org/T427391#11959646 (10Danya) [14:36:53] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-34 (2026-05-19 to 2026-06-02)), 07OKR-Work: Fix issue line highlighting in the linter - https://phabricator.wikimedia.org/T425935#11959823 (10KineticPelagic) 05Open→03In progress [14:50:51] (03open) 10kineticpelagic: fix(linter): Correct line highlighting and range reporting (T425935) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/23 (https://phabricator.wikimedia.org/T425935) [15:02:15] (03update) 10danyya: Draft: Resolve T427391 "Setup a version scheme" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/59 [15:05:21] (03update) 10danyya: Draft: Resolve T427391 "Setup a version scheme" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/59 [15:36:57] FIRING: ProbeDown: Service labweb-ssl:7443 has failed probes (http_labweb-ssl_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#labweb-ssl:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:37:57] FIRING: ProbeDown: Service labweb-ssl:7443 has failed probes (http_labweb-ssl_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#labweb-ssl:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:40:12] (03open) 10kineticpelagic: fix(linter): Pin version of Spectral [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/24 [15:41:20] (03update) 10kineticpelagic: fix(linter): Correct line highlighting and range reporting (T425935) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/23 (https://phabricator.wikimedia.org/T425935) [15:41:35] (03update) 10kineticpelagic: fix(linter): Pin version of Spectral [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/24 [15:43:54] RESOLVED: ProbeDown: Service labweb-ssl:7443 has failed probes (http_labweb-ssl_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#labweb-ssl:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:43:59] RESOLVED: ProbeDown: Service labweb-ssl:7443 has failed probes (http_labweb-ssl_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#labweb-ssl:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:45:58] 10Tool-humaniki-2: Add a notice banner whenever a website maintenance is planned - https://phabricator.wikimedia.org/T427412 (10Danya) 03NEW [15:46:01] 10Tool-humaniki-2: Add a notice banner whenever a website maintenance is planned - https://phabricator.wikimedia.org/T427412#11960181 (10Danya) [15:46:06] 10Tool-humaniki-2: Add a notice banner whenever a website maintenance is planned - https://phabricator.wikimedia.org/T427412#11960190 (10Danya) p:05Triage→03Medium [15:46:36] 10Tool-humaniki-2: Header items overflow on mobile screens - https://phabricator.wikimedia.org/T422344#11960192 (10Danya) p:05Medium→03High [16:02:33] 06cloud-services-team, 10Data-Services, 10PAWS: Can't connect to ToolsDB on PAWS - https://phabricator.wikimedia.org/T427417 (10Pppery) 03NEW [16:05:52] (03update) 10danyya: Draft: Resolve T427391 "Setup a version scheme" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/59 [16:07:47] (03update) 10danyya: Draft: Resolve T427391 "Setup a version scheme" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/59 [16:12:41] (03update) 10kineticpelagic: fix(linter): Pin version of Spectral [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/24 [16:15:13] (03update) 10kineticpelagic: fix(linter): Fix GitLab CI and pin version of Spectral [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/24 [16:22:43] (03update) 10kineticpelagic: fix(linter): Fix GitLab CI and pin version of Spectral [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/24 [16:24:33] (03approved) 10aghirelli: fix(linter): Fix GitLab CI and pin version of Spectral [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/24 (owner: 10kineticpelagic) [16:24:49] (03merge) 10aghirelli: fix(linter): Fix GitLab CI and pin version of Spectral [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/24 (owner: 10kineticpelagic) [16:29:33] (03update) 10fnegri: fix jobs load bug [repos/cloud/toolforge/jobs-api] (fix_oneoff_storage_error) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544) (owner: 10raymond-ndibe) [16:34:01] (03update) 10kineticpelagic: fix(linter): Correct line highlighting and range reporting (T425935) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/23 (https://phabricator.wikimedia.org/T425935) [16:40:37] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate Puppet CA: metricsinfra-puppetmaster-1.metricsinfra.eqiad1.wikimedia.cloud is about to expire in 17d 23h 58m 10s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [16:46:45] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support response body examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427361#11960535 (10BPirkle) [16:53:18] (03update) 10danyya: Draft: Resolve T427391 "Setup a version scheme" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/59 [16:54:51] (03update) 10danyya: Draft: Resolve T427391 "Setup a version scheme" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/59 [16:59:30] (03update) 10danyya: Resolve T427391 "Setup a version scheme" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/59 [16:59:38] (03merge) 10danyya: Resolve T427391 "Setup a version scheme" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/59 [17:00:18] (03update) 10fnegri: status: add StatusShort enum and basic job status detection [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/299 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [17:01:18] 10Tool-humaniki-2, 13Patch-For-Review: Setup a version scheme - https://phabricator.wikimedia.org/T427391#11960621 (10Danya) 05In progress→03Resolved [17:01:40] (03open) 10countcount: Move cargo audit ignores from CI flags to audit.toml [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/10 [17:01:47] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430 (10Danya) 03NEW [17:01:50] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430#11960634 (10Danya) p:05Triage→03High [17:01:54] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430#11960636 (10Danya) 05Open→03In progress [17:03:47] (03update) 10countcount: Move cargo audit ignores from CI flags to audit.toml [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/10 [17:04:30] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430#11960644 (10Danya) [17:04:33] 10Tool-humaniki-2, 13Patch-For-Review: Setup a version scheme - https://phabricator.wikimedia.org/T427391#11960643 (10Danya) [17:04:37] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430#11960647 (10Danya) [17:04:39] 10Tool-humaniki-2: Fix contribution guide - https://phabricator.wikimedia.org/T427149#11960646 (10Danya) [17:04:44] 10Tool-humaniki-2: Rename and update contribution guide - https://phabricator.wikimedia.org/T427124#11960650 (10Danya) [17:04:45] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430#11960651 (10Danya) [17:04:49] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430#11960653 (10Danya) [17:04:50] 10Tool-humaniki-2: Fix project/language selectors not disabled correctly - https://phabricator.wikimedia.org/T427209#11960652 (10Danya) [17:04:58] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430#11960655 (10Danya) [17:04:59] 10Tool-humaniki-2: Use transactions for snapshots - https://phabricator.wikimedia.org/T424903#11960654 (10Danya) [17:05:06] 10Tool-humaniki-2: Aggregate gender data in the database instead of client - https://phabricator.wikimedia.org/T425219#11960657 (10Danya) [17:05:07] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430#11960658 (10Danya) [17:05:11] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430#11960660 (10Danya) [17:05:13] 10Tool-humaniki-2: Integrate Denelezh and merge stats tables - https://phabricator.wikimedia.org/T422343#11960659 (10Danya) [17:05:18] 10Tool-humaniki-2: Remove unused bar chart properties in LanguagePages.vue - https://phabricator.wikimedia.org/T427218#11960661 (10Danya) [17:05:21] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430#11960662 (10Danya) [17:06:55] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430#11960668 (10Danya) [17:06:56] 10Tool-humaniki-2: Investigate best solution between ToolsDB and self-hosted database - https://phabricator.wikimedia.org/T425991#11960667 (10Danya) [17:07:00] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430#11960670 (10Danya) [17:07:01] 10Tool-humaniki-2: Migrate site to Cloud VPS? - https://phabricator.wikimedia.org/T425489#11960669 (10Danya) [17:07:32] 10Toolforge, 06tools-platform-team: ToolsDB disk space growing too fast - https://phabricator.wikimedia.org/T427187#11960671 (10fnegri) An update from today, `binlogs` are still growing but they should hopefully stop to grow in the next 24 hours as the retention window moves forward. The number of binlogs gene... [17:10:13] (03merge) 10countcount: Move cargo audit ignores from CI flags to audit.toml [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/10 [17:11:51] 10Tool-humaniki-2: Harmonize API routes - https://phabricator.wikimedia.org/T427215#11960694 (10Danya) 05Open→03In progress [17:12:04] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Support response body examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427361#11960697 (10BPirkle) The sitemap endpoints mentioned in the description are a little uncommon in that they return `appli... [17:12:34] 10Tool-humaniki-2: [VERSION] 0.1.0 - https://phabricator.wikimedia.org/T427430#11960700 (10Danya) [17:12:35] 10Tool-humaniki-2: Harmonize API routes - https://phabricator.wikimedia.org/T427215#11960699 (10Danya) [17:13:56] (03open) 10danyya: Draft: Resolve T427215 "Harmonize API routes" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/60 [17:15:14] (03update) 10renovatebot: Draft: Update Rust crate sqlx to 0.9.0 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/6 [17:15:19] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/7 [17:48:38] 10Tool-humaniki-2, 13Patch-For-Review: Harmonize API routes - https://phabricator.wikimedia.org/T427215#11960806 (10Danya) [17:49:00] 10Tool-humaniki-2, 13Patch-For-Review: Harmonize API routes - https://phabricator.wikimedia.org/T427215#11960807 (10Danya) [18:00:38] (03update) 10kineticpelagic: fix(linter): Fix GitLab CI and pin version of Spectral [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/24 [18:19:07] 10Tool-humaniki-2, 13Patch-For-Review: Harmonize API routes - https://phabricator.wikimedia.org/T427215#11960842 (10Danya) [18:20:21] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team: Generate a valid semver info.version for the default MediaWiki REST module - https://phabricator.wikimedia.org/T427359#11960844 (10BPirkle) I'm skeptical about using $wgVersion for this. For starters, [[ https://www.mediawiki.org/wiki/M... [18:25:35] 10Tool-humaniki-2, 13Patch-For-Review: Harmonize API routes - https://phabricator.wikimedia.org/T427215#11960867 (10Danya) [18:26:02] 10Tool-humaniki-2, 13Patch-For-Review: Harmonize API routes - https://phabricator.wikimedia.org/T427215#11960873 (10Danya) [18:27:19] 10Tool-humaniki-2, 13Patch-For-Review: Harmonize API routes - https://phabricator.wikimedia.org/T427215#11960877 (10Danya) [19:05:01] 10Tool-humaniki-2, 13Patch-For-Review: Harmonize API routes - https://phabricator.wikimedia.org/T427215#11960995 (10Danya) [19:10:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:25:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:39:42] 10Tool-humaniki-2: Add Python/TS type checking to CI and pre-commit hooks - https://phabricator.wikimedia.org/T427444 (10Danya) 03NEW [19:39:50] 10Tool-humaniki-2: Add Python/TS type checking to CI and pre-commit hooks - https://phabricator.wikimedia.org/T427444#11961109 (10Danya) [19:39:53] 10Tool-humaniki-2: Add Python/TS type checking to CI and pre-commit hooks - https://phabricator.wikimedia.org/T427444#11961111 (10Danya) p:05Triage→03High [19:48:36] 10Tool-humaniki-2: Store genders, countries, and professions into Pinia stores - https://phabricator.wikimedia.org/T427446 (10Danya) 03NEW [19:48:41] 10Tool-humaniki-2: Store genders, countries, and professions into Pinia stores - https://phabricator.wikimedia.org/T427446#11961132 (10Danya) [19:48:46] 10Tool-humaniki-2: Store genders, countries, and professions into Pinia stores - https://phabricator.wikimedia.org/T427446#11961134 (10Danya) p:05Triage→03High [19:48:59] 10Tool-humaniki-2: Store genders, countries, and professions into Pinia stores - https://phabricator.wikimedia.org/T427446#11961137 (10Danya) p:05High→03Medium [20:06:38] (03update) 10danyya: Draft: Resolve T427215 "Harmonize API routes" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/60 [20:07:08] (03update) 10danyya: Resolve T427215 "Harmonize API routes" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/60 [20:07:30] (03merge) 10danyya: Resolve T427215 "Harmonize API routes" [toolforge-repos/humaniki] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/60 [20:07:47] 10Tool-humaniki-2, 13Patch-For-Review: Harmonize API routes - https://phabricator.wikimedia.org/T427215#11961155 (10Danya) 05In progress→03Resolved [20:09:49] 06cloud-services-team, 10Toolforge: gitlab-webhooks: toolforge webservice logs -f choking on a log message - https://phabricator.wikimedia.org/T427449 (10dancy) 03NEW [20:12:53] 06cloud-services-team, 10Toolforge: gitlab-webhooks: toolforge webservice logs -f choking on a log message - https://phabricator.wikimedia.org/T427449#11961191 (10dancy) Noting that it doesn't always stop at the same message. For example, I ran again recently and it stopped at an earlier timestamp of `2026-05... [20:27:24] (03PS1) 10Majavah: StewardBot: Fix gblblock comment formatting [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1294404 [20:29:50] (03CR) 10Majavah: [C:03+2] StewardBot: Fix gblblock comment formatting [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1294404 (owner: 10Majavah) [20:30:34] (03Merged) 10jenkins-bot: StewardBot: Fix gblblock comment formatting [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1294404 (owner: 10Majavah) [20:57:47] 06cloud-services-team, 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T426547#11961325 (10Andrew) 05Open→03Resolved a:03Andrew I am not 100% convinced by the tests on https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Pywikibot_ima... [21:15:12] (03PS1) 10Majavah: StewardBox: Fix empty brackets for removed global blocks [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1294430 [21:20:42] (03CR) 10Majavah: [C:03+2] StewardBox: Fix empty brackets for removed global blocks [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1294430 (owner: 10Majavah) [21:22:56] (03Merged) 10jenkins-bot: StewardBox: Fix empty brackets for removed global blocks [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1294430 (owner: 10Majavah) [22:47:50] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26543153283 (https://github.com/cluebotng/component-configs/commits/b9f54eeee581a4f5a3788d3e16acbc76c7fa6fbb) [22:47:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL