[00:01:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:02:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:11:53] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:14:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/43 [00:14:30] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/43 [00:14:36] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/51 [00:14:45] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/51 [00:14:52] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/50 [00:15:02] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/50 [00:17:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/96 [00:17:22] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/96 [00:17:24] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/96 [00:18:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/72 [00:18:15] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/72 [00:18:18] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/72 [00:19:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/56 [00:19:02] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/56 [00:19:09] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/56 [00:19:53] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/44 [00:20:02] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/44 [00:20:04] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/44 [00:20:06] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/49 [00:20:20] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/49 [00:20:23] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/49 [00:42:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:53:55] 10Tool-community-activity-alerts-system, 03WikiIndaba-Hackathon-2026: Swahili transaltion of Community Activity Alerts documentation - https://phabricator.wikimedia.org/T444901#12563504 (10Lebu_Ayiga) >>! In T444901#12561689, @Swayam_Agrahari wrote: > Assigning @Lebu_Ayiga as this has not been started yet! C... [01:15:19] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/80 [01:15:21] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/80 [01:16:51] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/41 [01:16:53] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/41 [01:17:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:32:34] (03open) 10juniorb8: Search in Language Dropdowns (Source/Target) only matches localized script, not English names. [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/120 (https://phabricator.wikimedia.org/T123456 https://phabricator.wikimedia.org/T422997) [02:16:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:21:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:36:38] (03open) 10ibrahemqasim: Sync SQLAlchemy models with MySQL schema [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/41 [02:55:08] (03update) 10ibrahemqasim: Sync SQLAlchemy models with MySQL schema [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/41 [07:19:23] (03open) 10sadrettin: Wikidata kalitesi: Kulturenvanteri anıtları için öneriler ve bot hesabı [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/19 [07:22:36] (03merge) 10sadrettin: Wikidata kalitesi: Kulturenvanteri anıtları için öneriler ve bot hesabı [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/19 [07:42:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:48:56] (03open) 10sadrettin: Gereksiz P131: yalnız il düzeyi çıkarılır [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/20 [08:17:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:18:34] (03update) 10sadrettin: P131: yalnız il çıkarılır, yalnız il olana ilçe önerilir [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/20 [08:18:36] (03update) 10sadrettin: P131: yalnız il çıkarılır, yalnız il olana ilçe önerilir [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/20 [08:42:58] (03merge) 10sadrettin: P131: yalnız il çıkarılır, yalnız il olana ilçe önerilir [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/20 [09:01:45] (03open) 10sadrettin: İstatistikler: Wikidata kalitesi sayıları tablolarda [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/21 [09:01:46] (03update) 10sadrettin: İstatistikler: Wikidata kalitesi sayıları tablolarda [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/21 [09:12:12] 10Tool-itwiki: Quality assurance 2026 on itWiki repositories (Code of Conduct, CITATION.cff, codemeta.json, Software Heritage) - https://phabricator.wikimedia.org/T444000#12563972 (10valerio.bozzolan) [09:21:33] 10Tool-itwiki, 10Tool-itwiki-bot-cancellazioni, 10Tool-itwiki-orfanizzabot: Quality assurance 2026 on itWiki repositories (Code of Conduct, CITATION.cff, codemeta.json, Software Heritage) - https://phabricator.wikimedia.org/T444000#12563990 (10valerio.bozzolan) [09:22:36] 10Tool-itwiki, 10Tool-itwiki-bot-cancellazioni, 10Tool-itwiki-orfanizzabot: Quality assurance 2026 on itWiki repositories (Code of Conduct, CITATION.cff, codemeta.json, Software Heritage) - https://phabricator.wikimedia.org/T444000#12563993 (10valerio.bozzolan) a:05valerio.bozzolan→03None >>! In T444000#... [09:25:52] 10Tool-itwiki: Migrate itwiki's "markadmins" job from tool itwiki to dedicated tool - https://phabricator.wikimedia.org/T442501#12564018 (10valerio.bozzolan) p:05Triage→03Low [09:27:16] 10Tool-itwiki: Fix + harden itwiki tools after phpocalypse (October 2026) - https://phabricator.wikimedia.org/T442455#12564022 (10valerio.bozzolan) p:05High→03Low Nothing seems compromised. Lowering priority as this is "just" hardening postponed for decades. [09:28:35] 10Tool-itwiki: Fix + harden itwiki tools after phpocalypse (October 2026) - https://phabricator.wikimedia.org/T442455#12564024 (10valerio.bozzolan) [09:43:00] (03open) 10juniorb8: Link to the volunteer's own edits on My contributions [toolforge-repos/pulac] - 10https://gitlab.wikimedia.org/toolforge-repos/pulac/-/merge_requests/121 (https://phabricator.wikimedia.org/T123456 https://phabricator.wikimedia.org/T445005) [10:31:16] 10Striker, 06tools-platform-team, 03WMDE-TechWish-Kanban: Unable to request Toolforge membership - https://phabricator.wikimedia.org/T444782#12564545 (10MareikeHeuerWMDE) Thank you, yes, please remove the existing link pointing to MareikeHeuer90, I would like to use the LDAP account Mareike Heuer [11:56:13] 10Striker, 06tools-platform-team, 03WMDE-TechWish-Kanban: Unable to request Toolforge membership - https://phabricator.wikimedia.org/T444782#12564985 (10fnegri) 05Open→03Resolved a:03fnegri > Thank you, yes, please remove the existing link pointing to MareikeHeuer90 Done, you should now be able to... [11:58:05] (03update) 10sadrettin: İstatistikler ve ilçe kategorisi (P373) [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/21 [11:58:13] (03update) 10sadrettin: İstatistikler ve ilçe kategorisi (P373) [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/21 [12:02:45] (03update) 10sadrettin: İstatistikler, ilçe kategorisi (P373) ve tablo başlığında tümünü seç [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/21 [12:02:46] (03update) 10sadrettin: İstatistikler, ilçe kategorisi (P373) ve tablo başlığında tümünü seç [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/21 [12:06:56] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/46 [12:06:58] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/81 [12:06:59] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/54 [12:07:30] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1369898 (owner: 10L10n-bot) [12:28:06] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-test-nfs on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:35:29] (03merge) 10sadrettin: İstatistikler, ilçe kategorisi (P373) ve tablo başlığında tümünü seç [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/21 [13:02:57] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/46 (owner: 10l10n-bot) [13:03:16] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/46 (owner: 10l10n-bot) [13:06:42] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/54 (owner: 10l10n-bot) [13:06:55] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/54 (owner: 10l10n-bot) [13:26:18] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43875315038 (https://github.com/cluebotng/component-configs/commits/0865c1cccb5231f7c0473601427d10c7fad1cb34) [13:30:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:35:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:39:12] 10Striker, 06tools-platform-team, 10Charts, 06MediaWiki-API-Platform-Team, and 10 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12565535 (10Nikerabbit) [13:39:14] 06cloud-services-team: Agree how to track/find all WMCS tasks that have a common topic, but belong to different projects - https://phabricator.wikimedia.org/T336681#12565538 (10fnegri) 05Open→03Declined Marking as "Declined" as this has not come up as a need recently. We have been using a combination of... [13:43:18] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43878060764 (https://github.com/cluebotng/component-configs/commits/4b48c82422d710cfd8123b0815ef130df4f2d522) [13:43:20] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43878022011 (https://github.com/cluebotng/component-configs/commits/ca2b894f406aa98efa78494d28288dbfe481b5aa) [13:45:54] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/43878122858 (https://github.com/cluebotng/component-configs/commits/1f6d87176fa799cea1d16ae5d7e24e44749755de) [13:46:09] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/43878122946 (https://github.com/cluebotng/component-configs/commits/1f6d87176fa799cea1d16ae5d7e24e44749755de) [13:46:15] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43878204259 (https://github.com/cluebotng/component-configs/commits/c32fce82ce3092c002221a9d613ef42528c52636) [13:46:24] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43878040473 (https://github.com/cluebotng/component-configs/commits/bb94f721d6e087dd69bf0f07e01037c0fd664c19) [13:46:30] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43878040592 (https://github.com/cluebotng/component-configs/commits/bb94f721d6e087dd69bf0f07e01037c0fd664c19) [13:46:53] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43878089433 (https://github.com/cluebotng/component-configs/commits/918933281f2f16be5da23353082aeee869eb6d23) [13:47:16] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/43878264065 (https://github.com/cluebotng/component-configs/commits/f6ca387cf052638e639177a398a6d7f56e9f7daa) [13:48:44] !log tools.cluebot-documentation Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/43878263911 (https://github.com/cluebotng/component-configs/commits/f6ca387cf052638e639177a398a6d7f56e9f7daa) [13:51:05] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/43878263663 (https://github.com/cluebotng/component-configs/commits/f6ca387cf052638e639177a398a6d7f56e9f7daa) [13:51:11] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/43878263732 (https://github.com/cluebotng/component-configs/commits/f6ca387cf052638e639177a398a6d7f56e9f7daa) [13:52:30] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/43878514349 (https://github.com/cluebotng/component-configs/commits/4ddf69b018412e3551fa88b306c90c537aae4363) [13:52:33] !log tools.cluebotng-trainer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43878616353 (https://github.com/cluebotng/component-configs/commits/e4937110ed07604249b00d2932030b2eff16355c) [13:52:58] !log tools.cluebot-documentation Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/43878514308 (https://github.com/cluebotng/component-configs/commits/4ddf69b018412e3551fa88b306c90c537aae4363) [13:53:14] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43878942579 (https://github.com/cluebotng/component-configs/commits/a0581198660c2e03a15b3ef5c8500bf54b5340bd) [13:53:45] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43878514315 (https://github.com/cluebotng/component-configs/commits/4ddf69b018412e3551fa88b306c90c537aae4363) [13:53:59] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43878514553 (https://github.com/cluebotng/component-configs/commits/4ddf69b018412e3551fa88b306c90c537aae4363) [13:54:27] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43878942499 (https://github.com/cluebotng/component-configs/commits/a0581198660c2e03a15b3ef5c8500bf54b5340bd) [13:54:31] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43878514129 (https://github.com/cluebotng/component-configs/commits/4ddf69b018412e3551fa88b306c90c537aae4363) [13:54:47] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/43878942594 (https://github.com/cluebotng/component-configs/commits/a0581198660c2e03a15b3ef5c8500bf54b5340bd) [14:02:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:25:05] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-api] very slow to response to image list / [components-api] has constant timeouts to api.svc.tools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T442194#12565726 (10Raymond_Ndibe) [14:27:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:38:15] (03update) 10raymond-ndibe: README: link the embargoed security fix process wikitech doc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1522 (https://phabricator.wikimedia.org/T444635) [14:40:49] 06cloud-services-team, 10Toolforge, 10Tools, 06tools-platform-team: Determine if templatetiger is abandoned - https://phabricator.wikimedia.org/T253424#12565795 (10Ameisenigel) [14:41:15] (03update) 10raymond-ndibe: README: link the embargoed security fix process wikitech doc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1522 (https://phabricator.wikimedia.org/T444635) [14:52:28] (03update) 10raymond-ndibe: README: link the embargoed security fix process wikitech doc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1522 (https://phabricator.wikimedia.org/T444635) [14:53:25] (03update) 10raymond-ndibe: README: link the embargoed security fix process wikitech doc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1522 (https://phabricator.wikimedia.org/T444635) [14:53:30] (03update) 10fnegri: README: link the embargoed security fix process wikitech doc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1522 (https://phabricator.wikimedia.org/T444635) (owner: 10raymond-ndibe) [14:53:34] (03update) 10fnegri: README: link the embargoed security fix process wikitech doc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1522 (https://phabricator.wikimedia.org/T444635) (owner: 10raymond-ndibe) [14:53:34] (03approved) 10fnegri: README: link the embargoed security fix process wikitech doc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1522 (https://phabricator.wikimedia.org/T444635) (owner: 10raymond-ndibe) [14:53:40] (03update) 10fnegri: README: link the embargoed security fix process wikitech doc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1522 (https://phabricator.wikimedia.org/T444635) (owner: 10raymond-ndibe) [14:53:50] (03approved) 10raymond-ndibe: README: link the embargoed security fix process wikitech doc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1522 (https://phabricator.wikimedia.org/T444635) [14:54:00] (03merge) 10raymond-ndibe: README: link the embargoed security fix process wikitech doc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1522 (https://phabricator.wikimedia.org/T444635) [15:28:06] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate pcc-worker1002.puppet-diffs.eqiad1.wikimedia.cloud is about to expire in 25d 20h 24m 31s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [15:45:27] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1369898 (owner: 10L10n-bot) [15:57:55] 10Tool-inteGraality: Dashboard - inteGraality didn't write in wiki - https://phabricator.wikimedia.org/T444049#12566501 (10JeanFred) This is related to maxlag − the bot is getting write-throttled. [16:08:42] 10Tool-inteGraality: Replace the free-form `url` parameter with a wiki identity - https://phabricator.wikimedia.org/T438455#12566576 (10JeanFred) [16:12:07] 10Toolforge, 06tools-platform-team: document the deployment process of embargoed patches on toolforge - https://phabricator.wikimedia.org/T444635#12566598 (10Raymond_Ndibe) This was approved by @fnegri and @TLepage-WMF . marking as resolved. [16:12:39] 10Toolforge, 06tools-platform-team: document the deployment process of embargoed patches on toolforge - https://phabricator.wikimedia.org/T444635#12566604 (10Raymond_Ndibe) 05In progress→03Resolved [16:14:53] 10Toolforge, 06tools-platform-team: [components-api,beta] Config not updated from remote source - https://phabricator.wikimedia.org/T401868#12566606 (10Raymond_Ndibe) a:03Raymond_Ndibe [16:14:55] 10Toolforge, 06tools-platform-team: [components-api,beta] Config not updated from remote source - https://phabricator.wikimedia.org/T401868#12566608 (10Raymond_Ndibe) 05Open→03In progress [16:15:58] 06tools-platform-team: Retire wikidata-geo-dumps - https://phabricator.wikimedia.org/T443820#12566615 (10taavi) 05Open→03Invalid The tool account is already disabled, which will lead to it being automatically disabled after some time [16:18:29] 10Cloud-VPS, 06tools-infrastructure-team: Cleanup broken oidentd implementation from cloudnet hosts - https://phabricator.wikimedia.org/T445113 (10taavi) 03NEW [16:21:18] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Cleanup broken oidentd implementation from cloudnet hosts - https://phabricator.wikimedia.org/T445113#12566670 (10taavi) p:05Triage→03Low a:03taavi [16:47:04] (03update) 10fnegri: Dummy API endpoints for /firing & /history [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/16 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) (owner: 10tlepage) [16:53:16] 10Toolforge, 06tools-platform-team: Decide whether & how to sidecar all the things - https://phabricator.wikimedia.org/T445119 (10TLepage-WMF) 03NEW [16:58:04] 10Toolforge, 06tools-platform-team: Decide whether & how to sidecar all the things - https://phabricator.wikimedia.org/T445119#12567028 (10TLepage-WMF) [17:20:17] (03PS1) 10Majavah: README: Update example addresses [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1370025 [17:56:49] 10Cloud-VPS, 06tools-infrastructure-team: Cannot create EC2 credentials in OS - https://phabricator.wikimedia.org/T444826#12567416 (10brennen) [18:07:17] 10Tool-inteGraality: Replace the free-form `url` parameter with a wiki identity - https://phabricator.wikimedia.org/T438455#12567533 (10JeanFred) [18:11:05] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [18:15:24] 10Cloud-VPS, 06tools-infrastructure-team: Cannot create EC2 credentials in OS - https://phabricator.wikimedia.org/T444826#12567594 (10taavi) It seems like there have been a number of upstream fixes in this area recently (warning - suspiciously well-formatted plain text security reports ahead), which I suspect:... [18:20:21] 10Cloud-VPS, 06tools-infrastructure-team: Cannot create EC2 credentials in OS - https://phabricator.wikimedia.org/T444826#12567628 (10brennen) > warning - suspiciously well-formatted plain text security reports ahead Yeah, I did notice that the error message [[ https://docs.openstack.org/keystone/2026.2/_modu... [18:26:06] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [19:42:24] 10Toolforge, 06tools-platform-team: Decide the fate of liveness probes in Toolforge - https://phabricator.wikimedia.org/T445148 (10TLepage-WMF) 03NEW [19:47:43] 10Toolforge, 06tools-platform-team: Decide the fate of liveness probes in Toolforge - https://phabricator.wikimedia.org/T445148#12567964 (10TLepage-WMF) [20:39:26] (03approved) 10kineticpelagic: chore: run on Node.js 24 via the Toolforge Build Service [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/28 (https://phabricator.wikimedia.org/T442177) (owner: 10aghirelli) [20:40:35] (03merge) 10kineticpelagic: chore: run on Node.js 24 via the Toolforge Build Service [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/28 (https://phabricator.wikimedia.org/T442177) (owner: 10aghirelli) [21:34:23] (03PS1) 10Ryan Kemper: Add dummy analytics-experiment keytabs [labs/private] - 10https://gerrit.wikimedia.org/r/1370104 (https://phabricator.wikimedia.org/T416709) [21:39:58] (03CR) 10Ryan Kemper: [V:03+2 C:03+2] Add dummy analytics-experiment keytabs [labs/private] - 10https://gerrit.wikimedia.org/r/1370104 (https://phabricator.wikimedia.org/T416709) (owner: 10Ryan Kemper) [21:44:13] (03update) 10ibrahemqasim: Sync SQLAlchemy models with MySQL schema [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/41 [21:45:59] (03update) 10tlepage: Reject non-api-gateway traffic via NetworkPolicy [repos/cloud/toolforge/alerts-api] (dummy-api-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/20 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [21:46:14] (03update) 10tlepage: Dummy API endpoints for /firing & /history [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/16 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [22:14:33] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12568710 (101Veertje) I've updated the MediaActions and VideoCutTool gadgets. Is the ImageAnnotateTool a new gadget? [22:14:42] (03update) 10renovatebot: Update pnpm to v12.21.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/44 [22:14:44] (03open) 10renovatebot: Update pnpm to v12.21.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/44 [22:14:59] (03update) 10renovatebot: Update pnpm to v12.21.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/52 [22:15:01] (03open) 10renovatebot: Update pnpm to v12.21.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/52 [22:15:16] (03update) 10renovatebot: Update pnpm to v12.21.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/51 [22:15:19] (03open) 10renovatebot: Update pnpm to v12.21.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/51 [22:16:23] (03update) 10renovatebot: Update pnpm to v12.21.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/81 [22:16:26] (03open) 10renovatebot: Update pnpm to v12.21.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/81 [22:17:25] (03update) 10renovatebot: Update pnpm to v12.21.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/73 [22:17:28] (03open) 10renovatebot: Update pnpm to v12.21.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/73 [22:27:55] (03update) 10tlepage: Dummy API endpoints for /firing & /history [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/16 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [22:28:12] (03update) 10tlepage: Reject non-api-gateway traffic via NetworkPolicy [repos/cloud/toolforge/alerts-api] (dummy-api-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/20 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [22:30:16] (03update) 10tlepage: Dummy API endpoints for /firing & /history [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/16 (https://phabricator.wikimedia.org/T441144 https://phabricator.wikimedia.org/T441145) [23:06:15] (03update) 10ibrahemqasim: Sync SQLAlchemy models with MySQL schema [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/41 [23:08:10] (03merge) 10ibrahemqasim: Sync SQLAlchemy models with MySQL schema [toolforge-repos/mdwiki] - 10https://gitlab.wikimedia.org/toolforge-repos/mdwiki/-/merge_requests/41 [23:31:47] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] allow specifying `source_repo`+`ref` for the config - https://phabricator.wikimedia.org/T402764#12568979 (10Raymond_Ndibe) a:03Raymond_Ndibe [23:32:57] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] allow specifying `source_repo`+`ref` for the config - https://phabricator.wikimedia.org/T402764#12568980 (10Raymond_Ndibe) assigned to myself too, because this is a subtask of https://phabricator.wikimedia.org/T401868,... [23:33:20] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] allow specifying `source_repo`+`ref` for the config - https://phabricator.wikimedia.org/T402764#12568983 (10Raymond_Ndibe) 05Open→03In progress [23:44:14] (03update) 10raymond-ndibe: Expand `source` support in `ToolConfig` [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T402764) (owner: 10damian) [23:46:00] 10Toolforge, 06tools-platform-team: [components-api,beta] Config not updated from remote source - https://phabricator.wikimedia.org/T401868#12568999 (10Raymond_Ndibe) reminder to self: close when https://phabricator.wikimedia.org/T402764 closes