[00:06:36] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609639432 (https://github.com/cluebotng/component-configs/commits/8c2fccaaae357774084389157d9a305e72eccb20) [00:06:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [00:07:17] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609639469 (https://github.com/cluebotng/component-configs/commits/8c2fccaaae357774084389157d9a305e72eccb20) [00:07:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [00:33:10] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26610547573 (https://github.com/cluebotng/component-configs/commits/d647b7c087dff8de798d78a98082469fa41e7712) [00:33:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [00:36:36] 10Tools, 10MassMessage, 06WikiNYC Tech Committee: MassMessage toolkit - https://phabricator.wikimedia.org/T420847#11965908 (10Pharos) [00:54:44] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26611217389 (https://github.com/cluebotng/component-configs/commits/1c99ebab25e9cbf3bd595f45368b3a5e15489941) [00:54:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [02:31:14] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26614336971 (https://github.com/cluebotng/component-configs/commits/622f302a735e2a285b7ac4f46e726aaf538f8d93) [02:31:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [06:57:59] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-34 (2026-05-19 to 2026-06-02)), 07OKR-Work: Fix issue line highlighting in the linter - https://phabricator.wikimedia.org/T425935#11966145 (10KineticPelagic) 05In progress→03Resolved [07:43:17] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-34 (2026-05-19 to 2026-06-02)), 07OKR-Work: Improve and standardize linter messages - https://phabricator.wikimedia.org/T426821#11966208 (10KBach) 05In progress→03Resolved [07:56:45] 10Tool-Pageviews: Add ability to see pageviews of pages that transclude files in a given category - https://phabricator.wikimedia.org/T150507#11966223 (10Aklapper) [07:58:37] 10Tools: In Glamorous enable limiting the number of uses shown by default - https://phabricator.wikimedia.org/T379725#11966228 (10Aklapper) 05Open→03Invalid https://codeberg.org/magnusmanske/glamtools/issues seems to be the issue tracker for that piece of software; looks like this project does not use Ph... [07:58:39] 10Tools: In Glamorous the square brackets [ ] in the link cause these links to break in wikitext - https://phabricator.wikimedia.org/T379721#11966230 (10Aklapper) 05Open→03Invalid https://codeberg.org/magnusmanske/glamtools/issues seems to be the issue tracker for that piece of software; looks like this... [09:02:18] (03update) 10samwilson: Upgrade to Symfony 7 [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/9 [10:56:31] (03open) 10don-vip: Slava Ukraini [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/34 [11:00:29] (03update) 10don-vip: Slava Ukraini [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/34 [11:01:50] (03update) 10don-vip: Slava Ukraini [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/34 [11:02:26] (03update) 10vriaa: feat: add translations link as a selectable, styleable banner element [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/79 (https://phabricator.wikimedia.org/T423927) [11:02:35] (03merge) 10vriaa: feat: add translations link as a selectable, styleable banner element [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/79 (https://phabricator.wikimedia.org/T423927) [11:19:01] (03merge) 10don-vip: Slava Ukraini [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/34 [13:25:45] 06cloud-services-team, 10Toolforge: Unable to add MariaDB index to (large) table - https://phabricator.wikimedia.org/T427470#11967263 (10magnusmanske) [13:58:09] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 10Toolforge, 06tools-platform-team: [infra] Toolforge: migrate to Debian Bookworm or later - https://phabricator.wikimedia.org/T387005#11967535 (10aputhin) [14:04:32] 06tools-platform-team, 10Toolhub: Replace Bullseye base image with Trixie before August 2026 EOL deadline - https://phabricator.wikimedia.org/T425303#11967567 (10aputhin) >>! In T425303#11963125, @elukey wrote: > @aputhin Hi! Do you have a timeline about the completion of this work? The sooner we migrate away... [14:16:50] 10Toolforge, 06tools-platform-team: Unable to add MariaDB index to (large) table - https://phabricator.wikimedia.org/T427470#11967662 (10aputhin) [14:24:35] 10Toolforge, 06tools-platform-team: Unable to add MariaDB index to (large) table - https://phabricator.wikimedia.org/T427470#11967714 (10fnegri) 05Open→03In progress a:03fnegri I'm not sure what's causing the query to be interrupted. We do have a default limit of 1 hour that can be changed with `SET SESS... [14:34:08] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: gitlab-webhooks: toolforge webservice logs -f choking on a log message - https://phabricator.wikimedia.org/T427449#11967761 (10aputhin) [14:59:57] 06cloud-services-team, 10Openstack-Magnum: debian packaging for magnum-cluster-api - https://phabricator.wikimedia.org/T426431#11967858 (10Andrew) I have this working now, and somewhat documented. The only caveat is that everything is on the debian/trixie-wikimedia branch; not sure if that should be backported... [15:33:22] 06cloud-services-team, 10Openstack-Magnum: Investigate new Magnum drivers - https://phabricator.wikimedia.org/T393782#11968002 (10Andrew) [] ~~wait a bit for upstream to resolve the docker_volume_size thing~~ added this as patch to our debian package instead [x] redeploy in codfw1dev and make sure there aren... [15:38:22] 10Toolforge, 06tools-platform-team: Unable to add MariaDB index to (large) table - https://phabricator.wikimedia.org/T427470#11968026 (10magnusmanske) 05In progress→03Resolved That worked, many thanks! [15:46:12] (03PS1) 10Andrew Bogott: Add fake k3s config for eqiad1 magnum [labs/private] - 10https://gerrit.wikimedia.org/r/1295474 [15:48:08] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Add fake k3s config for eqiad1 magnum [labs/private] - 10https://gerrit.wikimedia.org/r/1295474 (owner: 10Andrew Bogott) [16:34:07] FIRING: ToolsDBReplicationLagIsTooHigh: ToolsDB replication is lagging - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationLagIsTooHigh [16:40:38] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate Puppet CA: metricsinfra-puppetmaster-1.metricsinfra.eqiad1.wikimedia.cloud is about to expire in 15d 23h 58m 10s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [16:44:07] RESOLVED: ToolsDBReplicationLagIsTooHigh: ToolsDB replication is lagging - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBReplicationLagIsTooHigh [17:40:22] FIRING: [3x] HAProxyBackendUnavailable: HAProxy service magnum-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:41:47] 10Toolforge, 06tools-platform-team: Unable to add MariaDB index to (large) table - https://phabricator.wikimedia.org/T427470#11968377 (10fnegri) Took 70 minutes in total: `lang=mysql MariaDB [s51434__mixnmatch_p]> ALTER TABLE `entry` ADD INDEX `user_timestamp` (`user`, `timestamp`), ALGORITHM=INPLACE, LOC... [17:45:22] RESOLVED: [3x] HAProxyBackendUnavailable: HAProxy service magnum-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [18:21:23] 10VPS-project-Codesearch, 06collaboration-services, 13Patch-For-Review: confd fails with "no such host" in SRV lookup from _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T417458#11968448 (10Dzahn) Figured out the fix is adding this to Hiera instead: ` profile::c... [18:21:45] 10VPS-project-Codesearch, 06collaboration-services, 13Patch-For-Review: confd fails with "no such host" in SRV lookup from _etcd-client-ssl._tcp.codesearch.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T417458#11968449 (10Dzahn) 05Open→03Resolved a:03Dzahn [18:44:22] FIRING: [3x] HAProxyBackendUnavailable: HAProxy service magnum-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [18:49:22] RESOLVED: [3x] HAProxyBackendUnavailable: HAProxy service magnum-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [19:13:16] 10Tool-humaniki-2: Migrate site to Cloud VPS? - https://phabricator.wikimedia.org/T425489#11968597 (10Danya) The JSON dump ingester is very slow on Toolforge: it took ~25h to parse 35.5 million out of ~120 million lines. Maybe we need to migrate it from Python to a faster language like Java, C, or C++. The issue... [19:19:51] 10Tool-humaniki-2: Migrate site to Cloud VPS? - https://phabricator.wikimedia.org/T425489#11968621 (10Danya) [19:36:37] 06cloud-services-team, 10Openstack-Magnum: Investigate new Magnum drivers - https://phabricator.wikimedia.org/T393782#11968673 (10Andrew) This has run into a new blocker, logged upstream as https://github.com/vexxhost/magnum-cluster-api/issues/1057 We can proceed and just rely on ubuntu worker nodes but it su... [20:32:17] (03update) 10raymond-ndibe: fix jobs load bug [repos/cloud/toolforge/jobs-api] (fix_oneoff_storage_error) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/287 (https://phabricator.wikimedia.org/T423544)