[00:02:17] 06cloud-services-team, 10Toolforge, 10Tools: Geohack tool frequently triggers the Toolforge front proxy's per-tool rate limit due to too much traffic - https://phabricator.wikimedia.org/T409185#11361550 (10bd808) >>! In T409185#11357778, @Magnus wrote: > Would it help if I rewrote it in Rust, with some cachi... [00:07:03] 06cloud-services-team, 10Toolforge: Track global and per-tool concurrent requests and 503 rate limitiing responses from Toolforge CDN edge - https://phabricator.wikimedia.org/T409794 (10bd808) 03NEW [00:37:29] FIRING: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [00:40:22] (03open) 10raymond-ndibe: tests: use harbor prebuilt images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/248 (https://phabricator.wikimedia.org/T409727) [00:40:25] (03update) 10raymond-ndibe: tests: use harbor prebuilt images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/248 (https://phabricator.wikimedia.org/T409727) [00:41:18] (03update) 10raymond-ndibe: tests: use harbor prebuilt images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/248 (https://phabricator.wikimedia.org/T409727) [00:46:01] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [builds-api,harbor,image-config] Move pre-built images to harbor - https://phabricator.wikimedia.org/T409727#11361615 (10Raymond_Ndibe) [00:47:54] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [builds-api,harbor,image-config] Move pre-built images to harbor - https://phabricator.wikimedia.org/T409727#11361621 (10Raymond_Ndibe) [00:49:05] 10VPS-project-Codesearch, 06collaboration-services, 13Patch-For-Review: Codesearch: Ensure logrotate for /var/log/account/pacct - https://phabricator.wikimedia.org/T408234#11361622 (10Dzahn) created and merged the change above. created some files like `dd if=/dev/random of=/var/log/account/pacct.3 bs=1M... [00:49:17] 10VPS-project-Codesearch, 06collaboration-services, 13Patch-For-Review: Codesearch: Ensure logrotate for /var/log/account/pacct - https://phabricator.wikimedia.org/T408234#11361623 (10Dzahn) 05Open→03Resolved a:03Dzahn [00:52:29] RESOLVED: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [00:53:38] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [builds-api,harbor,image-config] Move pre-built images to harbor - https://phabricator.wikimedia.org/T409727#11361626 (10Raymond_Ndibe) [00:54:00] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [builds-api,harbor,image-config] Move pre-built images to harbor - https://phabricator.wikimedia.org/T409727#11361627 (10Raymond_Ndibe) [01:30:02] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [builds-api] Add an endpoint to get all available images - https://phabricator.wikimedia.org/T409726#11361659 (10Raymond_Ndibe) `image-config` `configmap` has the below structure currently: **NOTE:** the below entry is not an exact example of what is... [01:40:04] (03open) 10raymond-ndibe: disable_tool.py: fix mypy error [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/25 [01:42:38] (03update) 10raymond-ndibe: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 [02:31:29] FIRING: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [02:36:29] RESOLVED: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [03:04:45] PROBLEM - Host clouddb1017 is DOWN: PING CRITICAL - Packet loss = 100% [03:06:22] FIRING: [4x] HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-s1 backend clouddb1017.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [03:12:26] RECOVERY - Host clouddb1018 is UP: PING OK - Packet loss = 0%, RTA = 0.53 ms [03:12:28] RECOVERY - Host clouddb1017 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [03:22:39] RESOLVED: [4x] HAProxyBackendUnavailable: HAProxy service wikireplica-db-analytics-s1 backend clouddb1017.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [03:57:51] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [03:58:05] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [03:58:09] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [07:59:29] FIRING: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [08:04:29] RESOLVED: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [08:08:30] 10Tool-yearinreview, 10MediaWiki-extensions-Translate, 06Wikipedia-Android-App-Backlog, 06LPL Essential (FY26 Q2), and 2 others: PLURAL syntax validator gets confused by other uses of equals signs in the message, as seen at [[Wikimedia:Wikipedia-android-st... - https://phabricator.wikimedia.org/T409655#11361977 [08:28:16] (03open) 10arthurtaylor: Add user agent to abide by the wikimedia bot policy [toolforge-repos/wmde-phabricator-charts] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-phabricator-charts/-/merge_requests/1 [08:28:39] (03merge) 10arthurtaylor: Add user agent to abide by the wikimedia bot policy [toolforge-repos/wmde-phabricator-charts] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-phabricator-charts/-/merge_requests/1 [09:12:04] (03open) 10dcaro: images: load refresh time from settings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/249 [09:19:21] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1203438 (owner: 10L10n-bot) [09:19:25] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1203436 (owner: 10L10n-bot) [09:20:29] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1203442 (owner: 10L10n-bot) [09:20:37] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1203440 (owner: 10L10n-bot) [09:20:47] (03open) 10dcaro: images: cache images retrieved from harbor [repos/cloud/toolforge/jobs-api] (image_use_setting) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/250 [09:47:27] (03update) 10dcaro: images: cache images retrieved from harbor [repos/cloud/toolforge/jobs-api] (image_use_setting) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/250 [10:36:29] FIRING: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [10:59:36] (03update) 10damian: Draft: core: normalize job images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/245 [10:59:42] (03close) 10damian: Draft: Move image resolution to core [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/244 [11:01:29] RESOLVED: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [11:46:43] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] ibdata1 growing on primary - https://phabricator.wikimedia.org/T409716#11362678 (10fnegri) @Usernamekiran I added `autocommit=True` to the db_connection in `enwiki/amp/amp_rc.py`, and restarted the `amp-rc` job... [12:08:19] (03open) 10dcaro: images: resolve the image every time [repos/cloud/toolforge/jobs-api] (use_cache_for_harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 [12:12:09] (03update) 10dcaro: images: resolve the image every time [repos/cloud/toolforge/jobs-api] (use_cache_for_harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 [12:49:29] FIRING: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [12:59:29] RESOLVED: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [13:00:49] (03update) 10dcaro: images: resolve the image every time [repos/cloud/toolforge/jobs-api] (use_cache_for_harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 [13:20:34] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266856634 (https://github.com/cluebotng/component-configs/commits/b0e9170597a778654185be762c580e2a6e19492f) [13:20:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [13:20:49] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266868378 (https://github.com/cluebotng/component-configs/commits/1179c63e9137cfff0d1cdc5282eb13246b57ef80) [13:20:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [13:20:56] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266879941 (https://github.com/cluebotng/component-configs/commits/7c158c111d99bc1219b8cd64ea79d80a40116c93) [13:20:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [13:22:01] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266902429 (https://github.com/cluebotng/component-configs/commits/d31ec1d2c99a6c00020790543c586a10bdb6a8a2) [13:22:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [13:24:36] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266909480 (https://github.com/cluebotng/component-configs/commits/bc0dd19078113c4e1cbe90a059fe26932fb70a29) [13:24:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [13:34:28] FIRING: TargetDown: Job toolsdb-mariadb is unreachable in project tools instance tools-db-4 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:34:31] FIRING: ToolsToolsDBReplicationError: ToolsDB replication is broken on tools-db-6 (errno 2003) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [13:34:31] FIRING: ToolsToolsDBWritableState: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [13:35:56] FIRING: SystemdUnitDown: The service unit disable-tool.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:40:29] (03update) 10dcaro: images: resolve the image every time [repos/cloud/toolforge/jobs-api] (use_cache_for_harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 [13:40:56] RESOLVED: SystemdUnitDown: The service unit disable-tool.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:41:20] (03update) 10don-vip: Draft: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10 [13:41:23] (03update) 10dcaro: images: resolve the image every time [repos/cloud/toolforge/jobs-api] (use_cache_for_harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 [13:41:26] FIRING: SystemdUnitDown: The service unit disable-tool.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:43:29] FIRING: ToolforgeToolviewsFailed: Toolviews processing failed - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsFailed [13:43:43] (03open) 10fnegri: Fail over tools-db [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/281 (https://phabricator.wikimedia.org/T409287) [13:44:12] (03approved) 10filippo: Fail over tools-db [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/281 (https://phabricator.wikimedia.org/T409287) (owner: 10fnegri) [13:44:39] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19267535662 (https://github.com/cluebotng/component-configs/commits/eae37aa10505a2ac3c773cdd4f85b7222cdae836) [13:44:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [13:45:15] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 13Patch-For-Review, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11362935 (10fnegri) Failing over now from tools-db-4 to tools-db-6, following https://wikitech.wikimedia.... [13:45:33] (03update) 10dcaro: images: resolve the image every time [repos/cloud/toolforge/jobs-api] (use_cache_for_harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 [13:45:39] (03merge) 10fnegri: Fail over tools-db [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/281 (https://phabricator.wikimedia.org/T409287) [13:45:54] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:46:38] !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [13:50:34] 06cloud-services-team, 10Toolforge: [webservice] returning default 404 even though webservice is healthy - https://phabricator.wikimedia.org/T403168#11362956 (10DamianZaremba) Noticed this again today during tools-db having an outage. Application is returning a server error as expected, since it cannot connec... [13:51:26] RESOLVED: SystemdUnitDown: The service unit disable-tool.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:11:11] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 13Patch-For-Review, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11363059 (10fnegri) tools-db-6 is now the new primary. Heartbeat enabled and working, I noticed the last... [14:13:41] (03update) 10dcaro: images: resolve the image every time [repos/cloud/toolforge/jobs-api] (use_cache_for_harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 [14:15:00] (03update) 10dcaro: images: cache images retrieved from harbor [repos/cloud/toolforge/jobs-api] (image_use_setting) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/250 [14:17:57] (03update) 10dcaro: images: cache images retrieved from harbor [repos/cloud/toolforge/jobs-api] (image_use_setting) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/250 [14:18:29] RESOLVED: ToolforgeToolviewsFailed: Toolviews processing failed - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsFailed [14:18:49] (03update) 10dcaro: images: load refresh time from settings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/249 [14:32:29] FIRING: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [14:40:05] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19268985739 (https://github.com/cluebotng/component-configs/commits/f88bf173399c2591eca2357bbc9bff54ee70b731) [14:40:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [14:42:29] RESOLVED: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [15:01:42] (03open) 10dcaro: images: support harbor-based pre-built images [repos/cloud/toolforge/jobs-api] (resolve_harbor_images_every_time) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/252 (https://phabricator.wikimedia.org/T409727) [15:14:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol1011:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:19:48] FIRING: [2x] PuppetFailure: Puppet has failed on cloudcontrol1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:24:16] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270254406 (https://github.com/cluebotng/component-configs/commits/5dfa0eff0c51e0d878863a5ac689e59f79d59876) [15:24:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [15:25:10] 06cloud-services-team (FY2025/26-Q1), 10Toolforge: [toolsdb] Add users and grants for IPv6, remove obsolete ones - https://phabricator.wikimedia.org/T409563#11363459 (10fnegri) I'm going to run the following commands on the current ToolsDB primary (tools-db-6), they will replicate automatically to the the repl... [15:25:25] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270276189 (https://github.com/cluebotng/component-configs/commits/040b83debcd71b207ee872095a3d2a1166ccc618) [15:25:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [15:25:30] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270264790 (https://github.com/cluebotng/component-configs/commits/531100511571d8262fc874f0f4cddfa2df3d6c2b) [15:25:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [15:25:59] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270264807 (https://github.com/cluebotng/component-configs/commits/531100511571d8262fc874f0f4cddfa2df3d6c2b) [15:26:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [15:26:34] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270303362 (https://github.com/cluebotng/component-configs/commits/df94fd299dd1ed07436488c348bad61d859da592) [15:26:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [15:27:16] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270263370 (https://github.com/cluebotng/component-configs/commits/d1674e8f4f6cec3b48e848137ce42585278d4a67) [15:27:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [15:27:32] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270301618 (https://github.com/cluebotng/component-configs/commits/f28dcaec8c5882b4a1b7d861fe7f5e400312a5b4) [15:27:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [15:27:40] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270285003 (https://github.com/cluebotng/component-configs/commits/e103f6ac56b26a2d6e3c0705c81c75a4419287cc) [15:27:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [15:28:24] !log volans@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.7 (T399313) [15:28:25] !log volans@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.7 (T399313) [15:28:28] T399313: Add tracing to understand Toolforge and CloudVPS usage and dependencies - https://phabricator.wikimedia.org/T399313 [15:28:43] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270313845 (https://github.com/cluebotng/component-configs/commits/900a30b8983bc80a537330fa4345e7952f2081b4) [15:28:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [15:28:45] !log volans@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.7 (T399313) [15:29:48] FIRING: [3x] PuppetFailure: Puppet has failed on cloudcontrol1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:30:09] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [components-api] Queue builds when the build queue is full - https://phabricator.wikimedia.org/T402568#11363472 (10DamianZaremba) Hit again today while bumping releases on nearly everything ` Deployment ID: 20251111-152523-4chqpjcf8c Created: 20251111-1... [15:30:35] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270301650 (https://github.com/cluebotng/component-configs/commits/f28dcaec8c5882b4a1b7d861fe7f5e400312a5b4) [15:30:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [15:31:17] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270294471 (https://github.com/cluebotng/component-configs/commits/df4e433c6a567df4484b7115ddf2c53fe1f9494f) [15:31:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [15:33:08] 06cloud-services-team (FY2025/26-Q1), 10Toolforge: [toolsdb] Add users and grants for IPv6, remove obsolete ones - https://phabricator.wikimedia.org/T409563#11363498 (10taavi) https://mariadb.com/docs/server/reference/sql-statements/account-management-sql-statements/create-user#host-name-component seems to sug... [15:36:46] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) [15:36:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [15:36:52] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642865 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) [15:36:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [15:37:31] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642885 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) [15:37:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [15:37:33] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642903 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) [15:37:33] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642926 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) [15:37:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [15:37:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [15:37:43] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270643026 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) [15:37:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [15:37:52] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642882 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) [15:37:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [15:38:04] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642949 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) [15:38:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [15:39:35] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642915 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) [15:39:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [15:41:55] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) [15:41:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [15:42:46] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642865 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c) [15:42:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [15:43:57] 06cloud-services-team, 10Cloud-VPS: Puppet fails on cloudcontrol when updating /srv/tofu-infra - https://phabricator.wikimedia.org/T373815#11363521 (10fnegri) 05Resolved→03Open This just happened again. [15:45:10] 06cloud-services-team, 10Cloud-VPS: Puppet fails on cloudcontrol when updating /srv/tofu-infra - https://phabricator.wikimedia.org/T373815#11363524 (10fnegri) a:05aborrero→03None [15:52:45] 06cloud-services-team (FY2025/26-Q1), 10Toolforge: [toolsdb] Add users and grants for IPv6, remove obsolete ones - https://phabricator.wikimedia.org/T409563#11363532 (10fnegri) > Your LIKE clause for the /56 has an additional colon, that should be 2a02:ec80:a000:00% instead. Otherwise LGTM. Good catch, fixed! [15:54:48] FIRING: [3x] PuppetFailure: Puppet has failed on cloudcontrol1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:58:48] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11363540 (10fnegri) > investigate[0][1] what's in the big ibdata1 in tools-db-4 I tried using innochecksum but it's taking hou... [15:59:41] (03open) 10don-vip: Draft: Add USGS Multimedia Gallery [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/11 [15:59:48] FIRING: [3x] PuppetFailure: Puppet has failed on cloudcontrol1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:04:36] (03update) 10volans: kind: add port 30004 for loki-tracing [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/294 (https://phabricator.wikimedia.org/T399313) [16:04:49] (03merge) 10volans: kind: add port 30004 for loki-tracing [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/294 (https://phabricator.wikimedia.org/T399313) [16:05:34] (03update) 10don-vip: Draft: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10 [16:05:48] (03update) 10don-vip: Draft: Add USGS Multimedia Gallery [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/11 [16:07:56] (03update) 10don-vip: Draft: Update to new NASA Photojournal [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/8 [16:09:30] 10Toolforge (Toolforge iteration 25): [maintain-kubeusers,maintain-dbusers] user homes are not readable by replica_cnf so it fails to create replica.my.cnf files - https://phabricator.wikimedia.org/T409847 (10dcaro) 03NEW [16:09:48] RESOLVED: [3x] PuppetFailure: Puppet has failed on cloudcontrol1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:12:13] 10Toolforge (Toolforge iteration 25): [maintain-kubeusers,maintain-dbusers] user homes are not readable by replica_cnf so it fails to create replica.my.cnf files - https://phabricator.wikimedia.org/T409847#11363598 (10taavi) > It might be that when the user logs in before maintain_kubeusers created the home dir... [16:16:28] 10Toolforge (Toolforge iteration 25): [maintain-kubeusers,maintain-dbusers] user homes are not readable by replica_cnf so it fails to create replica.my.cnf files - https://phabricator.wikimedia.org/T409847#11363616 (10dcaro) Populating the accounts seem to flop on the 29th of september: {F70115174} https://grafa... [16:25:46] (03open) 10don-vip: T389026 - remove field rev_sha1 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/12 [16:26:40] (03update) 10don-vip: T389026 - remove field rev_sha1 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/12 [16:28:59] 10Toolforge (Toolforge iteration 25): [maintain-kubeusers,maintain-dbusers] user homes are not readable by replica_cnf so it fails to create replica.my.cnf files - https://phabricator.wikimedia.org/T409847#11363701 (10fnegri) I noticed that `maintain-dbusers` logs contain many errors, maybe we should alert when... [16:31:10] 10Toolforge (Toolforge iteration 25): [maintain-kubeusers,maintain-dbusers] user homes are not readable by replica_cnf so it fails to create replica.my.cnf files - https://phabricator.wikimedia.org/T409847#11363718 (10fnegri) A user complained about this on Sep, 10th: {T404175} I fixed that one manually, but I... [16:32:34] (03approved) 10pepepiton: Make the footer stick to the bottom regardless of the page how short the page content is [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/98 (owner: 10kimbrenekakande) [16:35:29] FIRING: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [16:36:12] (03merge) 10don-vip: T389026 - remove field rev_sha1 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/12 [16:38:24] (03update) 10don-vip: Draft: Add USGS Multimedia Gallery [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/11 [16:38:35] (03update) 10don-vip: Draft: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10 [16:38:47] (03update) 10don-vip: Draft: Update to new NASA Photojournal [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/8 [16:45:29] RESOLVED: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [16:49:34] (03update) 10volans: logging: add tracing loki instance [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1040 (https://phabricator.wikimedia.org/T399313) [16:50:43] (03update) 10volans: logging: add tracing loki instance [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1040 (https://phabricator.wikimedia.org/T399313) [16:53:10] (03update) 10volans: tracing: add tracing loki instance [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1040 (https://phabricator.wikimedia.org/T399313) [17:00:34] (03update) 10volans: shared: add loki-tracing S3 buckets [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/92 (https://phabricator.wikimedia.org/T399313) [17:01:52] (03update) 10don-vip: Draft: Add USGS Multimedia Gallery [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/11 [17:02:02] (03approved) 10volans: shared: add loki-tracing S3 buckets [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/92 (https://phabricator.wikimedia.org/T399313) [17:03:17] (03update) 10don-vip: Draft: Add USGS Multimedia Gallery [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/11 [17:04:02] (03update) 10don-vip: Add USGS Multimedia Gallery [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/11 [17:07:59] 10Toolforge (Toolforge iteration 25): [maintain-kubeusers,maintain-dbusers] user homes are not readable by replica_cnf so it fails to create replica.my.cnf files - https://phabricator.wikimedia.org/T409847#11363833 (10dcaro) It seems there's currently ~19 accounts affected: ` root@cloudcontrol1007:~# journalctl... [17:11:44] (03merge) 10don-vip: Add USGS Multimedia Gallery [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/11 [17:26:51] 10Toolforge (Toolforge iteration 25): [maintain-kubeusers,maintain-dbusers] user homes are not readable by replica_cnf so it fails to create replica.my.cnf files - https://phabricator.wikimedia.org/T409847#11363862 (10dcaro) I think it's failing to commit that some users were already created, and recounting them... [17:34:55] 06cloud-services-team (FY2025/26-Q1), 10Toolforge: [toolsdb] Add users and grants for IPv6, remove obsolete ones - https://phabricator.wikimedia.org/T409563#11363899 (10fnegri) All grants applied, current situation: `lang=mysql MariaDB [(none)]> select user, host from mysql.user where host != '%' ORDER BY use... [17:35:17] 06cloud-services-team (FY2025/26-Q1), 10Toolforge: [toolsdb] Add users and grants for IPv6, remove obsolete ones - https://phabricator.wikimedia.org/T409563#11363902 (10fnegri) 05Open→03In progress p:05Triage→03High [17:42:49] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] ibdata1 growing on primary - https://phabricator.wikimedia.org/T409716#11363928 (10fnegri) >>! In T409716#11358231, @fnegri wrote: > @JeanFred @Multichill it looks like the `heritage` tool is using very long tr... [17:46:28] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [maintain-kubeusers,maintain-dbusers] user homes are not readable by replica_cnf so it fails to create replica.my.cnf files - https://phabricator.wikimedia.org/T409847#11363936 (10dcaro) I think this should avoid the current errors: https://gerrit.wikim... [17:55:27] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] ibdata1 growing on primary - https://phabricator.wikimedia.org/T409716#11363959 (10fnegri) [17:56:54] 06cloud-services-team, 10Toolforge: [toolsdb] Automatically terminate long transactions - https://phabricator.wikimedia.org/T409857 (10fnegri) 03NEW [17:57:00] 06cloud-services-team, 10Toolforge: [toolsdb] Automatically terminate long transactions - https://phabricator.wikimedia.org/T409857#11363986 (10fnegri) p:05Triage→03Medium [17:59:47] 06cloud-services-team (FY2025/26-Q1), 10Toolforge: [toolsdb] Add users and grants for IPv6, remove obsolete ones - https://phabricator.wikimedia.org/T409563#11364019 (10fnegri) Side note: I took the chance to collect all the passwords for the ToolsDB accounts above and stored them in pwstore under a new file `... [18:01:12] 06cloud-services-team, 10Toolforge: [toolsdb] Automatically terminate long transactions - https://phabricator.wikimedia.org/T409857#11364045 (10fnegri) [18:01:20] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] ibdata1 growing on primary - https://phabricator.wikimedia.org/T409716#11364044 (10fnegri) [18:06:18] 06cloud-services-team, 10PAWS: PAWS: Add WikibaseIntegrator to paws - https://phabricator.wikimedia.org/T408972#11364134 (10fnegri) p:05Triage→03Medium [18:10:29] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19274510349 (https://github.com/cluebotng/component-configs/commits/fea93378dd3d44f7cb93bc04bcdeb93272207f7e) [18:10:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [18:12:19] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [maintain-kubeusers,maintain-dbusers] user homes are not readable by replica_cnf so it fails to create replica.my.cnf files - https://phabricator.wikimedia.org/T409847#11364228 (10fnegri) 05Open→03In progress... [18:12:48] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11364234 (10fnegri) [18:15:01] (03open) 10don-vip: Add fallback to broken mp3 format detection [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/13 [18:22:00] 06cloud-services-team, 10Toolforge, 10import-500px: Tool seems not to work - https://phabricator.wikimedia.org/T324487#11364270 (10Olea) Not sure if this is the way to ping the Toolforge admins. [18:22:56] 06cloud-services-team, 10Toolforge, 10import-500px: Tool seems not to work - https://phabricator.wikimedia.org/T324487#11364274 (10Olea) ping @Chicocvenancio [18:24:10] (03merge) 10don-vip: Add fallback to broken mp3 format detection [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/13 [18:25:34] (03update) 10don-vip: Draft: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10 [18:26:55] (03update) 10don-vip: Draft: Update to new NASA Photojournal [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/8 [18:29:44] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19274968338 (https://github.com/cluebotng/component-configs/commits/11a18e10782c595ec28a055c4d1f4289f2276be9) [18:29:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [18:31:28] (03update) 10oluwatumininu: Improved layout and organized headings [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/52 [18:33:29] FIRING: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [18:38:29] RESOLVED: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [18:42:05] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [image-config] deprecate and move all data to builds-api - https://phabricator.wikimedia.org/T409728#11364354 (10Raymond_Ndibe) a:03Raymond_Ndibe [18:43:54] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [builds-api] Add an endpoint to get all available images - https://phabricator.wikimedia.org/T409726#11364356 (10Raymond_Ndibe) >>! In T409726#11361659, @Raymond_Ndibe wrote: > `image-config` `configmap` has the below structure currently: > **NOTE:**... [18:44:07] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [builds-api] Add an endpoint to get all available images - https://phabricator.wikimedia.org/T409726#11364357 (10Raymond_Ndibe) a:03Raymond_Ndibe [18:44:53] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [jobs-api,webservice] Fetch images from builds-api - https://phabricator.wikimedia.org/T409725#11364358 (10Raymond_Ndibe) a:03Raymond_Ndibe [19:02:08] 10Tool-paulina: Implement breadcrumb navigation for improved contextual navigation across Paulina pages - https://phabricator.wikimedia.org/T409868 (10Oluwatumininu.m) 03NEW [19:06:44] 10Toolforge (Toolforge iteration 25): [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images - https://phabricator.wikimedia.org/T409191#11364467 (10Raymond_Ndibe) >>! In T409191#11349625, @dcaro wrote: > I did not mean to unassign sorry, I think we both edited at the same ti... [19:30:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.992% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:00:34] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.987% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [20:18:25] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [docs] Update all toolforge repos in gitlab with contribution guidelines and license - https://phabricator.wikimedia.org/T408783#11364640 (10Raymond_Ndibe) 05Open→03In progress [20:33:37] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 10Wiki-Loves-Monuments-Database, 07Sustainability (Incident Followup): [toolsdb] ibdata1 growing on primary - https://phabricator.wikimedia.org/T409716#11364662 (10Multichill) Heritage is used for https://commons.wikimedia.org/wiki/Commons:Monuments_databa... [21:13:55] (03update) 10pepepiton: Make the footer stick to the bottom regardless of the page how short the page content is [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/98 (owner: 10kimbrenekakande) [21:19:16] (03update) 10pepepiton: Make the footer stick to the bottom regardless of the page how short the page content is [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/98 (owner: 10kimbrenekakande) [21:22:34] (03merge) 10pepepiton: Make the footer stick to the bottom regardless of the page how short the page content is [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/98 (owner: 10kimbrenekakande) [23:14:13] (03update) 10oluwatumininu: feat(loading): add consistent loading states for major async actions [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/168 (https://phabricator.wikimedia.org/T409535)