[00:16:22] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [docs] Update all toolforge repos in gitlab with contribution guidelines and license - https://phabricator.wikimedia.org/T408783#11347509 (10Raymond_Ndibe) a:03Raymond_Ndibe [01:04:53] !issync [01:04:54] Syncing #wikimedia-cloud-feed (requested by bd808) [01:04:55] No updates for #wikimedia-cloud-feed [01:06:52] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19121350626 (https://github.com/cluebotng/component-configs/commits/80f69f0ab7b09c2e3e5e208d847a954cb1975bc6) [01:06:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [01:13:42] 10Tool-iw, 13Patch-For-Review: IW ingress config does not work with nginx-ingress-controller:v1.13.3 - https://phabricator.wikimedia.org/T408570#11347624 (10bd808) @taavi should we make another task about all the other things that nginx-ingress-controller:v1.13.3 is going to break? [01:24:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudgw1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [01:29:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudgw1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:06:44] 06cloud-services-team (Kanban), 10Toolforge: Preserve the ability to make interwiki links to Toolforge tools under the host based routing scheme - https://phabricator.wikimedia.org/T247432#11347740 (10Ahecht) With my randomincategory tool (https://randomincategory.toolforge.org/), interwikilinks such as `[... [03:06:11] FIRING: [2x] SystemdUnitDown: The systemd unit hdfs_rsync_cirrus_index_dumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:06:52] 10Tool-wsindex, 10Wikisource Reader App, 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11347806 (10Bodhisattwa) [04:48:41] 10Toolforge (Toolforge iteration 25): [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images - https://phabricator.wikimedia.org/T409191#11347863 (10Raymond_Ndibe) a:03Raymond_Ndibe [04:50:25] 10Toolforge (Toolforge iteration 25): [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images - https://phabricator.wikimedia.org/T409191#11347864 (10Raymond_Ndibe) In all cases where both variants exist, **the `webservice` image is functionally a superset of the `jobs-framewo... [05:00:56] FIRING: SystemdUnitDown: The service unit hdfs_rsync_cirrus_index_dumps.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:00:56] FIRING: [2x] SystemdUnitDown: The systemd unit hdfs_rsync_cirrus_index_dumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:05:56] RESOLVED: [2x] SystemdUnitDown: The service unit hdfs_rsync_cirrus_index_dumps.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:05:56] RESOLVED: [2x] SystemdUnitDown: The systemd unit hdfs_rsync_cirrus_index_dumps.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:06:19] 10Tool-techcontribs: pie charts with overlapping/unreadable numbers - https://phabricator.wikimedia.org/T384557#11347885 (10Bugreporter2) >>! In T384557#11343745, @Chlod wrote: > I've long wanted to use a pie chart but with lines to denote which slice is which, but the library used for the charts do not support... [05:34:26] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [jobs-api] Allow customizing time to request Loki logs for - https://phabricator.wikimedia.org/T400917#11347933 (10Raymond_Ndibe) a:03Raymond_Ndibe [05:43:45] (03update) 10raymond-ndibe: [storage.kubernetes] remove _remove_none_values method [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/145 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) [07:07:27] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [07:50:55] 06cloud-services-team: Uprade cloudservices1005 and cloudservices1006 to MariaDB 10.11 - https://phabricator.wikimedia.org/T409395 (10Marostegui) 03NEW [08:00:30] 06cloud-services-team: Uprade cloudservices1005 and cloudservices1006 to MariaDB 10.11 - https://phabricator.wikimedia.org/T409395#11348215 (10fnegri) Thanks @Marostegui. @taavi had already noticed this and created {T402005}, which is assigned to me. I'll keep that one as it mentions both the db version and... [08:00:34] 06cloud-services-team: Uprade cloudservices1005 and cloudservices1006 to MariaDB 10.11 - https://phabricator.wikimedia.org/T409395#11348218 (10fnegri) →14Duplicate dup:03T402005 [08:00:37] 06cloud-services-team (FY2025/26-Q1), 10Cloud-VPS (Debian Bullseye Deprecation): Upgrade cloudinfra database hosts off of Bullseye - https://phabricator.wikimedia.org/T402005#11348220 (10fnegri) [08:01:10] 06cloud-services-team (FY2025/26-Q1), 10Cloud-VPS (Debian Bullseye Deprecation): Upgrade cloudinfra database hosts off of Bullseye - https://phabricator.wikimedia.org/T402005#11348222 (10fnegri) [08:03:01] 06cloud-services-team: Uprade cloudservices1005 and cloudservices1006 to MariaDB 10.11 - https://phabricator.wikimedia.org/T409395#11348237 (10fnegri) 05Duplicate→03Open Not true, that is a different server in cloud-vps. :/ [08:03:47] 06cloud-services-team (FY2025/26-Q1): Uprade cloudservices1005 and cloudservices1006 to MariaDB 10.11 - https://phabricator.wikimedia.org/T409395#11348245 (10fnegri) [08:03:48] 10Toolforge (Toolforge iteration 25): [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images - https://phabricator.wikimedia.org/T409191#11348242 (10dcaro) a:05Raymond_Ndibe→03None [08:05:09] 06cloud-services-team (FY2025/26-Q1), 10Cloud-VPS (Debian Bullseye Deprecation): Upgrade cloudinfra database hosts off of Bullseye - https://phabricator.wikimedia.org/T402005#11348248 (10fnegri) [08:06:17] 06cloud-services-team (FY2025/26-Q1), 10Cloud-VPS (Debian Bullseye Deprecation): Upgrade cloudinfra database hosts off of Bullseye - https://phabricator.wikimedia.org/T402005#11348257 (10fnegri) I thought this was the same db as {T409395} but I was wrong. I added the hostnames to the description. Sorry for the... [08:06:28] 06cloud-services-team (FY2025/26-Q1): Uprade cloudservices1005 and cloudservices1006 to MariaDB 10.11 - https://phabricator.wikimedia.org/T409395#11348259 (10fnegri) a:03fnegri [08:06:35] 06cloud-services-team (FY2025/26-Q1): Uprade cloudservices1005 and cloudservices1006 to MariaDB 10.11 - https://phabricator.wikimedia.org/T409395#11348260 (10fnegri) p:05Triage→03Medium [09:37:07] 06cloud-services-team, 10Toolforge, 07Sustainability (Incident Followup): Add filesystem space alerts for tools(db) - https://phabricator.wikimedia.org/T409404 (10fgiunchedi) 03NEW [09:39:46] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge.infra] run and monitor our own sample tools - https://phabricator.wikimedia.org/T357977#11348633 (10fgiunchedi) Mentioning it here as a followup to {T409244}: it is important we do monitor the ability to read/write toolsdb, and possibly page... [09:42:59] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11348662 (10fgiunchedi) [09:53:24] 10Toolforge, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11348677 (10fgiunchedi) Thank you @cmooney for the summary, I'll add a few thoughts I had while working on the Toolforge on Metal p... [10:05:34] (03PS1) 10Btullis: Add a summy secret for growthbook OIDC [labs/private] - 10https://gerrit.wikimedia.org/r/1202662 (https://phabricator.wikimedia.org/T409183) [10:05:51] (03CR) 10Btullis: [V:03+2 C:03+2] Add a summy secret for growthbook OIDC [labs/private] - 10https://gerrit.wikimedia.org/r/1202662 (https://phabricator.wikimedia.org/T409183) (owner: 10Btullis) [10:08:02] 06cloud-services-team (FY2025/26-Q1), 10Cloud-VPS: Uprade cloudservices1005 and cloudservices1006 to MariaDB 10.11 - https://phabricator.wikimedia.org/T409395#11348751 (10taavi) [10:08:58] 10Tool-wsindex, 10Wikisource Reader App: Add Arabic language to Wikisource reader app - https://phabricator.wikimedia.org/T409407#11348753 (10Bodhisattwa) 05Open→03Resolved a:03Bodhisattwa [10:21:47] (03open) 10fnegri: Add new data volume for tools-db-7 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/100 (https://phabricator.wikimedia.org/T409287) [10:21:48] (03update) 10fnegri: Add new data volume for tools-db-7 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/100 (https://phabricator.wikimedia.org/T409287) [10:23:10] (03update) 10fnegri: Add new data volume for tools-db-7 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/100 (https://phabricator.wikimedia.org/T409287) [10:33:06] 10Cloud-VPS (Quota-requests): Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365#11348841 (10fnegri) +1 [10:38:09] 10Cloud-VPS (Quota-requests): Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365#11348853 (10Volans) p:05Triage→03Medium [11:00:13] (03approved) 10volans: Add new data volume for tools-db-7 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/100 (https://phabricator.wikimedia.org/T409287) (owner: 10fnegri) [11:48:13] (03open) 10volans: flavors: add zuul to g4.cores8.ram32.disk20.4xiops [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/279 (https://phabricator.wikimedia.org/T409365) [11:49:23] 10Cloud-VPS (Quota-requests), 13Patch-For-Review: Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365#11349180 (10Volans) If I'm not mistaken the above patch should gran the usage of the requested instance type. I'm not sure if there is... [12:03:30] (03update) 10volans: flavors: add zuul to g4.cores8.ram32.disk20.4xiops [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/279 (https://phabricator.wikimedia.org/T409365) [12:04:44] (03update) 10volans: flavors: add zuul to g4.cores8.ram32.disk20.4xiops [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/279 (https://phabricator.wikimedia.org/T409365) [12:40:33] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1202693 (owner: 10L10n-bot) [13:07:09] (03open) 10system625: Add death year filter for author search [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/167 (https://phabricator.wikimedia.org/T409205) [13:16:11] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/31 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:16:24] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/149 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:16:46] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/35 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:16:51] (03update) 10system625: Resolve T397963 "Restore scholary works" [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/165 [13:21:59] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/58 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:22:12] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/78 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:23:11] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/28 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:23:28] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/40 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:23:36] (03update) 10damian: update_job - handle immutable fields [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/239 (https://phabricator.wikimedia.org/T409007) [13:23:40] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/149 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:23:54] (03approved) 10dcaro: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/99 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:24:10] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/67 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:24:26] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/foxtrot-ldap] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/merge_requests/12 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:25:08] (03approved) 10dcaro: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/21 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:25:15] (03approved) 10dcaro: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/45 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:25:20] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/16 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:25:32] (03approved) 10dcaro: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/9 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:26:12] (03update) 10damian: update_job - handle immutable fields [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/239 (https://phabricator.wikimedia.org/T409007) [13:26:17] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/24 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:26:33] (03approved) 10dcaro: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/12 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:26:43] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-gen-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-gen-cli/-/merge_requests/4 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:26:58] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/87 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:27:07] (03approved) 10dcaro: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/packaging-helmfile] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/packaging-helmfile/-/merge_requests/3 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:27:51] (03update) 10damian: images - support fully qualified harbor names [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/231 (https://phabricator.wikimedia.org/T408574) [13:43:59] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1059 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [13:45:51] (03update) 10damian: images - support fully qualified harbor names [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/231 (https://phabricator.wikimedia.org/T408574) [13:51:23] 10Cloud-VPS (Quota-requests), 13Patch-For-Review: Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365#11349616 (10fnegri) > not sure if there is a separate setting for the fast-iops bit My understanding is that volume types are only re... [13:54:08] 10Toolforge (Toolforge iteration 25): [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images - https://phabricator.wikimedia.org/T409191#11349625 (10dcaro) a:03Raymond_Ndibe I did not mean to unassign sorry, I think we both edited at the same time. Can you manually test th... [13:58:26] 10Toolforge (Toolforge iteration 25): [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images - https://phabricator.wikimedia.org/T409191#11349652 (10dcaro) Also, what does you mean with `doesn't exist in the toollabs-images repo, but setup is likely like the other node image`... [14:00:18] (03approved) 10dcaro: update_job - handle immutable fields [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/239 (https://phabricator.wikimedia.org/T409007) (owner: 10damian) [14:00:25] (03merge) 10dcaro: update_job - handle immutable fields [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/239 (https://phabricator.wikimedia.org/T409007) (owner: 10damian) [14:00:42] (03approved) 10dcaro: [dev] fix/add README.md, CONTRIBUTING.md [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/243 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [14:01:13] (03update) 10dcaro: k8s deployment object - reduce labels in selector [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/240 (https://phabricator.wikimedia.org/T409007) (owner: 10damian) [14:02:50] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.440-20251106140042-fb78d00c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1060 (https://phabricator.wikimedia.org/T409007) [14:21:36] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:27:38] (03approved) 10dcaro: k8s deployment object - reduce labels in selector [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/240 (https://phabricator.wikimedia.org/T409007) (owner: 10damian) [14:28:11] (03update) 10damian: images - support fully qualified harbor names [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/231 (https://phabricator.wikimedia.org/T408574) [14:33:10] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:33:46] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:43:21] 10Toolforge (Toolforge iteration 25): [elastic] add metrics - https://phabricator.wikimedia.org/T409047#11349925 (10dcaro) 05Open→03Resolved [14:46:38] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:48:49] 06cloud-services-team, 10Toolforge: [functional tests] leave a mess behind - https://phabricator.wikimedia.org/T409009#11349960 (10dcaro) p:05Triage→03Medium This was less of an issue is lima-kilo, where you rebuild it every now and then, but when running on a loop in prod becomes more relevant. [14:49:34] 06cloud-services-team, 10Toolforge: [functional tests] leave a mess behind - https://phabricator.wikimedia.org/T409009#11349964 (10dcaro) The cleanup of what's in the home dir can happen at the start of the tests, so in case anything fails, you still have some leftover to investigate. [14:49:46] (03approved) 10andrew: Replace only views that need updating [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/9 (https://phabricator.wikimedia.org/T351637) (owner: 10fnegri) [14:52:13] 10Toolforge (Toolforge iteration 25): [prometheus,infra] 2025-09-10 tools-prometheus-9 down - https://phabricator.wikimedia.org/T404199#11349971 (10dcaro) 05Open→03Resolved a:03dcaro It seems the memory limit has completely stopped the full outages, I'll close this as the main issue is "workedaround".... [14:52:19] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:52:19] (03update) 10damian: images - support fully qualified harbor names [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/231 (https://phabricator.wikimedia.org/T408574) [14:52:27] (03approved) 10dcaro: [storage.kubernetes] remove _remove_none_values method [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/145 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) (owner: 10raymond-ndibe) [15:01:37] (03approved) 10dcaro: images - support fully qualified harbor names [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/231 (https://phabricator.wikimedia.org/T408574) (owner: 10damian) [15:03:13] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:03:17] 06cloud-services-team, 10Toolforge: [functional tests] leave a mess behind - https://phabricator.wikimedia.org/T409009#11350004 (10DamianZaremba) > This was less of an issue is lima-kilo, where you rebuild it every now and then, but when running on a loop in prod becomes more relevant. I've had lima-kilo run... [15:03:31] (03approved) 10dcaro: jobs-api: bump to 0.0.440-20251106140042-fb78d00c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1060 (https://phabricator.wikimedia.org/T409007) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:03:35] (03merge) 10dcaro: jobs-api: bump to 0.0.440-20251106140042-fb78d00c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1060 (https://phabricator.wikimedia.org/T409007) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:04:10] (03merge) 10dcaro: k8s deployment object - reduce labels in selector [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/240 (https://phabricator.wikimedia.org/T409007) (owner: 10damian) [15:06:11] (03update) 10dcaro: ci: move to trixie [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/238 [15:06:25] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.442-20251106150420-1926a76c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1061 (https://phabricator.wikimedia.org/T409007) [15:06:45] (03update) 10damian: images - support fully qualified harbor names [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/231 (https://phabricator.wikimedia.org/T408574) [15:08:04] (03merge) 10dcaro: ci: move to trixie [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/238 [15:09:08] (03update) 10damian: images - support fully qualified harbor names [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/231 (https://phabricator.wikimedia.org/T408574) [15:09:41] 10Tool-paulina, 13Patch-For-Review: search authors for a specific death date year - https://phabricator.wikimedia.org/T409205#11350031 (10Pepe_piton) @Mind_Booster_Noori Thank you for your suggestion about this important feature, and @Oluwatumininu.m @System625 thank you for your contributions! Date of birth... [15:10:49] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.443-20251106150823-c3151d3f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1061 (https://phabricator.wikimedia.org/T409007 https://phabricator.wikimedia.org/T409058) [15:10:52] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.443-20251106150823-c3151d3f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1061 (https://phabricator.wikimedia.org/T409007 https://phabricator.wikimedia.org/T409058) [15:16:47] (03merge) 10dcaro: images - support fully qualified harbor names [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/231 (https://phabricator.wikimedia.org/T408574) (owner: 10damian) [15:18:04] 10Tool-paulina, 13Patch-For-Review: search authors for a specific death date year - https://phabricator.wikimedia.org/T409205#11350086 (10Oluwatumininu.m) >>! In T409205#11350029, @Pepe_piton wrote: > @Mind_Booster_Noori Thank you for your suggestion about this important feature, and @Oluwatumininu.m @System6... [15:19:01] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.449-20251106151655-61a26f8e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1061 (https://phabricator.wikimedia.org/T408574 https://phabricator.wikimedia.org/T409007 https://phabricator.wikimedia.org/T409058) [15:19:02] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.449-20251106151655-61a26f8e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1061 (https://phabricator.wikimedia.org/T408574 https://phabricator.wikimedia.org/T409007 https://phabricator.wikimedia.org/T409058) [15:21:20] (03update) 10damian: Apply nodeAffinity when mount=none [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38 (https://phabricator.wikimedia.org/T408707) [15:22:14] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:27:21] 06cloud-services-team, 10Data-Services, 06Data-Persistence: Set up replication on new hosts clouddb102[2-5] - https://phabricator.wikimedia.org/T408692#11350135 (10Marostegui) @fnegri do you want me to do this? [15:34:14] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:36:03] 06cloud-services-team, 10Toolforge, 07Sustainability (Incident Followup): Add filesystem space alerts for tools(db) - https://phabricator.wikimedia.org/T409404#11350198 (10dcaro) That sounds good to me yes, it's the similar to the other space. The predictive can be as silly as just 'when it hits 20%', if it... [15:45:55] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:57:50] 10Cloud-VPS (Quota-requests), 13Patch-For-Review: Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365#11350306 (10Andrew) >>! In T409365#11349615, @fnegri wrote: >> not sure if there is a separate setting for the fast-iops bit > > My u... [16:00:37] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [16:23:59] (03approved) 10dcaro: jobs-api: bump to 0.0.449-20251106151655-61a26f8e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1061 (https://phabricator.wikimedia.org/T408574 https://phabricator.wikimedia.org/T409007 https://phabricator.wikimedia.org/T409058) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:24:05] (03merge) 10dcaro: jobs-api: bump to 0.0.449-20251106151655-61a26f8e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1061 (https://phabricator.wikimedia.org/T408574 https://phabricator.wikimedia.org/T409007 https://phabricator.wikimedia.org/T409058) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:05:01] (03update) 10oluwatumininu: feat(search): add death-year filter for authors in advanced search [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/166 [17:23:41] (03open) 10damian: Move image resolution to core [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/244 [17:27:13] (03update) 10damian: Apply nodeAffinity when mount=none [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38 (https://phabricator.wikimedia.org/T408707) [17:27:44] (03update) 10damian: Draft: Move image resolution to core [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/244 [17:31:48] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#11350739 (10dcaro) [18:18:20] 10Toolforge, 06tools-infrastructure-team: Reduce tool breakage over new ingress-nginx annotation validation rules - https://phabricator.wikimedia.org/T409474 (10taavi) 03NEW [18:19:06] 10Tool-iw, 13Patch-For-Review: IW ingress config does not work with nginx-ingress-controller:v1.13.3 - https://phabricator.wikimedia.org/T408570#11350973 (10taavi) >>! In T408570#11347624, @bd808 wrote: > @taavi should we make another task about all the other things that nginx-ingress-controller:v1.13.3 is... [18:33:51] 06cloud-services-team (FY2025/26-Q1), 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE (2025.10.17 - 2025.11.07): Decide how to use the new clouddb hosts (clouddb102[2-5]) - https://phabricator.wikimedia.org/T401295#11351073 (10fnegri) Thanks @Marostegui for the additional considerations. Your... [18:38:19] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Fix MTU on single-NIC Ceph nodes - https://phabricator.wikimedia.org/T409294#11351077 (10cmooney) > Specifically: the cluster interface in the single nic case will have 8996 MTU (i.e. 9000 - 4 bytes for the vlan tag). Public interface stays at 9000 in s... [18:38:43] 06cloud-services-team, 10Data-Services, 06Data-Persistence: Set up replication on new hosts clouddb102[2-5] - https://phabricator.wikimedia.org/T408692#11351084 (10fnegri) @Marostegui no rush at all, but when you have some time you can start. My understanding is that the initial setup of the dbs is entirely... [18:43:47] (03merge) 10fnegri: Add new data volume for tools-db-7 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/100 (https://phabricator.wikimedia.org/T409287) [18:48:02] (03open) 10damian: runtime - move image from url logic to class [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/245 [18:48:58] (03open) 10damian: runtime - move image from url logic to class [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/246 [18:49:09] (03update) 10damian: runtime - move image from url logic to class [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/245 [18:52:44] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 13Patch-For-Review, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11351134 (10fnegri) Snapshot created from `tools-db-6` with the position below: ` MariaDB [(none)]> SELE... [18:52:50] 06cloud-services-team, 10Cloud-VPS: sso failure in codfw1dev (labtesthorizon.wikimedia.org) - https://phabricator.wikimedia.org/T409328#11351135 (10Andrew) a:03MoritzMuehlenhoff @MoritzMuehlenhoff has offered to take a look at this. [18:57:23] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 13Patch-For-Review, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11351148 (10fnegri) rsync in progress, this will take a few hours. (the full procedure is at https://wik... [19:11:58] 06cloud-services-team (Kanban), 10Toolforge: Preserve the ability to make interwiki links to Toolforge tools under the host based routing scheme - https://phabricator.wikimedia.org/T247432#11351169 (10bd808) >>! In T247432#11347740, @Ahecht wrote: > With my randomincategory tool (https://randomincategory.t... [19:18:31] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure on R450 hardware - https://phabricator.wikimedia.org/T407586#11351189 (10Andrew) For future debug research: We can prevent the final reboot after a reimage like this: - disable puppet on apt1002 - comment out the two reboot_in_p... [20:09:22] 06cloud-services-team, 10Tool-iw, 10Toolforge: iw.toolforge.org does not support URL-encoded query parameters ([[toolforge:foo?bar]]) - https://phabricator.wikimedia.org/T345783#11351385 (10Novem_Linguae) [20:54:28] (03PS1) 10Jean-Frédéric: Extract JSON payload to Toolforge API as its own variable [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202810 [20:54:28] (03PS1) 10Jean-Frédéric: Extract method to create a Toolforge job over the API [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202811 [20:54:28] (03PS1) 10Jean-Frédéric: Fix default variables sourcing in deploy-to-toolforge Shell script [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202812 [20:54:28] (03PS1) 10Jean-Frédéric: Extract Toolforge Docker images to use as default variables [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1202813 [21:10:36] 06cloud-services-team, 10Toolforge, 10MediaWiki-Interwiki, 10Wikimedia-Interwiki-links: Wikilinks to Toolforge now escape punctuation when they previously did not, breaking hundreds of thousands of links - https://phabricator.wikimedia.org/T409493 (10Tamzin) 03NEW [21:15:21] 06cloud-services-team, 10Toolforge, 10MediaWiki-Interwiki, 10Wikimedia-Interwiki-links: Wikilinks to Toolforge now escape punctuation when they previously did not, breaking hundreds of thousands of links - https://phabricator.wikimedia.org/T409493#11351618 (10Tamzin) [21:15:38] 06cloud-services-team, 10Tool-iw, 10Toolforge: Wikilinks to Toolforge now escape punctuation when they previously did not, breaking hundreds of thousands of links - https://phabricator.wikimedia.org/T409493#11351620 (10bd808) Interwiki links including query string data were never supposed to work: {T345783}... [21:16:49] 06cloud-services-team (Kanban), 10Toolforge: Preserve the ability to make interwiki links to Toolforge tools under the host based routing scheme - https://phabricator.wikimedia.org/T247432#11351627 (10Ahecht) @bd808 That was potentially a bad example, as the php script is doing some custom parsing of the U... [21:21:16] 06cloud-services-team, 10Tool-iw, 10Toolforge: iw.toolforge.org does not support URL-encoded query parameters ([[toolforge:foo?bar]]) - https://phabricator.wikimedia.org/T345783#11351653 (10Novem_Linguae) Is this a common enough use case? Maybe we should just not allow wikilinks to contain URL query paramete... [21:23:42] 06cloud-services-team, 10Tool-iw, 10Toolforge: Toolforge interwikli link handling no longer strips URL-encoding before redirecting when it previously did, breaking existing on-wiki links - https://phabricator.wikimedia.org/T409493#11351680 (10bd808) [21:25:52] 06cloud-services-team, 10Tool-iw, 10Toolforge: Toolforge interwikli link handling no longer strips URL-encoding before redirecting when it previously did, breaking existing on-wiki links - https://phabricator.wikimedia.org/T409493#11351681 (10bd808) [21:25:58] 10Tool-iw, 13Patch-For-Review: IW ingress config does not work with nginx-ingress-controller:v1.13.3 - https://phabricator.wikimedia.org/T408570#11351682 (10bd808) [21:30:39] 06cloud-services-team, 10Tool-iw, 10Toolforge: iw.toolforge.org does not support URL-encoded query parameters ([[toolforge:foo?bar]]) - https://phabricator.wikimedia.org/T345783#11351686 (10bd808) >>! In T345783#11351653, @Novem_Linguae wrote: > Is this a common enough use case? Maybe we should just not allo... [21:38:36] 06cloud-services-team, 10Tool-iw, 10Toolforge: Toolforge interwikli link handling no longer strips URL-encoding before redirecting when it previously did, breaking existing on-wiki links - https://phabricator.wikimedia.org/T409493#11351695 (10bd808) The lack of query string support in MediaWiki's internal li... [22:11:25] 06cloud-services-team (Kanban), 10Toolforge: Preserve the ability to make interwiki links to Toolforge tools under the host based routing scheme - https://phabricator.wikimedia.org/T247432#11351795 (10bd808) @Ahecht This discussion about a perceived regress should move to {T409493} where we will be able to... [22:13:17] 06cloud-services-team, 10Tool-iw, 10Toolforge: Toolforge interwikli link handling no longer strips URL-encoding before redirecting when it previously did, breaking existing on-wiki links - https://phabricator.wikimedia.org/T409493#11351803 (10bd808) Moving some discussion from {T247432} >>! In T247432#11347... [22:20:16] 06cloud-services-team, 10Tool-iw, 10Toolforge: Toolforge interwikli link handling no longer strips URL-encoding before redirecting when it previously did, breaking existing on-wiki links - https://phabricator.wikimedia.org/T409493#11351825 (10bd808) >>! In T247432#11351627, @Ahecht wrote: > @bd808 That was p... [23:24:56] 06cloud-services-team, 10Toolforge, 10Tools: Geohack tool frequently triggers the Toolforge front proxy's per-tool rate limit due to too much traffic - https://phabricator.wikimedia.org/T409185#11352006 (10bd808) [23:57:58] 06cloud-services-team, 10Toolforge, 10Tools: Geohack tool frequently triggers the Toolforge front proxy's per-tool rate limit due to too much traffic - https://phabricator.wikimedia.org/T409185#11352085 (10bd808) 05Invalid→03Open I am reopening this because it actually is a legitimate topic of discussion... [23:59:17] 06cloud-services-team, 10Toolforge, 10Tools: Geohack tool frequently triggers the Toolforge front proxy's per-tool rate limit due to too much traffic - https://phabricator.wikimedia.org/T409185#11352087 (10bd808) I would very much expect that a healthy portion of the traffic to geohack is unwanted bot reques...