[00:08:54] (03update) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [00:09:00] (03approved) 10bd808: gitlab: move to bookworm images [toolforge-repos/toolviews] - 10https://gitlab.wikimedia.org/toolforge-repos/toolviews/-/merge_requests/12 (owner: 10dcaro) [00:20:45] (03update) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [00:26:49] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests, 13Patch-For-Review: Enable protection indicators for wikitech - https://phabricator.wikimedia.org/T409785#11394629 (10bd808) 05Open→03Resolved a:03bd808 {F70307707, size=full} [00:43:09] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests: Enable sitenotice on mobile for Wikitech - https://phabricator.wikimedia.org/T410702 (10bd808) 03NEW [00:46:38] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests: Enable sitenotice on mobile for Wikitech - https://phabricator.wikimedia.org/T410702#11394666 (10bd808) Announced in the `#wikimedia-cloud` and `#wikimedia-operations` irc channels: `lang=irc [00:45] < bd808> I would like to have sitenotice enabled for m... [02:48:28] 10Tool-techcontribs: in GitHub section, include organizations that the user is a member of - https://phabricator.wikimedia.org/T393623#11394758 (10Chlod) 05Open→03In progress a:03Chlod [04:01:11] 10Tool-techcontribs, 07Upstream: hide mediawiki/wmf sub-groups from the gerrit groups section - https://phabricator.wikimedia.org/T393534#11394768 (10Chlod) Noting here that it's possible to build an inheritance tree with the Gerrit REST API [/groups/{group}?o=INCLUDES](https://gerrit.wikimedia.org/r/Documenta... [04:08:49] 10Tool-techcontribs, 07Upstream: hide mediawiki/wmf sub-groups from the gerrit groups section - https://phabricator.wikimedia.org/T393534#11394769 (10Chlod) 05Open→03In progress a:03Chlod On second thought, either of these would require one request per group anyway. 🤦 Did not think of the request logic t... [04:49:54] 10Tool-techcontribs, 07Upstream: hide mediawiki/wmf sub-groups from the gerrit groups section - https://phabricator.wikimedia.org/T393534#11394779 (10Chlod) 05In progress→03Stalled Putting this on stalled because this can't continue without a Gerrit account with authorization. The `/groups/{group}/group` e... [05:07:15] 10Tool-techcontribs: Add reporting on gitlab.wikimedia.org activity - https://phabricator.wikimedia.org/T383935#11394785 (10Chlod) 05Open→03In progress a:03Chlod [06:40:17] 10Tool-techcontribs: in GitHub section, include organizations that the user is a member of - https://phabricator.wikimedia.org/T393623#11394816 (10Chlod) 05In progress→03Resolved [08:25:37] 06cloud-services-team, 06SRE, 07Upstream: latest Trixie image (as of 2025-10-16) grub failure on R450 hardware - https://phabricator.wikimedia.org/T407586#11394895 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi I'm calling this one done since we have a workaround in place; I'll followup on the Deb... [09:07:50] 06cloud-services-team, 10Data-Services: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11395012 (10ayounsi) Thanks, then as it's only a few, option 1 seems best to me. Much less complex to setup and maintain. Setup the new FQDN, ask people to migrate, give the... [09:15:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.992% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:19:29] 06cloud-services-team, 10PAWS: [bug] - https://phabricator.wikimedia.org/T410712 (10Lollovand) 03NEW [09:20:24] 06cloud-services-team, 10PAWS: [bug] - https://phabricator.wikimedia.org/T410712#11395037 (10Lollovand) [09:21:05] 06cloud-services-team, 10PAWS: [bug] - https://phabricator.wikimedia.org/T410712#11395038 (10Lollovand) [10:02:59] (03update) 10tarrow: Added campwiz [toolforge-repos/wmde-github-phab-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-github-phab-bot/-/merge_requests/1 (owner: 10nokibsarkar) [11:46:50] (03update) 10fnegri: Stop managing all tools-db volumes with tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/102 [11:47:10] (03merge) 10fnegri: Stop managing all tools-db volumes with tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/102 [11:56:10] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [bastion] redis-cli is absent from tools bastion hosts - https://phabricator.wikimedia.org/T410102#11395484 (10taavi) 05Open→03Resolved a:03bd808 [11:58:14] 06cloud-services-team, 10Toolforge: [tofu-provisioning] Allow running manual "tofu state" commands - https://phabricator.wikimedia.org/T410720 (10fnegri) 03NEW [11:59:31] 06cloud-services-team, 10Toolforge: [tofu-provisioning] Allow running manual "tofu state" commands - https://phabricator.wikimedia.org/T410720#11395502 (10fnegri) [12:06:16] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11395512 (10fnegri) [12:06:54] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25), 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11395513 (10fnegri) 05In progress→03Resolved All done. [12:13:04] 10Toolforge, 06tools-infrastructure-team: Stand up an etcd cluster for testing requestctl in toolsbeta - https://phabricator.wikimedia.org/T410721 (10taavi) 03NEW [13:15:49] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.246% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:55:34] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.933% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:11:41] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11395975 (10fnegri) [14:23:45] (03PS1) 10Andrew Bogott: idp_clouddev: duplicate private settings from cloudweb2002-dev [labs/private] - 10https://gerrit.wikimedia.org/r/1208359 (https://phabricator.wikimedia.org/T410294) [14:24:51] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] idp_clouddev: duplicate private settings from cloudweb2002-dev [labs/private] - 10https://gerrit.wikimedia.org/r/1208359 (https://phabricator.wikimedia.org/T410294) (owner: 10Andrew Bogott) [15:05:35] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11396205 (10fnegri) > For this, can you please confirm once alert gets triggered, and visible at (https://prometheus-alerts.wmcloud.org/?q=... [15:13:24] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11396280 (10fnegri) 05Open→03In progress [15:13:35] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11396281 (10fnegri) 05In progress→03Stalled [15:13:40] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11396285 (10fnegri) 05Stalled→03In progress [15:18:08] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11396314 (10Reputation22) >>! In T409668#11396205, @fnegri wrote: >> For this, can you please confirm once alert gets triggered, and visibl... [15:39:10] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11396405 (10Reputation22) For beta-vct 1. CPU usage ` 100 * (sum by (mode) (irate(node_cpu_seconds_total{project="videocuttool",instance="... [15:42:04] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11396420 (10Reputation22) Also do we have different severity defined in alert-manager? (something like SEV-0, SEV-1, SEV-2 .etc) [16:09:24] 06cloud-services-team, 10Huggle: huggle-nfs volume filling up - https://phabricator.wikimedia.org/T402806#11396523 (10Petrb) Hello, I cleaned it up ` # du -ksh /mnt/nfs/labstore-secondary-home /mnt/nfs/labstore-secondary-project 350M /mnt/nfs/labstore-secondary-home 4.0K /mnt/nfs/labstore-secondary-pro... [16:09:32] 06cloud-services-team, 10Huggle: huggle-nfs volume filling up - https://phabricator.wikimedia.org/T402806#11396524 (10Petrb) 05Open→03Resolved a:03Petrb [16:12:57] 10Tools, 05PES1.3.3 WP25 Easter Eggs, 07Software-Licensing: wikipedia25-years-of-wikipedia tool loads and uses non-free JavaScript - https://phabricator.wikimedia.org/T410465#11396530 (10ATitkov) Thanks for the heads up, I will have a look at it as soon as possible [16:47:28] (03PS1) 10Urbanecm: Fix deployment script [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1208404 [16:48:03] (03CR) 10CI reject: [V:04-1] Fix deployment script [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1208404 (owner: 10Urbanecm) [16:54:43] (03PS2) 10Urbanecm: Fix deployment script [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1208404 [16:55:02] (03CR) 10CI reject: [V:04-1] Fix deployment script [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1208404 (owner: 10Urbanecm) [16:56:05] (03PS1) 10Urbanecm: [tox] Remove comments from tox.ini, where they are no longer expected [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1208405 [16:56:36] (03PS3) 10Urbanecm: Fix deployment script [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1208404 [17:10:01] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge, 07Sustainability (Incident Followup): Add filesystem space alerts for tools(db) - https://phabricator.wikimedia.org/T409404#11396689 (10fnegri) [17:10:05] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge, 07Sustainability (Incident Followup): Add filesystem space alerts for tools(db) - https://phabricator.wikimedia.org/T409404#11396690 (10fnegri) a:03fnegri [17:24:19] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE, and 2 others: Site: codfw 1 VM request for codfw1dev CAS test/dev, hostname: cloudidp2001-dev - https://phabricator.wikimedia.org/T410294#11396737 (10Andrew) (I'm moving this to a private address; lots of cookbook things to come) [17:38:08] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE, and 2 others: Site: codfw 1 VM request for codfw1dev CAS test/dev, hostname: cloudidp2001-dev - https://phabricator.wikimedia.org/T410294#11396759 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cum... [17:41:15] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge, 10Wiki-Loves-Monuments-Database, 07Sustainability (Incident Followup): [toolsdb] ibdata1 growing on primary - https://phabricator.wikimedia.org/T409716#11396763 (10fnegri) `ibdata1` is slowly growing again, it went form 115G to 134G. > @Usernamekiran I... [17:46:46] 06cloud-services-team, 10Cloud-VPS (Project-requests): CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11396772 (10Odinaldo) Dear @dcaro I think we managed to create and link the developer accounts to the accounts 1. and 2. above. I hope we followed the right procedure. Please let me... [17:50:23] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge, 10Wiki-Loves-Monuments-Database, 07Sustainability (Incident Followup): [toolsdb] ibdata1 growing on primary - https://phabricator.wikimedia.org/T409716#11396788 (10fnegri) An interesting video on the internals of InnoDB and undo logs: https://www.youtub... [17:53:51] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE, and 2 others: Site: codfw 1 VM request for codfw1dev CAS test/dev, hostname: cloudidp2001-dev - https://phabricator.wikimedia.org/T410294#11396797 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin20... [18:12:39] 06cloud-services-team, 10Toolforge: [toolsdb] Automatically terminate long transactions - https://phabricator.wikimedia.org/T409857#11396848 (10fnegri) 05Open→03In progress I tried setting `SET GLOBAL idle_transaction_timeout=300;` in the primary, to see if it helps with {T409716}, but I'm not too confiden... [18:14:55] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge, 07Sustainability (Incident Followup): Add filesystem space alerts for tools(db) - https://phabricator.wikimedia.org/T409404#11396855 (10fnegri) [18:14:58] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge, 10Wiki-Loves-Monuments-Database, 07Sustainability (Incident Followup): [toolsdb] ibdata1 growing on primary - https://phabricator.wikimedia.org/T409716#11396856 (10fnegri) [18:33:52] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge, 10Wiki-Loves-Monuments-Database, 07Sustainability (Incident Followup): [toolsdb] ibdata1 growing on primary - https://phabricator.wikimedia.org/T409716#11396932 (10fnegri) I pinged @Usernamekiran on [github](https://github.com/usernamekiran/KiranBOT/iss... [18:36:34] 06cloud-services-team, 10Toolforge: [toolsdb] Automatically terminate long transactions - https://phabricator.wikimedia.org/T409857#11396940 (10Volans) FYI in production there are already mechanism to automatically kill queries, so we might just be able to reuse them. [18:37:16] (03merge) 10bd808: gitlab: move to bookworm images [toolforge-repos/toolviews] - 10https://gitlab.wikimedia.org/toolforge-repos/toolviews/-/merge_requests/12 (owner: 10dcaro) [18:38:49] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11396949 (10fnegri) Thanks, I will create those next week! > will the alert comes when any one of the disk goes below the threshold Yes.... [18:47:35] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11396979 (10Reputation22) >>! In T409668#11396949, @fnegri wrote: > Thanks, I will create those next week! > >> will the alert comes when... [18:52:48] 06cloud-services-team, 10Toolforge: [toolsdb] Automatically terminate long transactions - https://phabricator.wikimedia.org/T409857#11396995 (10fnegri) @Volans I checked if `wmf-pt-kill` (used in clouddbs) did support killing sessions based on transaction time, but IIUC it only considers query time or idle tim... [20:18:30] 06cloud-services-team: cloudcephosd bios upgrades - https://phabricator.wikimedia.org/T410784 (10Andrew) 03NEW [20:21:27] 06cloud-services-team, 10Cloud-VPS: cloudcephosd bios upgrades - https://phabricator.wikimedia.org/T410784#11397327 (10taavi) [20:45:43] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE, and 2 others: Site: codfw 1 VM request for codfw1dev CAS test/dev, hostname: cloudidp2001-dev - https://phabricator.wikimedia.org/T410294#11397499 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cum... [20:49:04] 10VPS-project-Codesearch, 06collaboration-services: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11397507 (10Ladsgroup) So I don't have a way to point to the file. Since except the .git file, nothing ex... [20:51:05] (03PS1) 10Ladsgroup: [WIP] Switch off netbox-exports from diffusion [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1208445 (https://phabricator.wikimedia.org/T410624) [20:53:24] 10VPS-project-Codesearch, 06collaboration-services, 13Patch-For-Review: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11397521 (10Ladsgroup) (honestly it wouldn't be the end of the world if we just swi... [21:03:47] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE, and 2 others: Site: codfw 1 VM request for codfw1dev CAS test/dev, hostname: cloudidp2001-dev - https://phabricator.wikimedia.org/T410294#11397534 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin20... [21:14:38] (03update) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [21:21:22] (03update) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [21:27:02] (03CR) 10Andrew Bogott: [C:03+2] openstack: ensure_canary: Use Trixie for new instances [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1207176 (owner: 10Majavah) [21:31:00] (03Merged) 10jenkins-bot: openstack: ensure_canary: Use Trixie for new instances [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1207176 (owner: 10Majavah) [21:53:30] 06cloud-services-team, 10Toolforge: [tofu-provisioning] Allow running manual "tofu state" commands - https://phabricator.wikimedia.org/T410720#11397638 (10bd808) I was thinking about some of these gitops-related edge use cases as I built https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning. https:/... [22:03:39] (03update) 10don-vip: Draft: Update to Spring Boot 4.0.0 [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/14 [22:37:08] 10Tool-gitlab-bot, 10Wikibugs, 10GitLab (Integrations): Automate setup of comment, pipeline, and job webhooks for all GitLab projects - https://phabricator.wikimedia.org/T362940#11397755 (10bd808) >>! In T362940#9784524, @bd808 wrote: > It is on a timer, but that timer is on @thcipriani's laptop today. Let's... [23:17:22] (03open) 10bd808: phorge: Use user.search Conduit endpoint [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/63 (https://phabricator.wikimedia.org/T402454)