[07:30:43] 10Tool-yearinreview, 10MediaWiki-extensions-Translate, 06Wikipedia-Android-App-Backlog, 06LPL Essential (FY26 Q2), and 3 others: PLURAL syntax validator gets confused by other uses of equals signs in the message, as seen at [[Wikimedia:Wikipedia-android-st... - https://phabricator.wikimedia.org/T409655#11377639 [09:08:16] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: MTU setting in IPv6 VMs causes issues with Docker - https://phabricator.wikimedia.org/T408543#11377820 (10taavi) >>! In T408543#11375953, @Andrew wrote: > Today I'm draining a cloudvirt and I see this error in the logs (along with a failed migration): >... [09:18:13] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,keystone [09:18:52] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,keystone [09:36:49] (03update) 10dcaro: tracing: add tracing loki instance [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1040 (https://phabricator.wikimedia.org/T399313) (owner: 10volans) [09:53:39] (03approved) 10volans: Increase harbor quota for milhistbot [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1078 (https://phabricator.wikimedia.org/T409981) (owner: 10fnegri) [10:06:45] !log fnegri@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor (T409981) [10:06:50] T409981: Request increased build quota for MilHistBot Toolforge tool - https://phabricator.wikimedia.org/T409981 [10:07:02] (03approved) 10dcaro: flavors: add zuul to g4.cores8.ram32.disk20.4xiops [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/279 (https://phabricator.wikimedia.org/T409365) (owner: 10volans) [10:08:12] (03merge) 10fnegri: flavors: add zuul to g4.cores8.ram32.disk20.4xiops [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/279 (https://phabricator.wikimedia.org/T409365) (owner: 10volans) [10:10:31] !log fnegri@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor (T409981) [10:22:56] (03merge) 10fnegri: Increase harbor quota for milhistbot [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1078 (https://phabricator.wikimedia.org/T409981) [10:23:42] 10Toolforge (Quota-requests): Request increased build quota for MilHistBot Toolforge tool - https://phabricator.wikimedia.org/T409981#11378284 (10fnegri) 05Open→03Resolved `lang=shell-session tools.milhistbot@tools-bastion-15:~$ toolforge build quota Registry =================== Storage ----------- Avail... [10:25:13] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch (T409365) [10:25:20] T409365: Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365 [10:26:38] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch (T409365) [10:27:42] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch (T409365) [10:28:38] (03approved) 10dcaro: tracing: add tracing loki instance [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1040 (https://phabricator.wikimedia.org/T399313) (owner: 10volans) [10:29:57] 10Cloud-VPS (Quota-requests), 13Patch-For-Review: Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365#11378293 (10fnegri) 05Open→03Resolved * ✅ `high-iops` volume type * ✅ `g4.cores8.ram32.disk20.4xiops` instance flavor [10:30:37] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch (T409365) [10:30:44] T409365: Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365 [10:33:43] 06cloud-services-team, 10Cloud-VPS: [tofu-infra] tofu failing to retrieve DNS zones on codfw - https://phabricator.wikimedia.org/T410265 (10fnegri) 03NEW [10:52:10] 06cloud-services-team, 10Cloud-VPS (Project-requests): CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11378376 (10dcaro) >>! In T408387#11377235, @Odinaldo wrote: > Thank you Francesco (and Andrew). We will satisfy the requirements and ensure everything is transparent to allay any comm... [10:52:23] 06cloud-services-team, 10Cloud-VPS (Project-requests): CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11378377 (10dcaro) a:03dcaro [11:00:51] 10Cloud-VPS (Quota-requests): Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365#11378409 (10fnegri) > END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) This has no impact on this task, the change was applied successfully in... [11:46:11] 06cloud-services-team, 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11378533 (10fnegri) > there are currently no alerts defined on prometheus (https://prometheus-alerts.wmcloud.org/?q=project%3D~videocuttool) Please note tha... [11:50:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2010-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [12:35:52] 06cloud-services-team, 10Data-Services: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11378638 (10ayounsi) Could option 3 be something like what's currently being done for Gerrit ? https://phabricator.wikimedia.org/T365259 [12:37:34] 06cloud-services-team, 10Data-Services: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11378641 (10taavi) >>! In T306550#11378638, @ayounsi wrote: > @taavi Could option 3 be something like what's currently being done for Gerrit ? {T365259} That's the second o... [12:39:19] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/22 [12:50:03] 06cloud-services-team, 10Data-Services: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11378740 (10ayounsi) Oops, I'm still catching up. Sounds great to minimize user impact. do we know how many systems pull from our rsync ? Maybe it's not worth the hassle of... [13:02:24] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [13:05:09] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/22 (owner: 10l10n-bot) [13:05:12] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/22 (owner: 10l10n-bot) [13:14:49] (03open) 10dcaro: core: add prometheus counter for jobs synced from runtime [repos/cloud/toolforge/jobs-api] (use_custom_resources_in_code) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/253 [13:28:09] 06cloud-services-team, 10Cloud-VPS (Project-requests): CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11378843 (10Odinaldo) If we can have three admins, these would be them in this order of priority please: # https://phabricator.wikimedia.org/p/Odinaldo/ # https://phabricator.wiki... [13:34:27] !log dcaro@cloudcumin1001 prove START - Cookbook wmcs.vps.create_project for project prove in eqiad1 (T408387) [13:34:28] dcaro@cloudcumin1001: Unknown project "prove" [13:34:29] T408387: CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387 [13:35:00] !log dcaro@cloudcumin1001 prove END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project prove in eqiad1 (T408387) [13:35:00] dcaro@cloudcumin1001: Unknown project "prove" [13:43:12] 06cloud-services-team, 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11378887 (10Reputation22) >>! In T409668#11378532, @fnegri wrote: >> there are currently no alerts defined on prometheus (https://prometheus-alerts.wmcloud.o... [13:45:33] (03update) 10dcaro: core: add prometheus counter for jobs synced from runtime [repos/cloud/toolforge/jobs-api] (use_custom_resources_in_code) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/253 [13:46:25] (03merge) 10dcaro: images: load refresh time from settings [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/249 [13:46:28] (03update) 10dcaro: images: cache images retrieved from harbor [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/250 [13:46:53] (03merge) 10dcaro: toolforge_depoy: removed extension [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/296 [13:48:11] 06cloud-services-team, 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11378903 (10Reputation22) Based on this, I can provide you the threshold values for the expressions. Here's what we want based on the above mentioned grafana... [13:49:27] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.453-20251117134638-c2bc4111 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1079 [14:28:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet' [14:29:28] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1044.eqiad.wmnet' [14:30:45] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: MTU setting in IPv6 VMs causes issues with Docker - https://phabricator.wikimedia.org/T408543#11379070 (10Andrew) ec318e06-1ddc-4856-8e37-17a2a5aeb0b3 | tcp-proxy-test on cloudvirt1044 is showing the migration issue. ` 2025-11-17 14:29:24.723 2029263... [14:42:18] RESOLVED: PuppetFailure: Puppet has failed on cloudcontrol2010-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:47:37] (03merge) 10dcaro: global: only suport python3.13/trixie [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/67 [15:53:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [16:10:54] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [16:27:12] 06cloud-services-team, 10Data-Services: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11379659 (10bd808) >>! In T306550#11378740, @ayounsi wrote: > do we know how many systems pull from our rsync ? Maybe it's not worth the hassle of the tcp-proxy if the numbe... [16:44:17] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1045.eqiad.wmnet' [16:47:53] (03update) 10dcaro: core: add prometheus counter for jobs synced from runtime [repos/cloud/toolforge/jobs-api] (use_custom_resources_in_code) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/253 [17:04:07] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' [17:09:12] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet' [17:19:24] 06cloud-services-team, 10Cloud-VPS (Project-requests): CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11380061 (10dcaro) @Odinaldo you'll need to create developer accounts (https://www.mediawiki.org/wiki/Developer_account), or if you have one already, you'll have to link it to your pha... [17:20:13] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1046.eqiad.wmnet' [17:22:41] 06cloud-services-team, 06Infrastructure-Foundations, 06SRE, 10vm-requests: Site: 1 VM %request for codfw1dev CAS test/dev, hostname: cloudidp - https://phabricator.wikimedia.org/T410294 (10Andrew) 03NEW [17:23:29] 06cloud-services-team, 10Cloud-VPS, 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: sso failure in codfw1dev (labtesthorizon.wikimedia.org) - https://phabricator.wikimedia.org/T409328#11380091 (10Andrew) I'm leaning towards moving this service to a separate host. Ganeti request is T410294 [17:25:15] 06cloud-services-team, 06Infrastructure-Foundations, 06SRE, 10vm-requests: Site: codfw 1 VM request for codfw1dev CAS test/dev, hostname: cloudidp - https://phabricator.wikimedia.org/T410294#11380104 (10Andrew) [17:30:54] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' [17:36:37] (03update) 10dcaro: images: resolve the image every time [repos/cloud/toolforge/jobs-api] (use_cache_for_harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 [17:37:03] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [17:42:55] (03update) 10dcaro: README: update with deploy and local deploy instructions [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/4 [17:44:38] (03merge) 10dcaro: README: update with deploy and local deploy instructions [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/4 [17:45:51] (03update) 10dcaro: README: update deployment instructions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/230 [17:46:12] (03update) 10dcaro: README: update deployment instructions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/230 [17:46:43] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' [17:47:17] 10Toolforge (Toolforge iteration 25): [docs] update all readmes with the same deployment docs - https://phabricator.wikimedia.org/T407477#11380281 (10dcaro) [17:47:34] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: logs-api: bump to 0.0.7-20251117174451-6f8660fc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1080 (https://phabricator.wikimedia.org/T407477) [17:48:08] (03merge) 10dcaro: README: update deployment instructions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/230 [17:48:15] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [17:50:45] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.454-20251117174820-34492113 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1079 (https://phabricator.wikimedia.org/T407477) [17:50:46] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.454-20251117174820-34492113 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1079 (https://phabricator.wikimedia.org/T407477) [17:51:35] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [17:52:09] (03update) 10dcaro: readme: standardize deployment docs [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/29 [17:53:13] (03close) 10dcaro: loki: split into two components [repos/cloud/toolforge/toolforge-deploy] (loki-tracing) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1038 [17:55:29] (03merge) 10dcaro: readme: standardize deployment docs [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/29 [17:55:54] (03close) 10dcaro: DONOTMERGE projects: added project dcaro-test1 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/193 (https://phabricator.wikimedia.org/T375283) [17:59:09] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.71-20251117175543-3e629bb9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1081 [17:59:50] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet' [18:03:14] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [18:04:29] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [18:04:31] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [18:13:16] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [18:17:22] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [18:18:36] (03approved) 10dcaro: jobs-api: bump to 0.0.454-20251117174820-34492113 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1079 (https://phabricator.wikimedia.org/T407477) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:18:40] (03merge) 10dcaro: jobs-api: bump to 0.0.454-20251117174820-34492113 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1079 (https://phabricator.wikimedia.org/T407477) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:18:46] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [18:18:59] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [18:19:43] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' [18:19:55] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049.eqiad.wmnet' [18:24:31] 06cloud-services-team, 06Infrastructure-Foundations, 06SRE, 10vm-requests: Site: codfw 1 VM request for codfw1dev CAS test/dev, hostname: cloudidp - https://phabricator.wikimedia.org/T410294#11380505 (10MoritzMuehlenhoff) Looks good, but you definitely don't need 8G of RAM, 4G should be more than enough... [18:25:56] 06cloud-services-team, 06Infrastructure-Foundations, 06SRE, 10vm-requests: Site: codfw 1 VM request for codfw1dev CAS test/dev, hostname: cloudidp - https://phabricator.wikimedia.org/T410294#11380511 (10Andrew) >>! In T410294#11380505, @MoritzMuehlenhoff wrote: > Looks good, but you definitely don't need... [18:26:07] 06cloud-services-team, 06Infrastructure-Foundations, 06SRE, 10vm-requests: Site: codfw 1 VM request for codfw1dev CAS test/dev, hostname: cloudidp - https://phabricator.wikimedia.org/T410294#11380513 (10Andrew) [18:27:57] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [18:30:43] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [18:34:21] (03approved) 10dcaro: logs-api: bump to 0.0.7-20251117174451-6f8660fc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1080 (https://phabricator.wikimedia.org/T407477) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:34:25] (03update) 10dcaro: logs-api: bump to 0.0.7-20251117174451-6f8660fc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1080 (https://phabricator.wikimedia.org/T407477) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:34:29] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [18:34:50] (03merge) 10dcaro: logs-api: bump to 0.0.7-20251117174451-6f8660fc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1080 (https://phabricator.wikimedia.org/T407477) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:39:47] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' [18:39:56] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1050.eqiad.wmnet' [18:43:05] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [18:44:01] (03approved) 10dcaro: ingress-admission: bump to 0.0.71-20251117175543-3e629bb9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1081 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:44:04] (03update) 10dcaro: ingress-admission: bump to 0.0.71-20251117175543-3e629bb9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1081 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:44:35] (03merge) 10dcaro: ingress-admission: bump to 0.0.71-20251117175543-3e629bb9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1081 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:00:07] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1050.eqiad.wmnet' [19:01:33] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1051.eqiad.wmnet' [19:06:57] 10VPS-project-Codesearch: Codesearch "everything" index stuck in pre-start state; unable to search "everything" - https://phabricator.wikimedia.org/T410310 (10jsn.sherman) 03NEW [19:17:33] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' [19:56:42] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1052.eqiad.wmnet' [20:08:01] 06cloud-services-team, 06Infrastructure-Foundations, 06SRE, 10vm-requests: Site: codfw 1 VM request for codfw1dev CAS test/dev, hostname: cloudidp2001-dev - https://phabricator.wikimedia.org/T410294#11380957 (10Andrew) [20:12:52] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' [20:13:56] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' [20:21:17] 10VPS-project-Codesearch: Codesearch "everything" index stuck in pre-start state; unable to search "everything" - https://phabricator.wikimedia.org/T410310#11380991 (10Dzahn) ` Nov 17 20:20:46 codesearch9 docker[109888]: fatal: Unable to create '/data/data/vcs-dc1a443597790467097c6077343a73242284ded0/.git/index.... [20:32:08] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' [21:04:06] 10VPS-project-Codesearch: Codesearch "everything" index stuck in pre-start state; unable to search "everything" - https://phabricator.wikimedia.org/T410310#11381183 (10Dzahn) The /data/ path is mapped to /srv/hound. So this is actually an issue with the size of the "hound-search" index specifically. Even thou... [21:04:47] 10VPS-project-Codesearch: Codesearch "everything" index stuck in pre-start state; unable to search "everything" - https://phabricator.wikimedia.org/T410310#11381187 (10Dzahn) regardless of these comments: all services are UP now :) https://codesearch-backend.wmcloud.org/_health.json [21:05:09] 10VPS-project-Codesearch: Codesearch "everything" index stuck in pre-start state; unable to search "everything" - https://phabricator.wikimedia.org/T410310#11381194 (10Dzahn) 05Open→03Resolved a:03Dzahn [21:34:20] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: MTU setting in IPv6 VMs causes issues with Docker - https://phabricator.wikimedia.org/T408543#11381254 (10bd808) @xcollazo rediscovered this problem in {T408019}. See T408019#11380117 [22:19:15] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054.eqiad.wmnet' [22:39:11] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1054.eqiad.wmnet' [23:38:35] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054.eqiad.wmnet' [23:43:31] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1054.eqiad.wmnet' [23:43:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet' [23:58:59] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet'