[00:01:29] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node [05:01:51] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) [06:32:20] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: Phase out DSA keys for SSH access (ssh-dss) - https://phabricator.wikimedia.org/T177371#11130917 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff I think we can close this task. Any place where DSA keys are acti... [07:06:55] (03merge) 10taavi: Update repository URL [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/11 (https://phabricator.wikimedia.org/T403178) [08:13:57] (03open) 10taavi: channels: Add godod go -cloud channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/30 [08:14:02] (03update) 10taavi: channels: Add godod go -cloud channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/30 [08:20:31] (03update) 10taavi: channels: Add godod to -cloud channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/30 [08:20:36] (03update) 10taavi: channels: Add godod to -cloud channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/30 [08:24:21] (03merge) 10taavi: channels: Add godod to -cloud channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/30 [08:26:27] !issync [08:26:28] Syncing #wikimedia-cloud-feed (requested by Majavah) [08:26:29] Set /cs flags #wikimedia-cloud-feed godog +Afiortv [08:53:55] 06cloud-services-team, 10Cloud-VPS, 10DNS: Move some of wikimediacloud.org 185.15.56.0/23 to Netbox - https://phabricator.wikimedia.org/T268621#11131116 (10taavi) >>! In T268621#11129842, @ssingh wrote: > Can someone from WMCS comment on the status of this task? Currently the only manually managed record rem... [09:15:47] (03PS1) 10Majavah: Modernize code [labs/tools/github-pr-closer] - 10https://gerrit.wikimedia.org/r/1183078 [09:19:04] (03CR) 10Majavah: [C:03+2] Modernize code [labs/tools/github-pr-closer] - 10https://gerrit.wikimedia.org/r/1183078 (owner: 10Majavah) [09:19:27] (03Merged) 10jenkins-bot: Modernize code [labs/tools/github-pr-closer] - 10https://gerrit.wikimedia.org/r/1183078 (owner: 10Majavah) [09:36:40] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Add NEL headers to Cloud VPS and Toolforge managed proxies - https://phabricator.wikimedia.org/T400994#11131201 (10taavi) 05Open→03Resolved [09:56:18] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Rename terraform-cloudvps repo to tofu-cloudvps - https://phabricator.wikimedia.org/T403178#11131274 (10taavi) 05Open→03Resolved [11:41:06] (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505) [11:46:20] (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505) [11:56:45] (03merge) 10taavi: tofu-provisioning: Cache plugin directory [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/69 (https://phabricator.wikimedia.org/T403028) [12:02:11] (03open) 10taavi: dns: Reference floating IP module resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/71 [12:02:13] (03update) 10taavi: dns: Reference floating IP module resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/71 [12:10:59] (03update) 10samwilson: Migrate from GitHub to GitLab CI [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/1 (https://phabricator.wikimedia.org/T402505) [12:17:26] (03update) 10taavi: toolsbeta: Bump size of toolsbeta-docker-registry-data [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/72 [12:17:26] (03open) 10taavi: toolsbeta: Bump size of toolsbeta-docker-registry-data [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/72 [12:17:27] (03update) 10taavi: dns: Reference floating IP module resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/71 [12:17:35] (03update) 10taavi: dns: Reference floating IP module resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/71 [12:17:37] (03update) 10taavi: toolsbeta: Bump size of toolsbeta-docker-registry-data [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/72 [12:21:12] (03approved) 10fnegri: toolsbeta: Bump size of toolsbeta-docker-registry-data [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/72 (owner: 10taavi) [12:21:28] (03merge) 10taavi: toolsbeta: Bump size of toolsbeta-docker-registry-data [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/72 [12:21:31] (03update) 10taavi: dns: Reference floating IP module resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/71 [12:25:32] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudcephosd1004-10015 - https://phabricator.wikimedia.org/T402881#11131654 (10Jclark-ctr) a:03Jclark-ctr [12:26:23] (03update) 10taavi: dns: Reference floating IP module resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/71 [12:26:29] (03update) 10taavi: dns: Reference floating IP module resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/71 [12:29:52] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudcephosd1004-10015 - https://phabricator.wikimedia.org/T402881#11131690 (10Jclark-ctr) [12:32:59] (03update) 10taavi: dns: Reference floating IP module resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/71 [12:45:36] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [12:48:54] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudcephosd1004-10015 - https://phabricator.wikimedia.org/T402881#11131743 (10Jclark-ctr) [12:49:11] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudcephosd1004-10015 - https://phabricator.wikimedia.org/T402881#11131744 (10Jclark-ctr) 05Open→03Resolved [12:57:47] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [13:16:55] 06cloud-services-team, 10Toolforge: toolforge tofu-provisioning: Cache terraform-provider-openstack binary somewhere - https://phabricator.wikimedia.org/T403028#11131811 (10taavi) 05Open→03Resolved [13:23:15] 06cloud-services-team, 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T401077#11131819 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Done, I have followed and updated https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Pywikibot_image [13:30:46] 06cloud-services-team, 10Toolforge: Duplicated utils scripts - https://phabricator.wikimedia.org/T402949#11131832 (10fnegri) p:05Triage→03Medium [13:37:27] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,infra] Centralized logging for Toolforge infrastructure logs - https://phabricator.wikimedia.org/T97861#11131863 (10taavi) a:05taavi→03None Un-claiming for now. [13:38:21] (03update) 10samwilson: Add GitLab CI to test extension installation [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/2 [13:44:42] (03update) 10taavi: dns: Reference floating IP module resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/71 [13:44:42] (03update) 10taavi: toolsbeta: dns: Add MX records for toolsbeta.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/73 (https://phabricator.wikimedia.org/T394997) [13:44:43] (03update) 10taavi: dns: Unify emails in SOA records [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/74 [13:44:48] (03open) 10taavi: dns: Unify emails in SOA records [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/74 [13:44:49] (03open) 10taavi: toolsbeta: dns: Add MX records for toolsbeta.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/73 (https://phabricator.wikimedia.org/T394997) [13:45:03] (03update) 10taavi: dns: Reference floating IP module resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/71 [13:45:07] (03update) 10taavi: toolsbeta: dns: Add MX records for toolsbeta.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/73 (https://phabricator.wikimedia.org/T394997) [13:45:15] (03update) 10taavi: dns: Unify emails in SOA records [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/74 [13:47:21] (03update) 10taavi: dns: Unify emails in SOA records [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/74 [13:51:32] (03update) 10taavi: dns: Unify emails in SOA records [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/74 [13:51:38] (03update) 10taavi: toolsbeta: dns: Add MX records for toolsbeta.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/73 (https://phabricator.wikimedia.org/T394997) [13:54:44] 06cloud-services-team, 10Toolforge: [webservice] returning default 404 even though webservice is healthy - https://phabricator.wikimedia.org/T403168#11131903 (10DamianZaremba) I've managed to do some more troubleshooting as this has happened a few times. It appears the default route (fourohfour tool response)... [14:00:06] (03update) 10taavi: dns: Reference floating IP module resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/71 [14:00:08] (03update) 10taavi: toolsbeta: dns: Add MX records for toolsbeta.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/73 (https://phabricator.wikimedia.org/T394997) [14:00:08] (03update) 10taavi: dns: Unify emails in SOA records [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/74 [14:00:08] (03update) 10taavi: tools: dns: Drop temporary migration names [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/75 [14:00:11] (03open) 10taavi: tools: dns: Drop temporary migration names [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/75 [14:00:15] (03update) 10taavi: tools: dns: Drop temporary migration names [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/75 [14:05:46] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node [14:07:16] (03open) 10taavi: tofu-provisioning: Move plugin cache directory [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/70 (https://phabricator.wikimedia.org/T403028) [14:07:19] (03update) 10taavi: tofu-provisioning: Move plugin cache directory [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/70 (https://phabricator.wikimedia.org/T403028) [14:08:56] (03merge) 10taavi: tofu-provisioning: Move plugin cache directory [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/70 (https://phabricator.wikimedia.org/T403028) [14:18:18] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge tofu-provisioning: Cache terraform-provider-openstack binary somewhere - https://phabricator.wikimedia.org/T403028#11131999 (10taavi) This is in place now, but lack of {T365772} is making it not very effective. We'll see if that's enough o... [14:24:43] !log filippo@cloudcumin1001 wmf-research-tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [14:26:24] !log filippo@cloudcumin1001 wmf-research-tools END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [14:26:40] !log filippo@cloudcumin1001 globaleducation START - Cookbook wmcs.openstack.cloudvirt.vm_console [14:27:55] 06cloud-services-team, 10Cloud-VPS: Audit and potentially fix VMs not reachable by cloudcumin root key - https://phabricator.wikimedia.org/T402185#11132104 (10fgiunchedi) [14:29:42] !log filippo@cloudcumin1001 globaleducation END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [14:33:46] !log filippo@cloudcumin1001 collection-alt-renderer START - Cookbook wmcs.openstack.cloudvirt.vm_console [14:35:06] !log filippo@cloudcumin1001 collection-alt-renderer END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [14:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:45:45] 06cloud-services-team, 10Cloud-VPS: Audit and potentially fix VMs not reachable by cloudcumin root key - https://phabricator.wikimedia.org/T402185#11132214 (10fgiunchedi) All VMs but mediawiki2latex.collection-alt-renderer.eqiad1.wikimedia.cloud have been fixed, @Andrew mentioned the VM ip and DNS mismatch mig... [15:05:51] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: [wmf-sre-laptop] fetch public keys for Cloud bastions - https://phabricator.wikimedia.org/T329322#11132280 (10taavi) [15:08:15] 06cloud-services-team, 10Cloud-VPS: Audit and potentially fix VMs not reachable by cloudcumin root key - https://phabricator.wikimedia.org/T402185#11132294 (10Andrew) I repaired mediawiki2latex like this: ` root@cloudcontrol1011:~# openstack zone list --all-projects | grep " eqiad1.wikimedia.cloud." | 67603e... [15:08:43] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance abogott-nstesting [15:09:09] 06cloud-services-team, 10Cloud-VPS: Audit and potentially fix VMs not reachable by cloudcumin root key - https://phabricator.wikimedia.org/T402185#11132297 (10fgiunchedi) [15:09:39] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: [wmf-sre-laptop] fetch public keys for Cloud bastions - https://phabricator.wikimedia.org/T329322#11132299 (10taavi) For #Toolforge, https://tools-static.wmflabs.org/admin/fingerprints/ and https://toolsbeta-static.wmcloud.org/admin/fingerprints/ do exist alre... [15:09:39] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance abogott-nstesting [15:10:26] 06cloud-services-team, 10Cloud-VPS: Audit and potentially fix VMs not reachable by cloudcumin root key - https://phabricator.wikimedia.org/T402185#11132312 (10fgiunchedi) 05In progress→03Resolved All done here, thank you @Andrew for the DNS fix. Resolving for now, the more sustainable / long term fix i... [15:15:20] (03PS1) 10Majavah: Do not crash when the signature is missing [labs/tools/github-pr-closer] - 10https://gerrit.wikimedia.org/r/1183139 [15:15:33] (03CR) 10Majavah: [C:03+2] Do not crash when the signature is missing [labs/tools/github-pr-closer] - 10https://gerrit.wikimedia.org/r/1183139 (owner: 10Majavah) [15:16:01] (03Merged) 10jenkins-bot: Do not crash when the signature is missing [labs/tools/github-pr-closer] - 10https://gerrit.wikimedia.org/r/1183139 (owner: 10Majavah) [15:26:26] 10Cloud-VPS (Quota-requests), 10WLM-Italy (WLM-Italy-WebApp): Quota increase for Wiki Loves Monuments app - https://phabricator.wikimedia.org/T403165#11132380 (10fnegri) +1 [15:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:01:00] 06cloud-services-team, 06DC-Ops, 06SRE: cloudcephosd10[48-52] service implementation - https://phabricator.wikimedia.org/T395910#11132490 (10Andrew) [16:05:32] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 06SRE: cloudcephosd10[48-52] service implementation - https://phabricator.wikimedia.org/T395910#11132514 (10taavi) [16:42:16] 06cloud-services-team, 10Toolforge: [components-api] reuse_from components are not explicitly re-created in jobs-api - https://phabricator.wikimedia.org/T403285 (10DamianZaremba) 03NEW [16:43:46] (03open) 10damian: Ensure reuse_from components are re-run [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/125 (https://phabricator.wikimedia.org/T403285) [16:44:13] 06cloud-services-team, 10Toolforge: `webservice shell` sometimes loses output - https://phabricator.wikimedia.org/T403286 (10Anomie) 03NEW [16:44:43] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [components-api] reuse_from components are not explicitly re-created in jobs-api - https://phabricator.wikimedia.org/T403285#11132638 (10DamianZaremba) Original ticket: T401893 [16:50:27] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [components-api] reuse_from components are not explicitly re-created in jobs-api - https://phabricator.wikimedia.org/T403285#11132653 (10DamianZaremba) Missed this bit of logic sorry [16:53:36] 06cloud-services-team, 10Toolforge: [components-api] Validate `SourceReference` components point to `SourceBuild` components - https://phabricator.wikimedia.org/T403287 (10DamianZaremba) 03NEW [16:55:32] (03open) 10damian: Expand reuse_from validation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/126 (https://phabricator.wikimedia.org/T403287) [16:58:38] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [components-api] reuse_from components are not explicitly re-created in jobs-api - https://phabricator.wikimedia.org/T403285#11132678 (10DamianZaremba) [17:11:47] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [components-api] reuse_from components are not explicitly re-created in jobs-api - https://phabricator.wikimedia.org/T403285#11132729 (10DamianZaremba) Need to do some testing with this but don't have time right now. /cc @dcaro as the previous reviewer. [17:28:37] 06cloud-services-team, 10Toolforge: `webservice shell` sometimes loses output - https://phabricator.wikimedia.org/T403286#11132797 (10DamianZaremba) `webservice shell` basically calls `kubectl run` under the hood (https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/blob/main/toolsws/backends/ku... [18:23:45] 06cloud-services-team, 10Toolforge: `webservice shell` loses output when not run with a pty - https://phabricator.wikimedia.org/T403295 (10Anomie) 03NEW [19:05:06] 06cloud-services-team, 10Toolforge: `webservice shell` sometimes loses output - https://phabricator.wikimedia.org/T403286#11133024 (10Anomie) Running stuff via `webservice shell` instead of on the bastion seems to be the way Toolforge's current admins intend for {T360488} to be resolved. I have a lot of script... [19:06:00] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) [20:08:40] 06Toolforge-standards-committee: Define a process for keeping the committee membership "fresh" - https://phabricator.wikimedia.org/T379844#11133144 (10Soda) @bd808 Has there been any progress on this? [20:23:03] 06Toolforge-standards-committee: Define a process for keeping the committee membership "fresh" - https://phabricator.wikimedia.org/T379844#11133169 (10bd808) >>! In T379844#11133144, @Soda wrote: > @bd808 Has there been any progress on this? Not as much as I had hoped. I have been meaning to poke the discussion... [21:24:31] 06cloud-services-team, 10Toolforge: Build service tools built with --use-latest-versions are missing /etc/mime.types - https://phabricator.wikimedia.org/T403303 (10LucasWerkmeister) 03NEW [21:26:45] 06cloud-services-team, 10Toolforge: Build service tools built with --use-latest-versions are missing /etc/mime.types - https://phabricator.wikimedia.org/T403303#11133284 (10LucasWerkmeister) In #tool-wdactle, I’ve [worked around this](https://gitlab.wikimedia.org/toolforge-repos/wdactle/-/commit/69ac5a1656) by... [21:29:24] 06cloud-services-team, 10Toolforge, 07Upstream: Build service tools built with --use-latest-versions are missing /etc/mime.types - https://phabricator.wikimedia.org/T403303#11133286 (10bd808) Tagging as #upstream because we do not build these base images locally. [21:56:02] 06cloud-services-team, 10Toolforge, 07Upstream: Build service tools built with --use-latest-versions are missing /etc/mime.types - https://phabricator.wikimedia.org/T403303#11133326 (10LucasWerkmeister) Any volunteers for telling upstream about the issue? I can try to do it tomorrow, but I don’t know much ab... [23:13:48] 06cloud-services-team, 10Toolforge: [webservice] returning default 404 even though webservice is healthy - https://phabricator.wikimedia.org/T403168#11133440 (10bd808) >>! In T403168#11131903, @DamianZaremba wrote: > Health check is returning a good response: > ` > I have no name!@cluebotng-reviewer-95d85f65f-...