[00:01:41] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: MTU setting in IPv6 VMs causes issues with Docker - https://phabricator.wikimedia.org/T408543#11375975 (10Andrew) As @taavi predicted, a reboot --hard of that server reset the MTU and allowed it to migrate. So that's good, and suggests that maybe we onl... [00:08:54] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [00:08:58] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) [00:09:13] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [00:09:15] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) [00:09:35] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [00:09:38] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) [00:15:36] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1043.eqiad.wmnet' [00:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:23:17] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [00:28:17] RESOLVED: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [00:53:52] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1043.eqiad.wmnet' [00:58:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1043.eqiad.wmnet' [00:59:03] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1043.eqiad.wmnet' [01:00:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1043.eqiad.wmnet' [01:01:07] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' [02:26:42] (03open) 10jaredblumer: Add Terms of Service URL Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/2 [02:31:38] (03update) 10jaredblumer: Add Terms of Service URL Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/2 [02:31:58] (03update) 10jaredblumer: Add Terms of Service URL Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/2 [02:39:05] (03update) 10jaredblumer: Add Terms of Service URL Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/2 [09:34:05] (03update) 10blegodwin: Draft: Add initial /add-work page scaffold [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/163 [09:36:21] (03update) 10blegodwin: Add initial /add-work page scaffold [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/163 [09:40:29] 10Tool-techcontribs: Show admin groups in tech-contribs - https://phabricator.wikimedia.org/T393322#11376447 (10Chlod) 05In progress→03Resolved Now deployed (initially was going to wait for other tasks to be done but I thought I should get this out now). "Primary groups" have been renamed to just "groups... [09:44:07] 06cloud-services-team, 10Cloud-VPS, 10VideoCutTool: [alerting] Create alerts for cloud-vps/VideoCutTool app - https://phabricator.wikimedia.org/T409668#11376471 (10Aklapper) [15:47:13] 10Tool-techcontribs: Show admin groups in tech-contribs - https://phabricator.wikimedia.org/T393322#11376563 (10Novem_Linguae) Very nice. Two small suggestions: - remove "primary" from primary ldap groups. Im not aware of any "secondary" ldap groups - limit posix group descriptions to 95 characters, and cho... [16:40:47] 10Tool-techcontribs: Show admin groups in tech-contribs - https://phabricator.wikimedia.org/T393322#11376608 (10Chlod) >>! In T393322#11376563, @Novem_Linguae wrote: > Very nice. Two small suggestions: > - remove "primary" from primary ldap groups. Im not aware of any "secondary" ldap groups Your Toolforge t...