[06:39:41] XioNoX: I'm late for the answer but for completeness... [06:39:57] not late, still looking for a solution :) [06:40:25] 1) in Icinga there is a mgmt hostgroup, so you can downtime all mgmt hosts (and their services) at once from the UI. [spoiler alert, this is probably the best solution *right now*] [06:40:38] nice, TIL! [06:40:59] if it's for a short period (say 30m) I think it's ok, we could also modify a bit puppet to have them per site [06:41:23] yeah, should be 30min, + 30 extra if any unexpected issues [06:41:37] basically go to https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?hostgroup=mgmt&style=hostdetail [06:41:52] select all via the checkbox in the title row [06:42:04] I see [06:42:10] select drom the dropdown schedule downtime for selected hosts and services... [06:42:40] what about the hosts with already some longer downtime [06:42:52] will it shorten the downtime or ignore it? [06:42:53] doesn't matter, icinga adds downtimes [06:42:57] ok [06:43:44] 2) from the spicerack/cookbook side of things, it's an interesting problem for a couple of reasons [06:44:41] a) the current icinga-downtime script (in puppet) that spicerack executes on the Icinga host performs the SCHEDULE_HOST_SVC_DOWNTIME call, so downtime a host with all the services, and that's fine, just that mgmt are considered hosts in Icinga world [06:45:35] b) the downtime cookbook accept an arbitrary cumin's query, so that would allow to select all hosts in a site, *but* doesn't have a way to tell I want only the mgmt interfaces. [06:46:09] so right now it's not possible running a cookbook (but we can make a patch to allow it), but it's possible if you do it "hacking via a Python REPL" [06:46:15] discouraged but technically possible [06:47:36] is it because the icinga backend is only to do action, and not for "inventory" ? [06:48:49] we don't have an icinga "backend" in cumin. It's that you get a list of hosts from a cumin's query like A:codfw and you need to convert them to mgmt hostnames, spicerack can do that, let me show you how could be done [06:52:13] ah okay! [06:52:27] don't worry, the webUI trick should be good enough :) [06:57:39] it's slightly more tricky than I expected because Icinga doesn't have FQDN of the hosts but just hostname and for the mgmt has hostname.mgmt. So in spicerack's Icinga module there was an assumption in dowmtime_hosts(...) to pick the first part of the FQDN before the first dot. [06:58:34] so the helper method downtime_hosts() cannot be used as is, but can be done using a more lower level method, just slightly more work [07:00:01] volans: but icinga have the IP [07:00:17] no, the identified for a mgmt host is 'analytics1038.mgmt [07:00:19] ' [07:00:40] ofc icinga can reach them [07:00:53] but you can't downtime things with FQDN or IP or anything else, just with the identifier [07:01:32] yeah, but icinga stores the IP, I don't think it resolves the identifier at each check [07:03:01] for the host not, for service checks most likely for all of them [07:03:13] s/all/some/ [07:05:17] ok! [07:05:44] opened T226470 [07:05:54] https://phabricator.wikimedia.org/T226470 (no bot here) :D [07:06:18] got the email, thx :) [07:09:10] errata corrige to (a): that script calls both SCHEDULE_HOST_DOWNTIME and SCHEDULE_HOST_SVC_DOWNTIME