[16:37:05] godog: we are not the first people to encounter it https://community.librenms.org/t/lnms-migrate-results-in-sql-query-error-syntax-error-or-access-violation-1118-row-size-too-large/10648/2 [16:38:36] oh interesting [16:38:43] gotta run now, ttyl! [19:18:01] cdanis, volans, thx! https://www.irccloud.com/pastebin/PDookJTw/ [19:22:17] XioNoX: nice! [19:30:23] XioNoX: ehehe nice :) we need to decide how to puppetize it, if we want to go the separate config route or just the ssh_config route [19:30:46] what's the ssh_config route? [19:31:09] pro of the separate config is that we could set a different default backend once we have that has network devices (netbox), con is that you need to specify the different config both at CLI and from spicerack (that will need some tweak) [19:31:31] the ssh_config route is to just create an ssh_config sith the options based on the hostname, like the one we have locally on our laptops [19:32:13] I see [19:32:30] pro is that will be transparent, no need of special knowledge or input, con is that the default backend will be puppetdb so you have to specify either D{} or when we'll have it N{} for netbox [19:32:39] but the backend is decided by the selector letter, no? [19:32:40] also it seems to me that you cheated [19:33:06] volans: please, don't get salt-y [19:33:10] hahhaa [19:33:13] volans: why? [19:33:24] passing the identify file of the key [19:33:51] we should check with ssh -vvv what's happening there when in keyholder there are multiple keys [19:34:05] dunno, I followed the doc [19:34:44] keyholder has doc? since when :D [19:34:59] https://wikitech.wikimedia.org/wiki/Keyholder [19:35:20] yeah [19:35:28] but indeed it works without the "ssh_options" [19:39:20] basically the second route, is have a single ssh option that is an ssh_config file [19:39:36] and there separate the network devices [19:39:52] just for the different homer user? [19:40:20] yes and any other fine tuning we might need [19:40:38] you're saying that like network devices need special care [19:41:26] ahahah [19:41:42] atm are the prod servers that need special care [19:41:54] volans: where is the doc to implementa netbox backend? [19:42:02] :) [19:42:16] https://gerrit.wikimedia.org/r/c/operations/software/cumin/+/514840 [19:42:32] but it's pending the import of the dns names [19:42:55] niiiiice! [19:43:04] otherwise it will not know where to connect, strictly speaking might use the IP [19:43:13] nice [19:43:14] XioNoX: also the known hosts bits needs to be fine-tuned [19:43:33] volans: network devices already have DNS names :) [19:43:55] as the known host file is populated by puppet and we don't want to override it or to populate a custom one in root's home that will be outdated [19:44:08] and if we want to use strict host checking we need to know the fingerprint before hand somehow [19:47:01] volans: is something strictly blocking https://gerrit.wikimedia.org/r/c/operations/software/cumin/+/514840 ? [19:47:30] like if we don't have DNS for servers, then we just can't use the N{} selector [19:47:38] or it's more complicated? [19:47:58] that for sure, and probably get in back into shape, I totally forgot the current status [19:48:23] netbox API have also changed in the meanwhile (slightly but need some tweaking) [19:48:43] and are much slower, so we need to do something to optimize it, might need a netbox plugin to be written [19:48:58] to avoid to do 1 or 2 API call per host [19:49:37] and also the grammar was a simpler one to start with [19:51:11] ok [19:52:03] volans: so right now nothing prevents me to write an easy runbook if I can pass the cumin config as parameter and I only need to target 1 host at a time? [19:53:45] I think D{} will take multiple hosts as well [19:53:48] using https://doc.wikimedia.org/spicerack/master/api/spicerack.remote.html [19:53:57] XioNoX: you need to puppetize the config [19:54:13] once that's done yes, you can write a cookbook (not a runbook ;) ) [19:54:17] that uses it [19:54:39] if we go the config route might need some tweak on the spicerack side, if the ssh_config route surely not [19:54:43] ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕓🍵 sudo cumin 'D{cr[1-2]-eqiad.wikimedia.org}' [19:54:45] 2 hosts will be targeted: [19:54:47] cr[1-2]-eqiad.wikimedia.org [19:54:49] DRY-RUN mode enabled, aborting [19:55:08] D{} uses clustershell NodeSet [19:55:23] so all it's features are supported [19:55:27] *its [19:55:57] cool [19:56:01] yeah [20:00:21] volans: the ssh_config needs to be puppetized too? [20:00:30] yes ofc [20:00:34] I mean puppetized or dynamically generated [20:00:36] ? [20:00:43] puppetized [20:01:21] volans: is there any existing one already or it would be a new thing? [20:01:38] there was one for wmcs, it should still be there [20:01:40] can't be that hard to translate netbox api query into a .ssh/config ;) [20:01:55] cdanis: ? [20:02:17] oh I guess you can just glob match router hostnames, nevermind [20:02:28] yes like we have locally [20:02:31] it's all on wikitech [20:02:34] indeed [20:03:35] I'm looking for an existing one but can't find it [20:04:00] profile/openstack/eqiad1/cumin/ssh_config.erb [20:04:05] modules/profile/manifests/openstack/eqiad1/cumin/master.pp [20:04:29] ah, I was looking for it on cumin1001 [20:05:22] don't forget the known hosts stuff too ;) [20:06:18] no idea how to deal with that [20:06:27] static file manually updated? :) [20:07:49] 1) make sure that cumin will not write over the puppet generated one or generate a new one in the root's home [20:08:01] that allow to use it without strict host checking [20:08:23] 2) to enable strict checking, we need to generate a file with the fingerprints, how to do that TBD [20:15:43] volans: so for 1) I should pass a ssh config to either use dedicated file, or disable strict checking for now until 2) is figured out? [20:16:19] 1) almost [20:17:00] UserKnownHostsFile /dev/null and LogLevel QUIET [20:17:19] are the usual trick to make it silent and not spam files when using StrictHostKeyChecking no [20:17:28] but we'd like to use it as yes, so better figure out 2) ;) [20:19:02] I'll think about 2), it's so few hosts that something kind of manual would be ok [20:19:25] XioNoX: could just make it a raw file in puppet [20:19:27] it's an info present on librenms? [20:19:29] eg. running a script and commiting the diff in puppet [20:19:34] volans: no [20:19:35] or rancid [20:20:01] volans: neither [20:21:25] ok, then I guess static file it is for now [20:22:56] cool [20:23:22] so much progress, thx! I'll try to do some of it tomorrow [20:23:43] ofc we can use cumin with option 1) [20:23:45] to gather them ;) [20:24:49] volans: and once it's there we can use them with homer too :) [20:25:31] not sure I have the magic 1 liner to get them all with 1) [20:25:56] set a file, use strict host checking no [20:25:59] ssh into all of them [20:26:00] voila! [20:26:27] but that way you're "trusting" them implicitely [20:26:34] ah, I see [20:26:36] the other way is to ssh and get the fingerprint from within junos [20:26:52] with some comamnd I dunno [20:26:54] I tried to look quickly but there is no easy way [20:26:55] but maybe not worthed [20:27:13] there must be a way to go fetch the public key on the host but not worth it [20:27:39] you don't even need cumin [20:27:46] ✔️ cdanis@bast1002.wikimedia.org ~ 🕟🍵 ssh-keyscan cr{1,2,3,4}-{eqiad,eqsin,esams,knams,eqord,codfw,ulsfo}.wikimedia.org [20:27:48] done [20:28:02] cdanis: we need *all* the network devices [20:28:06] but sure [20:28:09] that too :D [20:31:05] I tried to run ssh-keyscan from my laptop, but it doesn't seem to listen to the ssh proxycommand [20:31:57] I was thinking of a bash script living in puppet/utils [20:32:14] yeah, i did it from the bastion host [20:32:37] that generates the file each time you run it, so you can see the diff, optionally inspect it before sending a CR [20:32:40] can be a script that lives on the puppetmasters and updates a file in the private repo [20:32:51] does it need to be private? [20:32:55] no [20:33:02] was easier to automate than gerrit ;) [20:33:04] script could also be a `ssh bast1002.wikimedia.org 'ssh-keyscan ...'` [20:33:29] the diff logic is in the update-known-hosts file if you want :-P [20:34:06] we can install ansible and run ssh-keyscan through it https://serverfault.com/questions/823687/ssh-keyscan-through-a-bastion [20:34:15] go away :D [20:34:19] :) [20:35:12] so you need to get the list of devices from netbox APIs [20:35:22] and then do an ssh keyscan with that list [20:36:02] i was sure i had some netbox api scrapes in my shell history but apparently i don't [20:36:25] cdanis: why bother? just go to https://netbox.wikimedia.org/api/docs/ :D [20:36:46] well yeah i got it from there [20:36:47] dcim/devices with a filter on device_group [20:37:11] sorry device_rolws [20:37:24] I'm sure cdanis is wondering if it can all fit in a 1 liner [20:37:33] I will -2 that :D [20:39:16] we can always install puppet for junos [20:39:30] paravoid: no :) [20:39:34] :D [20:39:43] omg that's a thing [20:39:46] the troll is going out of control :-P [20:39:54] cdanis: unfortunately yes [20:40:07] they used to have just jpuppet, these days you can even do a Docker container [21:40:39] and yet another netbox release :D [22:13:22] @#$ [22:13:25] of course :) [22:14:12] some fixes that may be required but no interesting enhancements [22:15:54] can we pay them to release less often? :)