[08:41:07] hnowlan: o/ [08:41:52] qq - is there any test running on restbase2009? I see only one cassandra systemd unit, but icinga complains about the missing -a,-b-c [09:58:17] I have updated the puppet compiler hosts facts for the new an-workers, and at first it seemed to have worked [09:58:24] (I saw the expected diff etc..) [09:58:47] but now it got back to failing (404s when I try to check the catalog diff errors) [09:59:55] for example https://puppet-compiler.wmflabs.org/compiler1003/27607/ [10:00:38] ah no wait PEBCAK [10:00:54] the change catalog leads to 404 [10:01:03] but the change errors/warnings do not [10:13:11] TIL: from https://repology.org/project/nginx/versions I learn that they do now software tracking in wikidata: https://www.wikidata.org/wiki/Q306144 [10:42:09] nice :) [10:43:33] and TIL that AIX is still a thing :) [10:47:30] elukey: no, it was reimaged a day ago after some hardware issues and needs setting up (which I'll do today come to think of it) [10:49:46] hnowlan: ah ok! thanks [11:20:36] dcaro: o/ [11:20:37] ok to merge? [11:22:20] arturo: if you are around can you check? [11:22:31] I'm here [11:22:50] elukey: what's up? pending puppet merge? [11:23:01] yep yep should be https://gerrit.wikimedia.org/r/c/operations/puppet/+/657780/ [11:23:08] elukey:yep [11:23:17] ah there you go sorry for the two pings :) [11:23:21] merging [11:23:27] 👍 [11:23:35] np 👍 [15:36:57] wind keeps knocking my power out and looks like it has managed to now knock out the my internet provider base station so im going to call it a day for now. enjoy the weekend all [15:37:49] have a good weekend :-) [15:37:57] oof, good luck jbond42, have a nice weekend [15:38:12] thx :) [18:16:46] heh, looks like Amazon forked ES [18:38:25] I very much appreciate the 'DO NOT USE THIS SERVER' alerts on e.g. icinga1001. I suspect I have mutante to thank for that — it saves me from many a dead end. [18:44:42] Jeff_Green: want me to merge 'Jeff Green: replace check_swap with check_memory globally in nsca_frack.cfg.erb' ? [18:44:59] andrewbogott: yes please [18:45:19] ok, done [18:46:02] ha re. icinga1001, I still have to log in there to have that message remind me where I actually need to go [18:58:54] Jeff_Green: alert1001 now [19:00:05] Jeff_Green: hey, since that is an icinga question and you are here, does frack still use the NSCA setup ? [19:00:12] mutante: my brain is stuck re. that, as many times as I've re-learned this, all I can remember is "not icinga1001" [19:00:18] mutante: yes [19:00:48] Jeff_Green: so the nsca::client puppet class that is in the prod puppetmaster .. when I say "used by frack" that was correct [19:01:28] I remember making that but it was probably 2012 , heh [19:01:51] we don't use prod puppet at all, just the nsca API [19:02:29] you have your own puppetmaster and copied that class over? [19:03:33] yeah, we forked puppet probably around 2012, and run our own puppetmasters [19:04:39] ok, so it's probably used but we can still remove it from prod repo. thanks! [19:04:53] there is some cleanup task about finding unused classes..that's why [19:05:35] I know once upon a time there were other things outside of frack that reported by nsca, but I don't recall what they were. [19:06:16] yea, but I think that got replaced/removed at some point [19:06:28] all we need for frack is the nsca server to keep collecting, and icinga/templates/nsca_frack.cfg.erb [19:06:36] cool [19:06:57] I remember we did passive checks specifically to make sure frack can report without prod connecting to it [19:07:02] yep [19:07:47] ok, if you do not care at all about the operations/puppet repo then we can delete it either way [19:08:23] yup! [19:08:31] unless.. it's your own master but still cloning stuff from that repo :) that's what I wanted to make sure [19:08:37] ack [19:10:27] no, the only way we use prod puppet is to understand prod and for ideas, usually when we implement something based on prod, we end up paring away everything unnecessary to keep the frack repo as simple as possible [19:11:12] *nod*, thanks [19:15:35] (.. even though we don't seem to use passive checks right now it could actually make sense if we start that (again), to reduce the load on the icinga server and turning a bunch of base checks into passive instead of all these NRPE connections could make sense.. so i'll not just simply nuke that... yet) [21:51:54] hey all, so we've got envoy set up for `wdqs` (the public-facing service), but currently not for `wdqs-internal` and I'm looking into getting envoy set up there as well so that our internal traffic is tls-encrypted [21:52:26] For `wdqs` (which is using envoy already), the config to set it up looks pretty barebones, just https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/role/manifests/wdqs/public.pp#14 and https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/hieradata/role/common/wdqs/public.yaml#43 afaict [21:52:50] Now I'm looking to do the same for `wdqs-internal` but I'm a bit confused about how certs work with `global_cert_name` [21:53:56] So in hiera for wdqs-public we're setting `profile::tlsproxy::envoy::global_cert_name: "wdqs.discovery.wmnet"`, does that mean that there's a `wdqs.discovery.wmnet` certificate pre-configured somewhere? basically I'm wondering how a `global_cert` gets provisioned [22:04:17] Ah, I think I might have figured it out: `sslcert::certificate` says that for a resource `foo`, the puppet file `files/ssl/foo.crt`will be written to `/etc/ssl/localcerts`, so the cert needs to already exist in puppet's files dir