[09:07:43] <_joe_>	 I just added the following line to apache's httpd configuration and I think it's amazing:
[09:07:45] <_joe_>	 SetEnvIf Request_URI "." SERVERGROUP=${SERVERGROUP}
[09:08:37] <_joe_>	 elukey: before you try to justify why I need to do it, please remember everyone else here is not affected by the httpd stockholm syndrome like us :P
[09:14:58] <apergos>	 is that . a regex??
[09:15:56] <_joe_>	 yes
[09:16:39] <apergos>	 !!
[09:17:01] <apergos>	 I can't wait to hear the explanatio
[09:17:02] <apergos>	 n
[09:17:06] <_joe_>	 it says: if the request uri exists, then set the environment variable SERVERGROUP to the value of the SERVERGROUP env variable
[09:17:24] <apergos>	 and is there an occasion where it would not exist??
[09:17:26] <_joe_>	 but "environment variable" in apache is a charged term
[09:17:29] <_joe_>	 no.
[09:17:40] <apergos>	 sssoooooo?
[09:18:04] * apergos grabs popcorn
[09:18:26] <_joe_>	 the issue is: env variables coming from the OS are not passed to the fcgi server we proxy to
[09:18:53] <_joe_>	 only variables set by SetEnv/SetEnvIf would
[09:19:13] <_joe_>	 but SetEnv gets set after RewriteRules are evaluated (IIRC, I should check)
[09:19:43] <_joe_>	 so we need to set the env var this way if we want to ensure it's always present in requests to the FastCGI server
[09:19:56] <apergos>	 oh geee :-/
[09:20:10] <apergos>	 "amazing" is a polite word for it, if that indeed turns out to be the case
[09:20:23] * elukey avoids to comment
[09:20:26] <apergos>	 lol
[09:21:05] <elukey>	 _joe_ SetEnv indeed is evaluated way after SetEnvIf
[09:22:01] <_joe_>	 elukey: I'm aware
[09:22:09] <elukey>	 I am trying to write something supporting httpd and cancelling every time, Joe knows me
[09:22:16] <_joe_>	 :D
[09:22:26] <_joe_>	 _joe_>	elukey: before you try to justify why I need to do it, please remember everyone else here is not affected by the httpd stockholm syndrome like us :P
[09:22:31] <elukey>	 yes yes :D
[09:22:32] <_joe_>	 that was supposed to be preemptive
[09:22:55] <elukey>	 but I thought it wasn't true, and I realized now it is
[09:23:12] <elukey>	 hence the "Joe knows me" :D
[14:36:36] <Krinkle>	 effie: btw, still have time until end of this month for gerrit dashboarding and stuff
[14:36:46] <Krinkle>	 can do later as well, but sort of "planned" for this quarter time asidd
[15:19:50] <hashar>	 _joe_: moritzm: I could use a script to restart php7.0 on doc1001 after we scap deploy php code. That is to clear out the opcache https://gerrit.wikimedia.org/r/c/operations/puppet/+/666309     It is still on php7.0 but we do plan to migrate to 7.2 / 7.3 eventually ;)
[15:20:16] <_joe_>	 hashar: what's wrong with "systemctl restart php7.0-fpm.service"?
[15:21:15] <hashar>	 Id like to have the php7.0 part to be defined in puppet next to the  php7.0 package installation
[15:21:40] <_joe_>	 not sure I get what you mean
[15:21:46] <hashar>	 and in the source repository scap just invoke the  thin shell script  wrapper and so I can forget about having to update it to match whatever php7.x is running on the server ;)
[15:22:54] <hashar>	 with   $php = 'php7.0'   being there to avoid repeating myself and ensuring we use the same for the packages installation, restart command etc. So later we just have to increment to php7.2 and we are set
[15:55:34] <effie>	 Krinkle: sure sure, that would be nice
[15:58:28] <Krinkle>	 effie: task is at https://phabricator.wikimedia.org/T263494
[15:58:42] <Krinkle>	 maybe we can start async on some of those (provocative?) questions, or we can meet?
[15:59:40] <effie>	 let's meet next week and upload something, I wonder if gitlab wil have some similar functionality \
[16:00:06] <effie>	 I will send you an invite
[21:04:36] <ryankemper>	 Anyone know if we use `cergen` for only a specific "type" of ssl/tls cert, or is it to be used for all service-related certs going forward?
[21:05:15] <cdanis>	 ryankemper: AIUI it's the current best way to make an internal service certificate
[21:05:16] <ryankemper>	 I've used cergen to generate a net-new cert before but in this case I have a service with an existing cert (but we replaced the hosts so we'll need a new cert now
[21:05:22] <ryankemper>	 )
[21:05:41] <cdanis>	 for anything externally-facing where you're using your own certs, and not just going via the traffic layer, you need to use acmechief
[21:05:58] <cdanis>	 yeah, you should use cergen for that, and most services already are, I think
[21:06:40] <ryankemper>	 Ah the internal distinction might be important part
[21:06:43] <mutante>	 whenever we turned an unencrypted connection from caching to backends into https, I would go to https://wikitech.wikimedia.org/wiki/Cergen#Cheatsheet
[21:07:08] <cdanis>	 ryankemper: cergen certs are signed by our puppetmaster's CA, not any 'real' CA, is the important bit
[21:07:41] <mutante>	 if it's public facing, acme-chief... as Chris already said
[21:08:23] <ryankemper>	 Ack, so for context this is for `relforge.svc.eqiad.wmnet`, so it sounds like I might want `acme-chief` here then
[21:08:37] <mutante>	 the good part is you basically just need to tell it which hosts should be allowed to request certs for what
[21:08:43] <mutante>	 and it's in puppet
[21:09:30] <ryankemper>	 It's a bit confusing because relforge is for "internal" use - for example the analytics network talks to it, not the outside world - but from the traffic perspective would that count as external or internal?
[21:09:35] <mutante>	 ryankemper: check out ./hieradata/role/common/acme_chief.yaml
[21:10:08] <cdanis>	 ryankemper: that would count as internal
[21:10:42] <cdanis>	 if 100% of the other hosts talking to it are also managed by Puppet, they have the Puppet CA installed -- and that sounds like it is the case here?
[21:10:52] <cdanis>	 .svc.*.wmnet addresses aren't reachable externally :)
[21:11:09] <ryankemper>	 ack (and @mutante there's no entry in `./hieradata/role/common/acme_chief.yaml` for `relforge.svc.eqiad.wmnet` so that lends support to the idea that it's internal only)
[21:11:37] <mutante>	 ryankemper: yea, in that case it is back to cergen and in the private repo
[21:11:39] <ryankemper>	 Heh, great point on them not even being externally reachable anyway :P
[21:12:35] <mutante>	 ryankemper: [puppetmaster1001:/srv/private/modules/secret/secrets/ssl] $ file relforge.svc.eqiad.wmnet.key
[21:12:44] <mutante>	 that is cergen
[21:12:55] <mutante>	 you need https://wikitech.wikimedia.org/wiki/Cergen#Update_a_certificate
[21:12:59] <mutante>	 to update an existing cert
[21:13:35] <mutante>	 that is .. if you need to have individual host name on that cert
[21:13:49] <mutante>	 often what we do is have a something.discovery.wmnet name on the cert
[21:14:07] <mutante>	 in some cases it will have both. discovery name and hostnames all on the cert
[21:14:12] <ryankemper>	 mutante: but I imagine cergen wasn't used to generate it initially because there's no `/srv/private/modules/secret/secrets/certificates/certificate.manifests.d/relforge.certs.yaml`, right?
[21:14:50] <ryankemper>	 The only relforge-related file I've found in private is `/srv/private/modules/secret/secrets/ssl/relforge.svc.eqiad.wmnet.key`
[21:15:18] <mutante>	 ryankemper: indeed. it seems like it wasn't created with the "official" workflow
[21:16:41] <ryankemper>	 mutante: Which brings me full circle to the question I was originally going to ask before realizing I needed to start more basic...are there any gotchas to watch out for when generating a cert for a service that wasn't using cergen previously?
[21:17:55] <ryankemper>	 I think if I understand how everything fits together properly, I can follow the usual `cergen` workflow and then need to change `hieradata/role/common/elasticsearch/relforge.yaml` to `s/relforge.svc.eqiad.wmnet/relforge.discovery.wmnet` (since the key will be named a bit differently now)
[21:18:00] <mutante>	 ryankemper: I just checked what exactly is on the existing cert
[21:18:01] <mutante>	                 DNS:relforge1001.eqiad.wmnet, DNS:relforge1002.eqiad.wmnet, DNS:relforge.svc.eqiad.wmnet
[21:18:17] <mutante>	 first thing I would ask is.. is it really needed to have the host names on the cert
[21:18:24] <mutante>	 or might it be enough to have the svc name
[21:18:38] <mutante>	 it could be there "just in case" but not really be used
[21:19:00] <mutante>	 then you wouldn't have to touch it for a new host name
[21:19:52] <mutante>	 if you do, i would say follow https://wikitech.wikimedia.org/wiki/Cergen#Cheatsheet  pretending everything is just new
[21:20:34] <mutante>	 what you can also do is  add relforge.discovery.wmnet to the cert.. and in DNS have a CNAME of your actual host to that.. that is what we do for many misc services
[21:20:50] <mutante>	 then a new host is a DNS change but not a cert change
[21:22:38] <mutante>	 you will have to revoke the old cert on the puppetmaster whether you make a new one or update an existing one
[21:23:05] <ryankemper>	 mutante: ack, thanks! I'll have to look into whether we actually care about the hostnames or not
[21:23:32] <mutante>	 yea, maybe you can test if it works without them
[21:23:58] <mutante>	 i know a bunch of certs have them "in case they are needed"
[21:24:13] <ryankemper>	 Yeah I would not be surprised if that were the case here
[21:24:29] <ryankemper>	 ack on revoking the old cert either way
[21:26:00] <mutante>	 the easiest test might be to just get one of the new servers into prod without touching that cert at all.. and see if it connects or not
[23:08:05] <twentyafterfour>	 anyone around willing to watch me deploy a kibana plugin? I don't want to do it without someone able to reboot kibana should something go wrong
[23:09:32] <shdubsh>	 twentyafterfour: o/
[23:10:47] <twentyafterfour>	 shdubsh: the updated dsh targets patch got merged so I think it's ready to go if you're ok with it
[23:11:58] <shdubsh>	 SGTM
[23:12:19] * shdubsh is ready
[23:14:33] <twentyafterfour>	 shdubsh: logging in and preparing to scap
[23:19:11] <twentyafterfour>	 shdubsh: doing it
[23:19:31] <shdubsh>	 🚀
[23:20:23] <twentyafterfour>	 shdubsh: no good :(
[23:20:48] <shdubsh>	 uh oh
[23:20:58] <shdubsh>	 error logs?
[23:22:04] <shdubsh>	 nothing appears down FWIW
[23:22:44] <twentyafterfour>	 second try worked (had to disable the "remove_plugin" before install)
[23:23:14] <twentyafterfour>	 BUT it doesn't appear to be installed
[23:25:18] <shdubsh>	 indeed
[23:25:33] <shdubsh>	 not seeing any indications of trying to run the kibana plugin install
[23:25:46] <twentyafterfour>	 indeed, trying it again
[23:27:27] <twentyafterfour>	 somehow my  sudo privs are no longer valid
[23:27:52] <shdubsh>	 haha, seeing the sudo first use message
[23:28:10] <shdubsh>	 "We trust you have received the usual lecture from the local System Administrator..."
[23:28:13] <twentyafterfour>	 it tries to do this: `/usr/bin/sudo -u kibana /usr/share/kibana/bin/kibana-plugin install file:///srv/deployment/releng/phatality/deploy/phatality-7.10.0.zip`
[23:28:45] <twentyafterfour>	 shdubsh: can you run the command since my deploy user can't do it anymore apparently?
[23:31:08] <shdubsh>	 this ought to be fixed
[23:31:24] <shdubsh>	 deploy-service apparently doesn't have sudo
[23:34:27] <twentyafterfour>	 shdubsh: I just submitted a patch for the sudo rules
[23:34:32] <twentyafterfour>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/667959
[23:34:38] <shdubsh>	 hah, beat me to it :)
[23:36:04] * shdubsh running puppet now
[23:37:51] <shdubsh>	 twentyafterfour: change is in place.  give it another go?
[23:38:25] <twentyafterfour>	 shdubsh: doing it
[23:39:28] <twentyafterfour>	 ok it succeeded but still no phatality tab in kibana
[23:40:11] <shdubsh>	 wasn't kibana service restart supposed to be a step?
[23:40:27] <twentyafterfour>	 I think it previously restarted automatically
[23:40:33] <twentyafterfour>	 or crashed actually
[23:40:43] <shdubsh>	 lol
[23:41:02] * shdubsh chalks this behavior up to an improvement
[23:41:08] <twentyafterfour>	 yeah indeed
[23:41:39] <twentyafterfour>	 I think plugin-install previously would auto-restart but the restart triggered OOM killer and just died
[23:41:51] <shdubsh>	 it's possible
[23:42:21] <twentyafterfour>	 should we add another sudoer rule or do you just wanna do the restart? I'm not sure if scap should do it
[23:43:12] <shdubsh>	 we'll want to add the sudoer rule, for sure
[23:43:21] <shdubsh>	 but I just finished the restart
[23:43:40] <twentyafterfour>	 ot
[23:43:44] <twentyafterfour>	 it's there! awesome
[23:43:57] <shdubsh>	 \o/
[23:44:07] <twentyafterfour>	 so what restart command?   `systemctl restart kibana`
[23:44:14] <shdubsh>	 yep that ^^
[23:48:12] <twentyafterfour>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/667964