[11:34:41] <kormat>	 hmm. it seems modern versions of Go do not appreciate tls certs that use CN and not SAN. this is the case for our puppet certs, from what i can see.
[11:34:45] <kormat>	 jbond42: is there any plan to fix that?
[11:39:29] <_joe_>	 kormat: if you create your certs via cergen that's taken care of, but yeah puppet host certs have that issue
[11:39:46] <kormat>	 _joe_: currently all mariadb instances use the puppet host cert
[11:39:49] <_joe_>	 and I guess what we'll do soon is moving away from using puppet certs for applications
[11:40:03] <_joe_>	 and towards using the pki infra john is building
[11:40:32] <jbond42>	 kormat: i was going to say probably in a recent version of puppet server but looks lke its stillan open bug https://tickets.puppetlabs.com/browse/SERVER-2338?jql=text%20~%20%22Subject%20Alternative%20Names%22
[11:40:37] <_joe_>	 so the plan to fix that is probably stop using puppet certs for other stuff
[11:40:53] <_joe_>	 and yes, that bug has been around forever
[11:41:13] <jbond42>	 fyi im looking for a dev service to test out the pki infrastructre if this relates to some none critical infrastructre
[11:41:49] <volans>	 luckily from maridb 10.4 you should have FLUSH SSL...
[11:41:53] <volans>	 me hides
[11:41:59] <kormat>	 sigh, ok
[11:43:15] <kormat>	 looking at cergen, that would be Very painful to start using in our case
[11:44:13] <volans>	 kormat: is this blocking orchestrator?
[11:45:34] <kormat>	 volans: ish. i think i might need to start building orchestrator with go1.14 instead of go1.15. i don't see another way around it for the moment.
[12:42:44] <klausman>	 Hmmm. When the VM creation cookbook fails (due to running in parallel with another one), I get leftover IPs in the repo when re-running it (after the other cookbook has completed DNS stuff). Is there some cleanup step? (reposting here because of SNR in -ops)
[12:44:57] <moritzm>	 I ran into that before, you need to clean out the stray assignment in Netbox before you re-run the cookbook, which hostname failed?
[12:45:10] <klausman>	 ml-etcd2001
[12:45:59] <klausman>	 So I just delete the addresses in NB?
[12:47:55] <moritzm>	 yeah, that's what I did back then when I ran into a traceback with the makevm cookbook. after that re-running the makevm cookbook will assign new addresses (which are likely the same anyway and the diff presented should only show your new names(
[12:50:16] <klausman>	 Roger!
[13:40:29] <volans>	 klausman: sorry for the trouble, there's a TODO to add a cleanup step in case of failure to the makevm and a wider one to allow to set distributed locks for certain operations
[13:41:03] <volans>	 if you didn't already solve it lmk and I can helo
[13:41:05] <volans>	 *help
[13:54:48] <klausman>	 Nah, it's all good.
[13:56:48] <volans>	 ack
[15:10:14] <cdanis>	 kormat: can I naively ask why it would be painful?  you still get certs that are signed by the puppet CA
[15:10:40] <kormat>	 cdanis: from looking at the doc it sounded like it would need to be manually done for each of the 200 servers
[15:10:48] <_joe_>	 cdanis: because they need that ^^
[15:10:49] <kormat>	 is that not the case?
[15:11:00] <_joe_>	 and for every new server, they'd need to add it to cergen
[15:11:01] <volans>	 and all pre 10.4 mariadb need a restart of the mysql process
[15:11:06] <kormat>	 _joe_: yep
[15:11:10] <cdanis>	 ah
[15:11:15] <volans>	 not too many but we still have a bunch
[15:11:19] <cdanis>	 yeah you want jbond's cfssl :)
[15:12:41] <kormat>	 it looks like the existing CA processes assuming service-level certs, rather than host-level certs
[15:12:57] <kormat>	 *assume
[15:13:07] <_joe_>	 kormat: yeah and that's true for most stuff :)
[15:13:27] <cdanis>	 also the things that want a host-level cert mostly just use the puppet host cert :)
[15:13:39] <kormat>	 cdanis: and the puppet host cert is bad, because puppet
[15:13:46] <cdanis>	 it is what it is
[15:19:31] <_joe_>	 and it is pretty sad :)
[15:19:46] <_joe_>	 but still better than when we had no internal certs for anything :)
[16:11:47] <kormat>	 jbond42: as i couldn't find an existing task that covers this clearly, i opened https://phabricator.wikimedia.org/T273637 and assigned it to you. please feel free to unassign it and put it where ever makes sense.
[16:14:43] <jbond42>	 kormat: ack
[19:00:30] <elukey>	 It took a little while but thanks to Razzi and Dan our dear Turnilo now has a nice URL shortner :)
[19:04:02] <XioNoX>	 nice!
[19:04:04] <apergos>	 \o/
[19:05:29] <sukhe>	 elukey: very nice! thank you Razzi and Dan!
[19:05:39] <sukhe>	 trying to figure out where it is :P
[19:06:21] <elukey>	 sukhe: top right corner
[19:06:41] <elukey>	 there is the share icon
[19:07:25] <sukhe>	 ah ha yeah thanks!
[19:07:36] <sukhe>	 very useful feature _/\_
[19:12:39] <cdanis>	 nice!
[19:12:48] <cdanis>	 glad to not need to copy & paste :)
[20:02:01] <Krinkle>	 legoktm: mwdebug1 special version "7.2.31-1+0~20200514.41+debian9~1.gbpe2a56b+wmf1+icu63 (fpm-fcgi)" vs mwdebug3: "7.2.31-1+0~20200514.41+debian9~1.gbpe2a56b+wmf1+buster1 (fpm-fcgi)" so exact same PHP version, best I can tell.
[20:02:16] <Krinkle>	 I do notice "10.4.15-MariaDB-log" vs "10.1.43-MariaDB"
[20:02:17] <legoktm>	 yes
[20:02:18] <Krinkle>	 not sure if a glitch
[20:03:03] <legoktm>	 a replica using a different version?
[20:03:16] <Krinkle>	 also "10.4.12-MariaDB-log"
[20:03:21] <Krinkle>	 yeah, I guess that's just different replicas
[20:03:27] <Krinkle>	 not deterministic/related
[20:03:27] <Krinkle>	 nbm
[20:03:28] <Krinkle>	 nvm
[20:03:29] <legoktm>	 https://phabricator.wikimedia.org/T273312#6793061 has the diff of `phpinfo()` across stretch/buster
[20:06:09] <Krinkle>	 ack, nice
[20:06:27] <Krinkle>	 legoktm: btw regarding the two hourly flamegraphs there, they have less than a 100 samples so yeah, I'd say that's random noise more or less
[20:06:51] <Krinkle>	 remembering that from any given prod web req, we take less than 1 sample for the entire request response duration
[20:06:58] <Krinkle>	 on average
[20:09:13] <legoktm>	 ack
[20:09:59] <Krinkle>	 for the main excimer/stretch pipeline there also aren't many hits in a given hour (~300), that's just because of how fast and well-cached it usually is (~<50ms at p50, 99.97% varnish hit). will need a day to get good traffic
[20:10:38] <Krinkle>	 I think the staggered random interval is currently around 60 seconds, so most reqs get no samples.
[20:13:08] <legoktm>	 in theory all the benchmarking traffic I made should show up too
[20:13:50] <legoktm>	 'https://kk.wikipedia.org/w/load.php?debug=false&lang=kk&modules=ext.3d.styles%7Cext.cite.styles%7Cext.uls.interlanguage%7Cext.visualEditor.desktopArticleTarget.noscript%7Cext.wikimediaBadges%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.page.gallery.styles%7Cmediawiki.skinning.interface%7Cmediawiki.toc.styles%7Cskins.vector.styles%7Cwikibase.client.init&only=styles&skin=vector'