[15:59:32] <mutante>	 _joe_: so first follow-up was i needed the has_lvs for wtp2001 explicitely and now it's "invalid secret" for the parsoid.svc.codfw.wmnet.key  file (did eqiad but not codfw i guess) will fix it
[16:00:19] <_joe_>	 eheheh
[16:00:21] <_joe_>	 ok
[16:00:27] <_joe_>	 it seems overall a small problem
[16:00:37] <_joe_>	 if you consider how much complexity you added on those servers
[16:00:38] <mutante>	 we will see what comes after that one
[16:00:55] <mutante>	 is the staff meeting now? oops
[16:10:46] <mutante>	 i created "parsoid.discovery.wmnet.key" and cert and added all the SANs on it
[16:10:55] <mutante>	 but puppet wants parsoid.svc.codfw.wmnet.key
[16:11:01] <mutante>	 as in the file name
[16:11:46] <mutante>	 that seems to imply a separate cert for each DC
[16:16:49] <mutante>	 or copying everything simply to new names
[16:20:58] <wikibugs>	 10serviceops, 10Citoid: Alert on 0 zotero requests from zotero - https://phabricator.wikimedia.org/T234544 (10akosiaris)
[16:21:09] <wikibugs>	 10serviceops, 10Citoid: Alert on 0 zotero requests from zotero - https://phabricator.wikimedia.org/T234544 (10akosiaris) p:05Triage→03Normal
[16:24:15] <wikibugs>	 10serviceops: Alert on coreDNS misbehaving - https://phabricator.wikimedia.org/T234545 (10akosiaris)
[16:24:25] <wikibugs>	 10serviceops: Alert on coreDNS misbehaving - https://phabricator.wikimedia.org/T234545 (10akosiaris) p:05Triage→03High
[16:53:51] <_joe_>	 mutante: yeah we had two certs, one per dc
[16:54:02] <_joe_>	 each including the local svc name as CN
[18:24:37] <wikibugs>	 10serviceops, 10Growth-Team, 10Notifications, 10Operations, 10CPT Initiatives (Multi-DC Echo Notification Storage): Provision Kask for Echo timestamp storage in k8s - https://phabricator.wikimedia.org/T234376 (10Eevans)
[18:25:09] <wikibugs>	 10serviceops, 10Growth-Team, 10Notifications, 10Operations, and 2 others: Echostore service endpoints - https://phabricator.wikimedia.org/T234464 (10Eevans)
[19:15:56] <wikibugs>	 10serviceops, 10CPT Initiatives (Multi-DC (TEC1)): k8s liveness check(?) generating session storage log noise - https://phabricator.wikimedia.org/T227514 (10Eevans) Deployed to staging, the log output now looks like:  ` {"msg":"Initializing Kask dbb8ec9 (Go version: go1.11.5, Build host: 87a0b5ccf9a6, Timestam...
[20:15:09] <mutante>	 _joe_: watching the puppet run on wtp2001 now. i had it disabled because still fixing the cert situation..it was a bit of a hassle but now it should work
[20:15:34] <mutante>	 there is a single parsoid.yaml that describes 3 certs using 3 keys. one each DC and a unified one 
[20:15:52] <mutante>	 (now)
[20:18:51] <mutante>	 which fixed the puppet run because it expects the correct key name too per DC, cant just be one key for 2 certs
[20:21:51] <wikibugs>	 10serviceops, 10CPT Initiatives (Multi-DC (TEC1)): k8s liveness check(?) generating session storage log noise - https://phabricator.wikimedia.org/T227514 (10Eevans) I think there were (implicitly) two issues related to this open task: a) a superfluous log message (aka log spam), and b) unstructured log message...
[20:40:21] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10Dzahn) certficate issues fixed.    wtp2001 after the mediawiki roles have been applied now:  https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=wtp2001...
[21:39:08] <wikibugs>	 10serviceops, 10observability: Alert on coreDNS misbehaving - https://phabricator.wikimedia.org/T234545 (10Dzahn)
[21:39:20] <wikibugs>	 10serviceops, 10Citoid, 10observability: Alert on 0 zotero requests from zotero - https://phabricator.wikimedia.org/T234544 (10Dzahn)