[09:41:56] <marostegui>	 Emperor: not sure if you saw it, but thanos-fe1001:9100 has been alerting for the weekend
[09:42:01] <marostegui>	  (SystemdUnitFailed) firing: (3) swift_dispersion_stats.service Failed on thanos-fe1001:9100
[09:49:59] <Emperor>	 sigh, thanks.
[09:51:22] <Emperor>	 Hm, cert fail, and it's a puppet7 host
[09:53:48] <Emperor>	 moritzm: thanos-fe1001 is one of the nodes you migrated to puppet7 and now swift_dispersion_stats is failing because of cert errors, are these likely related? 'urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)>'
[09:54:48] <Emperor>	 swift-dispersion-report fails with a lengthy backtrace ending thus
[09:59:13] <Emperor>	 moritzm: swift-dispersion-report is failing on all the thanos frontends with the certificate verify failed error
[09:59:55] <Emperor>	 (ms-fe1014 is the p7 ms frontend where we don't see that problem, so maybe it's a consequence of having moved the backends too? I don't know, but I have seen signs of TLS funkyness around the p7 migration)
[10:25:42] <Emperor>	 openssl s_client -connect 10.2.2.54:443 -showcerts </dev/null 2>/dev/null
[10:26:01] <Emperor>	 ^-- from a thanos-frontend this gets Verification error: unable to verify the first certificate
[10:26:11] <Emperor>	 whereas e.g. cumin1001 can verify this OK
[10:27:28] <Emperor>	 that IP is thanos-swift.svc.eqiad.wmnet (and strace suggests to me that's the cert verification that's breaking swift-dispersion-report)
[12:59:31] <jbond>	 marostegui: where you able to test the two db servers and if so do you have some roles i can try migrating?
[12:59:43] <marostegui>	 jbond: which two db servers?
[13:00:12] <jbond>	 you mentioned db1124 and db1133 yuo mentioned them last week
[13:00:19] <marostegui>	 meeting
[13:00:34] <jbond>	 ok no probs gonna grab some food in the mean time :)
[13:20:24] <marostegui>	 jbond: All the ones we did last week went fine
[13:20:28] <marostegui>	 Can you do pc1014 too?
[13:22:54] <jbond>	 marostegui: sure ill do pc1014 now
[13:35:09] <jbond>	 marostegui: pc1014 is migrated
[13:35:49] <marostegui>	 Cool I'll do some testing and next we should probably go for a DC master
[13:36:12] <jbond>	 ack sgtm just tell me what and when :)
[13:36:24] <marostegui>	 We just got an alert for stats.service 
[13:36:29] <marostegui>	 On pc1014
[13:36:42] <marostegui>	 jbond: ^
[13:36:47] <jbond>	 ac looking
[13:36:55] <marostegui>	 Puppet agent stats service that is
[13:37:58] <jbond>	 marostegui: do you mean puppet-agent-timer.timer
[13:38:07] * jbond i don't see any failed units
[13:38:29] <marostegui>	 14:34:49 <jinxer-wm> (SystemdUnitFailed) firing: (4) prometheus_puppet_agent_stats.service Failed on pc1014:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:38:33] <marostegui>	 That's what we got
[13:39:35] <jbond>	 marostegui: ack you can ignore that there is a race condition, patch is allready out.  also loks like its allready cleared
[13:39:56] <marostegui>	 Ah cool
[16:23:16] <Emperor>	 Opened T351653 re the TLS sadness on thanos-fe*
[16:23:17] <stashbot>	 T351653: thanos internal TLS failure after puppet 7 update - https://phabricator.wikimedia.org/T351653