[19:05:12] <greg-g>	 tgr: anomie: I'm looking at these huge spikes in fatalmonitor at 15:55 UTC (and a few later) today and they don't appear to be any of the usual suspects. I suspect it's a ton of the CentralAuthUser and CryptHash errors (lots of groups of 4, is my guess). Does that make sense to you? https://logstash.wikimedia.org/#dashboard/temp/AVX_ZGGzw3dCNxx2VxHi
[19:05:26] * anomie looks
[19:05:41] <greg-g>	 I can't get the right kibana incantation to group those with wildcards
[19:05:48] <greg-g>	 otherwise I would know more definitively
[19:11:30] <tgr>	 greg-g: the kibana thing is T136849
[19:11:31] <stashbot>	 T136849: normalized_message is a JSON dump of the whole event for exceptions in beta logstash - https://phabricator.wikimedia.org/T136849
[19:13:30] <greg-g>	 ty
[19:15:26] <tgr>	 the CryptHash thing is just a consequence, the user update which would set a token fails due to a CAS error and the crypt call receives NULL for a token
[19:17:07] * greg-g nods
[19:17:31] <anomie>	 As for those CryptHash errors, it's already fixed for newly-created users in wmf.11, and I have T140478 to remind me to clean up existing users so it won't happen anymore.
[19:17:32] <stashbot>	 T140478: Populate gu_auth_token for existing users - https://phabricator.wikimedia.org/T140478