[21:24:40] Can anyone explain to me how it can happen that there are more rows in text table than there are in the content table; is anything else referencing the text table? [21:25:07] e.g. on testwiki there are 574601 rows in the content table and 647692 rows in the text table [21:34:51] zabe: details about abuse filter hits? [21:34:57] this would be my first guess [21:35:51] new ones use URIs like `es:DB://cluster30/7932`, old ones use tt:462696 [21:36:33] oh, thats a a good one [21:36:53] yeah, searching for "storeBlob" unveiled it: https://codesearch.wmcloud.org/search/?q=storeBlob&files=&excludeFiles=&repos= [21:37:32] and structured data on commons would be my guess if you said commonswiki, heh :) [22:12:21] I've updated our Logstash dash to now include `channel:authentication` or `channel:CentralAuth` and under `channel:exception` we now include `trace:CentralAuth` as well. [22:12:49] The "should" nesting DSL takes a bit of getting used to, it's documented at https://opensearch.org/docs/latest/query-dsl/compound/bool/ [22:13:22] The dash now went from <50/day to >80,000/day due to two errors that are appear very frequently at the moment: [22:13:22] > 70,000x [CentralAuth] Expected key {key} not found due to timeout. [22:13:22] > 8,000x [CentralAuth] Account autocreation denied for {name} by ClosedWikiProvider See also: [22:14:48] for the second one we could probably change https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/wmf-config/CommonSettings.php#4580 to something less severe then error [22:15:08] These don't count as production errors (they're diagnostic with level=error, not an uncaught error or some such), but likely of interest, or possibly to retune to info if they're not actionable for us. [22:15:25] Yeah, that seems like somethign that's expected to happen by default on a closed-wiki domain? [22:21:15] yes, the second was what i was actually getting at. it's not really an error, but intentional that not masses of accounts are auto created in closed wikis where they are completely useless. [22:31:40] I can see this being an error if there is an expectation that autocreation isn't attempted by higher-level code for users without this right. But, afaik we call this unconditionally from the CentralAuth hooks. This is where we make that decision, not somewhere else as well. So it's an expected condition to reach, not an error yeah [22:31:55] Wanna write a patch? :D [22:42:47] sure [22:47:36] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1068879