[08:08:42] In case other people ends up wondering, no it's not only editing that's broken. It also affects things like resource/profile. (re @lucaswerkmeister: reported at https://phabricator.wikimedia.org/T417839) [09:24:18] This now appears to be fixed and deployed ⭐️ (re @lucaswerkmeister: reported at https://phabricator.wikimedia.org/T417839) [13:45:56] Can anyone take a look at, or shall I ask in a different channel? https://phabricator.wikimedia.org/T417889 [13:54:56] wurgl: looking [13:57:20] dhinus: at first sight it seems that the table there is missing the PRIMARY KEY (`il_from`,`il_to`) that is in prod [13:57:49] but the query is using only il_from so it would not use the key anyway [13:58:42] it will, as il_from is the first part of the primary [13:59:05] ref const, rows 20 [14:00:13] explain gives me "Using where; Using index" [14:01:05] where are you doing the explain? I'm comparing that with the prod table ;) [14:01:42] explain in the replica view (commonswiki_p) [14:01:43] I tried --cluster web and --cluster analytics both are suddenly slow (fast until ~10 am GMT, then my bot had a timeout) [14:07:02] ok sorry I was doing the explain on a different query... it's not using the index [14:07:59] there's also 18h of lag in the replicas [14:09:20] might be related to T415786 [14:09:21] T415786: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786 [14:09:38] yes there's a big alter table running on imagelinks [14:10:18] that probably explains why it's slower since this morning. but it doesn't really explain why it was not slow before :D [14:18:42] ok we have an explanation also for the second part: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1222603/7/sql/mysql/patch-imagelinks-pk.sql [14:19:03] the primary key was dropped and will be recreated, but right now is not there in the replicas [14:20:32] Okay. Case closed. [14:20:37] Thx [14:22:02] np