[13:47:02] I'm chasing a dumps bug, quick Q for pmiazga. My notes [1] led me to try $store->getBlob( "tt:331925" ) and see a failure [13:47:02] [1] https://www.mediawiki.org/wiki/SQL/XML_Dumps/Debugging_PHP [13:47:45] using the es: address for that revision, es:DB://cluster5/188443/03d8995c4b8b437077225ee9dbeabb11, works well [13:48:28] so the question is, why is dumps trying the old id? I don't find it anywhere in the database, and I've been trying to read the related change https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1020948 without finding an answer [13:54:59] The answer is https://phabricator.wikimedia.org/T183490 [13:55:19] The text table row 331925 got deleted during this migration [13:56:25] zabe: do you think it's possible that deletion happened mid-dump, so like by the time dumps got around to reading it it wasn't there anymore? [13:59:28] I am not really sure what your bug is, but at least for normal prod this shouldn't be an issue. The text table row is referenced in the content_address field of the content table. Prior to the migration its value was 'tt:331925' and the corresponding text table row then pointed to the external storage. Now content_address directly points to external storage. But it certainly could be the case that the dump and the migration collided, e.g. the dump [13:59:28] fetched 100 rows of the content table and then fetched the corresponding text table rows one by one, but the migration touched those meanwhile (to be clear, I do not really know how dumps work). [14:04:42] But I need to say this migration only started 4 weeks ago, so if your issue is older, then this is not the cause for your dumps bug, but only for the failure you see following https://www.mediawiki.org/wiki/SQL/XML_Dumps/Debugging_PHP [14:15:06] (btw. I known I only explained why fetching tt:331925 does not work now, not why dumps tries to do it) [14:29:37] (btw from above, https://phabricator.wikimedia.org/T378603 filed by Xabriel for the blank rev_timestamps) [14:30:28] no, that lines up exactly, zabe, the issues started about 4 weeks ago [14:31:50] it makes sense, my vague understanding of dumps tells me your theoretical "fetch 100 -> migration modifies -> dumps errors" scenario could be pretty close to reality [14:37:45] ok, but in https://phabricator.wikimedia.org/T377594#10261898 you wrote that wikidatawiki is affected. For wikidatawiki, however, I only started the migration afterwards (on October 26 or 27). [15:22:35] Ben wrote that, I think there was some misunderstanding. Wikidatawiki has its own issues but I didn't see a significant number of these kinds of failures there. [15:22:46] There were lots on enwiki, frwiki, and a bunch on others like itwiki [15:22:58] @zabe: when do you think the migration will be done? [17:22:41] Amir1: any guess on when this migration ^ will finish? Right now the guess is that we can't run dumps until its database is done with the migration [17:24:12] milimetric: so the thing is that it shouldn't even try to load tt: entries. It will take a couple of months I'm sure [17:25:37] the error message in the logs is like this: `getting/checking text tt:331925 failed` [17:25:57] all access must go through SqlBlobStore (or a class like that) [17:26:02] so somehow it's getting the old ids. Got some time to talk db cluster topology with me to figure this out? [17:26:03] that encapsulate it [17:26:17] milimetric: sure. Where? [17:26:22] to the batcave! [17:26:27] https://bit.ly/a-batcave