[13:31:22] tgr|away: I meant to keep $wgUseSiteJs disabled on auth.wikimedia.org, that seems worthwhile if we dont' have a reason yet. [13:31:37] But I'll let you decide how we launch, can always revisit in the future. [15:44:36] Amir1: Are you aware of some low level limit around pagination existing around 10,000? I noticed that both https://commons.wikimedia.org/w/index.php?title=Special:PagesWithProp/wikibase_item&limit=50&offset=9990&namespace=14 and https://commons.wikimedia.org/w/index.php?title=Special:PagesWithProp/wikibase_item&limit=50&offset=9990 seem to imply there are exactly 10,000 results which seems suspicious. [15:44:47] I'm not seeing anything in Rdbms or QueryPage, so I'm guessing there's a limit elsewhere. [15:45:06] Context being that this would be a reasonable alternative to https://commons.wikimedia.org/wiki/Category:Uses_of_Wikidata_Infobox, given that each of those generates a pageprop anyway [15:45:11] https://commons.wikimedia.org/wiki/Special:ApiSandbox#action=query&format=json&prop=pageprops&titles=Category%3AViews_from_Miradouro_do_Convento_dos_Grilos&formatversion=2 [15:45:53] 10,000 is the limit for rows returned after rdbms transaction profiler emits warnings [15:46:19] Krinkle: it's the CirrusSearch search limit [15:46:21] I think it's a good limit, there are some problems with large chunk of results being returned from the server [15:46:37] ok, let me see if I can rule those two out. [15:46:44] elastic is not queried [15:47:01] trxProfiler is not triggered, it returns only 10 rows, limit of 50. [15:47:24] it is cursoring on the database side for offset 9990, limit 50 and not going beyodn result #10,000 [15:47:27] also QueryPage has a total row count limit I think? [15:47:48] $wgQueryCacheLimit [15:47:54] Found QueryPage::getMaxResults, that says 10K is the limit because some/most query pages are cached in the querycache table, which must have a limit of course. [15:48:24] However that shouldn't apply to simple query pages like PagesWithProp that search a table directly based on user input criteria. [15:49:10] Yeah, it's configurable via wgQueryCacheLimit, but default/prod is 10K indeed [15:49:19] hm, but the efficient way to query Special:PagesWithProp indefinitely would not be with a numerical index, right? [15:49:25] Or 5000 in wmf-config. [15:49:28] you’d want the offset to encode the last (or next) row somehow [15:49:35] // Max of 10000, unless we store more than 10000 in query cache. [15:49:35] return max( $this->getConfig()->get( MainConfigNames::QueryCacheLimit ), 10000 ); [15:49:54] maybe we should set limits to 10000+rnd(100), would be easier to identify which one is occurring [15:49:58] Right, it returns upto 10K, or more if you have a higher config, but never less. Well, it returns less for those that are cached and have less. [15:50:23] The limit is applied virtually to the query, so OFFSET 9999, LIMIT 50 is written to LIMIT 1 [15:52:01] Lucas_WMDE: yeah, that could easily use a more reasonable pager like page_id or page_ns/page_title like API conitnuation does already for large result sets. [15:52:04] but it doesn't right now [15:52:11] *nods* [15:52:25] and QueryPage might not be built to allow for that kind of continuation (for all I know) [15:55:04] Yeah, you'd want a different Pager to be used in the body of the special page [15:55:25] Looks like some of the API equivalants have been rewritten already but the UIs haven't. [15:55:32] e.g. Special:LinkSearch also uses a crude offset pager still [15:55:47] but the API uses scalable continuation IDs [15:55:57] ApiQueryExtLinksUsage [15:56:39] That might be good enough for what I need, since nobody is going to manually go through 6 million results in the UI [15:56:50] it's the API we care about [15:57:19] ApiQueryPagesWithProp uses a primary key offset as well [19:03:54] Amir1: I BOLD'ly invented a timeline for the Wikitech SUL leftovers and announced it to the mailing lists. I am going to be working on scripting the rename+attach+talk page notice steps after my lunch today. [19:33:03] Thank you so much! [19:36:11] with the migration of the same users with same emails, we got around 20% of the users done