[00:13:20] 10DBA, 10Operations, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3666645 (10jrbs) >>! In T174370#3666568, @Reedy wrote: > Just needs scheduling, patches rebasing and someone to deploy it And, uh. How does one do all of thes... [00:14:00] 10DBA, 10Operations, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3666646 (10Reedy) >>! In T174370#3666645, @jrbs wrote: >>>! In T174370#3666568, @Reedy wrote: >> Just needs scheduling, patches rebasing and someone to deploy... [00:18:27] 10DBA, 10Operations, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3559677 (10Dzahn) @jrbs: re: patches rebasing: git review -d ; git rebase -i origin/master; then if it fails "git rebase --continu... [00:20:18] 10DBA, 10Operations, 10Patch-For-Review, 10Wiki-Setup (Create): Create elections committee private wiki - https://phabricator.wikimedia.org/T174370#3666654 (10Dzahn) added the mobile domain to DNS: https://electcom.m.wikimedia.org/ [14:09:20] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 8 others: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3667090 (10jcrespo) p:05High>03Unbreak! a:03jcrespo... [14:33:16] jynus_: I think, https://github.com/wikimedia/mediawiki-extensions-Wikidata/commits/master/extensions/Wikibase/client/includes/Changes/InjectRCRecordsJob.php [14:33:26] It appeared back in July [14:33:38] Which is about right for ~3 months [14:36:26] But that's only actually a refactoring.. [14:36:26] https://github.com/wikimedia/mediawiki-extensions-Wikibase/commit/2987fe5bcf6223d03835670f7a6832e6a41a51ef#diff-486c6a731894db38fa8ad29967b6df2e [14:42:42] lol [15:37:47] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 8 others: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3667147 (10Ladsgroup) I'm around to help anything about t... [15:38:17] Reedy: okay, how we can fix this :D [15:38:31] Amir1: I'm really not sure [15:38:46] It needs disabling, at least till we get things under control [15:40:35] it will make debugging super difficult, I think this should be done in a way that can keep the situation in one small wiki and see what's going on [15:41:06] That would probably work [15:41:16] At least, a temporary config variable would allow that [17:33:32] Reedy, do you support my fist on the table? [17:33:51] I complained about 400GB revision table before [17:33:53] this is worse [17:34:01] jynus_: Yeah. We can't let this take the wikis down [17:34:12] speciall worrying is the disk space [17:34:13] It's a massive data growth [17:34:17] yeah [17:34:26] and it is not a definitive step [17:34:31] And even worse, if like you say, it wasn't run by the DBAs [17:34:33] we revert because something is wrong [17:34:50] then once thing are working we thing slowly a way to move forward [17:34:56] Reverting maybe easier said than done [17:35:02] I know [17:35:14] but worse case scenario, I create a job to purge rows :-) [17:35:19] heh [17:35:28] hope I do not have to reach that :-) [17:36:06] disconnecting, I just connected to fix a related rc breakage on dbstore1001 [17:36:38] Ideally, we need to get the wikidata guys to identify what we need to turn off [17:36:50] I think it is not wikidata [17:36:57] but who knows [17:37:11] wikidata as wikidata team [17:37:34] I think it was some functionality by community tech [17:37:42] also, we only need to disable it on s4 and s6 [17:37:48] maybe cebwiki [17:37:56] the rest seem to be okish for now [17:38:01] Oh? [17:38:07] I thought it was wikidata at fault... [17:38:16] Do you have some example RC rows so I can have a look around? [17:38:17] yes, the wikidata data [17:38:32] but not sure they created the new functionality [17:38:45] the worse cases is commonswiki and ruwiki [17:38:53] where 90% of the rcs are now wikidata [17:39:08] which happen to be the pathological ones [17:39:15] even when pooled our fasters servers [17:39:39] after all, rc table is supposed to be light table to do bad queries [17:39:47] instead of doing them on the full revision table [17:39:51] plus denormalized [17:40:01] if rcs are as large as revision, it makes not much sense [17:41:34] as an example of practical problems- our backup server got an index corruption [17:41:36] on rc [17:41:49] now I have to recreate the index- not a big deal [17:42:06] except that the table is 400GB in data and it will take days [17:42:14] huge fail [17:42:34] then people complain our schema change take too much time :-D [17:42:49] heh [17:44:17] sorry, but I cannot stand users getting impacting if it is avoidable [17:44:45] they give their hard work, the least we can do is to serve them the best we can [17:47:13] I've got a feeling just killing the wikibase-InjectRCRecords job might be enough to stop all the crap going into the table for now [17:48:03] feel free to comment with that [17:48:14] killing it on ruwiki and commons will be an inprovement [17:48:27] comment on the ticket [17:48:53] s3 size incresed in the last 3 months from 700GB to 1.8 TB [17:49:00] probably related to that :-( [17:49:14] That's very unsustainable [17:49:18] yep [17:49:26] 800 wikis, it is not linear [17:49:27] Of course... as we don't keep RC long... It should plataeu, eventually [17:49:31] it is quadratic [17:49:37] but it was not the only reason [17:49:38] Until more wikidata changes happen [17:49:50] we can have the same data if it is necesary [17:49:57] but on a separate table or something [17:50:05] or not replicated 20 times per replica set [17:50:14] there are many way to make it work [17:50:38] if only people talked to us beforehand! :-) [17:51:17] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 8 others: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3667193 (10Bawolff) >>! In T171027#3667090, @jcrespo wrot... [17:52:26] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 8 others: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3667194 (10jcrespo) We can do it on commons and ruwiki, w... [17:53:23] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 8 others: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3667195 (10Reedy) I believe, the `wikibase-InjectRCRecord... [17:53:33] Oh, bawolff reckons this is a setting... [17:53:34] $wgWBClientSettings['injectRecentChanges'] [17:54:05] note this is unbreaknow- let's merge on monday [17:54:15] not unbreaknow- let's do it on the weeked [17:54:31] so we can wait a couple of days, just not sustainable for weeks [17:55:07] so no need to do hurry deploys [17:55:20] but I would not wait longer than monday [17:55:33] disconnecting, see you next week [17:56:07] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 8 others: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3667196 (10Reedy) >>! In T171027#3667193, @Bawolff wrote:... [18:02:45] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 8 others: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3667197 (10Reedy) https://github.com/wikimedia/mediawiki-... [18:03:46] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 8 others: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3667198 (10Ladsgroup) Yeah, I'm building a new one, hold on [18:12:03] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 9 others: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3667201 (10Reedy) I've done this as a revert in https://g... [18:14:35] Reedy: Did you see my patch? https://gerrit.wikimedia.org/r/#/c/382990/ [18:14:56] I did [18:15:04] But it seems cleaner to just revert the commits that removed it [18:16:23] And cleanup the bugs that were removed at the point of reverting [18:16:26] yeah I was worried it might interfere with some old config vars [18:16:28] *removal [18:16:44] so I introduced a new config one [18:16:50] but meh [18:23:26] Reedy: who is going to deploy it? me or you? [18:23:39] we have to deploy a config change as well, yay [18:23:50] jynus said it's fine to wait till monday [18:25:41] oh okay [18:26:03] We'd be aswell just disabling it everywhere, and stop it putting anything into recent changes, and letting all the rows disappear [18:26:26] Based on what he's said, this stuff isn't gonna be going back into RC unless in a much different format anyway [18:26:52] [18:49:50] we can have the same data if it is necesary [18:26:52] [18:49:57] but on a separate table or something [18:26:52] [18:50:04] or not replicated 20 times per replica set [18:26:52] [18:50:14] there are many way to make it work [18:26:52] [18:50:38] if only people talked to us beforehand! :-) [18:42:02] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 9 others: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3667221 (10Ladsgroup) Let's merge and deploy yours in Monday [22:29:45] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 9 others: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3556549 (10Lydia_Pintscher) @jcrespo I'm sorry. I wasn't... [22:31:50] 10DBA, 10Commons, 10Contributors-Team, 10MediaWiki-Watchlist, and 9 others: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis - https://phabricator.wikimedia.org/T171027#3667462 (10Lydia_Pintscher)