[11:42:18] <marostegui>	 Amir1: It is partitioned on the rc slaves as far as I know, let me check
[11:43:08] <marostegui>	 Amir1: Yeah, it is partitioned on the rc slaves (like every big wiki)
[12:04:12] <Amir1>	 marostegui: it's used to be 100 times bigger, back then it made sense, now if you want to revisit it, it'd be great. It's 6m rows only
[12:06:34] <marostegui>	 Amir1: As far as I know logging table is partitioned on tables.sql for the special slaves?
[12:06:43] <marostegui>	 Not sure about it
[12:07:02] <marostegui>	 Amir1: It is now consistent in the sense that enwiki rc slaves is also partitioned
[12:07:47] <wikibugs>	 10DBA: replication broken on db1124:3318 on wikidata.pagelinks - https://phabricator.wikimedia.org/T209521 (10Marostegui) This has happaned today again with `pagelinks`. I have had to add a few rows and delete multiple rows after it that were giving duplicate entry for another pl_from value. I have waited until...
[12:10:34] <wikibugs>	 10DBA: Recheck wikidatawiki.pagelinks table across all hosts in s8 - https://phabricator.wikimedia.org/T212574 (10Marostegui) p:05Triage→03High
[12:17:44] <Amir1>	 marostegui: I see. Thanks
[12:21:02] <Amir1>	 ROW-based replications break all the time :D
[12:25:42] <marostegui>	 I know…but it is a good thing actually
[12:25:53] <marostegui>	 Cause it ensures consistency
[12:31:22] <Amir1>	 Since you're planning to move everything to row-based replication, I think your work will be a lot :( I hope this gets better
[12:31:49] <wikibugs>	 10DBA: Recheck wikidatawiki.pagelinks table across all hosts in s8 - https://phabricator.wikimedia.org/T212574 (10Marostegui) a:03Marostegui
[12:32:11] <marostegui>	 Amir1: Nah, there are many blockers we have to solve to move to row based
[12:32:22] <marostegui>	 One of them, schema changes
[12:32:32] <marostegui>	 And another one, yes, this consistency issues
[12:32:54] <marostegui>	 We are in a lot better state now, I believe this problem with pagelinks is due to the s8 issues we had when we moved back from codfw
[12:33:14] <marostegui>	 We actually ran s5 (by mistake) with row based replication for a few months and it went very well actually
[12:33:41] <Amir1>	 fingers crossed, I hope I can help in any case. Regarding automation of schema changes, I really love to see it
[12:33:42] <marostegui>	 But with row based, adding or dropping columns would need to be done directly on the master, with replication, or otherwise replication breaks
[12:33:45] <marostegui>	 And that is a no go
[12:33:55] <marostegui>	 We cannot run alters on the master and let them replicate
[12:34:07] <marostegui>	 That would generate massive lag
[12:34:21] <Amir1>	 oh yes, what's the plan for that then?
[12:34:36] <marostegui>	 Not move to row based yet :)
[12:34:52] <marostegui>	 Mariadb will have instand add/drop column in the next releases
[12:34:57] <marostegui>	 So that will probably unblock tht
[12:35:12] <marostegui>	 https://mariadb.com/kb/en/library/instant-add-column-for-innodb/
[12:35:26] <marostegui>	 On 10.3, we are on 10.1, but we will most likely go from 10.1 to 10.3 and won't go thru 10.2
[12:35:37] <Amir1>	 Nice
[12:35:50] <marostegui>	 That needs lots of testing before we will be brave enough to run that directly on a master XD
[12:35:53] <Amir1>	 That would make schema changes really quick <3 <3
[12:36:01] <marostegui>	 Indeed!
[12:36:40] <marostegui>	 This is also a nice one from 10.3: https://mariadb.com/kb/en/library/invisible-columns/
[12:37:02] <marostegui>	 And this I would love mariadb to implement sooner rather than later: https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.html
[12:37:38] <Amir1>	 Maybe migrating to MySQL :D
[12:38:00] <marostegui>	 Not totally discarded! :p
[12:38:20] <marostegui>	 https://phabricator.wikimedia.org/T193226
[12:38:21] <marostegui>	 :)
[12:40:05] <marostegui>	 I am going to get some food!
[12:41:35] <Amir1>	 Have fun!
[12:41:40] <Amir1>	 Happy Holidays 
[14:06:20] <wikibugs>	 10DBA, 10Wikimedia-Site-requests: Remove 'editusercssjs' and 'sendemail-new-users' from global groups - https://phabricator.wikimedia.org/T203464 (10Framawiki) Database action required per task desc.
[15:57:22] <wikibugs>	 10DBA: Recheck wikidatawiki.pagelinks table across all hosts in s8 - https://phabricator.wikimedia.org/T212574 (10Marostegui)
[17:08:38] <wikibugs>	 10DBA: Recheck wikidatawiki.pagelinks table across all hosts in s8 - https://phabricator.wikimedia.org/T212574 (10Marostegui) I have run the following check for `pagelinks`: `./compare.py wikidatawiki pagelinks pl_from db1087 db1071 db1092 db1099:3318 db1101:3318 db1104 db1109  db1116:3318` Basically db1087 (san...
[18:25:45] <wikibugs>	 10DBA: Recheck wikidatawiki.pagelinks table across all hosts in s8 - https://phabricator.wikimedia.org/T212574 (10Marostegui) The check between db1087 and db1124:3318 reported multiple differences for `pagelinks` ` Execution ended, a total of 301 chunk(s) are different. `  We could try to manually fix those, but...