[06:48:54] 10DBA: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#3035718 (10Marostegui) The dbstore2001 issue never happened again. It only happened once and I tested pt-table-checksum several times during the day so, if it was something in particular i... [07:02:32] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: Change rack for servers in s1 in codfw - https://phabricator.wikimedia.org/T156478#3035728 (10Marostegui) db2070 has been repooled. Thanks everyone for the help to move all these three servers! [07:02:43] 10DBA, 07Epic: Meta DBA ticket for the DC switchover - https://phabricator.wikimedia.org/T155099#3035730 (10Marostegui) [07:02:46] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: Change rack for servers in s1 in codfw - https://phabricator.wikimedia.org/T156478#3035729 (10Marostegui) 05Open>03Resolved [08:36:17] 10DBA, 13Patch-For-Review: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418#3035817 (10Marostegui) TL;DR: gtid_domain_id + gtid works fine with multiple switchovers just using GTID. Note: this remains broken for multisource+gtid. We can go ahead and start deploying: htt... [10:56:03] 10DBA, 13Patch-For-Review: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#3035959 (10Marostegui) I have populated all the dsns tables, excluding db1069, db1095 and dbstore servers. Basically whatever is on the .hosts files but those hosts.... [13:50:03] 10DBA, 13Patch-For-Review: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#3036450 (10jcrespo) Not needed now, probably x1 will be needed. Also we should start thinking about merging the hosts lists into a dynamic table + maintaining that up... [14:42:39] 10DBA, 13Patch-For-Review: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#3036523 (10Marostegui) I am thinking about running this first iteration over s2 and a concrete database next week at some point: nlwiki. (s2) Why nlwiki? because it i... [14:49:49] 10DBA, 13Patch-For-Review: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485#3036527 (10jcrespo) We have semisync on core, I would expect much less impact, lag-wise. [15:17:16] 10DBA, 06Operations, 10ops-eqiad: Replace BBU for db1060 - https://phabricator.wikimedia.org/T158194#3036566 (10Marostegui) @Cmjohnson sorry to push, but were you able to see if there's a replacement BBU? I wouldn't like to leave the server with WriteBack forced without the BBU as we might lose data if there... [19:57:51] 10DBA, 10fundraising-tech-ops: fundraising database tuning - https://phabricator.wikimedia.org/T158446#3037278 (10Jgreen) [20:00:21] 10DBA, 10fundraising-tech-ops: fundraising database tuning - https://phabricator.wikimedia.org/T158446#3037294 (10Jgreen) 1) current, low load "show status" {F5676034} 2) "show variables" {F5676038} 3) my.cnf {F5676045} [23:07:56] 10DBA: enwiki database on db1065 missing rev_timestamp index - https://phabricator.wikimedia.org/T158454#3037574 (10EBernhardson)