[04:49:33] 10DBA, 10ArchCom-RfC, 10MediaWiki-Database, 07RfC: Should we bump minimum supported MySQL Version? - https://phabricator.wikimedia.org/T161232#3125897 (10tstarling) >>! In T161232#3150649, @Reedy wrote: > I'm not sure we need an RfC (maybe we do?) but ArchCom guys should have a say for sure Yes, I think i... [05:38:55] 10DBA, 06Labs, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3159545 (10Marostegui) db1070 has been compressed. [06:04:06] 10DBA, 13Patch-For-Review: Defragment s4: db1091, db1084, db1081, d1059 and probably the rest - https://phabricator.wikimedia.org/T161088#3159566 (10Marostegui) db1081 has been defragmented and migrated to file per table. It is now catching up. [06:06:19] 10DBA, 13Patch-For-Review: Unify revision table on s7 - https://phabricator.wikimedia.org/T160390#3159567 (10Marostegui) db2047 is done: ``` root@neodymium:/home/marostegui# for i in `cat s7_T160390`; do echo $i; mysql --skip-ssl -hdb2047.codfw.wmnet $i -e "show create table revision\G" | egrep "KEY";done arwi... [06:09:03] 10DBA, 13Patch-For-Review: Unify revision table on s7 - https://phabricator.wikimedia.org/T160390#3159568 (10Marostegui) db2040, the rc slave already looks like a rc slave, so no need to alter it: ``` root@neodymium:/home/marostegui# for i in `cat s7_T160390`; do echo $i; mysql --skip-ssl -hdb2040.codfw.wmnet... [06:17:50] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: setup tempdb2001(WMF6407) - https://phabricator.wikimedia.org/T162290#3159572 (10Marostegui) Thanks Rob! We will take it from here! [06:54:27] 10DBA, 10ArchCom-RfC, 10MediaWiki-Database, 07RfC: Should we bump minimum supported MySQL Version? - https://phabricator.wikimedia.org/T161232#3159587 (10jcrespo) Two comments: * utf8mb4 only is up to date to the latest unicode standards on the latest MySQL versions. Not 100% sure I would recommend utf8mb4... [07:13:23] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: setup tempdb2001(WMF6407) - https://phabricator.wikimedia.org/T162290#3159628 (10Marostegui) a:03Marostegui [07:46:43] 10DBA: How many revision comments are exactly the same? Get some stats. - https://phabricator.wikimedia.org/T162138#3159663 (10jcrespo) So I created: ``` root@dbstore2002[ops]> SHOW CREATE TABLE comment_summary2\G *************************** 1. row *************************** Table: comment_summary2 Crea... [09:33:28] 10DBA: convert dbstore1001 to InnoDB compressed by importing db shards to it - https://phabricator.wikimedia.org/T159430#3159890 (10Marostegui) So we were hitting https://jira.mariadb.org/browse/MDEV-9027 with the table locks. After switching the tables to InnoDB and running the same query(s), the server doesn't... [09:50:52] 10DBA: convert dbstore1001 to InnoDB compressed by importing db shards to it - https://phabricator.wikimedia.org/T159430#3159944 (10jcrespo) I agree with that not being a bug, but more reasons to go for gap-based locking on InnoDB in repeatable read mode, at least until we have ROW. [11:36:21] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup first 10 DB servers - https://phabricator.wikimedia.org/T162159#3160117 (10Marostegui) [12:17:48] 10DBA, 10ArchCom-RfC, 10MediaWiki-Database, 07RfC: Should we bump minimum supported MySQL Version? - https://phabricator.wikimedia.org/T161232#3160189 (10Reedy) Any advances on 5.5.8? We seem to have generally decided that the 5.5 branch is where we want to be (at least, to take forward to the RfC stage).... [12:24:16] 10DBA, 10ArchCom-RfC, 10MediaWiki-Database, 07RfC: Should we bump minimum supported MySQL Version? - https://phabricator.wikimedia.org/T161232#3160226 (10MoritzMuehlenhoff) All distros with security support are updating to the latest 5.5.x release (since Oracle doesn't release enough information to backpor... [12:34:29] 10DBA, 10ArchCom-RfC, 10MediaWiki-Database, 07RfC: Should we bump minimum supported MySQL Version? - https://phabricator.wikimedia.org/T161232#3160239 (10Reedy) 5.5.48 would be the first release of 2016, and 5.5.54 (and as of writing, currently released version) would be the last. None in 2017 yet Conside... [12:52:39] 10DBA, 10ArchCom-RfC, 10MediaWiki-Database, 07RfC: Should we bump minimum supported MySQL Version? - https://phabricator.wikimedia.org/T161232#3160257 (10jcrespo) I think support and recommend are different stories. The recommendation is always to run the latest or some of the latest versions within a majo... [12:58:57] 10DBA, 10ArchCom-RfC, 10MediaWiki-Database, 07RfC: Should we bump minimum supported MySQL Version? - https://phabricator.wikimedia.org/T161232#3160264 (10Marostegui) I believe we should not recommend but support, and for that I would say we support 5.5.8 (or the first GA). There is nothing stopping us from... [13:26:49] wasn't there a programmed deployment or something today? [13:28:27] 10DBA, 13Patch-For-Review: Unify revision table on s7 - https://phabricator.wikimedia.org/T160390#3160295 (10Marostegui) codfw master (db2029) is done, so the whole codfw DC is now done: ``` root@neodymium:/home/marostegui# for i in `cat s7_T160390`; do echo $i; mysql --skip-ssl -hdb2029.codfw.wmnet $i -e "sho... [13:34:16] the cronjob thingy no? [13:34:26] it was supposed to be between 14:00 and 16:00 [13:34:29] that is what he said [13:35:27] ah, so in half an hour [13:36:27] yeah, he mentioned between 14:00 and 16:00 [13:36:29] so we will see [13:36:35] don't know if he meant UTC :) [13:37:24] https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170406T1400 [13:37:42] then he did :) [13:59:38] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: setup tempdb2001(WMF6407) - https://phabricator.wikimedia.org/T162290#3160374 (10Marostegui) tempdb2001 is now replicating from db2033. It is running 10.0.30 (mysql_upgrade has been run), SSL is enabled. Right now it is quite delayed: ``` Seconds_Behi... [14:00:31] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: setup tempdb2001(WMF6407) - https://phabricator.wikimedia.org/T162290#3160378 (10Marostegui) [14:00:49] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: setup tempdb2001(WMF6407) - https://phabricator.wikimedia.org/T162290#3160379 (10jcrespo) Cool! We need to let it replicate for a while before pooling it to confirm is is ok. [14:01:25] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: setup tempdb2001(WMF6407) - https://phabricator.wikimedia.org/T162290#3160381 (10Marostegui) >>! In T162290#3160379, @jcrespo wrote: > Cool! We need to let it replicate for a while before pooling it to confirm is is ok. Yes! I will leave the ticket op... [14:45:38] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: setup tempdb2001(WMF6407) - https://phabricator.wikimedia.org/T162290#3160496 (10Marostegui) Too early to say, but we might have IO problems on this host: {F7309725} If I `set global innodb_flush_log_at_trx_commit = 0;` the server starts to catch up... [14:50:55] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: setup tempdb2001(WMF6407) - https://phabricator.wikimedia.org/T162290#3160517 (10jcrespo) No hardware RAID like the other servers- we need to set innodb_flush_log_at_trx_commit = 0; sync_binlog=0; and other stuff to reduce syncronous IO. If it goes dow... [15:09:13] 10DBA, 10Analytics-EventLogging, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Improve eventlogging replication procedure - https://phabricator.wikimedia.org/T124307#3160560 (10Nuria) [15:09:13] 10DBA, 10Analytics-EventLogging, 06Analytics-Kanban, 05WMF-NDA: Drop tables with no events in last 90 days. - https://phabricator.wikimedia.org/T161855#3145199 (10Nuria) 05Open>03Resolved [15:11:34] 10DBA, 10Analytics-EventLogging, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Improve eventlogging replication procedure - https://phabricator.wikimedia.org/T124307#1952524 (10Nuria) 05Open>03Resolved [15:49:59] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup first 10 DB servers - https://phabricator.wikimedia.org/T162159#3160661 (10Papaul) @Marostegui we can not do a7 and b7 because we have 10G switch in a7 and b7 or the serves have only 1GB NIC's. Please relocate db2091 and db2092. Thanks. [15:51:18] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup first 10 DB servers - https://phabricator.wikimedia.org/T162159#3160662 (10Marostegui) [15:51:51] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup first 10 DB servers - https://phabricator.wikimedia.org/T162159#3154083 (10Marostegui) >>! In T162159#3160661, @Papaul wrote: > @Marostegui we can not do a7 and b7 because we have 10G switch in a7 and b7 or the serves have only 1GB NIC... [16:11:06] 10DBA, 13Patch-For-Review: Unify revision table on s7 - https://phabricator.wikimedia.org/T160390#3160716 (10Marostegui) db1079 is done: ``` root@neodymium:/home/marostegui# for i in `cat s7_T160390`; do echo $i; mysql --skip-ssl -hdb1079.eqiad.wmnet $i -e "show create table revision\G" | egrep "KEY";done arwi... [16:21:19] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup first 10 DB servers - https://phabricator.wikimedia.org/T162159#3160759 (10Papaul) @Marostegui db2089 needs to move as well. [16:35:16] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup first 10 DB servers - https://phabricator.wikimedia.org/T162159#3160833 (10Marostegui) [16:35:29] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup first 10 DB servers - https://phabricator.wikimedia.org/T162159#3154083 (10Marostegui) >>! In T162159#3160759, @Papaul wrote: > @Marostegui db2085 needs to move as well. Updated [16:42:33] 10DBA, 06Operations, 10ops-codfw, 13Patch-For-Review: codfw rack/setup first 10 DB servers - https://phabricator.wikimedia.org/T162159#3160873 (10Papaul) @Marostegui Just for your information asw-a2-codfw asw-a7-codfw asw-b2-codfw asw-b7-codfw asw-c2-codfw asw-c7-codfw asw-d2-codfw asw-d7-codfw are 1... [18:40:01] 10DBA, 13Patch-For-Review: run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038) - https://phabricator.wikimedia.org/T154485#3161379 (10jcrespo) Differences on dbstore1002: ``` itwiki: --where "up_user BETWEEN 0 AND 1009999" nowiki: --where "up_user BETWE... [21:45:38] 10DBA, 06Labs, 13Patch-For-Review, 07Regression: Tool Labs: Add skin, language, and variant to user_properties_anon - https://phabricator.wikimedia.org/T152043#3161912 (10chasemp) >>! In T152043#3152818, @Andrew wrote: > I merged the puppet change, but maybe this needs to be run by hand -- I've never done...