[06:00:05] 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s4 - https://phabricator.wikimedia.org/T166206#3297449 (10Marostegui) db1097 is now done: ``` root@neodymium:/home/marostegui# for i in `cat s4_tables`; do echo $i; mysql --skip-ssl -hdb2019.c... [06:00:20] 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s4 - https://phabricator.wikimedia.org/T166206#3297450 (10Marostegui) [06:14:15] 10DBA, 10Analytics, 10Analytics-EventLogging: db1047 has been restarted - needs another restart - https://phabricator.wikimedia.org/T166452#3297456 (10Marostegui) I have restarted it and able to set the replication filter on the s1 channel: ``` Replicate_Wild_Ignore_Table: enwiki.__wmf_checksums ``` I am go... [06:24:48] 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: Unify revision table on s3 - https://phabricator.wikimedia.org/T166278#3297461 (10Marostegui) [06:57:51] ack dbstore1002 [06:58:19] oh yes [06:58:22] will do right now [06:58:54] sorry [06:59:56] I am going to stop db1095 and labsdb1009 to take a backup before starting the process of importing s5 [06:59:59] just in case [07:00:31] actually only db1095 for now [07:27:10] 10DBA, 06Operations, 13Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3297517 (10Marostegui) [07:27:12] 10DBA, 06Operations, 10ops-eqiad: Decommission db1023 - https://phabricator.wikimedia.org/T166486#3297505 (10Marostegui) [07:55:10] 10DBA, 06Community-Tech, 10MediaWiki-User-management: Do test queries for range contributions to gauge performance of using different tables - https://phabricator.wikimedia.org/T156318#3297550 (10jcrespo) For the query you are doing, there are 2 main indexes, ipc_hex and ipc_rev_timestamp. When one index get... [08:28:26] https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=codfw%20prometheus%2Fops&var-server=db2049&from=1495793965043&to=1495974377121 [08:29:39] Not bad! [08:30:12] We should increase the binlog_cache_size [08:33:12] binlog_cache_size | 1048576 [08:33:16] what do you have in mind? [08:36:10] 10DBA, 06Operations, 10ops-eqiad, 13Patch-For-Review: Decommission db1023 - https://phabricator.wikimedia.org/T166486#3297610 (10Marostegui) a:03Cmjohnson [08:36:33] 10DBA, 06Operations, 10ops-eqiad, 13Patch-For-Review: Decommission db1023 - https://phabricator.wikimedia.org/T166486#3297505 (10Marostegui) This is ready for @Cmjohnson to take over whenever he can [09:01:56] 10DBA, 06Reading-Web-Backlog: Drop Gather tables from wmf wikis - https://phabricator.wikimedia.org/T166097#3297666 (10Marostegui) 05Open>03Resolved a:03Marostegui I have removed the tables from production as after almost a week there were no issues with replication/log errors on the renamed hosts. [09:01:58] 10DBA, 07Epic, 07Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3297669 (10Marostegui) [11:14:36] 10DBA, 06Analytics-Kanban, 06Operations, 10ops-eqiad: Degraded RAID on db1046 - https://phabricator.wikimedia.org/T166422#3298035 (10Volans) [11:16:15] 10DBA, 06Analytics-Kanban, 06Operations, 10ops-eqiad: Degraded RAID on db1046 - https://phabricator.wikimedia.org/T166422#3298040 (10Marostegui) @elukey @Ottomata maybe this can be done along with: T166141 [11:17:45] 10DBA, 06Analytics-Kanban, 06Operations, 10ops-eqiad: Degraded RAID on db1046 - https://phabricator.wikimedia.org/T166422#3298063 (10elukey) +1 [11:23:15] forgot to add DBA to this one --^, volans <3 [11:23:27] yw :) [13:57:50] 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: Unify revision table on s3 - https://phabricator.wikimedia.org/T166278#3298321 (10Marostegui) [15:18:53] 10DBA, 10Analytics, 10Analytics-EventLogging: db1047 has been restarted - needs another restart - https://phabricator.wikimedia.org/T166452#3296553 (10Nuria) This is a slave machine that is not used. [15:20:38] 10DBA, 10Analytics, 10Analytics-EventLogging: db1047 has been restarted - needs another restart - https://phabricator.wikimedia.org/T166452#3298512 (10Marostegui) Hey Nuria! But we still need to maintain it, right? As in, it is not going to be decommissioned soon but it is a backup server just in case? Tha... [15:22:20] 10DBA, 10Analytics, 10Analytics-EventLogging: db1047 has been restarted - needs another restart - https://phabricator.wikimedia.org/T166452#3296553 (10Ottomata) @marostegui, correct. [16:16:49] 10DBA, 06Labs, 13Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3298667 (10Marostegui) After transferring all the ibd and cfg files and run md5sum over them I am now importing wikidatawiki tablespace on db1095. Once it... [16:17:31] 10DBA, 06Operations: Adapt wmf-mariadb101 package for stretch and adapt its service to systemd - https://phabricator.wikimedia.org/T116903#3298671 (10jcrespo) a:03jcrespo [16:17:44] 10DBA, 06Operations: Adapt wmf-mariadb101 package for stretch and adapt its service to systemd - https://phabricator.wikimedia.org/T116903#1761444 (10jcrespo) p:05Low>03Normal [16:18:24] 10DBA, 06Operations: Adapt wmf-mariadb101 package for stretch and adapt its service to systemd - https://phabricator.wikimedia.org/T116903#1761444 (10jcrespo) we got some progress: {P5501} [16:19:16] ^ [16:19:21] very nice [16:19:52] that is only for creating the package conditionally for stretch, now the systemd unit has to work [16:59:41] 10DBA, 06Operations: Adapt wmf-mariadb101 package for stretch and adapt its service to systemd - https://phabricator.wikimedia.org/T116903#3298736 (10jcrespo) Some more progress, after removing galera stuff: P5501#29701 [17:46:03] 10DBA, 06Operations, 10ops-eqiad: Degraded RAID on db1094 - https://phabricator.wikimedia.org/T166518#3298788 (10Volans) [20:05:11] Cyberpower678: I'd suggest to split it into two in the software and then merge the result. Also remember that MySQL can use only one index per query (per table, apart some particular cases)