[07:17:06] 10DBA, 06Operations, 10ops-codfw: db2035: RAID disk about to fail - https://phabricator.wikimedia.org/T150511#2788294 (10Marostegui) [07:25:29] 10DBA, 13Patch-For-Review: codfw: Fix S4 commonswiki.templatelinks partitions - https://phabricator.wikimedia.org/T149079#2788309 (10Marostegui) db1068 is finished ``` MariaDB PRODUCTION s4 localhost commonswiki > show create table templatelinks\G *************************** 1. row ***************************... [09:50:25] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2788425 (10Marostegui) The data copy finished and after running mysql_upgrade I have started replication and the slaves are catching up nicely with the master. I forgot to include the RAID config... [09:51:21] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2788426 (10Marostegui) @Papaul the disks still need to be wiped, is that something you can do or something we have to do? I will leave this ticket open until you let us know. Thanks [09:58:29] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2788432 (10Marostegui) @Papaul let me know when you want/could proceed and upgrade its firmware Thank you! [10:05:13] 10DBA: Import S5 to dbstore2001 and dbstore2002 + compression - https://phabricator.wikimedia.org/T150518#2788434 (10Marostegui) [10:19:05] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search (Current work), 13Patch-For-Review: CirrusSearch SQL query for locating pages for reindex performs poorly - https://phabricator.wikimedia.org/T147957#2788463 (10dcausse) @jcrespo thanks for the investigation, I agree with you and Erik this tool is ra... [11:04:39] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search (Current work), and 2 others: MySQL chooses poor query plan for link counting query - https://phabricator.wikimedia.org/T143932#2788508 (10dcausse) Thanks! We reverted the code that uses the db for this process and switched back to elastic to compute... [12:25:14] 10DBA, 13Patch-For-Review: Unify commonswiki.revision - https://phabricator.wikimedia.org/T147305#2788668 (10Marostegui) dbstore1002 is done ``` MariaDB DBSTORE localhost commonswiki > show create table revision\G *************************** 1. row *************************** Table: revision Create Tab... [12:44:46] 10DBA, 13Patch-For-Review: Unify commonswiki.revision - https://phabricator.wikimedia.org/T147305#2788690 (10Marostegui) Summary of what is pending after this week: Needed a PK change + indexes: labsdb1001 labsdb1003 ``` ./software/dbtools/osc_host.sh --host=xxx --port=3306 --db=commonswiki --table=revision... [13:42:22] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search (Current work), and 2 others: MySQL chooses poor query plan for link counting query - https://phabricator.wikimedia.org/T143932#2788835 (10dcausse) Sorry ignore the above gerritbot comment, wrong bug id. [14:18:14] 10DBA, 10CirrusSearch, 06Discovery, 06Discovery-Search (Current work), and 2 others: MySQL chooses poor query plan for link counting query - https://phabricator.wikimedia.org/T143932#2788885 (10dcausse) Avg time for this cirrus job decreased from an unstable ~250ms to ~100ms by using elastic: https://graph... [14:25:52] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2788886 (10chasemp) Are these https://gerrit.wikimedia.org/r/#/c/320804/2/modules/role/templates/labsdb/maintain-views.yam... [16:38:51] 10DBA, 13Patch-For-Review: Reimage dbstore2001 as jessie - https://phabricator.wikimedia.org/T146261#2789018 (10Marostegui) [16:38:53] 10DBA, 06Operations, 10ops-codfw: install new disks into dbstore2001 - https://phabricator.wikimedia.org/T149457#2789017 (10Marostegui) [16:53:39] 10DBA, 06Labs, 10Labs-Infrastructure, 07Availability: Decide between proxysql and haproxy for labsdbproxy service - https://phabricator.wikimedia.org/T149844#2789064 (10chasemp) Where I'm at after thinking on it for a bit: * We have the ecosystem to effectively standup HAProxy here (monitoring, operating... [18:18:41] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2789166 (10AlexMonk-WMF) >>! In T148561#2788886, @chasemp wrote: > Are these https://gerrit.wikimedia.org/r/#/c/320804/2/m... [19:13:44] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2789262 (10chasemp) sure, thanks, I understand. I meant something more like: is this a table to be exposed on a single wi... [19:37:10] 10DBA, 06Labs: Reduce watchlist_count threshold - https://phabricator.wikimedia.org/T150548#2789278 (10Dispenser) [21:53:35] 10DBA, 06Labs, 10Labs-Infrastructure, 07Availability: Decide between proxysql and haproxy for labsdbproxy service - https://phabricator.wikimedia.org/T149844#2766141 (10bd808) Having to keep two authentication systems in sync seems like something that will break and consume a lot of operational debugging t... [22:20:59] 10DBA, 06Labs: High replication activity filled up labsdb1004 with binlogs - https://phabricator.wikimedia.org/T150553#2789420 (10fgiunchedi) [23:16:20] 10DBA, 06Labs, 10Tool-Labs-tools-Other: High replication activity filled up labsdb1004 with binlogs - https://phabricator.wikimedia.org/T150553#2789559 (10Volans) Just to clarify, this is because of row-based replication, for each 100MB relay log there was ~93k updated rows for the table `s51370__isbntool.nr... [23:23:51] 10DBA, 06Labs, 10Tool-Labs-tools-Other: High replication activity filled up labsdb1004 with binlogs - https://phabricator.wikimedia.org/T150553#2789567 (10Volans) To the `isbn` tool group: Looks like you have some issue in the queries you're performing because I can see that you're updating always the same ~...