[05:47:03] 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s2 - https://phabricator.wikimedia.org/T166205#3334652 (10Marostegui) [05:51:55] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s4 - https://phabricator.wikimedia.org/T166206#3334654 (10Marostegui) db1056 is done: ``` root@neodymium:/home/marostegui# for i in `cat s4_tables`; do echo $i; mysql --skip-ssl -hdb1056 commo... [05:52:22] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s4 - https://phabricator.wikimedia.org/T166206#3334655 (10Marostegui) [05:59:53] 10DBA, 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3334659 (10Marostegui) Please, have a plan B just in case this host doesn't come back up, it is a very old server and we know that sometimes, old servers once powered o... [06:40:24] 10DBA, 10Operations: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3334686 (10Marostegui) We need to test myloader using compressed tables, as during T153743 it took almost 24 hours to load: ``` root@labsdb1009:/srv/tmp/dewiki# du -sh * 37G e... [07:09:09] 10DBA, 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3334712 (10elukey) Thanks @Marostegui, I didn't think the situation was so desperate :D If there could be the risk of a bigger failure I'd change idea about the BBU an... [07:09:48] 10DBA, 10Operations: Investigate slow servermon updating queries on db1016 - https://phabricator.wikimedia.org/T165674#3273350 (10Marostegui) I know this is old but maybe this was affected by: T166344 and its performance degradation? [07:11:00] 10DBA, 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3334717 (10Marostegui) I don't want to be pessimistic, but I have had issues with old servers in the past, so just wanted to give a heads up to make sure you guys have... [07:18:20] 10DBA, 10Operations: Investigate slow servermon updating queries on db1016 - https://phabricator.wikimedia.org/T165674#3334734 (10akosiaris) I doubt that. Looking at tendril for the last 72 hours (>2 days before T166344 happened again), it's clear that this is happening around the time the `make_updates` cron... [07:19:26] 10DBA, 10Operations: Investigate slow servermon updating queries on db1016 - https://phabricator.wikimedia.org/T165674#3334735 (10Marostegui) Cool - just wanted to make sure we had that in mind :-) [08:03:48] 10DBA, 10Labs, 10Tool-Labs: s51127__dewiki_lists (merlbot) database using 13G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133325#2228208 (10Marostegui) Thanks @Luke081515 - we expanded the partitions some months ago so disk space isn't an issue here anymore, but it is always good to clean up... [08:05:19] 10DBA, 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: [Discuss] Hosting the monthly article quality dataset on labsDB - https://phabricator.wikimedia.org/T146718#3334831 (10Marostegui) Sounds good thank you! [08:16:57] 10DBA, 10MediaWiki-extensions-Babel, 10Wikimedia-General-or-Unknown: Create and populate babel database table on Wikimedia wikis - https://phabricator.wikimedia.org/T145366#3334849 (10Amire80) Are there any blockers for this? Just the //script that queries those pages, and then runs the refreshlinks code on... [08:19:13] 10DBA, 10MediaWiki-extensions-Babel, 10Wikimedia-General-or-Unknown: Create and populate babel database table on Wikimedia wikis - https://phabricator.wikimedia.org/T145366#2627711 (10Marostegui) Please ping the DBAs before running the script so we can be aware of this (also if this is a long running script... [11:24:55] 10DBA, 10Operations: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3335077 (10Marostegui) Looks like not having compression makes a big difference. On labsdb1011: 139G in 5 hours which is way faster than the other two hosts with dewiki (which... [13:06:32] 10DBA, 10Analytics, 10Analytics-EventLogging: Add index on event_action, event_isAnon and event_namespaceId to NavigationTiming tables - https://phabricator.wikimedia.org/T70396#3335198 (10Marostegui) Is this still needed? [13:06:44] 10DBA, 10Analytics, 10Analytics-EventLogging: Add index on event_action, event_isAnon and event_namespaceId to NavigationTiming tables - https://phabricator.wikimedia.org/T70396#3335199 (10Marostegui) a:05Springle>03None [13:34:19] 10DBA, 10Labs, 10Labs-Infrastructure: Database upgrade MariaDB 10: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#731912 (10Marostegui) For the record, this is the current status of the discrepancies of this table (based on tables.sql: https://phabricator.wik... [13:38:47] 10DBA, 10Fundraising-Analysis: Consider installing pt-deadlock-logger - https://phabricator.wikimedia.org/T128011#3335261 (10Marostegui) 05Open>03declined a:05Eileenmcnaughton>03None I will close this for now as per your comment. We will reopen if this is needed in the future. Thanks for the suggestion... [13:47:12] 10DBA, 10Labs: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#2872990 (10Marostegui) @Halfak what do you want to do with this ticket. The views are failing because the tables do not exist anymore as per: T153213#2873200 Thanks! [14:46:34] 10DBA, 10Labs: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#3335483 (10Halfak) Seems like the views for tables that do not exist should get cleaned up, I suppose. [14:57:59] 10DBA, 10Labs: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#3335515 (10Marostegui) @chasemp is that something you guys can do? [16:35:03] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1089: update RAID controller firwmare - https://phabricator.wikimedia.org/T166935#3335699 (10Marostegui) @Cmjohnson you think you will have time for this sometime next week? Thanks! [19:42:58] 10DBA, 10Analytics, 10Analytics-EventLogging: Add index on event_action, event_isAnon and event_namespaceId to NavigationTiming tables - https://phabricator.wikimedia.org/T70396#3336282 (10Nuria) Resolving, EL and dashboards have much changed , data from navigation timing can be found in graphana: https://gr... [19:43:12] 10DBA, 10Analytics, 10Analytics-EventLogging: Add index on event_action, event_isAnon and event_namespaceId to NavigationTiming tables - https://phabricator.wikimedia.org/T70396#3336285 (10Nuria) 05Open>03declined [19:57:12] 10DBA, 10Operations: Create less overhead on bacula jobs when dumping production databases - https://phabricator.wikimedia.org/T162789#3336319 (10Marostegui) It is inserting around 6k rows every 10 seconds for the revision dewiki table: ``` root@labsdb1009:/srv/sqldata/dewiki# strace -p 27245 -tT-f -s100000 -o... [22:52:27] 10DBA, 10Labs: Missing data on labs replica database - https://phabricator.wikimedia.org/T167554#3336634 (10Ankry) [23:14:49] 10DBA, 10Labs: Missing plwikisource_p.revision data on labs replica database - https://phabricator.wikimedia.org/T167554#3336667 (10bd808) [23:17:47] 10DBA, 10Labs: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3336668 (10bd808) Copied from {T167554} by @Ankry: > As this query shows (https://quarry.wmflabs.org/query/19383) > Revisions from 1147609 to 1147764 are missing from plwikisource_p.revision table in labs database. > > They... [23:17:59] 10DBA, 10Labs: Missing plwikisource_p.revision data on labs replica database - https://phabricator.wikimedia.org/T167554#3336634 (10bd808) [23:18:02] 10DBA, 10Labs: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3336674 (10bd808)