[03:33:39] 10DBA, 10MediaWiki-Database, 10MW-1.32-release-notes (WMF-deploy-2018-06-26 (1.32.0-wmf.10)), 10Multi-Content-Revisions (MCR-SDC Storage Layer - phase 1), 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10CCicale... [04:45:01] 10DBA, 10Operations, 10ops-eqiad: db1069 bad disk - https://phabricator.wikimedia.org/T199056 (10Marostegui) 05Resolved>03Open Actually it this disk has smart errors too. Was this a re-used or a new disk, @Cmjohnson? ``` PD: 0 Information Enclosure Device ID: 32 Slot Number: 0 Drive's position: DiskGrou... [06:01:00] 10Blocked-on-schema-change, 10MediaWiki-Database, 10Patch-For-Review: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 (10Marostegui) [06:18:38] 10Blocked-on-schema-change, 10MediaWiki-Database, 10Patch-For-Review: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 (10Marostegui) [06:28:54] 10DBA, 10Patch-For-Review, 10Schema-change: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 (10Marostegui) [06:28:56] 10DBA, 10Patch-For-Review, 10Schema-change: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 (10Marostegui) [06:29:24] 10Blocked-on-schema-change, 10MediaWiki-Database, 10Patch-For-Review: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 (10Marostegui) [06:47:24] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Prepare and check storage layer for amwikimedia (including dropping s7 version of the wiki) - https://phabricator.wikimedia.org/T176043 (10jcrespo) 05Open>03Resolved a:05jcrespo>03bd808 Only multisource hosts have now amwik... [08:37:50] marostegui: I have deployed mysql.py to neodymium/sarin [08:38:15] \o/ [08:38:16] I will now delete /home/jynus/mysql.py, but only when you tell me you are ok with that [08:38:29] yeah, let do a grep on my scripts [08:38:54] let me do I mean [08:39:05] I can also ln -s it [08:39:10] nah not needed [08:39:13] but I would prefer not to [08:39:29] You can remove it now (or move it to another name) [08:39:36] It is now on path [08:39:50] yeah, I see it [08:39:54] or on /usr/local/sbin if you want an absoulte reference [08:40:05] yeah, that is what I am doing [08:40:10] feel free to remove it from your home now [08:40:34] done [08:42:18] I've also removed section [08:42:24] it is on the repo now [08:42:46] what do you mean removed section? [08:44:55] https://gerrit.wikimedia.org/r/#/c/operations/software/+/444798/2/dbtools/section [08:45:00] I had this on my home [08:45:06] now it is on dbtools [08:45:22] Ah right, yeah, I saw that [08:45:29] I was confused with the "removed" part [08:45:32] all clear now! thanks :) [08:45:40] removed the copy on my home [08:45:50] actually added it to production, my fault [08:46:02] all clear, thank you! :) [09:53:26] 10DBA, 10Patch-For-Review: Gather statistics about the backups on a database - https://phabricator.wikimedia.org/T198987 (10jcrespo) So I have the first backup-specific statistics: ``` root@db1115.eqiad.wmnet[zarcillo]> select * FROM backup_files LIMIT 20; +------+---------+---------------------+------------... [10:15:57] 10DBA, 10Patch-For-Review: Gather statistics about the backups on a database - https://phabricator.wikimedia.org/T198987 (10Marostegui) Some more weird empty password grants cleaned up: db1097:3314 ``` root@db1097.eqiad.wmnet[mysql]> show grants for 'wikiuser@208.80.154.%'@'%'; +------------------------------... [10:16:28] gah [10:37:12] 10DBA, 10Patch-For-Review: Gather statistics about the backups on a database - https://phabricator.wikimedia.org/T198987 (10jcrespo) False alarm- empty tables do not get a data dump- we have to compare with schema dumps only, and they match: ``` mysql.py -BN -h db1115 zarcillo -e "SELECT DISTINCT SUBSTRING_IND... [12:02:18] 10DBA, 10Patch-For-Review: Gather statistics about the backups on a database - https://phabricator.wikimedia.org/T198987 (10jcrespo) more useful stats (size is after compression): ``` root@db1115.eqiad.wmnet[zarcillo]> select source, section, backup_date, sum(size) from backup_files GROUP BY source, section, b... [12:24:47] 10DBA, 10MediaWiki-Platform-Team, 10Structured-Data-Commons, 10Wikidata, and 4 others: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044 (10daniel) [13:17:15] 10DBA, 10Structured-Data-Commons, 10Wikidata, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), and 2 others: MCR schema migration stage 0: create tables - https://phabricator.wikimedia.org/T183486 (10daniel) 05Resolved>03Open Re-opening: the tables where dropped again per T190153, and SAL does not ind... [13:25:38] 10DBA, 10Structured-Data-Commons, 10Wikidata, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), and 2 others: MCR schema migration stage 0: create tables - https://phabricator.wikimedia.org/T183486 (10Marostegui) Did you check if the are there? Because I can see them, maybe @anomie forgot to !log them - so... [13:29:35] 10DBA, 10Structured-Data-Commons, 10Wikidata, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), and 2 others: MCR schema migration stage 0: create tables - https://phabricator.wikimedia.org/T183486 (10daniel) 05Open>03Resolved Seems like i mistyped when checking the log or something. It's there 10DBA: Test database master switchover script on codfw - https://phabricator.wikimedia.org/T199224 (10jcrespo) [13:35:34] 10DBA: Test database master switchover script on codfw - https://phabricator.wikimedia.org/T199224 (10jcrespo) p:05Triage>03High [13:36:00] 10DBA: switchover es1014 to es1017 - https://phabricator.wikimedia.org/T197073 (10jcrespo) [13:36:02] 10DBA: Test database master switchover script on codfw - https://phabricator.wikimedia.org/T199224 (10jcrespo) [13:36:04] 10DBA, 10Patch-For-Review: Failover db1052 (s1) db primary master - https://phabricator.wikimedia.org/T197069 (10jcrespo) [13:38:27] 10DBA: Test database master switchover script on codfw - https://phabricator.wikimedia.org/T199224 (10Marostegui) Reminder, we have 4 hosts as spare now in eqiad: T196376 [14:32:03] 10DBA, 10Operations, 10ops-eqiad: db1069 bad disk - https://phabricator.wikimedia.org/T199056 (10Marostegui) @Cmjohnson disk #0 is now offline, feel free to replace it when you can. [14:39:48] 10DBA, 10Operations, 10ops-eqiad: db1069 bad disk - https://phabricator.wikimedia.org/T199056 (10Marostegui) Disk replaced by Chris, let's see if this time it turns out fine! ``` root@db1069:~# megacli -PDRbld -ShowProg -PhysDrv [32:0] -a0 Rebuild Progress on Device at Enclosure 32, Slot 0 Completed 1% in 1... [14:57:29] 10DBA, 10Operations, 10ops-eqiad: db1069 bad disk - https://phabricator.wikimedia.org/T199056 (10Marostegui) [16:35:22] 10DBA, 10Operations, 10ops-eqiad: db1069 bad disk - https://phabricator.wikimedia.org/T199056 (10Marostegui) 05Open>03Resolved All good this time ``` root@db1069:~# megacli -LDPDInfo -aAll Adapter #0 Number of Virtual Disks: 1 Virtual Drive: 0 (Target Id: 0) Name : RAID Level :... [16:47:04] 10DBA, 10JADE, 10Operations, 10Scoring-platform-team (Current), 10User-Joe: Extension:JADE scalability concerns due to creating a page per revision - https://phabricator.wikimedia.org/T196547 (10mark) [19:40:14] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Umherirrender) Before a...