[07:24:13] 10DBA, 13Patch-For-Review: Unify commonswiki.revision - https://phabricator.wikimedia.org/T147305#2798123 (10Marostegui) labsdb1003 is finished ``` root@neodymium:~# mysql -hlabsdb1003 -A commonswiki -e "show create table revision\G" *************************** 1. row *************************** Table:... [07:48:50] 10DBA: Compress InnoDB on S5 (wikidatawiki, dewiki) - https://phabricator.wikimedia.org/T150747#2798132 (10Marostegui) dewiki from 365G to 200G wikidatawiki from 788G to 607G So in total we went from 1.2T to 807G [07:50:39] 10DBA, 13Patch-For-Review: Import S5 to dbstore2001 and dbstore2002 + compression - https://phabricator.wikimedia.org/T150518#2798134 (10Marostegui) S5 caught up and was compressed. I have started the transfer to dbstore2002 now. [08:13:36] 10DBA, 06Community-Tech, 06Labs, 10MediaWiki-extensions-PageAssessments: Replicate page_assessments and page_assessments_projects tables on Labs - https://phabricator.wikimedia.org/T150832#2798143 (10kaldari) [08:14:04] 10DBA, 06Labs, 10Labs-Infrastructure: LabsDB replica service for tools and labs - issues and missing available views (tracking) - https://phabricator.wikimedia.org/T150767#2798156 (10kaldari) [08:14:06] 10DBA, 06Community-Tech, 06Labs, 10MediaWiki-extensions-PageAssessments: Replicate page_assessments and page_assessments_projects tables on Labs - https://phabricator.wikimedia.org/T150832#2798155 (10kaldari) [08:15:23] 10DBA, 06Community-Tech, 06Labs, 10MediaWiki-extensions-PageAssessments: Replicate page_assessments and page_assessments_projects tables on Labs - https://phabricator.wikimedia.org/T150832#2798143 (10Marostegui) Note: I checked the tables on enwiki and they are present in labs (ie: labsdb1003) already, but... [08:15:36] 10DBA, 06Community-Tech, 06Labs, 10MediaWiki-extensions-PageAssessments: Replicate page_assessments and page_assessments_projects tables on Labs - https://phabricator.wikimedia.org/T150832#2798159 (10kaldari) Apparently, they are already being replicated, but need to be added to the maintain-views config i... [10:14:05] 10DBA, 13Patch-For-Review: db2042 reimage - https://phabricator.wikimedia.org/T150334#2798381 (10jcrespo) 05Open>03Resolved [10:22:17] 10DBA, 06Operations, 10ops-codfw: db2034 crashes meta ticket - https://phabricator.wikimedia.org/T150233#2798398 (10Marostegui) p:05Triage>03High [10:23:16] 10DBA, 06Labs, 10Tool-Labs-tools-Other, 13Patch-For-Review: High replication activity filled up labsdb1004 with binlogs - https://phabricator.wikimedia.org/T150553#2798400 (10jcrespo) 05Open>03stalled [10:23:30] 10DBA, 06Labs, 10Tool-Labs-tools-Other, 13Patch-For-Review: High replication activity filled up labsdb1004 with binlogs - https://phabricator.wikimedia.org/T150553#2789420 (10jcrespo) p:05Triage>03Low [10:41:15] https://wikitech.wikimedia.org/wiki/MariaDB#Account_handling [10:42:18] \o/ [10:43:09] nice one [13:27:22] nice :) [13:27:42] that's great to hear [13:28:29] (sorry, lagging behind :) [13:37:04] 10DBA: Compress InnoDB on S5 (wikidatawiki, dewiki) - https://phabricator.wikimedia.org/T150747#2798881 (10Marostegui) dewiki finished. From 365G to 144G [13:42:02] 10DBA: Decouple roles from mariadb.pp into their own file - https://phabricator.wikimedia.org/T150850#2798888 (10Marostegui) [13:42:42] 10DBA, 07Epic: Decouple roles from mariadb.pp into their own file - https://phabricator.wikimedia.org/T150850#2798900 (10Marostegui) p:05Triage>03Low [13:44:17] 10DBA: Moving backup and otrs role into their own .pp - https://phabricator.wikimedia.org/T150851#2798902 (10Marostegui) [14:39:24] 10DBA, 13Patch-For-Review: Import S5 to dbstore2001 and dbstore2002 + compression - https://phabricator.wikimedia.org/T150518#2799210 (10Marostegui) dbstore2002 got all the tables imported from dewiki and wikidatawiki and it is now catching up with the master. [16:08:38] 10DBA, 13Patch-For-Review: Import S5 to dbstore2001 and dbstore2002 + compression - https://phabricator.wikimedia.org/T150518#2799433 (10Marostegui) dbstore2002 finished catching up. I am now compressing dewiki and wikidatawiki [16:11:07] are you folks aware of the predictive failure alert on db2042? [16:11:30] yes [16:11:41] there are 2 of them [16:14:42] And db2035 is also with predictive failure: https://phabricator.wikimedia.org/T150511 [16:38:27] 10DBA, 06Labs, 10Labs-Infrastructure: LabsDB replica service for tools and labs - issues and missing available views (tracking) - https://phabricator.wikimedia.org/T150767#2799553 (10chasemp) [16:38:35] 10DBA, 06Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-ORES, and 3 others: Replicate ores_classification and ores_model tables in labs - https://phabricator.wikimedia.org/T148561#2799550 (10chasemp) 05Open>03Resolved ```--- /etc/maintain-views.yaml 2016-11-02 19:12:41.326458322 +0000 +++ /tmp/pup... [16:51:30] 10DBA, 06Operations, 10ops-codfw: db2035: RAID disk about to fail - https://phabricator.wikimedia.org/T150511#2799602 (10Papaul) Dear Mr Papaul Tshibamba, Thank you for contacting Hewlett Packard Enterprise for your service request. This email confirms your request for service and the details are below. Yo... [17:13:28] 10DBA, 13Patch-For-Review: db2034: investigate its crash and reimage - https://phabricator.wikimedia.org/T149553#2799693 (10Papaul) @Robh discard my comment I was able to download the file. Thanks [17:24:50] 10DBA, 10Edit-Review-Improvements-RC-Page, 06Collaboration-Team-Triage (Collab-Team-Q2-Oct-Dec-2016), 13Patch-For-Review: Implement functionality for RC page 'Experience level' filters - https://phabricator.wikimedia.org/T149637#2799749 (10SBisson) Pinging @jcrespo and @Marostegui For this ticket, we join... [17:28:18] 10DBA, 06Operations, 10ops-codfw: db2049 overheated and restarted - https://phabricator.wikimedia.org/T150876#2799779 (10jcrespo) [17:29:25] 10DBA, 06Operations, 10ops-codfw: db2049 overheated and restarted - https://phabricator.wikimedia.org/T150876#2799779 (10Marostegui) I am checking the fans logs and they look fine. [20:01:22] db1034 was showing intermittent lag, I see a slow query adding a page_lang column to page ? [20:03:52] ah I guess that's T69223 [20:03:52] T69223: Schema change for page content language - https://phabricator.wikimedia.org/T69223 [20:04:49] also please log in to neodymium as your user and then sudo as root, makes things easier to see who's doing what [22:26:23] 10DBA, 06Community-Tech, 06Labs, 10MediaWiki-extensions-PageAssessments, 13Patch-For-Review: Replicate page_assessments and page_assessments_projects tables on Labs - https://phabricator.wikimedia.org/T150832#2800790 (10kaldari)