[07:17:56] 10DBA, 06Operations, 10ops-codfw: es2004 has a dead disk, but it is not under warranty - https://phabricator.wikimedia.org/T143220#2578260 (10jcrespo) Ok, now things are nice: it says `Unconfigured(good), Spun Up` rather than Failed. [07:30:15] 10DBA, 06Operations, 10ops-codfw: es2004 has a dead disk, but it is not under warranty - https://phabricator.wikimedia.org/T143220#2578268 (10jcrespo) Now rebuilding, I didn't know the drive didn't rebuild automatically. ``` root@es2004:~$ megacli -Pdgetmissing -a0 A... [09:04:49] 10DBA: Test InnoDB compression - https://phabricator.wikimedia.org/T139055#2578363 (10jcrespo) db2034 was cloned from db1073; it now also uses compression with a reduction to 1/2-1/3 of the original size. [09:07:20] 10DBA: Test InnoDB compression - https://phabricator.wikimedia.org/T139055#2578366 (10jcrespo) https://grafana-admin.wikimedia.org/dashboard/db/server-board?panelId=17&fullscreen&from=1471853467356&to=1471901762667&var-server=db2034&var-network=eth0 [11:25:01] 10DBA, 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Database, 07WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#2578701 (10Nikerabbit) [12:17:27] 10DBA, 06Operations, 10ops-codfw: es2004 has a dead disk, but it is not under warranty - https://phabricator.wikimedia.org/T143220#2578791 (10jcrespo) ``` RECOVERY - MegaRAID on es2004 is OK: OK: optimal, 1 logical, 2 physical ``` [12:17:35] 10DBA, 06Operations, 10ops-codfw: es2004 has a dead disk, but it is not under warranty - https://phabricator.wikimedia.org/T143220#2578792 (10jcrespo) 05Open>03Resolved [13:16:30] 10DBA, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: Notice: Undefined variable: wmfMasterDatacenter in /srv/mediawiki/wmf-config/db-eqiad.php - https://phabricator.wikimedia.org/T143784#2578877 (10hashar) [13:16:46] 10DBA, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: Notice: Undefined variable: wmfMasterDatacenter in /srv/mediawiki/wmf-config/db-eqiad.php - https://phabricator.wikimedia.org/T143784#2578889 (10hashar) Spotted that just now. No idea how it is triggered though. [13:24:16] 10DBA, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: Notice: Undefined variable: wmfMasterDatacenter in /srv/mediawiki/wmf-config/db-eqiad.php - https://phabricator.wikimedia.org/T143784#2578877 (10jcrespo) `CommonSettings.php:$wmfMasterDatacenter = 'eqiad';` Maybe this is loaded before that one? [13:26:16] 10DBA, 10Wikimedia-Site-requests, 07Wikimedia-log-errors: Notice: Undefined variable: wmfMasterDatacenter in /srv/mediawiki/wmf-config/db-eqiad.php - https://phabricator.wikimedia.org/T143784#2578932 (10jcrespo) The change is most likely due to the changes on lag detection, but I do not see at first glance a... [18:22:14] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2579863 (10dduvall) [20:00:24] 10Blocked-on-schema-change, 10DBA, 10ArchCom-RfC, 10Wikimedia-Site-requests, and 2 others: Schema change for page content language - https://phabricator.wikimedia.org/T69223#2580336 (10RobLa-WMF) We plan to discuss this at E263 in an hour