[08:11:30] DBA, Continuous-Integration-Config, Datasets-General-or-Unknown, Release-Engineering-Team, and 2 others: Automatize the check and fix of object, schema and data drifts between production masters and slaves - https://phabricator.wikimedia.org/T104459#2469832 (ArielGlenn) [10:08:20] DBA, Operations: db2056 freezed - https://phabricator.wikimedia.org/T140598#2470054 (jcrespo) [10:14:24] DBA, Operations: db2056 freezed - https://phabricator.wikimedia.org/T140598#2470079 (jcrespo) ``` Slot 0 HP Smart Array P420i Controller (1 GB, v6.00) 1 Logical Drive 1719-Slot 0 Drive Array - A controller failure event occurred prior to this power-up. (Previous lock up code = 0x13) 1792-S... [10:24:39] DBA, Operations: db2056 freezed - https://phabricator.wikimedia.org/T140598#2470110 (jcrespo) No mysql-related logs before the freeze: ``` 160524 12:34:13 [Note] Slave I/O thread: connected to master 'repl@db2017.codfw.wmnet:3306',replication starts at GTID position '0-171970567-2955826593' 160718 10:20... [10:36:27] DBA, Operations: db2056 freezed - https://phabricator.wikimedia.org/T140598#2470128 (jcrespo) No syslog/kernel messages. But I found this on the management logs: ``` hpiLO-> show record6 status=0 status_tag=COMMAND COMPLETED Mon Jul 18 10:34:35 2016 /system1/log1/record6 Targets... [10:38:27] DBA, Operations: db2056 RAID controller (temporary) failure - https://phabricator.wikimedia.org/T140598#2470134 (jcrespo) [12:40:47] DBA, Continuous-Integration-Config, Datasets-General-or-Unknown, Release-Engineering-Team, and 2 others: Automatize the check and fix of object, schema and data drifts between production masters and slaves - https://phabricator.wikimedia.org/T104459#2470388 (ArielGlenn) Couple more followup chang... [14:12:05] DBA, Flow, Collab-Team-2016-Apr-Jun-Q4, Collaboration-Team-Archive-2015-2016, Patch-For-Review: Cleanup ptwikibooks conversion - https://phabricator.wikimedia.org/T119509#2470798 (jcrespo) [14:17:04] DBA, Dumps-Generation, Patch-For-Review: db1053 transient replica lag during dumping - https://phabricator.wikimedia.org/T132279#2470825 (jcrespo) "Blocked" only because waiting on updates of those changes, see what happens. [15:46:58] DBA, MediaWiki-Database: Special:Block request makes the database timeout on db1055 - https://phabricator.wikimedia.org/T140650#2471271 (jcrespo) [16:35:31] DBA, MediaWiki-Database, Performance: Query for Special:Contributions?contribs=newbie&dir=prev is slow on enwiki - https://phabricator.wikimedia.org/T140537#2471419 (jcrespo) [16:40:06] DBA, MediaWiki-Database, Performance: Query for Special:Contributions?contribs=newbie&dir=prev is slow on enwiki - https://phabricator.wikimedia.org/T140537#2471429 (jcrespo) I am setting #DBA tag as "Blocked external" to note that this is on my radar, and that I can provide support, reviews and comm... [16:41:28] DBA, MediaWiki-Database, MediaWiki-Special-pages, Performance: Query for Special:Contributions?contribs=newbie&dir=prev is slow on enwiki - https://phabricator.wikimedia.org/T140537#2471432 (matmarex) a:matmarex Let's try cutting off at 30 days in the past, I guess. @Bsadowski1 Do you think t... [20:04:20] DBA, MediaWiki-Database, MediaWiki-Special-pages, Patch-For-Review, Performance: Query for Special:Contributions?contribs=newbie&dir=prev is slow on enwiki - https://phabricator.wikimedia.org/T140537#2472542 (matmarex) p:Triage>Low [20:14:10] DBA, Wikimedia-General-or-Unknown, MW-1.28-release-notes, Patch-For-Review: Entries in revision table where rev_len is NULL - https://phabricator.wikimedia.org/T135414#2472592 (matmarex) p:Triage>Low a:matmarex