[06:36:22] <wikibugs>	 10DBA, 10Continuous-Integration-Config, 10Datasets-General-or-Unknown, 06Release-Engineering-Team, and 3 others: Automatize the check and fix of object, schema and data drifts between production masters and slaves - https://phabricator.wikimedia.org/T104459#2504400 (10ArielGlenn) Sounds great!  Care to lin...
[06:45:06] <wikibugs>	 10DBA, 10Incident-20150205-SiteOutage, 06Operations, 05Wikimedia-Incident: sleeper database connection surges during outage - https://phabricator.wikimedia.org/T88770#2504411 (10jcrespo) 05Open>03Resolved This is resolved:  1) All important boxes are using mariadb10 and pool of connections 2) There are...
[06:49:57] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: db2069 is down - https://phabricator.wikimedia.org/T141601#2504417 (10jcrespo)
[06:52:32] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: db2069 is down - https://phabricator.wikimedia.org/T141601#2504429 (10jcrespo)
[06:57:20] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: db2069 crashed due to RAID controller - https://phabricator.wikimedia.org/T141601#2504431 (10jcrespo)
[06:57:39] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: db2069 crashed due to RAID controller - https://phabricator.wikimedia.org/T141601#2504417 (10jcrespo) And I was right:   ``` </system1/log1>hpiLO-> show record5  status=0 status_tag=COMMAND COMPLETED Fri Jul 29 06:35:23 2016    /system1/log1/record5   Targets   Properties...
[08:20:15] <wikibugs>	 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team: Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades - https://phabricator.wikimedia.org/T139755#2504553 (10jcrespo) @Dzahn @demon can db1042 be wiped?
[10:33:08] <wikibugs>	 10DBA, 13Patch-For-Review, 05Prometheus-metrics-monitoring: Prepare mysql account and options for prometheus - https://phabricator.wikimedia.org/T128185#2504776 (10jcrespo) This is done, but only has been deployed to s1. Only pending its deploy to the other shards.
[11:39:43] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: db2069 crashed due to RAID controller - https://phabricator.wikimedia.org/T141601#2504844 (10jcrespo) 05Open>03Resolved a:03jcrespo Replication went back to normal and GTID reliable replication avoided corruption- resolving, keeping an eye on the data.
[13:31:37] <wikibugs>	 10DBA, 10Gerrit, 06Operations, 06Release-Engineering-Team: Need snapshot of 'reviewdb' on spare machine to test gerrit schema upgrades - https://phabricator.wikimedia.org/T139755#2505059 (10demon) Ah yep, we're done. Wipe away!
[16:56:30] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10MediaWiki-API, 05MW-1.28-release-notes, and 2 others: ApiQueryRecentChanges::run is spiking, nuking API servers - https://phabricator.wikimedia.org/T140108#2505651 (10jcrespo) This is almost done, I have to do a second pass to redo the tables that failed to apply the ch...
[17:39:19] <wikibugs>	 10DBA, 10MediaWiki-Redirects, 13Patch-For-Review: Moving a page over the redirect leaves an abandoned row in the revision table - https://phabricator.wikimedia.org/T106119#2505801 (10Bugreporter)
[18:29:53] <wikibugs>	 10DBA, 10Analytics-EventLogging, 10ImageMetrics: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#2505951 (10Jdforrester-WMF) Thank you!
[23:56:57] <wikibugs>	 10DBA, 10MediaWiki-Redirects, 13Patch-For-Review: Moving a page over the redirect leaves an abandoned row in the revision table - https://phabricator.wikimedia.org/T106119#2507269 (10MGChecker) @brion: I don't know if it is already included, but I think it would be useful to have an own log_action for this s...