[06:22:11] 10DBA, 10Patch-For-Review: Prepare and indicate proper master db failover candidates for all database sections (s1-s8, x1) - https://phabricator.wikimedia.org/T186321#3957478 (10Marostegui) For s1 probably the right candidate is db1067: not a 512G host, old master, different row. [06:44:46] 10DBA, 10Data-Services: labsdb1010 crashed - https://phabricator.wikimedia.org/T186579#3957519 (10Marostegui) ``` Last_SQL_Error: Could not execute Delete_rows_v1 event on table srwiki.geo_tags; Can't find record in 'geo_tags', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's ma... [06:55:24] 10DBA, 10Data-Services: labsdb1010 crashed - https://phabricator.wikimedia.org/T186579#3957540 (10Marostegui) ``` Last_SQL_Error: Could not execute Delete_rows_v1 event on table wikidatawiki.wb_changes; Can't find record in 'wb_changes', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the... [07:21:47] 10DBA, 10Data-Services: labsdb1010 crashed - https://phabricator.wikimedia.org/T186579#3957583 (10Marostegui) ``` Last_SQL_Error: Could not execute Delete_rows_v1 event on table wikidatawiki.wb_changes; Can't find record in 'wb_changes', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the... [07:27:39] 10DBA, 10Data-Services: labsdb1010 crashed - https://phabricator.wikimedia.org/T186579#3957585 (10Marostegui) Replication is now flowing, but it will most likely fail during the weekend again :-(, so we need to rebuild it as soon as we have some time [09:14:58] 10DBA, 10cloud-services-team: labsdb1004 replication broken: s51999__merge_candidates.image_candidates - https://phabricator.wikimedia.org/T186863#3957697 (10Marostegui) [09:15:10] 10DBA, 10cloud-services-team: labsdb1004 replication broken: s51999__merge_candidates.image_candidates - https://phabricator.wikimedia.org/T186863#3957708 (10Marostegui) 05Open>03Resolved p:05Triage>03Normal [09:22:40] 10DBA: Drop flaggedrevs tables from mediawikiwiki - https://phabricator.wikimedia.org/T186865#3957724 (10demon) p:05Triage>03Low [09:30:17] 10DBA: Drop contest* tables from mediawikiwiki - https://phabricator.wikimedia.org/T186867#3957759 (10demon) p:05Triage>03Low [12:05:53] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Perform schema change to add externallinks.el_index_60 to all wikis - https://phabricator.wikimedia.org/T153182#3958007 (10Marostegui) a:03Marostegui I will combine this alter table with T185128 as we have to depool host by host. [12:06:23] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Perform schema change to add externallinks.el_index_60 to all wikis - https://phabricator.wikimedia.org/T153182#3958011 (10Marostegui) [12:18:58] 10DBA, 10Patch-For-Review: Productionize 2 new eqiad database servers - https://phabricator.wikimedia.org/T184161#3958036 (10Marostegui) a:05Marostegui>03None [12:22:14] 10DBA, 10Patch-For-Review: Replace codfw x1 master (db2033) (WAS: Failed BBU on db2033 (x1 master)) - https://phabricator.wikimedia.org/T184888#3958044 (10Marostegui) I will try to move the dbstore codfw servers under db2034 and make it the master next week. [12:22:17] 10DBA, 10Patch-For-Review: Replace codfw x1 master (db2033) (WAS: Failed BBU on db2033 (x1 master)) - https://phabricator.wikimedia.org/T184888#3958045 (10Marostegui) I will try to move the dbstore codfw servers under db2034 and make it the master next week. [22:12:33] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1030 - https://phabricator.wikimedia.org/T184397#3959698 (10RobH) a:05Cmjohnson>03RobH [22:25:02] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1030 - https://phabricator.wikimedia.org/T184397#3959738 (10RobH) a:05RobH>03Cmjohnson [22:27:44] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1039 - https://phabricator.wikimedia.org/T184262#3877626 (10RobH) a:05Cmjohnson>03RobH [22:29:07] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1029 and db1031 - https://phabricator.wikimedia.org/T184054#3959750 (10RobH) a:05Cmjohnson>03RobH [22:29:41] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1034 - https://phabricator.wikimedia.org/T182556#3959752 (10RobH) a:05Cmjohnson>03RobH