[02:19:37] 10DBA, 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4089990 (10aaron) The... [08:21:06] twentyafterfour: ack, looking [09:06:52] 10DBA, 10Analytics-Kanban, 10Contributors-Analysis, 10MW-1.27-release (WMF-deploy-2016-02-02_(1.27.0-wmf.12)), and 2 others: Edit schema needs purging, table is too big for queries to run (500G before conversion) {oryx} - https://phabricator.wikimedia.org/T124676#4090482 (10Neil_P._Quinn_WMF) p:05Normal>... [09:07:01] 10DBA, 10Cloud-Services, 10Contributors-Analysis, 10Security: Replicate editor_month table from analytics-store to Labs - https://phabricator.wikimedia.org/T143955#4090485 (10Neil_P._Quinn_WMF) p:05Normal>03Triage [09:09:19] 10DBA: labsdb1004: broken replication - https://phabricator.wikimedia.org/T191001#4090586 (10Volans) [09:09:29] 10DBA: labsdb1004: broken replication - https://phabricator.wikimedia.org/T191001#4090597 (10Volans) p:05Triage>03High [10:43:02] 10DBA: labsdb1004: broken replication - https://phabricator.wikimedia.org/T191001#4090849 (10Volans) I've found the missing 20 rows that were missing on labsdb1004 in this delete that was deleting 14677 rows, re-added them to labsdb1004 and re-started the replication. All those rows were properly deleted and th... [13:08:37] 10DBA: labsdb1004: s51541_sulwatcher.logging is out of sync - https://phabricator.wikimedia.org/T191020#4091131 (10Volans) [13:09:31] 10DBA: labsdb1004: broken replication - https://phabricator.wikimedia.org/T191001#4090586 (10Volans) 05Open>03Resolved Replication lag back to zero, no other errors, the two tables are different though. I've opened T191020 for tracking it, while resolving this one. [16:51:23] 10DBA, 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4091856 (10greg) @aaro... [18:38:09] 10DBA, 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4092548 (10mmodell) @a... [18:41:28] 10DBA, 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4092549 (10mmodell) Th... [21:35:46] 10DBA, 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4093091 (10greg) Just... [21:38:39] 10DBA, 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4093104 (10aaron) I do... [21:44:07] 10DBA, 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4093159 (10mmodell) @a... [21:45:30] 10DBA, 10MW-1.31-release-notes (WMF-deploy-2018-04-03 (1.31.0-wmf.28)), 10Patch-For-Review, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4093160 (10mmodell) I...