[00:04:09] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4086816 (10Krinkle) [00:06:30] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4086827 (10Krinkle) [00:06:32] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#4086825 (10Krinkle) 05Resolved>03Open It seems the `click_tracking_events` table w... [00:06:53] 10DBA, 10MediaWiki-extensions-ClickTracking, 10Operations: Drop the tables old_growth, hitcounter, click_tracking, click_tracking_user_properties from enwiki, maybe other schemas - https://phabricator.wikimedia.org/T115982#4086828 (10Krinkle) [00:14:50] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4086851 (10Krinkle) [00:14:52] 10DBA: Drop localisation and localisation_file_hash tables, l10nwiki databases too - https://phabricator.wikimedia.org/T119811#4086849 (10Krinkle) 05Resolved>03Open The `localisation` table seems to still exists, at least on enwiki/db1080 where I checked. [00:15:51] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3137204 (10Krinkle) [00:22:13] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4086854 (10Krinkle) [00:23:11] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3221129 (10Krinkle) [00:25:37] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4086860 (10Krinkle) [00:26:34] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#3221161 (10Krinkle) [04:04:53] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4086990 (10MZMcBride) Since the "Removable" table column is not sortable, perhaps splitting the table into multiple tables would be useful? The current mix... [07:23:51] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921#4087074 (10jcrespo) @MZMcBride I suggested moving it to mediawiki.org or wikitech, help is welcome! [07:36:44] 10DBA, 10Schema-change: Schema changes to site_stats - https://phabricator.wikimedia.org/T190780#4087083 (10jcrespo) @EddieGP Contact someone with access so it can be verified, and also check things are working smoothly there (checking it is working as intended should be trivial). When that is done, I will add... [07:48:17] 10DBA, 10Release-Engineering-Team (Watching / External): Certain wiki databases missing from replicas? - https://phabricator.wikimedia.org/T132838#4087090 (10jcrespo) p:05High>03Low This is arguably not an issue- those databases are deleted, so of course they are missing from public/analytics replicas. Low... [08:27:19] 10DBA, 10Wikimedia-Incident: Improve regular production database backups handling - https://phabricator.wikimedia.org/T138562#4087146 (10jcrespo) [08:27:22] 10DBA, 10Patch-For-Review: Faster backups by starting mydumper execution of different sections in parallel when they are about to finish and are running on a limited number of threads - https://phabricator.wikimedia.org/T189384#4087143 (10jcrespo) 05Open>03Resolved a:03jcrespo Backup of s* sections took... [15:07:03] 10DBA, 10Operations, 10hardware-requests, 10ops-codfw: Decommission db2030 - https://phabricator.wikimedia.org/T187768#4088271 (10Papaul) switch port information asw-b6-codfw ge-6/0/13 [15:07:38] 10DBA, 10Operations, 10hardware-requests, 10ops-codfw: Decommission db2030 - https://phabricator.wikimedia.org/T187768#4088272 (10Papaul) [15:19:32] 10DBA, 10Operations, 10hardware-requests, 10ops-codfw: Decommission db2030 - https://phabricator.wikimedia.org/T187768#4088298 (10Papaul) [17:43:35] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db1011 - https://phabricator.wikimedia.org/T184703#4088779 (10Cmjohnson) [18:19:25] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1030 - https://phabricator.wikimedia.org/T184397#4088932 (10Cmjohnson) [18:25:46] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1001 - https://phabricator.wikimedia.org/T190262#4088955 (10Cmjohnson) [18:28:21] 10DBA, 10Operations, 10hardware-requests, 10ops-eqiad, 10Patch-For-Review: Decommission db1011 - https://phabricator.wikimedia.org/T184703#4088963 (10Cmjohnson) [19:59:45] 10DBA, 10Train Deployments, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4089198 (10mmodell) [20:03:23] 10DBA, 10Train Deployments, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4089230 (10mmodell) [20:05:07] 10DBA, 10Train Deployments, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4089198 (10mmodell) [20:06:30] 10DBA, 10Train Deployments, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4089236 (10mmodell) p:05Triage>03Unbreak! [20:06:58] 10DBA, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4089239 (10greg) [20:07:28] 10DBA, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4089198 (10greg) @aaron as our DBAs are out, can you help here or not? [20:07:41] 10DBA, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4089244 (10mmodell) @Niharika found suspect queries in tendril: https://tendril.wikimedia.org/report/slow_queri... [20:10:09] https://phabricator.wikimedia.org/T190960 is a critical train blocker because 1.31.0-wmf.27 caused a huge increase in temp tables and that resulted in mysql connection errors [20:10:18] had to roll back to wmf.26 [20:17:05] 10DBA, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4089264 (10mmodell) [21:57:51] 10DBA, 10Patch-For-Review, 10Wikimedia-log-errors: 1.31.0-wmf.27 rolled back due to increase in fatals: "Replication wait failed: lost connection to MySQL server during query" - https://phabricator.wikimedia.org/T190960#4089487 (10mmodell) @aaron: Can you explain how your patch addresses the issue? I'm not c...