[00:00:36] (03CR) 10Milimetric: "Joseph's instinct was right, I think. I de-refactored the code back and gave unique names to all the variables (a bit obsessively) and ra" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/480796 (https://phabricator.wikimedia.org/T210543) (owner: 10Joal) [00:36:09] 10Analytics, 10Community-Tech, 10Event Metrics: Review category queries - https://phabricator.wikimedia.org/T206783 (10jmatazzoni) [01:06:59] 10Analytics, 10Product-Analytics: Standardize datetimes/timestamps in the Data Lake - https://phabricator.wikimedia.org/T212529 (10Neil_P._Quinn_WMF) [01:47:36] (03PS1) 10Neil P. Quinn-WMF: Add VisualEditorFeatureUse schema to EventLogging whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/485354 (https://phabricator.wikimedia.org/T212588) [08:48:17] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) Table `frwiki.echo_notification` (x1) might need reimporting: ` Last_SQL_Error: Could not execute Update_rows_v1 event on table frwiki.echo_notification; Ca... [09:06:50] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) And now MySQL process has crashed. [09:15:52] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Convert Aria tables to InnoDB on dbstore1002 - https://phabricator.wikimedia.org/T213706 (10Marostegui) MySQL process has crashed again, I think the ALTERs are messing up with the general stability of the server. I think once we are ready to move the users to t... [09:33:16] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) MySQL still doing recovery [09:42:06] 10Analytics, 10Analytics-Kanban, 10Operations, 10Product-Analytics: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) [12:11:50] dbstore1002 crashed again :( [12:11:53] it is up and running now [12:11:54] sigh [12:27:42] :( [13:37:44] 10Analytics, 10Analytics-Kanban, 10Operations, 10Product-Analytics: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) After all the crashes, MySQL was able to start at around 10:18:11 (UTC). @elukey start replication on all slaves at around 12:07:56 (UTC). x1 replication w... [17:05:51] 10Analytics, 10Analytics-Kanban, 10DBA, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) >>! In T210478#4800102, @Banyek wrote: > About the `staging` db: > > Analytics team uses it, but I am not... [17:58:30] 10Analytics, 10Analytics-Kanban, 10DBA, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10elukey) I thought that we could have had one mysql instance of the staging DB on one dbstore node, so option c) but no... [18:18:38] 10Analytics, 10Analytics-Kanban, 10DBA, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) Sure, we can do that if that's what's the preferred solution. You would have a spof though. There is no w... [18:29:35] 10Analytics, 10Analytics-Kanban, 10DBA, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10elukey) Yep we are aware, one of the follow ups that I'd like to do after the migration is to create a regular backup... [18:33:33] 10Analytics, 10Analytics-Kanban, 10DBA, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) Sure, we can discuss backups strategies!