[05:29:55] 10DBA, 10cloud-services-team (Kanban): Create a new nova database on m5 named 'nova_cell0' - https://phabricator.wikimedia.org/T239170 (10Marostegui) So I would assume the steps needed are: - Create the database itself on the master - Create the puppet patch for the grants mirroring the ones for `nova` - App... [05:47:01] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) [05:47:09] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [05:50:29] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) s1 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1003 [] db1139 [] db1134 [] db1124 [] db1119 [] db1118 [] db1106 [] db1105 [] db1099 []... [05:50:33] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) s1 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1003 [] db1139 [] db1134 [] db1124 [] db1119 [] db111... [05:57:54] https://phabricator.wikimedia.org/T238046#5691991 [06:19:33] 10DBA, 10Operations, 10Patch-For-Review, 10User-notice: Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC - https://phabricator.wikimedia.org/T238044 (10Marostegui) [06:30:04] 10DBA, 10Operations: Decommission db1061-db1073 - https://phabricator.wikimedia.org/T217396 (10Marostegui) [06:30:06] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) [06:30:08] 10DBA, 10Operations, 10Patch-For-Review, 10User-notice: Switchover s7 primary database master db1062 -> db1086 - 26th Nov 06:00 - 06:30 UTC - https://phabricator.wikimedia.org/T238044 (10Marostegui) 05Open→03Resolved This was done successfully. Read only starts: 06:00:24 Read only stops: 06:01:09 Tot... [06:30:23] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) [06:46:26] 10DBA: Remove ar_comment from sanitarium triggers - https://phabricator.wikimedia.org/T234704 (10Marostegui) [06:53:03] 10DBA: Decommission db1062.eqiad.wmnet - https://phabricator.wikimedia.org/T239188 (10Marostegui) [06:53:15] 10DBA: Decommission db1062.eqiad.wmnet - https://phabricator.wikimedia.org/T239188 (10Marostegui) p:05Triage→03Normal [06:53:30] 10DBA, 10Operations: Decommission db1061-db1073 - https://phabricator.wikimedia.org/T217396 (10Marostegui) [06:54:27] 10DBA: Decommission db1062.eqiad.wmnet - https://phabricator.wikimedia.org/T239188 (10Marostegui) [07:03:19] 10DBA, 10Patch-For-Review: Decommission db1061.eqiad.wmnet - https://phabricator.wikimedia.org/T238624 (10Marostegui) [07:13:11] 10DBA, 10Patch-For-Review: Decommission db1062.eqiad.wmnet - https://phabricator.wikimedia.org/T239188 (10Marostegui) [07:17:35] 10DBA, 10Patch-For-Review: Decommission db1062.eqiad.wmnet - https://phabricator.wikimedia.org/T239188 (10Marostegui) [08:10:56] 10DBA, 10Patch-For-Review: Productionize db213[2-5} - https://phabricator.wikimedia.org/T238183 (10Marostegui) [11:17:34] akosiaris: quick ok on https://gerrit.wikimedia.org/r/c/operations/puppet/+/553084 if I update https://wikitech.wikimedia.org/wiki/Bacula#Adding_a_new_client ? [11:17:54] (which was already outdated, BTW) [11:25:51] s8 wikidatamaster is having issues [11:25:55] with a schema change I think [11:30:29] 10Blocked-on-schema-change, 10DBA, 10Wikidata: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language - https://phabricator.wikimedia.org/T237120 (10Marostegui) Impossible to alter this table on the master while it is a master. I just got a bunch of metadata locking for lots of... [11:32:04] 10Blocked-on-schema-change, 10DBA, 10Wikidata: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language - https://phabricator.wikimedia.org/T237120 (10Marostegui) 05Open→03Stalled [11:33:27] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [11:36:43] I am going to alter db1139:3311 (backup source for s1) there is no on-going snapshotting on it [11:38:41] 10Blocked-on-schema-change, 10DBA, 10Wikidata: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language - https://phabricator.wikimedia.org/T237120 (10Marostegui) Probably a good idea to coordinate this required failover with: T238966 as that one will also probably require a failov... [11:43:17] cool [12:13:20] jynus: yup, +1ed in essence (minus my pedantic self). [12:16:04] I will do later a cleanup of all deprecated ones [14:01:11] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui) [14:08:31] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) [14:08:56] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) [14:18:22] 10DBA: Switchover s8 primary database master db1109 -> db1104 - Date TBD - https://phabricator.wikimedia.org/T239238 (10Marostegui) [14:18:44] 10DBA: Switchover s8 primary database master db1109 -> db1104 - Date TBD - https://phabricator.wikimedia.org/T239238 (10Marostegui) [14:18:46] 10Blocked-on-schema-change, 10DBA, 10Wikidata: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language - https://phabricator.wikimedia.org/T237120 (10Marostegui) [14:19:03] 10DBA: Switchover s8 primary database master db1109 -> db1104 - Date TBD - https://phabricator.wikimedia.org/T239238 (10Marostegui) p:05Triage→03Normal [14:19:50] 10DBA: Switchover s8 primary database master db1109 -> db1104 - Date TBD - https://phabricator.wikimedia.org/T239238 (10Marostegui) [15:54:49] 10DBA, 10cloud-services-team (Kanban): Create a new nova database on m5 named 'nova_cell0' - https://phabricator.wikimedia.org/T239170 (10Andrew) Those steps sound right to me. Backups would be nice -- I think the nova db is backed up but I'm not positive. [16:12:51] 10DBA, 10cloud-services-team (Kanban): Create a new nova database on m5 named 'nova_cell0' - https://phabricator.wikimedia.org/T239170 (10jcrespo) > Does this need backups? cc @jcrespo To add misc hosts to backups (assuming they are existing backed hosts), the only thing needed is to provide SELECT GRANTS on... [16:17:42] 10DBA, 10cloud-services-team (Kanban): Create a new nova database on m5 named 'nova_cell0' - https://phabricator.wikimedia.org/T239170 (10Marostegui) >>! In T239170#5693933, @jcrespo wrote: >> Does this need backups? cc @jcrespo > > To add misc hosts to backups (assuming they are existing backed hosts), the o... [16:39:38] hi all, can i get a +1 on https://gerrit.wikimedia.org/r/c/operations/puppet/+/548241 just so i know i have buy in from you dba's [16:42:03] 10DBA, 10cloud-services-team (Kanban): Create a new nova database on m5 named 'nova_cell0' - https://phabricator.wikimedia.org/T239170 (10jcrespo) > It was more a question for @Andrew (which he already answered) I commented also so he also is aware and he could even do it himself if he doesn't want to get blo... [18:34:30] 10DBA, 10Analytics: add wbc_entity_usage to all labs projects? - https://phabricator.wikimedia.org/T239261 (10Nuria) [19:28:17] jbond42: I will check tomorrow, I wasn't added to that patchset [19:30:33] thanks marostegui [19:32:20] sorry for not adding you that was definetly an oversite and if you have issue its not a problem to hold off [19:33:03] there is/was some concern about having to restart all mysql services although i dont see right now where that descussion happened [19:33:54] either way i dont think this change makes that problem worse its just a case of being aware that daemons may need to get restarted ater this change to make the certificate refreshe complete [19:36:36] https://phabricator.wikimedia.org/T236277#5631255 [20:58:03] 10DBA, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Extend Puppet CA Expiry date - https://phabricator.wikimedia.org/T236277 (10Marostegui) @jbond maybe it is a good idea to disable puppet on all databases before merging the change and then trying a manual run on a single host to see how it...