[03:35:12] 10DBA, 10Patch-For-Review: Productionize x2 databases - https://phabricator.wikimedia.org/T269324 (10Marostegui) >>! In T269324#6785507, @aaron wrote: >>>! In T269324#6772394, @Marostegui wrote: >> This is all done - hosts are ready to start getting data. > > I was thinking that these would be setup just like... [03:38:06] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10Marostegui) Thank you Rob - they all look good! [07:09:18] 10DBA, 10mariadb-optimizer-bug: Investigate possible optimizer regression on 10.4.17 with DELETE statements - https://phabricator.wikimedia.org/T268457 (10Marostegui) Cross posting from what I just wrote on the mariadb bug: //For what is worth, after a few days of running 10.4.18 on one of the hosts, it is per... [08:27:22] 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Productionize clouddb10[13-20] - https://phabricator.wikimedia.org/T267090 (10Marostegui) I am starting to change buffer pool sizes on all the clouddb hosts to make sure we are using 403 out of 512GB of RAM (which is what we use at... [09:31:24] 10Data-Persistence-Backup, 10SRE, 10Goal, 10Patch-For-Review: Followup to backup1001 bacula switchover (misc pending tasks) - https://phabricator.wikimedia.org/T238048 (10jcrespo) [09:37:28] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) db1175 is now replicating, won't pool it until Monday though. [09:48:34] 10Data-Persistence-Backup, 10SRE, 10decommission-hardware, 10ops-codfw, 10Patch-For-Review: decommission heze and heze-array1 - https://phabricator.wikimedia.org/T273051 (10jcrespo) [09:48:36] 10Data-Persistence-Backup, 10SRE, 10decommission-hardware, 10ops-eqiad, 10Patch-For-Review: decommission helium.eqiad.wmnet and helium-array - https://phabricator.wikimedia.org/T273049 (10jcrespo) [10:17:31] 10Data-Persistence-Backup, 10SRE, 10decommission-hardware, 10ops-eqiad: decommission helium.eqiad.wmnet and helium-array - https://phabricator.wikimedia.org/T273049 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jynus@cumin1001 for hosts: `helium.eqiad.wmnet` - helium.eqiad.wmnet (**PA... [10:54:14] 10Data-Persistence-Backup, 10SRE, 10decommission-hardware, 10ops-codfw: decommission heze and heze-array1 - https://phabricator.wikimedia.org/T273051 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jynus@cumin1001 for hosts: `heze.codfw.wmnet` - heze.codfw.wmnet (**PASS**) - Downtimed... [10:58:41] 10Data-Persistence-Backup, 10SRE, 10decommission-hardware, 10ops-codfw: decommission heze and heze-array1 - https://phabricator.wikimedia.org/T273051 (10jcrespo) [10:59:00] 10Data-Persistence-Backup, 10SRE, 10decommission-hardware, 10ops-eqiad: decommission helium.eqiad.wmnet and helium-array - https://phabricator.wikimedia.org/T273049 (10jcrespo) [11:04:28] 10Data-Persistence-Backup, 10SRE, 10decommission-hardware, 10ops-eqiad: decommission helium.eqiad.wmnet and helium-array - https://phabricator.wikimedia.org/T273049 (10jcrespo) a:05jcrespo→03Cmjohnson This is ready for full decommission, many people will be happy to get rid of these 2 boxes. > reassig... [11:04:40] 10Data-Persistence-Backup, 10SRE, 10decommission-hardware, 10ops-eqiad: decommission helium.eqiad.wmnet and helium-array - https://phabricator.wikimedia.org/T273049 (10jcrespo) [11:07:20] 10Data-Persistence-Backup, 10SRE, 10decommission-hardware, 10ops-codfw: decommission heze and heze-array1 - https://phabricator.wikimedia.org/T273051 (10jcrespo) [11:07:25] 10Data-Persistence-Backup, 10SRE, 10decommission-hardware, 10ops-codfw: decommission heze and heze-array1 - https://phabricator.wikimedia.org/T273051 (10jcrespo) a:05jcrespo→03Papaul This is ready for full decommissioning, a lot of people will be happy to get rid of these 2 boxes. [11:08:02] 10Data-Persistence-Backup, 10SRE, 10decommission-hardware, 10ops-codfw: decommission heze and heze-array1 - https://phabricator.wikimedia.org/T273051 (10jcrespo) [11:08:05] 10Data-Persistence-Backup, 10DC-Ops, 10SRE, 10Patch-For-Review: decom helium and heze - https://phabricator.wikimedia.org/T260717 (10jcrespo) [11:08:28] 10Data-Persistence-Backup, 10SRE, 10decommission-hardware, 10ops-eqiad: decommission helium.eqiad.wmnet and helium-array - https://phabricator.wikimedia.org/T273049 (10jcrespo) [11:08:31] 10Data-Persistence-Backup, 10DC-Ops, 10SRE, 10Patch-For-Review: decom helium and heze - https://phabricator.wikimedia.org/T260717 (10jcrespo) [11:12:47] 10Data-Persistence-Backup, 10SRE, 10Goal, 10Patch-For-Review: Followup to backup1001 bacula switchover (misc pending tasks) - https://phabricator.wikimedia.org/T238048 (10jcrespo) [11:13:27] 10Data-Persistence-Backup, 10DC-Ops, 10SRE, 10Patch-For-Review: decom helium and heze - https://phabricator.wikimedia.org/T260717 (10jcrespo) 05Open→03Resolved All non-dc ops steps (#data-persistence-backups) have been completed, dc-ops steps filed on separate tasks T273049 and T273051. Resolving this. [11:18:13] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10jcrespo) helium and heze (jessie) hosts were sent for decommissioning today: T260717 [11:46:31] 10Data-Persistence-Backup, 10SRE, 10Goal, 10Patch-For-Review: Followup to backup1001 bacula switchover (misc pending tasks) - https://phabricator.wikimedia.org/T238048 (10jcrespo) [11:47:57] 10Data-Persistence-Backup, 10SRE: print a list of backed up directories in the MOTD of production servers - https://phabricator.wikimedia.org/T272686 (10jcrespo) 05Open→03Resolved a:03jcrespo I consider this solved, as I got not negative feedback on it. We can reevaluate printing more detailed stats once... [11:48:01] 10Data-Persistence-Backup, 10SRE, 10Goal, 10Patch-For-Review: Followup to backup1001 bacula switchover (misc pending tasks) - https://phabricator.wikimedia.org/T238048 (10jcrespo) [13:40:12] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [13:59:06] 10DBA, 10SRE, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) db1166 is now replicating but won't pool it until Monday though [16:01:14] 10Data-Persistence-Backup, 10SRE, 10Patch-For-Review, 10Puppet: Missing dependency on bacula-fd Puppet setup - https://phabricator.wikimedia.org/T256454 (10jcrespo) 05Open→03Resolved a:05jcrespo→03jbond Oh, this may not be needed, as it was added on the above notifies. Resolving unless we see it ha... [23:11:25] 10DBA, 10Patch-For-Review: Productionize x2 databases - https://phabricator.wikimedia.org/T269324 (10aaron) >>! In T269324#6786272, @Marostegui wrote: >>>! In T269324#6785507, @aaron wrote: >>>>! In T269324#6772394, @Marostegui wrote: >>> This is all done - hosts are ready to start getting data. >> >> I was t...