[05:58:44] 10DBA, 10Patch-For-Review: Decommission db2070.codfw.wmnet - https://phabricator.wikimedia.org/T239684 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: `db2070.codfw.wmnet` - db2070.codfw.wmnet (**PASS**) - Downtimed host on Icinga - Downtimed management... [06:01:31] 10DBA, 10Operations: Decommission db2043-db2070 - https://phabricator.wikimedia.org/T228258 (10Marostegui) [06:02:05] 10DBA, 10Operations: Decommission db2043-db2070 - https://phabricator.wikimedia.org/T228258 (10Marostegui) 05Open→03Resolved All these hosts have been decommissioned or pending the last on-site decommissioning steps. [06:02:07] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui) [06:02:54] 10DBA, 10Goal: Address Database infrastructure blockers on datacenter switchover & multi-dc deployment - https://phabricator.wikimedia.org/T220170 (10Marostegui) 05Open→03Resolved Everything is now done [07:03:04] 10.4.11 and 10.3.21 released [07:59:12] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [08:07:23] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [08:36:38] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [08:42:54] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [08:44:28] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1123 - https://phabricator.wikimedia.org/T240534 (10Marostegui) p:05Triage→03High This is 3 primary master, can we make sure we get this replaced before the holidays break? [08:45:15] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1123 - https://phabricator.wikimedia.org/T240534 (10Marostegui) a:03wiki_willy Assigning to @wiki_willy to make sure it is under #dc-ops radar [08:53:21] "This is 3 primary master"? maybe you meant s3 or something else? [08:53:39] ups, yes, I meant s3 :) [08:53:42] fixing [08:53:56] done - thanks [09:04:31] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [09:24:14] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [14:24:25] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [15:50:08] 10DBA, 10Performance-Team, 10conftool, 10Patch-For-Review: #dbctl: manage 'externalLoads' data - https://phabricator.wikimedia.org/T229686 (10CDanis) a:05aaron→03CDanis [16:02:01] about to do a dbctl release and schema migration, no pushes for a few minutes please [16:24:14] ok! a little longer than expected due to a CI bug i set myself up for, but, all good [16:25:14] nice! [17:05:17] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1123 - https://phabricator.wikimedia.org/T240534 (10wiki_willy) a:05wiki_willy→03Jclark-ctr @Jclark-ctr - looks like this one is still under warranty, so you should be able to just RMA it. Thanks, Willy [22:44:16] marostegui: good news, looks like we can stop updating host:port entries in the hostsByName section of the PHP files as well [22:45:23] this is assuming that mediawiki never needs to contact a database server cross-DC, which I am not sure about [22:45:35] (and ofc if we wanted to put all servers in both hostsByName lists, that would not be hard)