[04:56:49] 10DBA, 10Operations, 10Patch-For-Review: Decommission db2053.codfw.wmnet - https://phabricator.wikimedia.org/T231407 (10Marostegui) [04:58:46] 10DBA, 10Operations, 10Patch-For-Review: Decommission db2053.codfw.wmnet - https://phabricator.wikimedia.org/T231407 (10Marostegui) [05:02:07] 10DBA, 10Operations: Decommission db2043-db2069 - https://phabricator.wikimedia.org/T228258 (10Marostegui) [05:34:40] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [07:29:37] 10DBA, 10Data-Services, 10Operations: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 - https://phabricator.wikimedia.org/T231520 (10Marostegui) [07:29:48] 10DBA, 10Data-Services, 10Operations: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 - https://phabricator.wikimedia.org/T231520 (10Marostegui) p:05Triage→03Normal [07:30:01] 10DBA, 10Data-Services, 10Operations: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 - https://phabricator.wikimedia.org/T231520 (10Marostegui) [07:31:52] 10DBA: Remove grants for the old dbproxy hosts from the misc databases - https://phabricator.wikimedia.org/T231280 (10Marostegui) [07:31:57] 10DBA, 10Data-Services, 10Operations: Replace labsdb (wikireplicas) dbproxies: dbproxy1010 and dbproxy1011 - https://phabricator.wikimedia.org/T231520 (10Marostegui) [07:48:46] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui) [08:17:30] 10DBA, 10MediaWiki-Cache, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10Marostegui) pc1010 is done with all the cleanups+optimizations ` ssh pc1010.eqiad.wmnet df -hT /srv Filesystem... [08:31:24] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) s3 done - set it to 2: ` # dbctl -s eqiad section s3 get ; dbctl -s codfw section s3 get { "s3": { "master": "db1075", "min_replicas": 2, "readonly": false,... [08:31:39] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) [08:36:35] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) s4 done - set it to 4: ` # dbctl -s eqiad section s4 get ; dbctl -s codfw section s4 get { "s4": { "master": "db1081", "min_replicas": 4, "readonly": false,... [08:36:47] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) [08:37:08] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) [08:44:18] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) s2 done - set it to 4: ` # dbctl -s eqiad section s2 get ; dbctl -s codfw section s2 get { "s2": { "master": "db1066", "min_replicas": 4, "readonly": false,... [08:44:44] 10DBA, 10conftool: set min_replicas on database sections in dbctl - https://phabricator.wikimedia.org/T231019 (10Marostegui) [10:47:11] 10DBA, 10Operations: Switchover m1 primary master: db1063 to db1135 - https://phabricator.wikimedia.org/T231403 (10akosiaris) Sep 10 16:00UTC sounds okish to me as far as `bacula` goes. We will probably have a couple of full backups (it's the start of the month, when full backups happen and they might not have... [10:50:54] 10DBA, 10Operations: Switchover m1 primary master: db1063 to db1135 - https://phabricator.wikimedia.org/T231403 (10Marostegui) Thanks @akosiaris! As spoken on IRC, no need to re-schedule because of bacula. [10:54:38] 10DBA, 10Operations: Switchover m1 primary master: db1063 to db1135: Tuesday 10th September at 16:00 UTC - https://phabricator.wikimedia.org/T231403 (10Marostegui) [11:04:12] 10DBA, 10Operations: Drop puppet database from m1 - https://phabricator.wikimedia.org/T231539 (10Marostegui) [11:04:39] 10DBA, 10Operations: Drop puppet database from m1 - https://phabricator.wikimedia.org/T231539 (10Marostegui) p:05Triage→03Normal [14:24:20] 10DBA, 10MediaWiki-Cache, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10Marostegui) pc1010 is done: ` root@pc1010:/srv/sqldata-cache/parsercache# df -hT /srv/ Filesystem Type Siz...