[05:04:21] 10DBA, 10Operations: db2097 (codfw s1&s6 source backups) mariadb@s6 *process* (10.1.39) crashed on 2019-06-08 - https://phabricator.wikimedia.org/T225378 (10Marostegui) 05Open→03Resolved a:05Marostegui→03jcrespo As spoken, I am going to close this as the scope of the ticket is done. I will create a new... [05:04:23] 10DBA, 10Goal: Implement database binary backups into the production infrastructure - https://phabricator.wikimedia.org/T206203 (10Marostegui) [05:20:58] 10DBA: Re-build db2097 s1 and s6 - https://phabricator.wikimedia.org/T228613 (10Marostegui) [05:21:09] 10DBA: Re-build db2097 s1 and s6 with Debian Buster and 10.3 - https://phabricator.wikimedia.org/T228613 (10Marostegui) p:05Triage→03Normal [05:23:18] 10DBA, 10Math: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055 (10Marostegui) So, for the table drop what I will do will be. Drop the table from a enwiki on a codfw replica (passive DC) to make sure there are no writes (if there are, replication will break and we'll know). Wil... [07:05:03] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [07:05:26] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) I have provisioned dbproxy2001 into m1 codfw - with notifications disabled as it is not an active proxy (or even service) [07:17:20] 10DBA, 10Math: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055 (10Marostegui) [07:17:37] 10DBA, 10Math: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055 (10Marostegui) [07:18:10] 10DBA, 10AbuseFilter: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) [07:52:26] 10DBA, 10AbuseFilter, 10Patch-For-Review: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) I have altered db1134, and will monitor the errors to make sure nothing complains about this column being gone. ` root@db1134.eqiad.wmnet[enwiki]> show create ta... [07:52:46] 10DBA, 10AbuseFilter, 10Patch-For-Review: Drop abuse_filter_log.afl_log_id in production - https://phabricator.wikimedia.org/T226851 (10Marostegui) [08:10:23] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [08:34:33] 10DBA, 10Math: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055 (10Marostegui) I have renamed the table on db2116: enwiki: ` root@db2116.codfw.wmnet[enwiki]> show tables like 'TO_DRO%'; +----------------------------+ | Tables_in_enwiki (TO_DRO%) | +----------------------------+... [08:34:55] 10DBA, 10Math: Remove table `math` from the database - https://phabricator.wikimedia.org/T196055 (10Marostegui) [09:09:08] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) a:03Marostegui [12:46:44] 10DBA, 10Goal: Address Database infrastructure blockers on datacenter switchover & multi-dc deployment - https://phabricator.wikimedia.org/T220170 (10Marostegui) [13:14:34] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) [13:15:23] 10DBA, 10Patch-For-Review: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4] - https://phabricator.wikimedia.org/T202367 (10Marostegui) dbproxy1014 is now ready in m1 (as standby) and as a replacement of dbproxy1006 which can be decommissioned. [16:06:44] 10DBA, 10Goal: Address Database infrastructure blockers on datacenter switchover & multi-dc deployment - https://phabricator.wikimedia.org/T220170 (10Papaul) [16:12:56] 10DBA, 10Operations, 10ops-codfw: pc2010 possibly broken memory - https://phabricator.wikimedia.org/T227552 (10Papaul) a:05Papaul→03Marostegui DIMM replaced and Firmware upgrade Before Operating System Version Service Tag DHPR0S2 Asset Tag DHR0S2 Express Service Code 29369346482 BIOS Version 1.5.4 Lif... [16:19:18] marostegui: "Wait until after Tuesday 30th for the s8 master migration" <-- that's accurate, right? [16:19:50] cdanis: correct! [16:20:26] ty [16:20:27] 10DBA, 10Operations, 10ops-codfw: pc2010 possibly broken memory - https://phabricator.wikimedia.org/T227552 (10Marostegui) Thanks Papaul. Memory looking good. ` root@pc2010:~# free -m total used free shared buff/cache available Mem: 257392 674 256516... [17:08:29] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Papaul) [18:24:15] 10DBA, 10Operations, 10ops-codfw, 10Goal: rack/setup/install db21[21-30].codfw.wmnet - https://phabricator.wikimedia.org/T227113 (10Papaul)