[06:00:23] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [06:04:14] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) db1068 (s4 master) has too much concurrency on the `page` table to be able to alter it live (metada locking issues) [06:04:54] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) [06:05:19] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [06:05:31] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) [06:25:12] 10DBA, 10Operations, 10ops-eqiad, 10User-Marostegui: rack/setup/install db11[26-38].eqiad.wmnet - https://phabricator.wikimedia.org/T211613 (10Marostegui) Hey @robh! Thanks for putting up an initial racking plan. It is a bit more complicated than just replacing the masters, as we also have candidate mast... [06:47:34] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Marostegui) a:03Marostegui I have renamed the tables on db1078 for now, to make sure nothing really breaks. Renamed t... [06:48:54] addshore: db1111 has finished importing wb_terms again (still replicating to db1112 - the slave) [08:37:38] 10DBA: labsdb1004 replication broken for linkwatcher_linklog table - https://phabricator.wikimedia.org/T211210 (10Marostegui) 05Open>03Resolved a:03Marostegui labsdb1004 finally caught up and so far everything is fine. The table seems to be in sync. ` for i in labsdb1004 labsdb1005; do echo $i; mysql.py -h... [10:58:57] 10DBA, 10User-Banyek, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 (10Urbanecm) Doesn't look like a site request to me. [11:02:21] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Zoranzoki21) Everything looks good.. [13:50:26] 10DBA, 10User-Banyek, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 (10Banyek) a:03Banyek [14:30:22] I format and add the `/dev/tank/data` logical volume on the new dbstore hosts. [14:39:53] 10DBA, 10Analytics, 10Analytics-Kanban, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Banyek) While the section install schema being discussed on T172410 I created and mounted the data directories on the new hosts. [15:09:44] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Banyek) on labsdb1010 I stared to create materialized views from all the ` comment` views, except enwiki - as we... [20:41:59] 10DBA, 10Operations, 10decommission, 10ops-codfw: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) - https://phabricator.wikimedia.org/T209858 (10Papaul) ` papaul@asw-b-codfw> show interfaces ge-5/0/35 descriptions Interface Admin Link Description ge-5/0/35 down do... [20:42:25] 10DBA, 10Operations, 10decommission, 10ops-codfw: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) - https://phabricator.wikimedia.org/T209858 (10Papaul) [20:46:40] 10DBA, 10Operations, 10decommission, 10ops-codfw: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) - https://phabricator.wikimedia.org/T209858 (10Papaul) @Marostegui any reason why production DNS is still showing for pc2004? 170 1H IN PTR pc2004.codfw.wmnet. [20:56:07] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) - https://phabricator.wikimedia.org/T209858 (10Papaul)