[00:19:14] 10DBA, 10Core-Platform-Team, 10monitoring, 10Performance-Team (Radar): Improve database application performance monitoring visibility - https://phabricator.wikimedia.org/T177778 (10CCicalese_WMF) [00:21:33] 10DBA, 10Core-Platform-Team, 10Structured-Data-Commons, 10Wikidata, and 4 others: Deploy MCR storage layer - https://phabricator.wikimedia.org/T174044 (10CCicalese_WMF) [05:04:49] 10DBA, 10Patch-For-Review, 10Schema-change: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 (10Marostegui) [05:05:03] 10DBA, 10Patch-For-Review, 10Schema-change: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 (10Marostegui) [05:05:05] 10Blocked-on-schema-change, 10MediaWiki-Database, 10Patch-For-Review: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 (10Marostegui) [05:05:19] 10DBA, 10MediaWiki-Database, 10PostgreSQL, 10Schema-change: Some tables lack unique or primary keys, may allow confusing duplicate data - https://phabricator.wikimedia.org/T17441 (10Marostegui) [05:05:21] 10Blocked-on-schema-change, 10MediaWiki-Database, 10Patch-For-Review: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 (10Marostegui) 05Open>03Resolved All done [05:05:35] 10DBA, 10Patch-For-Review, 10Schema-change: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 (10Marostegui) 05Open>03Resolved All done [05:05:38] 10DBA, 10Patch-For-Review, 10Schema-change: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 (10Marostegui) 05Open>03Resolved All done [05:06:04] 10DBA, 10MediaWiki-API, 10MediaWiki-Database, 10MW-1.32-release-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), and 3 others: ApiQueryExtLinksUsage::run query has crazy limit - https://phabricator.wikimedia.org/T59176 (10Marostegui) [05:09:09] 10DBA, 10Schema-change: Convert UNIQUE INDEX to PK in Production - https://phabricator.wikimedia.org/T199368 (10Marostegui) a:03Marostegui [05:10:12] 10DBA, 10Wikidata: Optimize recentchanges and wbc_entity_usage table across wikis - https://phabricator.wikimedia.org/T187521 (10Marostegui) [05:10:40] 10DBA, 10Wikidata: Optimize recentchanges and wbc_entity_usage table across wikis - https://phabricator.wikimedia.org/T187521 (10Marostegui) 05Open>03Resolved The lists of pending wikis have been done [05:12:57] 10Blocked-on-schema-change, 10DBA, 10Schema-change: Truncate SHA-1 indexes - https://phabricator.wikimedia.org/T51190 (10Marostegui) a:03Marostegui [05:16:06] 10Blocked-on-schema-change, 10DBA, 10Wikidata, 10Schema-change: Drop eu_touched in production - https://phabricator.wikimedia.org/T144010 (10Marostegui) a:03Marostegui I will go ahead and start deploying this change. It has been renamed for more than a year without any issues (so nothing is writing to it... [07:06:41] 10DBA, 10Patch-For-Review: Failover db1052 (s1) db primary master - https://phabricator.wikimedia.org/T197069 (10Marostegui) [07:27:29] 10DBA, 10Patch-For-Review: Failover db1052 (s1) db primary master - https://phabricator.wikimedia.org/T197069 (10Marostegui) So, I have restarted db1083 with binlog format = STATEMENT. This host is ready to be the candidate master once db1067 is the new master. Right now in s1 we have two candidate masters d... [07:27:42] 10DBA, 10Patch-For-Review: Failover db1052 (s1) db primary master - https://phabricator.wikimedia.org/T197069 (10Marostegui) [07:49:08] 10DBA, 10Patch-For-Review: Failover db1052 (s1) db primary master - https://phabricator.wikimedia.org/T197069 (10Johan) [08:53:36] can I reimage db1087? [08:53:56] yep! [08:54:06] jynus: if you wait ~20m I'll finish my test with the reimage [08:54:13] as I've merged the changes this morning ;) [08:54:28] should be good but I know you don't want to be the guinea pig [08:54:33] with DBs [08:56:12] oh, don't worry [08:56:23] it takes me 3 hours just to depool that server [08:56:36] and most likely it will not finish depooling until tomorrow [08:56:57] because mediawiki [08:57:53] jynus, marostegui: any objections against me doing puppet clean/deactivate for db2064? per T195228 the hardware won't be salvaged and it's causing ongoing errors in debdeploy as it's trying to connect to a broken server [08:57:54] T195228: db2064 crashed and totally broken - decommission it - https://phabricator.wikimedia.org/T195228 [08:58:26] no problem [08:58:35] however, I can see that being an issue [08:58:40] in other cases [08:58:45] moritzm: yeah, fine by me, maybe also worth commeting on the ticket so rob maybe can speed up decommissioning for that particularhost [08:59:09] maybe you may need a "disable client" flag on puppet or on app [09:02:22] jynus: yeah, I have a TODO to blacklist lists, but I'll still take it down as an interim hotfix [09:02:25] marostegui: ack, will do [09:02:59] moritzm: I have sent him a private message too [09:19:11] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: db2064 crashed and totally broken - decommission it - https://phabricator.wikimedia.org/T195228 (10MoritzMuehlenhoff) [13:05:17] Heads up: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/445179 has been merged meaning next week most of slow queries for change_tag table (=Special:Tags) will be gone [13:18:23] \o/ [15:26:54] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: db2064 crashed and totally broken - decommission it - https://phabricator.wikimedia.org/T195228 (10RobH) [15:32:47] 10DBA, 10Operations, 10decommission, 10ops-codfw: db2064 crashed and totally broken - decommission it - https://phabricator.wikimedia.org/T195228 (10RobH) [15:33:01] 10DBA, 10Operations, 10decommission, 10ops-codfw: db2064 crashed and totally broken - decommission it - https://phabricator.wikimedia.org/T195228 (10RobH) a:05RobH>03Papaul [16:05:46] 10DBA, 10MediaWiki-API, 10Performance: list=logevents slow for users with last log action long time ago - https://phabricator.wikimedia.org/T71222 (10Anomie) [20:36:34] 10DBA, 10decommission: Decommission db1059 - https://phabricator.wikimedia.org/T196606 (10RobH)