[04:58:50] 10DBA, 10Patch-For-Review: Fix revision special indexes and partitions on db1103:3314 and db1113:3316 - https://phabricator.wikimedia.org/T221782 (10Marostegui) 05Open→03Resolved db1113:3316 frwiki, jawiki and ruwiki got the revision table fixed and now have the proper indexes and PK: ` PRIMARY KEY (`rev... [05:00:52] 10DBA, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission parsercache hosts: pc1004.eqiad.wmnet pc1005.eqiad.wmnet pc1006.eqiad.wmnet - https://phabricator.wikimedia.org/T210969 (10Marostegui) @RobH @Cmjohnson I have removed the spare role entries, so only pending the DNS entries... [05:01:02] 10DBA, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission parsercache hosts: pc1004.eqiad.wmnet pc1005.eqiad.wmnet pc1006.eqiad.wmnet - https://phabricator.wikimedia.org/T210969 (10Marostegui) a:05Cmjohnson→03RobH [05:10:57] 10DBA, 10Patch-For-Review: Decommission 2 codfw x1 hosts db2033 and db2034 - https://phabricator.wikimedia.org/T219493 (10Marostegui) db2045 has been compared with db2080 and it is the same. It can now be moved to x1 to replace db2034. [13:01:26] ok with me setting up 1 or 2 instances at T220572? [13:01:27] T220572: Productionize eqiad and codfw source backup hosts & codfw backup test host - https://phabricator.wikimedia.org/T220572 [13:02:02] sure! [13:02:57] do you have thoughts on mirroring eqiad or avoid mirroring the distribution of sections? [13:04:08] I think mirroring eqiad is what would make more sense from an organizational point of view no? [13:04:57] so I ask because that is one valid thought [13:05:45] but the opposite would be "redundancy by avoiding combine the same sections on the same host" [13:06:01] like we do for special replicas on production [13:06:17] yeah, but eqiad and codfw use different hosts for everything [13:06:28] if a source host is down in eqiad for s1 and s3, it won't impact codfw [13:07:07] I wanted to hear your thoughts and opinion [13:07:49] So as codfw and eqiad are fully independent I think it makes sense to mirror configs [13:08:33] if I wanted to be devil's advocate, my argument would be: "a maintenance script/disk space goes wrong on s4" [13:09:07] but because on eqiad is with e.g. s5 and on codfw with e.g. s7, it is better? [13:09:35] on the other side, it can be hard to organize backups and recovering between dcs [13:10:50] With your example, we might gain a few more hours, but I am not sure it is worth the hassle of not mirroring things [13:11:09] As in the possibility of that happening isn't big and it might not save us from much [13:11:16] thanks, you convinced me [13:11:20] haha [13:11:24] Glad to hear! [13:11:48] I think I will setup s1 and s6 first [13:16:59] 10DBA: Productionize eqiad and codfw source backup hosts & codfw backup test host - https://phabricator.wikimedia.org/T220572 (10jcrespo) a:03jcrespo [13:33:23] 10DBA, 10Patch-For-Review: questions about standalone wmf-mariadb103 - https://phabricator.wikimedia.org/T221463 (10Marostegui) a:03Andrew Assigning this back to you to decide what you want to do :-) Our opinion has been given :-) T221463#5127855 T221463#5130471 Basically: better to use 10.1 on stretch and... [13:35:25] 10DBA, 10Patch-For-Review: questions about standalone wmf-mariadb103 - https://phabricator.wikimedia.org/T221463 (10Reedy) [14:05:42] 10DBA, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission parsercache hosts: pc1004.eqiad.wmnet pc1005.eqiad.wmnet pc1006.eqiad.wmnet - https://phabricator.wikimedia.org/T210969 (10Marostegui) So DNS is now clean Are the switches ports also cleaned up? T210969#4795686 [14:09:05] 10DBA, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission parsercache hosts: pc1004.eqiad.wmnet pc1005.eqiad.wmnet pc1006.eqiad.wmnet - https://phabricator.wikimedia.org/T210969 (10Marostegui) [14:21:22] 10DBA, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission parsercache hosts: pc1004.eqiad.wmnet pc1005.eqiad.wmnet pc1006.eqiad.wmnet - https://phabricator.wikimedia.org/T210969 (10Marostegui) [14:21:53] 10DBA, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission parsercache hosts: pc1004.eqiad.wmnet pc1005.eqiad.wmnet pc1006.eqiad.wmnet - https://phabricator.wikimedia.org/T210969 (10Marostegui) I have tried to set these servers as unracked, but I have failed to do so on Netbox (I... [14:22:09] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) - https://phabricator.wikimedia.org/T209858 (10Marostegui) I have tried to set these servers as unracked, but I have failed to do so on Netbox (I guess I don'... [14:43:22] 10DBA, 10Operations, 10decommission, 10ops-codfw, 10Patch-For-Review: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) - https://phabricator.wikimedia.org/T209858 (10Marostegui) Nevermind that comment, I read https://wikitech.wikimedia.org/wiki/Server_Lifecycle#States wrongly.... [14:44:09] 10DBA, 10Operations, 10decommission, 10ops-eqiad, 10Patch-For-Review: Decommission parsercache hosts: pc1004.eqiad.wmnet pc1005.eqiad.wmnet pc1006.eqiad.wmnet - https://phabricator.wikimedia.org/T210969 (10Marostegui) 05Open→03Resolved I read https://wikitech.wikimedia.org/wiki/Server_Lifecycle#State... [15:36:27] 10DBA, 10Core Platform Team, 10MediaWiki-Database, 10MediaWiki-Logging, and 6 others: Special:Log on commons -- entire web request took longer than 60 seconds and timed out - https://phabricator.wikimedia.org/T221458 (10EvanProdromou) So, it looks like this ticket is done, and we're just waiting for it to... [15:37:30] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging, 10Operations, and 5 others: Special:Log on commons -- entire web request took longer than 60 seconds and timed out - https://phabricator.wikimedia.org/T221458 (10EvanProdromou) [15:38:43] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging, 10Operations, and 5 others: Special:Log on commons -- entire web request took longer than 60 seconds and timed out - https://phabricator.wikimedia.org/T221458 (10Marostegui) I believe so [18:32:44] 10DBA, 10MediaWiki-API, 10Patch-For-Review: Slow query "ApiQueryLogEvents::execute" after actor rollout - https://phabricator.wikimedia.org/T220999 (10Anomie)