[04:38:59] 10DBA, 10Core Platform Team Backlog (Watching / External), 10Performance: Review special replica partitioning of certain tables by `xx_user` - https://phabricator.wikimedia.org/T223151 (10Marostegui) >>! In T223151#5198820, @Anomie wrote: >>>! In T223151#5198160, @Marostegui wrote: >> @Marostegui moved this... [05:13:56] 10DBA, 10Goal, 10Patch-For-Review: Productionize db2[103-120] - https://phabricator.wikimedia.org/T222772 (10Marostegui) [05:36:16] 10DBA: Decommission db2041 - https://phabricator.wikimedia.org/T223950 (10Marostegui) [05:46:57] 10DBA, 10Patch-For-Review: Decommission db2041 - https://phabricator.wikimedia.org/T223950 (10Marostegui) [05:47:23] 10DBA, 10Patch-For-Review: Decommission db2041 - https://phabricator.wikimedia.org/T223950 (10Marostegui) p:05Triage→03Normal [05:48:05] 10DBA: Decommission db1064 - https://phabricator.wikimedia.org/T223217 (10Marostegui) [05:49:46] 10DBA, 10Patch-For-Review: Decommission db2041 - https://phabricator.wikimedia.org/T223950 (10Marostegui) [07:33:24] 10DBA, 10Goal, 10Patch-For-Review: Productionize db2[103-120] - https://phabricator.wikimedia.org/T222772 (10Marostegui) [09:01:46] 10DBA, 10Operations, 10serviceops, 10HHVM, 10PHP 7.2 support: Increased instability in MediaWiki backends (according to load balancers) - https://phabricator.wikimedia.org/T223952 (10Joe) Having confirmed pybal is not an issue here, while databases might have a part in the issue, I'll change the tags acc... [09:02:35] 10DBA, 10Goal, 10Patch-For-Review: Address Database infrastructure blockers on datacenter switchover & multi-dc deployment - https://phabricator.wikimedia.org/T220170 (10Marostegui) [09:50:08] 10DBA, 10Operations, 10Performance-Team, 10serviceops, and 2 others: Increased instability in MediaWiki backends (according to load balancers) - https://phabricator.wikimedia.org/T223952 (10Marostegui) From the DB land this is what I have seen regarding s1: - Small increase on query response time starting... [10:03:55] 10DBA, 10Goal, 10Patch-For-Review: Productionize db2[103-120] - https://phabricator.wikimedia.org/T222772 (10Marostegui) [14:56:56] 10DBA, 10Operations, 10Performance-Team, 10serviceops, and 2 others: Increased instability in MediaWiki backends (according to load balancers) - https://phabricator.wikimedia.org/T223952 (10Marostegui) [15:01:58] 10DBA, 10Operations, 10Performance-Team, 10serviceops, and 2 others: Increased instability in MediaWiki backends (according to load balancers) - https://phabricator.wikimedia.org/T223952 (10CDanis) We saw one of these events at 14:48 today and pybal reported fetch failures for -- and wanted to depool -- ba... [15:02:53] 10DBA, 10Operations, 10Performance-Team, 10serviceops, and 2 others: Increased instability in MediaWiki backends (according to load balancers) - https://phabricator.wikimedia.org/T223952 (10Marostegui) >>! In T223952#5201169, @CDanis wrote: > We saw one of these events at 14:48 today and pybal reported fet... [21:27:45] 10DBA, 10MediaWiki-Database, 10MediaWiki-Logging, 10Operations, and 4 others: Special:Log on commons -- entire web request took longer than 60 seconds and timed out - https://phabricator.wikimedia.org/T221458 (10Ladsgroup)