[03:52:27] 10DBA, 06Labs, 10Tool-Labs: Replication seems to be halted for multiple databases - https://phabricator.wikimedia.org/T142310#2531012 (10Peachey88) [10:22:38] 10DBA, 10MediaWiki-extensions-Renameuser, 13Patch-For-Review, 07Performance, 07Wikimedia-Incident: Rename user creates lag on enwiki - https://phabricator.wikimedia.org/T116425#2531281 (10Glaisher) There was slave lag on Meta-Wiki (s7) causing writes to be locked for more than three minutes a little whil... [10:23:57] 10DBA, 10MediaWiki-extensions-Renameuser, 13Patch-For-Review, 07Performance, 07Wikimedia-Incident: Rename user creates lag on enwiki - https://phabricator.wikimedia.org/T116425#2531295 (10Glaisher) It's happening again now on s7. [10:41:56] 10DBA, 10MediaWiki-extensions-Renameuser, 13Patch-For-Review, 07Performance, 07Wikimedia-Incident: Rename user creates lag on enwiki - https://phabricator.wikimedia.org/T116425#2531312 (10aaron) I don't think 7k rows matter. I do however see a huge spike in write query time warnings in DBPerformance log... [11:15:02] 10DBA, 10MediaWiki-extensions-Renameuser, 13Patch-For-Review, 07Performance, 07Wikimedia-Incident: Rename user creates lag on enwiki - https://phabricator.wikimedia.org/T116425#2531318 (10aaron) I made https://gerrit.wikimedia.org/r/303417 for the above problem (the jobs triggered a hook that triggered... [11:29:44] 10DBA, 10MediaWiki-extensions-Renameuser, 13Patch-For-Review, 07Performance, 07Wikimedia-Incident: Rename user creates lag on enwiki - https://phabricator.wikimedia.org/T116425#2531326 (10jcrespo) I can confirm the long running writes on the master were from queries such as: ``` UPDATE /* RevisionReviewF... [15:32:05] 10DBA, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2531513 (10AlexMonk-WMF)