[00:13:43] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350#4320761 (10Yann) There are at least 150 files without a page (and probably more). Starting from https://commons.wikimedia.org... [00:14:03] so apparently phadmin@phab1001 is no longer authorized to modify phabricator's schema [00:17:15] 10DBA, 10Deployments, 10Phabricator: Mysql Access denied to 'phadmin'@'10.64.0.198' - https://phabricator.wikimedia.org/T198367#4320781 (10mmodell) [00:19:03] 10DBA, 10Deployments, 10Phabricator: Mysql Access denied to 'phadmin'@'10.64.0.198' - https://phabricator.wikimedia.org/T198367#4320793 (10mmodell) p:05Triage>03High Marking high priority because there are security related patches that need to be deployed asap. [00:25:20] 10DBA, 10Deployments, 10Phabricator: Mysql Access denied to 'phadmin'@'10.64.0.198' - https://phabricator.wikimedia.org/T198367#4320781 (10Paladox) that user + ip are in the grants https://github.com/wikimedia/puppet/blob/a7883098b37ecb23d1291d404857a8105a193fee/modules/role/templates/mariadb/grants/producti... [01:03:05] 10DBA, 10Deployments, 10Phabricator: Mysql Access denied to 'phadmin'@'10.64.0.198' - https://phabricator.wikimedia.org/T198367#4320867 (10mmodell) Well maybe the grants didn't get applied properly to the new database master? I'm really not sure. [04:12:49] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350#4320899 (10AlexisJazz) Assuming this was not a case of "It compiles, ship it!" I am curious as to why this wasn't noticed whe... [04:35:03] 10DBA, 10Deployments, 10Phabricator: Mysql Access denied to 'phadmin'@'10.64.0.198' - https://phabricator.wikimedia.org/T198367#4320781 (10Marostegui) Can you try again? [04:37:54] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4320909 (10Marostegui) s8 eqiad progress [x] dbstore1002 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] db1109 [] db1104... [04:38:07] 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10Schema-change: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926#4320910 (10Marostegui) s8 eqiad progress [x] dbstore1002 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] db1109 [] db1104 [] db1124 []... [04:38:10] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Patch-For-Review, 10Wikidata-Ministry-Of-Magic: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193#4320911 (10Marostegui) s8 eqiad progress [x] dbstore1002 [] labsdb1011 [] labsdb1010... [04:38:13] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#4320912 (10Marostegui) s8 eqiad progress [x] dbstore1002 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] db1109 [] db1104 [] db1124 [] db1101 [] d... [04:39:17] 10DBA, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Patch-For-Review, 10Schema-change: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316#4320913 (10Marostegui) [04:39:29] 10DBA, 10Multi-Content-Revisions, 10Patch-For-Review, 10Schema-change: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926#4320914 (10Marostegui) [04:39:49] 10Blocked-on-schema-change, 10MediaWiki-Change-tagging, 10MediaWiki-Database, 10Patch-For-Review, 10Wikidata-Ministry-Of-Magic: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193#4320915 (10Marostegui) [04:40:00] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737#4320916 (10Marostegui) [04:59:55] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350#4320941 (10Marostegui) >>! In T198350#4320639, @daniel wrote: > Is that stack trace representative? is it always INSERT INTO... [05:29:25] 10DBA, 10Deployments, 10Phabricator: Mysql Access denied to 'phadmin'@'10.64.0.198' - https://phabricator.wikimedia.org/T198367#4320947 (10Marostegui) This seems to be working for me from phab1001, but I would like to get your confirmation: ``` root@phab1001:~# mysql --skip-ssl -hdb1072 -p -uphadmin -e "sele... [05:32:32] 10DBA, 10Deployments, 10Phabricator: Mysql Access denied to 'phadmin'@'10.64.0.198' - https://phabricator.wikimedia.org/T198367#4320964 (10Marostegui) Also works from phab1002: ``` root@phab1002:~# mysql --skip-ssl -hdb1072 -p -uphadmin -e "select now()" Enter password: +---------------------+ | now()... [07:06:09] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#4320989 (10jcrespo) @Ladsg... [08:39:05] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350#4321099 (10Tgr) There has always been a slow but steady stream of lock timeouts: https://logstash.wikimedia.org/app/kibana#/d... [08:40:11] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350#4321101 (10Addshore) >>! In T198350#4320941, @Marostegui wrote: > From what I can see it was not only related to that table a... [08:43:33] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350#4320155 (10jcrespo) @Addshore Most of those you point happen all the time, unlike the ones @Marostegui pointed, which are new. [08:51:48] o/ marostegui [08:52:08] o/ [08:52:21] Re the above ticket, I guess this magical performance schema tables might be able to tell us if full table locks were happening due to any queries during that period? [08:54:35] no [08:54:47] but I already know the problem [08:54:53] because it is not the first time it happened [08:55:24] a transaction is blocking the comment tables, creating too much contention [08:55:48] because someone user a very high consistency level cross-tables [08:56:03] but that doesn't scale for the comment table [08:56:58] addshore: see https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/425283/ [09:00:46] basically, comment table is an append only table, so lower level of consistency can be used for it [09:01:12] no need to lock potentially millions of rows that refer to them, just the other tables [09:02:00] Right, and so in the MCR refactoring this little bit has probably gone missing and no longer happens [09:02:12] ? [09:02:34] jynus: your saying that that patch fixed the issue before? or? [09:02:50] I don't know what code is changed, I am exptrapolating a similar thing is happening [09:02:57] ahhh, okay [09:02:59] but to a different part of the code [09:03:08] that definitely fixed that code [09:03:31] but there are potentially many other parts of the code that just did the join new tables [09:03:46] so CommentStore:insertInternal is called when a new revision is saved, and it stores a new revid + commentid pair [09:03:52] whithout considering new features == new scalability threads [09:03:56] *threats [09:04:02] yup, okay, well, the code in the patch you linked to was touched in the last branch so I guess I'll start looking there [09:04:07] in theory that should never conflict as no other process can own the same revision ID [09:04:41] that's different from actor / comment tables where the same row can belong to many edits [09:06:42] sorry, but for code questions, you may want to ask someone that has written actual mediawiki code- I am an ignorant on that regard, sorry :-( (I have 0 commits on mediawki) [09:06:51] I am also have 0 involvement on MCR [09:08:56] actually, I may 1 one commit, written with gerrit code editor [09:09:00] *have [09:09:01] jynus: btw is there a way to automatically log what's the other query causing the lock timeout? I imagine running SHOW ENGINE STATUS in a relatively unprivileged MediaWiki application connection is a non-starter [09:10:22] we have those metrics available, but because public prometheus we were asked to turn those off [09:10:43] I can give you something we have on tendril [09:10:47] let me see [09:11:26] that would be great, knowing what call made the lock is a huge help in figuring out what's going on [09:11:42] well, normally it is the call itself [09:11:45] self-block [09:12:13] as I said, the other time it was a contention issue, not "one call breaking the others" [09:13:20] which wiki had the issue? [09:15:17] mostly commonswiki [09:16:15] 10DBA, 10Deployments, 10Phabricator: Mysql Access denied to 'phadmin'@'10.64.0.198' - https://phabricator.wikimedia.org/T198367#4321166 (10mmodell) the upgrade script is using m3-master.eqiad.wmnet. When I try that: ``` twentyafterfour@phab1001:/srv/phab$ mysql --skip-ssl -h m3-master.eqiad.wmnet -P3323 -p... [09:16:46] Internal locking is performed within the MySQL server itself to manage contention for table contents by multiple threads. This type of locking is internal because it is performed entirely by the server and involves no other programs. See Section 8.11.1, “Internal Locking Methods”. [09:16:51] tgr: this is what I have https://phabricator.wikimedia.org/P7314 [09:17:09] sorry, I always mess up the Linux dual clipboard thing [09:17:13] if you try to write the same row twice [09:17:25] or if you do select for update on the same rows [09:17:25] INSERT INTO revision_comment_temp (revcomment_rev,revcomment_comment_id) VALUES (X,Y) [09:17:29] is the affected query [09:17:40] those are cases in which you will get an innodb lock [09:17:58] you don't need an explicit lock [09:18:20] with the whole table being a PK + a unique index on revcomment_rev [09:18:22] (please note that is private data - the content of the queries [09:18:35] so do not publish those as is [09:18:55] AIUI the only way to get contention there is if multiple processes try to insert with the same revcomment_rev [09:19:04] noted, thanks [09:19:13] I see ​SELECT FOR UPDATE [09:19:20] I would be more worried about that [09:19:31] FOR UPDATE == lock [09:19:47] and transactions are larger than a single query [09:20:43] for example, SELECT /* WikiPage::doDeleteArticleReal */ 1 FROM `revision`, `revision_comment_temp` WHERE rev_page = '29419032' FOR UPDATE [09:20:49] was up for 244 seconds [09:20:57] others for 19 seconds [09:21:03] and it's a delete, it could be running long - image usage needs to be updated and whatever [09:21:06] https://tendril.wikimedia.org/report/slow_queries_checksum?checksum=1e575c4bdbc2ee81bcfdf42ce46bd297&host=db1068&user=wikiuser&schema=commonswiki&hours=15 [09:21:19] I know this is not good metrics [09:21:32] but I think it should be enough to start a proper research [09:21:36] what I don't get is how can a delete and a comment insertion happen for the same revision ID [09:21:58] a bad indexed table [09:22:06] can lead to locks on more than one record [09:22:18] not saying it is that [09:22:24] just avoiding suppositions [09:22:41] and it affects all of its transactions parts [09:27:17] 10DBA, 10Deployments, 10Phabricator: Mysql Access denied to 'phadmin'@'10.64.0.198' - https://phabricator.wikimedia.org/T198367#4320781 (10jcrespo) Can you try now, I have a theory. [09:32:45] tgr I am adding all relevant queries to the phaste [09:33:04] sadly, normally this is not a question of "single call"'s fault [09:33:14] but the combination that explodes [09:33:29] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350#4321248 (10Addshore) [09:33:53] jynus: thanks! what you say about deletes locking the rows makes sense [09:34:26] it's probably not related to bad indexing, just people editing and deleting something at the same time [09:34:51] yes, but we are supposed to make that very improbable [09:35:00] if that's the case, it's an easy fix, there is no real reason why the rows should be deleted from revision_comment_temp instantly [09:35:22] or just last a few seconds- I think that lasted a few minutes [09:35:52] yeah, I have no idea what's up with that long-running delete [09:36:11] tgr: I can tell you this- the issue is page got locked for 15 + seconds [09:36:13] did not have many revisions, did not have much global usage as far as I can tell [09:36:34] new inserts to page were failing [09:36:42] which should almost never happen [09:36:56] specially for new records [09:37:22] it is the insert ignore though, so they where most likely existing records [09:37:29] but they got blocked anyway [09:38:08] what I cannot tell you is which ones are the cause are which are the consequence [09:38:50] focus on https://phabricator.wikimedia.org/P7314#42317 [09:44:43] 10DBA, 10Deployments, 10Phabricator: Mysql Access denied to 'phadmin'@'10.64.0.198' - https://phabricator.wikimedia.org/T198367#4321265 (10jcrespo) Yeah, let's elevate this to a security incident: ``` $ mysql -h m3-master.eqiad.wmnet -P 3306 -uphadmin -e "select now()" -p Enter password: +-----------------... [10:43:09] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10Tgr) Here's a stack trace for a Commons error, which is more typical (probably. We really need a frequent stack traces vie... [11:06:54] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10Tgr) The errors in T197464#4321254 are less strange, they all fail in the page lock step. However, starting the process wi... [11:09:41] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Ladsgroup) >>! In T1044... [11:39:12] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10DerHexer) I got the first error when I used Special:Nuke on Wikimedia Commons. Is it possible that something has not been... [11:50:08] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10jcrespo) {icon heart} Tgr analysis [12:08:36] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10matmarex) >>! In T198350#4320899, @AlexisJazz wrote: > Assuming this was not a case of "It compiles, ship it!" I am curiou... [12:36:15] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10daniel) I created a patch that reverts the MCR patches related to RevisionStore, but keeps the chang... [12:40:06] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10daniel) >>! In T198350#4321620, @matmarex wrote: >>>! In T198350#4320899, @AlexisJazz wrote: >> Assu... [12:45:46] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10Aklapper) (As this is related to revisions, could the latest examples in T179884, T197464#4321254, T... [12:52:38] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10daniel) @Aklapper As far as I can see, recent instance of the first two issues T179884 and T197464... [13:03:38] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External), and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) Thanks Amir... [13:41:18] jynus, marostegui: FYI, I'm merging the microcode patch for DB serves in a bit [13:41:31] \o/ [13:50:25] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10Tgr) >>! In T198350#4321620, @matmarex wrote: > The errors also did not appear (or at least not in n... [14:22:44] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10daniel) The DBPerformance log shows a spike during the time wmf-10 was deployed on group1: 10DBA, 10MediaWiki-Database, 10Wikidata, 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10Tgr) [[https://logstash.wikimedia.org/goto/43a44ac599ce220cfdcac775c674179d|There are]] about 1000 e... [14:43:08] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10daniel) After staring at the code a bit, my best guess is: The MCR refactoring introduced doAtomicS... [14:44:37] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for satwiki - https://phabricator.wikimedia.org/T198401 (10Urbanecm) [14:45:17] 10DBA, 10Cloud-Services, 10User-Urbanecm: Prepare and check storage layer for satwiki - https://phabricator.wikimedia.org/T198401 (10Marostegui) Let us know when it is created so we can sanitize it [15:31:08] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10daniel) Here are a few things I poked at, without finding anything relevant: @tgr suspects that the... [15:44:13] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10greg) a:05dduvall>03None [16:12:04] 10DBA, 10Operations-Software-Development, 10Patch-For-Review: Debmonitor: request for misc DB allocation - https://phabricator.wikimedia.org/T192875 (10Volans) @jcrespo, You asked to not resolve it for now, anything left here on your side? Anything I can help with? [16:21:08] 10DBA, 10Operations-Software-Development, 10Patch-For-Review: Debmonitor: request for misc DB allocation - https://phabricator.wikimedia.org/T192875 (10jcrespo) We should create a ticket to document the bug and have some kind of workaround before closing it. [16:21:58] 10DBA, 10MediaWiki-Database, 10Wikidata, 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10daniel) The above patch is an alternative attempt to fix the lock retention issue. It's the best I c... [16:26:55] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10Aklapper) [16:53:29] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10Tgr) ``` $ mwscript shell.php mediawikiwiki >>> $mainPage = Title::... [17:27:26] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10dduvall) It seems there are two competing patches here, and given t... [17:43:03] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10aaron) >>! In T198350#4321988, @daniel wrote: > The DBPerformance l... [17:49:40] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10daniel) @dduvall never mind my patches. The second one won't fix th... [17:50:47] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10daniel) Mentioning for reference: {T191892} and T191875#4120050. Th... [18:03:20] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10dduvall) Thanks again for mobilizing! And thanks for the fix, @tgr!... [18:57:28] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10daniel) For the record, here is when and why I added the offending... [19:01:13] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10Jarekt) Another side-effect of this issue are couple of hundred of... [19:50:29] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10Jarekt) I just got [WzU7SwpAME8AAGa@HYEAAAAC] 2018-06-28 19:47:39:... [19:57:26] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10Yann) #metoo: [WzU7GQpAICoAAIy18VQAAABW] 2018-06-28 19:46:51: Fatal... [21:01:13] 10DBA, 10Continuous-Integration-Infrastructure, 10MediaWiki-Database, 10Quibble, 10Release-Engineering-Team (Kanban): Enable MariaDB/MySQL strict mode on CI db hosts - https://phabricator.wikimedia.org/T119371 (10hashar) @jcrespo we had MediaWiki tests running with `sql_mode = 'TRADITIONAL'` . I clearly... [21:01:17] 10DBA, 10Continuous-Integration-Infrastructure, 10MediaWiki-Database, 10Quibble, 10Release-Engineering-Team (Kanban): Enable MariaDB/MySQL strict mode on CI db hosts - https://phabricator.wikimedia.org/T119371 (10hashar) [21:12:28] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10greg) >>! In T198350#4322905, @Jarekt wrote: > Another side-effect... [21:14:28] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10dduvall) @tgr's fix worked. Thanks again, @tgr! I'll remove this as... [21:14:38] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10dduvall) [21:56:19] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10Tgr) >>! In T198350#4323192, @greg wrote: > @Tgr @daniel others: Th... [22:02:38] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10Tgr) As a possible followup, DB error logging could be improved. It... [22:16:49] 10DBA, 10MediaWiki-Database, 10Multi-Content-Revisions (MCR-SDC phase 1), 10Patch-For-Review, 10Wikimedia-log-errors: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment - https://phabricator.wikimedia.org/T198350 (10greg) >>! In T198350#4323348, @Tgr wrote: > As a possible followup,...