[08:40:38] o/ tgr [08:40:52] any chance you could re share that logstash dashboard you just posted in the phab ticket? [08:41:02] but use the "share" link so others can open it? :) [08:41:22] doesn't that work with a plain link? [08:41:26] nope [08:41:38] I just get Error: Unable to completely restore the URL, be sure to use the share functionality. [08:42:27] huh. I was under the impression you could do that [08:42:39] it was just a custom time range + search query [08:44:51] anyway, fixed [08:44:55] thanks! [08:45:15] So, I remember seeing some "sub optimal queries" in the logs from beta, but they looked like they generally always happened [08:45:24] I didn't spot any while things were being on group0 [08:45:50] *Sub-optimal transaction [08:47:36] https://www.irccloud.com/pastebin/WyvA8oJ2/ [08:47:48] these happen on inserts so AIUI the only two ways are if two processes try to insert with the same unique key (revision ID) or some select does not use the index and locks the whole table [08:48:23] both seem impossible for the comment table, it's just a key-value store [08:48:40] "some select does not use the index and locks the whole table" > I think tendril or some other DB tool can tell us if that was happening [08:49:15] I could never figure out how to use tendril [08:50:57] So, things seems to happen on commons wiki the most, https://tendril.wikimedia.org/report/slow_queries?host=%5Edb&user=wikiuser&schema=commonswiki&qmode=eq&query=&hours=15 shows the spike in slow queries there, basically all the CommentStore::insertInternal call [08:52:13] Commons is the largest of the group1 wikis by far, right? [08:52:20] or is Wikidata also group1 [08:52:21] ? [08:54:07] here's a more useful dashboard: https://logstash.wikimedia.org/goto/7a32b85858cc1177d342ae2ce63c2adb [08:54:19] most of these seem to happen in the job queue [08:54:27] which is surprising [08:54:45] tgr: wikidata is also group1 [08:55:27] and those have no stack trace for whatever reason :/ [08:56:03] I think that is because they are the DBQuery logs, you need to find the matching DBPerformance log ? [08:57:19] tgr: you can find the matching ones using the reqid, for example https://logstash.wikimedia.org/goto/58d54ff00568b504cffc7e497d722a01 [08:57:26] we have a dbquery log which does not include the stack trace? that's... not helpful [08:57:36] looking at the surrounding logstash documents will probably also reveal it? [08:58:22] tgr: get rid of the "DBQuery" filter and it will show the log lines with the stacktraces too for the lock timeouts [08:59:02] oh, tgr in #wikimedia-databases jynus just said https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/425283/ is probably the cause [09:00:05] that was merged 3 months ago [09:00:14] or rather, that was the patch that fixed this issue before... and in the refactor that fix got undone? [09:01:13] tgr: https://usercontent.irccloud-cdn.com/file/ZbDPoWVY/image.png [09:01:19] thats what you missed :) [09:01:55] in any case, jobqueue DB errors are a false positive, they all happened during a previous spike [09:02:05] thx [09:29:31] I wonder what's up with that 240sec delete [09:29:49] it only had 9 revisions, no huge file usage as far as I can tell [18:23:34] tgr: i'll leave it to addshore and anomie to answer your question regarding multi-dimentional table specis in getQueryInfo() [18:23:53] i didn't quite get it myself, i just know that it broke the array_intersect stuff :) [18:24:34] the conversation is on one of the old patch sets of https://gerrit.wikimedia.org/r/c/mediawiki/core/+/416465 I think [18:25:50] meh, at some point i squashed and rebased a bunch of code, no i can't find the old patch sets :P [18:29:06] tgr: select() does support multi-dimensional table specs. Look for "parenthesized group" in Database::tableNamesWithIndexClauseOrJOIN [18:29:35] that's what the patch was originally using. for some reason I didn't quite get. [18:31:46] oh wow, it really does [18:31:52] haven't seen that before [18:32:41] I guess the conversation is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/416465/8/includes/Storage/RevisionStore.php#2113 [18:33:02] doing a grouped left join [18:34:06] but flattening that will break the query completely [18:34:26] if it is needed in the future, the correct filtering would be something like this: https://gist.github.com/benjamw/1690140 [18:35:17] the idea was to ignroe the groups. that was intentional [18:49:11] tgr: found the origin of all this: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/406595/44/includes/page/WikiPage.php#2880 [18:50:19] thanks [18:50:39] sadly, PHP's handling of associative arrays is not great [18:57:59] Anyone using PHPStorm by JetBrains on OS/X and has gotten xdebug to work with php 7.2.0 on PHPStorm [20:14:34] tgr, addshore: there are a couple of things sitting in the review column on https://phabricator.wikimedia.org/project/view/3373/ [20:14:56] they are all pretty small [20:15:13] if you have half an hour, review would be lovely [20:15:48] i'll look some more at the diff stuff and the wikibase integration tomorrow. [22:57:46] Reedy: hm.. phpunit/phpunit key is set twice? https://github.com/wikimedia/operations-mediawiki-config/commit/07d621ea76fce10c483f84611fe9f213548e7abe [22:57:55] composer warns "Key phpunit/phpunit is a duplicate in ./composer.json at line 12"