[05:07:19] is there a way, using the API, to determine if a page was moved to its current location? (Besides Linkshere, because redirects can be deleted, or Revisions, because that's an unlimited number of revisions to crawl) [06:05:01] Hi enterprisey. [06:05:11] We have a move log. [06:05:15] Surely you've seen it. [06:05:19] You could just check that. [06:19:20] Savannah: There's no way to query that by destination title, unless I'm missing something in the documentation. [06:23:13] enterprisey: What do you mean? [06:23:23] You just want to see if "Barack Obama" has been moved previously? [06:30:34] https://en.wikipedia.org/w/index.php?title=Special:Log&page=Pulo+Aceh [06:30:41] Our move logs are pretty garbage, huh. [06:36:40] https://phabricator.wikimedia.org/T152829 [06:50:44] enterprisey: I left a note on that task. This is an embarrassing bug tbh. [07:25:55] Ah, nice. Yeah, for now I'll query the logging table and check that the destination page is a substring of the log_params column entry. [07:40:23] I have a weird issue with an image that was uploaded, then moved, and just this one particular image is now missing some thumbnails, but only those used by the File: page itself, and the image listing special page [07:40:41] the weirdest part is that the File: page does not even have an tag for it [07:42:07] (it does have the "Size of this preview" line, however, and the thumbnail links work; the Open in Media Viewer link pops up "Error: could not load image from...") [07:42:36] a similar file uploaded and moved at the same time has none of these issues [11:37:56] Hey all, I got a php memory error while running runJobs.php, "PHP Fatal error: Allowed memory size of 157286400 bytes exhausted (tried to allocate 131072 bytes) in .../extensions/SemanticMediaWiki/src/Utils/HmacSerializer.php on line 151", but I'm confused on where the limit is coming from. The memory exhaustion is saying exactly 150 MB but I have [11:37:56] no idea what the source of that limit is. My PHP process memory limit is 256 MB in php.ini. What might I be overlooking here? [11:39:46] justinl: just to make sure that you have looked at the correct php.ini? sometimes there are different ini files for webserver and cli [11:40:09] Ah yeah, I only tweak it in the fpm file. That's probably it, thanks! [11:40:20] Well I'll go check to be sure... ;) [11:43:01] Hmm, the cli memory_limit is -1, so no limit there. [11:43:27] Unless it's a system process limit... [11:44:48] Not according to ulimit. [11:44:53] hmmm [11:45:59] "php -r 'echo ini_get("memory_limit") . "\n";'" returns -1 as well? [11:46:30] yes [11:47:47] oh...hmmm... https://lists.wikimedia.org/pipermail/mediawiki-l/2018-March/047351.html - job queue default mem limit? [11:48:28] yep https://www.mediawiki.org/wiki/Manual:RunJobs.php [11:48:47] that seems to explain it [11:48:58] so I need to use runJobs.php --memory-limit=whatever if I need more [11:49:21] yes, that seems to be the solution [11:52:05] lol https://github.com/wikimedia/mediawiki/blob/master/maintenance/runJobs.php#L58 [15:39:29] oh my, that is odd indeed. [15:39:35] I understand where it's coming from though [15:39:44] DefaultSettings.php specifies 50M [15:39:48] so it's 3x higher than that [15:40:03] but it fails to account for other sources of mem settings, like web requests do. [15:40:15] it probably should consider that as well and take the max() of both. [15:40:31] also relevant is that maintenance/CLI scripts have the default of max-memory=max (unlimited) [15:40:42] and runJobs.php opts-out from that. [15:45:30] duesen: Regarding T251457, the reason I'm asking about a change is because it's a regression from this weeks train. It's fine when we rolled back. Finding the cause may be hard, and maybe it's fine to find a general way of fixing it, but just wanted to make sure it was intentionally skipped to find the cause, no worries either way. [15:45:30] T251457: LoadBalancer: Transaction spent [n] second(s) in writes, exceeding the limit of [n] - https://phabricator.wikimedia.org/T251457 [15:47:10] https://logstash.wikimedia.org/goto/97be5c4889714f235c14dc012610a4f3 [15:47:32] Krinkle: I'm still trying to find the cause. To bisect, I'd need a way to reliably detect the issue. I'm working on that right now. the Datbase object doesn't expose all the info about transactions that I'd like, though... [15:47:48] might be in TrxProfiler [15:47:55] but thanks, that makes sense now :) [15:47:59] Basically, some code shouldn't run inside a transaction. If it does, I want to warn about it, but I also want to log what function started the transaction. [15:48:32] duesen: https://phabricator.wikimedia.org/T251457#6098270 [15:48:46] the "sub-optimal" warning shows where the idle time was in user land [15:49:02] or... no, this one didn't maybe he removed it [15:49:04] while pasting [15:49:30] but that won't tell me what started the transaction, right? [15:49:56] tryFname has that info. But I see no way to extract it. [15:51:30] The first query naturally started it [15:51:36] then the "pending" time is between to queries [15:51:42] that narrows it down quite a bit [15:53:45] These are the messages from 1 edit as an example: https://logstash.wikimedia.org/goto/cdbe878232840d2a996185ed17d90a0e [15:54:31] For this one there wasn't any notable time in PHP land, there was no "pending" chunk [15:54:36] it's only the lock itself which is interesting [15:54:56] looks like maybe we're seeing two distinct issues that might share a common root cause in terms of change [15:55:08] one where get_LOCK is slow due to contention perhaps or something in another thread being slow? [15:55:16] and one where there is a big chunk of "pending" between two queries [15:56:01] The individual "GET_LOCK" query also has its own log message there with a stack trace [15:56:07] (PageEditStash->getAndWaitForStashVal0 [15:56:29] Krinkle: my interpretation is that GET_LOCK is slow because something else has the lock. That lock exists to make a process that wants to read the stash wait for the process that is currently parsing to provide the stash. [15:56:42] So a slow parse in one process makes get_lock in the other process wait long. [15:56:48] But that's working as expected. [15:57:04] What confuses my is what that get_lock happens within a transaction. it shouldn't. [15:57:34] duesen: Why/how is that significant? [15:57:36] Krinkle: can you remind me of the proper way to commit any automatic transaction that may be open? Is it just commit()? [15:57:56] in general any web code is not allowed to commit trx ever. [15:57:58] as a rule. [15:58:32] it doesn't have the authority/ticket to do that, it doesn't know what else is there that can still fail ahead of the current code and must roll back together in case of failure to protect db integrity. [15:58:43] the first query starts the trx, and at pre-send we commit. [15:58:49] The get_lock in the try is significant because it makes that trx time out, no? [15:59:06] then each deferred update and each job gets its own trx ticket where it cn do what it wants, either commit manually or no-trx auto-commit. [15:59:52] duesen: right, yes. So the alternative is to open a secondary connection to the same database, separate from the overall web trx. [15:59:55] so what happens if there is an auto-trx, and then i start an explicit trx? does it just keep the same transaction open? [16:00:19] auto-trx = no transaction, every query immediately commits. [16:00:23] that's how CLI scripts behave [16:00:47] well that's called "auto commit" [16:01:04] I gues by auto-trx you mean "trx mode" which is first query starts the trx and and of request/pre-commit commits it [16:01:10] DBO_TRX starts "automatic" transactions, see $trxAutomatic [16:01:12] nothing else is allowed to call commit, it will throw if you try [16:01:15] Yeah [16:01:22] that'S what I'm referring to. [16:03:49] duesen: it won't change our problem though, it's not about whether it's part of the trx [16:04:03] Then I don't understand the problem [16:04:13] if you give EditStashits own db master connection, without any transaction (or its own) it won't change the fact that the main trx is still being held open in real time. [16:04:41] yes, but the error is about the time spent in writes, no? [16:04:45] However the good news is that for the overall trx we have a higher limit. [16:04:46] Right [16:04:59] so yeah that might work [16:05:05] but also, why did it change. [16:05:09] maybe we're missing something [16:06:05] duesen: The lock select queries have an "IGNORE TRX" flag set, which means if they are the first query, it won't start the trx just for that [16:06:20] so perhaps something ahead of it changed and started a trx [16:06:28] by calling startAtomic, or doing any other kind of select [16:06:44] also, this isn't a write query [16:06:59] why is is marked as a write query [16:07:03] we have separate limits for reads [16:07:09] I think that's the problem. [16:07:22] Yes, maybe... in my mind, the get_lock taking long is working as designed. We are waiting for a slow parse [16:07:33] Yup [16:07:51] I wasassuming get_lock counts as a write [16:08:03] the problem is most likely 1) something caused locks to be marked as write instead of non-write, or if that was already broken, it got exposed because 2) something is starting the trx earlier than before. [16:08:47] right. is the desision whether locks are "write" ours? Or is this coming from the db? [16:08:58] duesen: ours [16:08:59] https://github.com/wikimedia/mediawiki/commit/27cf5ace4507f4f5eaae8bdeb72cb6128d9853fc [16:09:01] Found it [16:09:05] $flags = self::QUERY_IGNORE_DBO_TRX | self::QUERY_CHANGE_ROWS; [16:09:06] $res = $this->query( "SELECT GET_LOCK($encName, $timeout) AS lockstatus", $method, $flags ); [16:09:20] QUERY_CHANGE_ROWS got added by Aaron as a way to avoid having to use regexes to determine whether something is a write [16:09:22] which is nice [16:09:31] but also, the regex wasnt considering this as a write before afaik [16:09:43] lock() has $flags = self::QUERY_IGNORE_DBO_TRX | self::QUERY_CHANGE_ROWS; [16:09:51] yes, as of a few days ago [16:09:58] to optimise isWriteQuery() [16:10:00] ah, right, you were right ahead of me :) [16:10:13] You familiarise with isWriteQuery, and I'll CR? [16:10:59] Sounds good, I have to go and fix dinner soon though. [16:11:19] It seems like the self::QUERY_CHANGE_ROWS flag should just not be there. [16:11:29] Can't we just remove it? [16:12:49] Krinkle: can we just set it to QUERY_CHANGE_NONE? [16:13:15] duesen: LGTM :) [16:13:47] Yeah use QUERY_CHANGE_NONE to avoid getting to QUERY_CHANGE_NONE implicitly by running regex match on "SELECT" [16:13:59] which is the else branch in isWrite if neither is set [16:21:44] Krinkle: patch is up, going to cook dinner now [17:47:38] Hi there, with [[Extension: Page Forms]] how do I split {{{field|…|propert=…|input type=checkboxes}}} so that each check box is in a different list item ? [17:49:07] HTML LI element == list item, that is what I mean [19:10:42] adfeno: a change in includes/forminputs/PF_CheckboxInput.php around line 79 I think [19:12:49] adfeno: but one could also play with CSS [19:13:41] Indeed [19:14:20] I managed the issue by making checkboxLabel display: list-item. [19:28:03] saper: on second thought, I reverted that and now am discussing possibily to make the huge list as multi-select list box. [21:38:34] Hi all. I'm running into problems with the installer script, specifically where it tries to GRANT ALL to the wiki user. MediaWiki is on an EC2 instance and the DB is an AWS RDS Maria DB instance. It is expected that this will fail as the RDS admin user has limited privileges and can't grant all privileges to another use. [21:40:04] I see that I can go in and grant the necessary privileges (CRUD and create) myself but I suspect I'm still missing the things that are after the user privileges part of the installer. Is there a way to get the installer to skip user setup, or resume after that point? [21:50:15] Reading through MysqlInstall.php it looks like that happens no matter what. [21:55:31] It does look like it should potentially check for the grants before trying to do it [21:55:38] The same way as it won't create a user if it doesn't exist [21:58:32] Hmm. I could be looking at the wrong function or file, I'm brand new to the MW codebase. I'm looking at setupUser in MysqlInstaller.php [21:59:13] I don't see a conditional around that [21:59:59] if ( $tryToCreate ) { [22:00:07] Look at the else clause [22:00:17] https://github.com/wikimedia/mediawiki/blob/master/includes/installer/MysqlInstaller.php#L514-L553 [22:00:26] It has no else clause [22:05:58] Ah oops, the one above [22:05:59] https://github.com/wikimedia/mediawiki/blob/master/includes/installer/MysqlInstaller.php#L509 [22:06:10] This else clause sets grantable names even if you're not creating. [22:09:54] https://phabricator.wikimedia.org/T251650 [22:12:02] I'm going to comment it out for myself so I can move forward, then maybe take a poke at that task. Thanks. [22:12:48] There's a few things that need adjusting it seems [22:20:52] My security brain twitches a bit at GRANT ALL too.