[00:37:36] I have an extension that creates pages and it will add a page category at the bottom. the code needs a table mapping info it retrieves online to category names -- basically three letter codes to human readable text. [00:37:50] what would be a good way to incorprate that mapping table into the extension code? [00:37:57] just a php array literal or something else? [01:15:43] hrm... so now I have it saved to a JSON file in the extension dir [01:16:15] but is that something I can access from the extension? [01:33:15] ok, it seems to start from the root dir of mediawiki and you can load files as 'extensions/...' [03:42:26] Hello, I'm getting a 500 from a fresh install of 1.32.0 when trying to save a page in the MediaWiki: namespace. It seems like the page is saved, but the revision never shows up in RecentChanges. https://dpaste.de/v2vp [03:45:41] This happens with memcached enabled, but everything works fine with it disabled. Any suggestions? [08:29:05] ST47: I approved your phab account btw [15:39:22] legoktm: I saw, thanks! [17:58:25] ST47: I seem to have issues with memcached on 1.32 too [17:58:37] but some different kind [17:59:11] ("Thumbnail generation failed. File missing" for valid files with existing originals and all thumbnails present) [17:59:40] Why is that memcached? [18:00:10] because switching to APCu fixes the issue [18:00:20] restarting memcached also fixes the issue [18:01:28] I'd check if switching just the parser cache to APCu fixes it but the wiki is used heavily, and users complain a lot [18:20:16] I'll poke about with settings some more this evening [18:20:54] I did notice a warning in the manual that Memcached isn't recommended for low-traffic wikis, because it can cause some issues. So maybe I'll leave it disabled. [18:21:19] if you run a single wiki, there's really not much point in using memcached [18:21:37] if you run a wiki farm, APCu is not necessarily a replacement for memcached [18:23:34] anyway chances are memcached is erroring out, and some wiki code doesn't differentiate between a cache error and a cache miss [18:23:35] we have a lot of that [18:23:47] so maybe check memcached error logs [19:16:02] how bad is it considered, from a stylistic perspective, to use the ParserBeforeInternalParse hook to do global replacements on wikitext before conversion to html? [19:16:14] these are things I don't want to end up in the stored wikitext, just to optimize html rendering. [19:17:10] that is -- I'm returning true from the hook, but modifying the text [19:17:54] maybe I should use InternalParseBeforeSanitize [19:18:16] You probably shouldn't do that [19:18:42] I mean, unless you're sure what your doing [19:18:49] bawolff: I am just trying to remove some unwanted whitespace [19:18:54] that's kind of hooking very solidly into MW internals [19:19:23] I want to make it possible to do "foo\n\n..." and have it render as if there was one newline, not two or more [19:20:35] Its not super terrible per-se. I wouldn't exactly reccomend it, but i wouldn't say its so terrible it burns my eyes or anything like that [19:20:56] Also, those hooks might change a bit in upcoming parser changes (kind of unclear) [19:21:00] yeah I know [19:21:44] is there a better way to achieve this whitespace thing? [19:22:59] basically, for the purpose of editing, it's nice to have the extra newlines, but they cause rendering issues [19:43:09] I'm back. An update for Skizzers that yeah, 1.32 does render [[News:]] as "[<'news' uri>]", as suspected. [19:44:50] Otherwise, the process from 1.22 -> 1.32 went smoothly. The critical failure message lead to an extension that needed updated, but that was straightforward. [19:46:06] Skizzerz, that is. [19:46:44] yay [19:48:55] And you, Reedy, although it was Skizz who seemed more curious about using News as a namespace. [19:49:27] Tho you asked "why would you do this", which is always a valid question. ;) [19:51:53] Maybe I'm just a three year old [19:52:01] So "Why?" is always a valid question [19:52:54] As long as "Because" is always a valid answer. [19:53:31] heh [21:10:50] is there anything that turns wikitext into a parse tree? rather than html output? [21:12:47] you could look into parsoid, although that isn't exactly a parse tree either, it has annotations that help you understand the context of the stuff [21:13:50] Skizzerz: right, I think parsoid is both too much and too little [21:14:08] so basically, I have a js interface I want to add on top of the regular edit page. not a new editor, more an overlay [21:14:24] and it will make changes to the text in the text box [21:15:00] there's a small set of tag extensions it will modify. nothing else. but obviously, using regex to parse the text in the edit box looking for ... can be problematic [21:15:04] it may be sufficient, we'll see [21:18:12] ningu: did you check out https://www.mediawiki.org/wiki/API:Expandtemplates ? [21:18:51] there's a prop=parsetree param you can pass into that [21:18:55] dunno if it'll get you what you want [21:19:04] hmmmmm [21:19:07] thanks [21:19:10] I will take a look! [21:19:18] having to query it from js is fine [21:19:29] the kludgy way may also be fine [21:19:37] it depends on how much engineering we want to put into this basically [21:19:53] if we do the kludgy way it will limit us a little but that may still be all we need [21:52:35] tgr_: I ran memcached because APCu in php 7.x was a little broken on FreeBSD, now it seems to be working fine so I will see how it goes. What's weird is that there were no such issues in 1.30 :\ [21:53:32] ran everything with memcached for at least half a year without any complaints, the moment I upgraded to 1.32 complaints appeared [21:54:05] (and sadly 1.32 still does not fix the deadlock with PostgreSQL because seemingly every single operation on the wiki just has to update module_deps) [21:55:51] is that a resourceloader thing? maybe you can just increase RL cache times [21:56:14] but yeah postgres does not get a lot of love [21:57:49] UPDATE /* Wikimedia\Rdbms\Database::upsert */ "module_deps" SET md_deps = '["resources/src/mediawiki.editfont.less"]' WHERE ((md_module = 'mediawiki.editfont.styles') OR (md_skin = 'vector|en-gb')) [21:57:50] this [21:58:24] transactions end up deadlocked [21:58:55] does not seem to break anything because I am pretty sure all of this is already there [21:59:31] but it sure spams my otherwise empty postgres logs lol [22:14:48] tgr_: the worst part is having Cargo (users demanded it) on PostgreSQL, that thing is simply broken beyond reason :( [22:15:58] Remilia: no comment on Cargo but you might be interested in https://phabricator.wikimedia.org/T191231 [22:17:37] you should file a bug about the deadlocks though [22:17:53] tgr_: thank you, looks very relevant (and I am slowly fixing Cargo one line at a time, will probably try to submit patches to the author as the issues should happen with mysql with certain settings like wgDBprefix) [22:18:22] lock contention is hard to avoid with resourceloader since it gets invoked in several parallel threads for a single request, one for every asset group [22:18:36] oh there is already a task regarding that [22:18:40] but deadlocks would be a design mistake [22:18:48] I am guessing MySQL simply ignores it? [22:19:07] since transactions are not something it is particularly good at [22:19:25] in theory it does have deadlock detection, but I imagine particulars might differ [22:20:02] in my experience MySQL would just shove things in and hope it works :\ [22:20:23] (spoiler: it did not work in several cases) [22:20:49] anyway thank you for your input, it is late over here and I will continue research tomorrow [22:21:01] having structured logging helps a ton [22:21:11] no, you can certainly do the classic deadlock where two transactions are waiting on each others' lock [22:21:40] but exactly what gets locked during a DB write is very engine-dependent [22:21:54] actually re: Postgres deadlocks, if I am remembering correctly it is some kind of a schema issue [22:22:32] I'll probably bump my older report about it [22:23:02] the pgsql schema is very different [22:23:30] there was some effort a year ago to make it more uniform: T164898 [22:23:31] T164898: PostgreSQL schema change for consistency with MySQL - https://phabricator.wikimedia.org/T164898 [22:23:56] I guess that's on hold due to the abstract schema RfC [22:24:29] I experienced a really annoying schema issue during upgrade: T221496 [22:24:29] T221496: 1.32's update.php is broken for PostgreSQL - https://phabricator.wikimedia.org/T221496 [22:25:13] still thinking about how to fix this in a proper way (i. e. just shoving "SET DEFAULT 1" is likely not a good idea) [22:29:17] IGNORE is an option for the database wrapper (which happens to be named after the mysql option because that was comfortable for the developers who mainly deal with mysql) [22:30:06] if it's not automatically translated to the correct thing, that sounds like a bug in DatabasePostgres [22:30:13] Remilia: That's a dupe [22:30:31] https://github.com/wikimedia/mediawiki/commit/1444e5306a27c88185139cc9bbef2dea4d6ed81b [22:30:35] there seems to be at least an effort to do so [22:30:56] Brad fixed it in master, and it's already backported to REL1_32 [22:30:59] Will be in 1.32.2 [22:49:41] ugh... so it seems like the parsetree that api.php gives you at action parse/expandtemplates can't be converted back to wikitext? [22:49:47] why go only one direction? [22:50:25] ningu: MediaWiki's current (legacy) parser is a one-way-only affair. [23:10:13] James_F: so if you had wikitext with a custom tag ... and wanted to edit the contents of it, but only the contents of that tag and nothing else ... what would you do? [23:10:54] I am considering doing it with a regex but I know the pitfalls there [23:18:44] ningu: In what part of MediaWiki do you want to "edit" it? On user submission? On the DB? [23:19:17] (In general, this isn't do-able sanely in MediaWiki yet, which is an example of why we're replacing the parser.) [23:26:53] James_F: in the edit box on the regular edit page [23:27:09] via a custom editor that will be added to the edit page (but will not replace it) [23:29:18] is there a page I can visit to read about current work on the parser? [23:29:58] ningu: Start from https://www.mediawiki.org/wiki/Parsoid [23:30:14] We're only eight years into the project. [23:30:25] oh, wait, parsoid is the new parser? [23:30:31] Yup. [23:30:31] I thought it was just an addon thing [23:30:36] Nope. [23:30:43] Well, it was [23:31:00] could I use parsoid for this case, then? [23:31:02] But now they're working to make it properly replace it, and not need a node dependancy tree [23:31:13] I have no problem hosting it myself [23:31:15] Reedy: It was always intended to be the replacement parser. [23:31:21] Sure [23:31:31] But it has been just an addon for this time [23:31:50] ningu: Yes, Parsoid will give you the content in a tag – see https://www.mediawiki.org/wiki/Specs/HTML [23:32:20] I thought parsoid only converted wikitext to html output? [23:32:26] if it has parse trees or similar, that's better [23:33:03] Reedy: I always get this error when trying to run action=visualeditoredit [23:33:05] "apierror-visualeditor-docserver-http" [23:33:24] ningu: It can do both ways [23:33:27] ningu: E.g. for the bar tag it'll give you …; you can replace the value in the data-mw tag and push it back into the API and it'll get saved. [23:33:41] cool [23:33:55] James_F: can I just get wikitext back, though? that's all I actually need for this (addmitedly hacky) use case [23:34:01] ASammour: You absolutely should not ever ever ever try to use the visualeditoredit internal API. It's flagged as such. [23:35:06] James_F: Why?. Is there another way to handle it? [23:35:06] ningu: If you've got RESTbase installed, there's the nicely-documented https://www.mediawiki.org/api/rest_v1/#/Transforms/post_transform_html_to_wikitext endpoint. If not, Parsoid has that endpoint, but it's not as nice (and won't do staging of edits over time). [23:35:53] ASammour: Handle what? It's like saying you have difficulty controlling the thrust on a plane when you spark wires in the engine cowling. You're massively bypassing the process. [23:36:12] James_F: ok cool, I will try it out [23:36:28] it looks like this may work ok [23:37:01] ASammour: If you want to transform some HTML into wikitext, use the RESTbase transform API call. If you want to save the resulting wikitext, use a response promise and MediaWiki's JS API for editing. [23:37:24] just want transform [23:38:04] James_F: This one maybe can achieve the same process? https://www.mediawiki.org/api/rest_v1/#/Transforms/post_transform_html_to_wikitext__title_ [23:38:39] ASammour: Yes, there a several different transform end-points for slightly different use-cases. [23:44:55] James_F: Thanks a lot. Could you please see why this edit add the hidden categories, and destroy the format of templates? [23:44:58] https://ar.wikipedia.org/w/index.php?title=%D9%85%D8%B3%D8%AA%D8%AE%D8%AF%D9%85:ASammour/test&diff=prev&oldid=34843073 [23:45:43] I use transform API to convert from html to wikitext [23:47:47] https://www.mediawiki.org/w/api.php?action=parse&format=json&titles=User_talk%3AGryllida&prop=wikitext§ion=1 <-- why doesn't it see section one? [23:50:02] ASammour: That's not designed for that. There's half a million lines of code that you're bypassing by not using VisualEditor and RESTbase. The things you're finding aren't there are because you're bypassing that code.:-) [23:53:48] Sveta: You want https://www.mediawiki.org/w/api.php?action=parse&format=jsonfm&page=User_talk%3AGryllida&prop=text§ion=1 [23:54:03] Well, probably not format=jsonfm. ;-) [23:54:24] Sveta: https://www.mediawiki.org/w/api.php?action=parse&format=json&page=User_talk%3AGryllida&prop=wikitext§ion=1 [23:56:21] James_F: Didn't understand a thing :). I'm sorry. All I want to do is retrieve the html parsoid of a page. Then make a few modifications to it. Then convert it to wikitext. Is that possible?. [23:57:17] Because I use action=visualeditor before. But when convert it to wikitext it destroy the whole format. [23:57:20] ASammour: Yes. But it's very, very hard. I would strongly recommend you instead write a VE gadget that adds a button to the VE toolbar that does the change you want. [23:59:36] James_F: Ah!. Actually I need it for a bot. There is a problem we suffer from in almost all wikipedias. Which is the red files.