[08:18:44] Looks like we may need to consider moving to another IRC network... [08:37:13] hello, urgently need help. i need to block account creation on my wiki (spammers). $wgGroupPermissions['*']['createaccount']=false; does not help for some reason, what should i do? [08:39:32] cloudcell: That should prevent account creations for unregistered users. If you're logged-in you may still create an account. Have you tried creating an account while not logged-in? (using a private browser session for example) [08:40:02] Vulpix: I'll try now [08:42:28] Vulpix: for some reason I can create an account (but I have two computers running on the same ip address, one is logged in) [08:43:24] i dont know what to do :( [08:44:00] That's very strange. Have you put that line at the end of LocalSettings.php? Just be sure it's not overwritten further down that file [08:44:30] Also be sure you're editing the right LocalSettings.php file (in case you have multiple installations, or leftover files from past upgrades) [08:44:40] well, there's another line with [*][edit]=false; [08:44:44] below [08:46:24] edit and createaccount are not related [08:51:10] i think i copied the instructions from stackoverflow and the quotes were replaced by apostrophes... [08:52:54] Vulpix: ^^ that was the error [08:53:02] thanks for the support :) [08:53:47] everything works now \o/ [08:54:31] Well, that's... strange. It should break your wiki displaying a parse error... [08:55:11] ...unless it takes it like an undefined constant with unicode characters being part of the constant name. Oh, wel... [09:13:08] cloudcell: you can check the permissions at Special:ListGroupRights to verify your configuration works the way you think it does [09:13:19] thanks tgr_ [09:15:53] Iamthehuman: check the page length in the revision and recentchanges tables, one of those got corrupted probably [09:24:12] tgr_: thanks for that information. I'm looking at https://www.mediawiki.org/wiki/Manual:Recentchanges_table and https://www.mediawiki.org/wiki/Manual:Revision_table. I have access to the MariaDB. What should I check for? [09:26:15] I'm worried about how serious this is and can it be fixed. See I've been upgrading this MediaWiki since March 2003 and so far so good, but now running into this kind of mismatch [09:29:47] I don't quite get this... The diff shows everything was replaced with the text "testing", but that is clearly not the case as the page looks like it's been looking like for years [09:33:34] are you running any extensions like flaggedrevs/pending changes/approved revs? [09:38:34] p858snake: nope [11:17:57] Hello and thanks for the MediaWiki. Yesterday I noticed that one of my MediaWikis is somehow broken. This diff shows that all content was replaced with "Testing" https://develop.consumerium.org/w/index.php?title=Lists_of_price_comparison_services&diff=next&oldid=54500 but looking at the content before and after the edit there is clearly an addition of +85 bytes (counted with wc). The edit that is supposed to be in the diff was an edit of the [11:17:59] heading ==== Jämför priser för husdjursförsäkring ==== [11:19:12] So the person clicked "edit" on the heading, added 85 bytes and saved his change, but the diff looks like that. [11:21:24] Iamthehuman: This looks like a polluted diff/revision cache. Maybe the cache is shared between instances and someone actually replaced a page with "testing" on another instance... or the same instance was replaced by a backup but the cache wasn't cleared [11:23:07] Vulpix: thanks for that info. The developement wiki and test wiki together are a wikifamily of 2, sorry to not have mentioned this as this could be the cause [11:24:48] Vulpix: yeah, I replaced the wiki with a 2 days old backup on 2021-05-17. How do I clear cache? which cache? [11:25:24] I have memcached, but this is not about that cache, right? [11:25:43] it most likely is. Just restart memcached [11:25:59] ok.. hold on a few secs and I login [11:27:12] Whatever your $wgMainCacheType is configured, or $wgObjectCaches [11:28:43] restarting memcached did not help with the diff mismatch (?) ... should I revert to backup and then restart the memcached? [11:30:06] is memcached actually in use? Maybe you're configured it to store data in the database. If in doubt, truncate the objectcache database table. Note that it can log-out all current logged-in users [11:30:43] $wgMainCacheType = CACHE_MEMCACHED; [11:31:04] it is a just one server setup, so the memcached is listening on localhost only [11:31:47] How do I truncate objectcache database table? In MariaDB or..? [11:33:06] Maybe there's something wrong for that page. Probably the page_latest doesn't point to the last revision, or the last content text [11:33:32] Since the diff is also wrong when comparing from a different revision [11:33:48] Maybe what's wrong is the last version of the page and not the diff [11:34:23] the latest revision contains the 85 bytes added that RC reports [11:37:21] This could be relevant: The 3 follwing tables are shared between the two wikis: 'ipblocks', 'user_groups' and 'interwiki'. When I run upgrade.php on the testwiki, I disable those shared tables [11:40:13] It is not a big deal if I need to go back an backup or few, but it is a big deal the development wiki cannot be fixed to a working state. It has been going since 2003 and I've made it this far [11:43:48] Could the erronous state be due to me having forgotten to restart memcached after doing an 'sudo apachectl graceful' to make the apache find the reverted directory? At least my understanding is that when I 'mv w w-YYYY-MM-DD' and put the reverted 'w' in place the apache will keep on running the version in w-YYYY-MM-DD till graceful restart [11:45:42] This could be caused by me learning only a month or two ago to drop the database that is in place at the backup verification server before loading a new backup, to avoid weird shit from happeing re database [11:46:22] I can't say for sure... maybe [11:48:15] up till then I was just loading the .sql-dumps into MariaDB on the backup verfication server without first dropping and recreating the database. [11:48:53] the page table contains pointers to the latest revision and latest "text", or "content" (this last point may have changed since the MCR changes) [11:48:57] !db page [11:48:57] See http://www.mediawiki.org/wiki/Manual:page_table [11:49:14] You may look into that table and see if it matches the latest revision data [11:49:34] lookin [11:50:00] When you create a new revision, the page table gets updated to point to the latest one (for efficiency, so you don't have to guess what's the latest revision of a page) [11:50:13] Maybe something went wrong when that edit was published [11:50:28] And it saved a wrong revision [11:54:50] Vulpix: Here is the result of 'select * from page where page_id=4722; https://paste.debian.net/1198095/ [11:55:43] the information in that row seems correct to my eyes [11:57:38] I'm guessing it is more likely that the erronous table is https://www.mediawiki.org/wiki/Manual:Revision_table and not https://www.mediawiki.org/wiki/Manual:Recentchanges_table or the page table [12:01:44] could the culprit really be that made apache to do a graceful restart after switching in the backupped w-directory and the old database? There have been 2 edits since I did that, so I can easily recreate the edits, if reverting again might help make the error state go away [12:02:45] I meant: could forgetting to restart memcached after restart of apache be causing that diff to be broken in that way? [12:06:53] On 2021-05-15 I do remember doing an edit with the visualeditor (which worked for a while and then broke) where the addition was "Testing", but it was to the Sandbox-page. As I stated I reverted the /var/www/... and the DB on 2021-05-17 and could have forgotten to restart memcached [12:07:30] hi all, does someone have a pointer how to troubleshoot a "Error in LDAP seach: Bad search filter" ? I have been looking on the internet and found a few posts but nothing fixed my issue. [12:13:39] the revision table contents seem ok too: https://paste.debian.net/1198100/ ... 85 bytes added in the last edit. Is there perhaps a maintenance script I could try to run [12:19:35] I think I may have an idea what this https://develop.consumerium.org/w/index.php?title=Lists_of_price_comparison_services&curid=4722&diff=54504&oldid=54500 is about. I may have forgotten to restart memcached after reverting to backup and as a result the diff shows the whole contents replaced with "Testing", because I did an edit with the content "Testing" when I was testing the VisualEditor and that revision did not get reverted out of [12:19:36] existence, because memcached was holding it [12:19:56] Could this be the cause? [12:23:55] I could try to do another edit and see what happens with that one? Maybe that would yield a clearer pattern of what's wrong. I don't care how much time and effort I need to expend to get this fixed, coz this wiki is really really important to me [12:28:10] I did an edit and indeed the diff is off by a couple of integer notches. Now the diff claims I added removed almost everything and added the text "Testing with mobile" and that one also I remember doing on 2021-05-15. https://develop.consumerium.org/w/index.php?title=Consumerium:Sandbox&curid=3094&diff=54505&oldid=54215 [12:31:00] So I have confirmed that somehow the edits that I was intending to revert out have somehow hung around. Could someone confirm my theory that they've stuck around because memcached was not started right after 'sudo apachectl graceful' ? [12:46:04] Yeah, clearly the revert to 2015-05-15 backup did not go as planned, because the edits that were supposed to be removed are now showing up in the diffs, because some counter adjusting mechanism is off by 3-4 integers [12:56:07] This is my bad it seems... the edits I made while testing VisualEditor are showing up in diffs of edits made now, so obviously the recovery from the 2015-05-15 backup did not go as planned, if stuff added to the database after the backup point keeps on popping up and messing up the wiki [13:04:32] I just looked and this exhibits on all 4 wikis I reverted or more like tried to revert. I am going to try the recovery from backup again. Should I perhaps this time just stop memcached at some point in time and then restart it after gracefully restarting the apache to get the receovered version of the wiki directory into play? [13:06:37] Iamthehuman: That's very strange. I don't think new edits should continue to exhibit this problem, but I don't know really what could have gone wrong here... Restarting memcached should force it to read the database to get the latest data, at least to fix it for new edits [13:08:31] I just looked at 'last' and it looked chill [13:12:09] I'm going to make a backup of these wikis in this state and put it aside in case I need to look at them again in this state and then recover the last "seems to be working without problems state" and hope that that is a no problems state, this time paying attention to stop memcached before restarting apache to get the new dir to be served by apache and then restarting memcached and seeing if it works this time around [13:32:56] Did the another go at recovery and an edit. Memcached was stopped before that. It seems that some pointer is off by one, because doing this first edit in Consumerium:Sandbox after the recovery attempt https://develop.consumerium.org/w/index.php?title=Consumerium:Sandbox&curid=3094&diff=54503&oldid=54215 shows on the right side of the diff the whole contents of the last edit to last edited page, before I took the database dump on 2015-05-15 [13:33:04] Before that moment all diffs look fine [13:37:45] Last backup before that was on 2021-05-02. Work needed to reproduce additions since is next to neglible. I need to think and if someone has some clearer thoughts about this case, please help. [14:13:13] Oh noes... the 2021-05-02 backup yields the same situation. When editing a page, its diff will show the entire contents of the last edited page before the mysqldump was taken [14:18:08] If I try to edit Consumerium:Sandbox again the content in the edit-box is the contents of the last page edited before I ran the mysqldump back then [14:24:59] so this appears to me like an autoincrement somewhere regarding what is the latest revision of a page is behind by one. I am thinking this because the contents of the last page edited before the mysqldump was done shows up in the diff of the first edit done after loading the old database [14:26:38] I'm not aware of any counter stored somewhere, except for the latest revision count in the statistics table (the one that holds the number of pages, revisions, etc) [14:27:00] And also I'm not sure if that one is used for doing the increment [14:27:14] yeah, my lingo bad.. I don't mean counter I mean an autoincrement thingy [14:32:33] Iamthehuman: I don't have time to read back but 1) check if the data in the revision table is correct (depends on your setup but usually the revision and content tables hold the metadata, slot table connects the two, text table holds the page contents) [14:33:30] 2) check if the page table points to the right revision 3) check if the recentchanges table points to the right revision (although if the problem is only with #3 then page history would work correctly) [14:33:58] tgr_: all seemed fine in the revision table https://paste.debian.net/1198100/ [14:34:25] that was the +85 bytes edit when I noticed that the wiki is broken [14:35:22] if the tables are fine, it's either some sort of caching issue, or an extensions does something unexpected. You can try disabling all extensions. [14:35:38] Also set caches to CACHE_NONE. [14:40:07] ok, I will try to check out all those things now that I have not edited anything after restoring the 2021-05-15. [14:41:34] Looking at the table 'page' and the entry of the last edited page the column page_latest points to the latest version correctly as far as I can see [14:49:37] I dunno why 'select * from recentchanges where rc_id=(select max(rc_id));' returned 298 rows ... [14:55:04] the things in tables 'page,' 'revision' and 'recentchanges' seem to have valid values now that I have not made a single edit after restoring the database from 2021-05-15 [14:56:35] check also slot and content tables [14:56:43] and text [14:59:50] The DB does not have a 'slot'-table. In the 'content'-table I don't know what's 'content_id' nor 'content_address' ... content size looks ok [15:03:54] content_address is (unless you're using external storage or something equally exotic) in the form of tt: where number corresponds to the appropriate old_id in the text table [15:09:21] In the 'text'-table with the row with the highest old_id looks to be the latest revision of the correct page [15:11:50] it shouldn't necessarily be 1 to 1 with the revision table. Some other places may be storing text outside of the revision (AbuseFilter for example) [15:12:49] If in doubt, follow the table documentation on mediawiki.org [15:15:44] I do not have abusefilter installed. So the restored state looks just fine, until one makes one edit and the diff of that edit is broken, if make a 2nd edit, then the edit box will be filled with the contents of the last page edited before the backup [15:16:21] Like I said, AbuseFilter is an example, not the only thing that causes that [15:17:06] Also, I'm pretty sure when you revert a page, the same text id is reused instead of creating a new text record with the same contents of the previous revision, which can cause you to have less text records than revisions, for example [15:17:12] oh sorry, I do have abuse filter installed [15:19:55] How should I proceed in trying to solve this case? [15:21:46] first you should find where the IDs start to get out of sync, by following previous recommendations [15:22:15] The ids are in sync untill I do a new edit after the database has been reverted [15:23:22] the state restored from backup seems to be ok... all things I checked point to the correct last edit [15:41:14] After making an edit, the latest entry in the 'text'-table is good, latest entry in the 'revision'-table looks good too and in the 'page'-table the entry for the page I edited looks good too, but the diff is messed up and if you try to edit the same page again the edit box will contain the page that was last edited before the backup, not the sandbox I am trying to edit [16:03:38] Iamthehuman: I'd try to disable all extensions at this point. Some hook might be messing up revision loading. [16:03:57] what version of MediaWiki are you using? [16:04:41] tgr_: MediaWiki 1.35.2 [16:06:57] I'll check for what was the last extension that was added and when [16:08:15] are history links also broken? e.g. if the diff link is index.php?oldid=123&diff=456 then try index.php?oldid=123 and index.php?oldid=456 too [16:08:21] It was wfLoadExtension( 'ReplaceText' ); on 2021-04-12 ... also did $wgEnableScaryTranscluding = true; that day but could not get it to work [16:11:06] tgr_: in the history all diffs to current show wrong, prev works for older, but not for the edit that was made after the database was restored [16:14:49] On Sunday 2021-04-11 I upgraded from 1.35.1 to 1.35.2. I have both pre-and-post state backups of that upgrade should the need arise [16:21:51] Everyone seems to agree that this brokenness is surprising, I'm not a huge corporation so I don't have the extensive intrusion detection systems or such ... I'm thinking what if something / someone managed to change the MediaWiki code itself, because the database looks fine, but the diffs and editing properly [16:25:02] I'd sooner suspect broken extension code. Are the non-diff links in the history (when you click on the timestamp) wrong too? [16:26:53] tgr_: everything in history is as it should be until one makes an edit on the restored backup. The prev is wrong and all the cur links are wrong. This is the one edit that breaks it https://develop.consumerium.org/w/index.php?title=Consumerium:Sandbox&curid=3094&diff=54503&oldid=54215 [16:31:50] tgr_: you're probably right, that this is a problem with some extension not working properly [16:31:53] Do you figure that a clean reinstall using git clone to get the extensions and git pull to update the existing ones would do any good? [16:32:22] .. or is something like that that I could try in-place? [16:36:59] Iamthehuman: if the extension is broken in the repository, that won't change anything. As has been suggested already, disable all extensions and try to edit again... [16:42:11] Yeah, well my reasoning was that maybe it has been fixed in the repository, but I just don't have the newest changes. I've never disabled all extensions, but allright. Let me do this on a low importance wiki, with the same problem, now restoring the db from backup and then gonna disable all extensions [16:49:46] ok... all extensions disabled and ready to try an edit and see if it breaks in the same way [16:50:48] https://develop.consumerium.org/w/index.php?oldid=54215&action=raw [16:50:53] https://develop.consumerium.org/w/index.php?oldid=54503&action=raw [16:51:02] the diff itself is correct [16:51:26] so one of those revisions has different content in the DB? [16:52:02] nope, disabling all the extensions did not affect the situation (in another wiki, sorry about that, but too freaked out about breaking the important one) [16:54:28] tgr_: the latter is the text of https://develop.consumerium.org/wiki/User:Jukeboksi/Log/2021 the last article I edited before I did the mysqldump [16:59:10] right. I am just trying to understand whether your problem is that a revision is displayed with content different from what was saved, or the edit box was mispopulated, or both. [16:59:16] the actual edit I did was an addition of text "Testing to find out where the software goes wrong" to the Consumerium:Sandbox article [17:01:22] Iamthehuman: Actually, that text was the edit summary, not the text added to the page [17:01:32] Vulpix: It was both [17:01:41] Apparently not https://develop.consumerium.org/w/index.php?title=Consumerium%3ASandbox&type=revision&diff=54503&oldid=54502 [17:02:34] Providing wrong information to people trying to help just adds to the confusion and complexity of the problem [17:05:29] Vulpix: sorry, trying to provide the right info but that one is curious and should not be happening. I mean the addition of ":: Backups made and verified" obviously should not be in the backup. I have remembered to drop the databases and to recreate them in order to make sure that the right .sql is added to the database. I'm really confused [17:06:02] Iamthehuman: maybe you can just describe step by step what you did after restoring the backup? [17:06:37] the thing to figure out is whether the edit UI is broken or the process os saving the new revision or the process of loading the new revision. [17:08:03] Also, it may be possible that backups are not consistent (depending on how do you take the backups), and maybe an edit was made in the middle of the backup, causing some tables being dumped before the edit and others after the edit, leaving it inconsistencies [17:12:43] Steps to reproduce: #1 restore database from mysqldump 21 make an edit to https://develop.consumerium.org/wiki/Consumerium:Sandbox, edit goes through just fine and the rendered page shows the new content, but #3 visit the diff and it shows you the diff like the page contents got replaced with the contents of https://develop.consumerium.org/wiki/User:Jukeboksi/Log/2021 and it also shows up in the edit box if you try to edit edit again the [17:12:44] correctly rendering newest version of the Sandbox-article [17:15:24] sorry, clarification: the Consumerium:Sandbox renders correctly after I press "publish" for the first time, but returning to the article and the contents have been replaced with the other page [17:20:50] Vulpix: This link that you kindly sent to me https://develop.consumerium.org/w/index.php?title=Consumerium%3ASandbox&type=revision&diff=54503&oldid=54502 I don't quite understand how that message about backups being made and verified is in the database at all. I mean it was written after the upgrades were done. Also I have tried the 2021-05-02 backup in addition to the 2021-05-15 and this brokenness appears in that backup too [17:21:30] sorry, my language is falling apart. not "upgrades", but "backups" [17:31:20] It looks like your backups are messed up, then [17:57:23] I think something is wrong with my MariaDB ... either the production one or the one used to verify the dumps. I did the procedure again and the result is the same, all seems fine after revert to backup until one does an edit, so I would guess the backup verification MariaDB has gone bad. I'm trying the procedure on my laptop instead of the server [18:08:21] nope, same result [18:23:23] tried again with the older backup, same result ... I try one notch older backup. [18:46:33] the even older one wouldn't result in a functioning wiki, always broken in the same way and I cannot figure out where the addition ":: Made, verified and stashed backups" (in the sandbox article where it should not appear) appears from. [18:47:52] So confused.. [18:50:27] I am certain that I have added the note about backups being complete only after I had loaded them onto the backup verification server [19:00:14] I think I'm going to call it a day. I've restored the latest backup and am closing the wikis for editing and then tomorrow I'm probably going to try doing a fresh install and migrating the old db, uploads and LocalSettings.php to the fresh install, to see if that works. Could consider purging and reinstalling the MariaDB too because either of these is not working right. Tried the backups from 2021-05-15, 2021-05-02 and 2021-04-22 and throuhout [19:00:15] all these the wikis were working fine [19:08:27] thanks for bearing with me verbalising on what I think may be the problem. Tomorrow I try those things in that order, unless someone has a better suggestion [19:50:08] One more thing that could be helpful towards resolving my problem is that I have snapshots of the whole server and I could spin off a new server from one of those if that will help me in investigating this [20:33:11] Iamthehuman: you may want to create a new backup from a snapshot and be sure to use the --single-transaction when running mysqldump [20:36:52] Thank you Vulpix. If I spin up a new server from the snapshot, slightly alter the /etc/apache2/sites-enabled and maybe even set up DNS for it, I can test if the most recent snapshot is in such a state that edits are possible and if yes, then I can work from that [20:41:13] also recorded the "be sure to use the --single-transaction when running mysqldump" in my notes in my intrawiki [22:28:05] Reedy: [[mw:Compatibility]] says PHP 8 is not yet supported. Is this still true? [22:28:20] it seems the tracking task is all but resolved, except for a (minor?) deprecation warning [22:28:20] I think so [22:28:55] https://phabricator.wikimedia.org/T274966 [22:29:22] I think most of it is probably working [22:32:05] https://phabricator.wikimedia.org/project/view/4652/ [22:39:57] I dunno how well it's tested on 8 though [23:05:02] Krinkle: https://phabricator.wikimedia.org/T274966#7099577 [23:06:02] * Reedy looks at REL1_36 [23:12:58] Aye, the same bunch more or less. [23:13:23] Maybe worth tagging a few more components/teams to cover the areas [23:16:13] 1.36 is worse [23:16:15] A lot worse [23:16:32] 1.35 [23:16:32] 00:03:22 Tests: 18497, Assertions: 64616, Errors: 8, Failures: 11, Skipped: 28, Incomplete: 1. [23:16:32] 1.36 [23:16:33] 00:12:28 Tests: 22064, Assertions: 74226, Errors: 88, Failures: 5, Skipped: 30, Incomplete: 1. [23:17:35] though, a lot more are libxml_disable_entity_loader ? [23:19:44] Oh, right, we were talking about back-porting a 'fix' to disable that stuff. [23:20:08] Yeah, but we haven't got a proper fix in master [23:20:18] Yeah. #priorities [23:20:22] heh [23:26:34] Reedy: The 'fun' question is whether T283208 is a subset or superset of T274966. Probably a disjoint set. [23:26:35] T274966: Make MW 1.35 tests pass on PHP 8.0 - https://phabricator.wikimedia.org/T274966 [23:26:35] T283208: Make MW 1.36 tests pass on PHP 8.0 - https://phabricator.wikimedia.org/T283208 [23:26:44] yes [23:26:58] Part of me wants to remove avro from 1.35 [23:27:05] but probably not worth it [23:27:17] (just mark the test skipped on PHP 8?) [23:27:17] Too much of a breaking change now it's released? [23:27:27] We dropped it for 1.36, right? [23:27:33] pretty sure [23:27:50] Yeah, just skip everything, I guess. Certainly we won't be doing feature/fix backports for avro to REL1_35. [23:28:02] If anything does come up, we'll probably have to actually drop it.