[01:52:41] maffblaster: WMF just renames accounts to "Vanished user xxxxxx" where xxxxxx is some random number. Accounts are never outright deleted [01:52:55] for a single wiki (not part of a wiki farm), Extension:RenameUser can be used for that purpose [01:55:00] if the goal is just database pruning, MediaWiki ships with a removeUnusedAccounts.php maintenance script which outright deletes accounts that have never edited [02:27:38] Jdlrobson: if you're around, I'm narrowing down a repro case for https://phabricator.wikimedia.org/T264376 [16:56:54] Hi. Is it just me, or is the new MW 1.35 release slightly faster than the previous release? [16:59:33] There have been numerous performance improvements across the board indeed. Glad its noticeable :) [17:00:11] Yea. I am happy =) 100 points on pagespeed for desktop and 96 for mobile: https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fwiki.tnonline.net%2Fw%2FMain_Page&tab=mobile [17:00:18] Do you have numbers comparing before and af.. [17:00:37] 98 for desktop I think it was. not sure about mobile. [17:00:52] So not a huuge difference. Still :) Feels snappy [17:01:28] I am thinking about doing the preload for fonts, but I have pretty long cache time for fonts, so is it _that_ nessecary? [17:01:37] A little improvement is better than a regression ;) [17:02:04] This biggest gains this year were on the backend. Time to save and rendering pages for simple MW installs. Fewer CDB files. Optimised mine data etc [17:02:21] mime* [17:02:40] And various things removed from setup and webstart [17:03:37] I use my MW as a blog and personal wiki, so seems perfect for me. [17:03:37] Forza: font display optional would be best [17:04:04] That would mean it is only used from the second page view or on very fast connections [17:04:10] I have font-display: swap; [17:04:31] Right so it eventually swaps [17:04:47] Yea, so I wonder if pagespeed is wrong there. [17:05:08] That means load end time (browser spinner) and visual complete time (last paint) will be pushed back [17:06:20] But on the whole these are not captured well by PsI/lighthouse. Overall it's pretty snappy now and this might not make it faster depending om thee things holding it back to that point in time for other reasons [17:06:56] s/om thee/on the other/ [17:13:12] I could do http2 push, but problem is that i might push to users that already have the font files cached, for example. I _think_ http2 can cancel a push request, but I am not sure if it helps? [18:13:17] Hey, since updating to 1.35 I'm having trouble using the shell.php in maintenance. Does anybody ran into this already? [18:14:14] It would come very handy to help me debug some issues with the update, but is giving this syntax error msg that makes no sense at all. [18:14:14] having trouble? [18:15:39] If I do e.g echo 'foo'; it prints PHP Error: syntax error, unexpected '@', expecting ',' or ';' in Psy Shell code on line 1 [18:16:00] any input I give I get the same. [18:18:35] seems to work fine for me [18:18:36] Psy Shell v0.10.4 (PHP 7.4.3 — cli) by Justin Hileman [18:19:54] >>> echo 'foo'; [18:19:55] foo⏎ [18:19:55] >>> [18:20:44] I'm also on 0.10.4, but PHP 7.3. But I don't think this should be an issue. Well thanks for checking Reedy [18:21:01] It shouldn't indeed [18:22:20] This will probably fix itself after I've dealt with the other issues that updating has caused :) [18:22:34] heh [18:23:25] Depending on what you're trying to do... eval.php might suffice for now [18:24:02] Uhmm, good idea. I forgot about eval.php [18:27:10] Certainly, if you're still having issues with shell.php when other issues are fixed up, it could be worth digging into further for sure [18:42:42] Forza: I don't recommend push indeed. Preload is better if you want to go that route [18:42:55] Because it considers local cache first [18:43:28] But beware that both can slow you down because preload is given a high bandwidth priority [18:43:47] Which means it could make css and other above the fold slower [18:44:05] So your total time would be lower maybe but longer blank page [18:51:40] Just to share, Shell.php working again after I fixed an error with my extension.json files. [18:51:58] cool :) [18:52:06] must be some weird referred error [18:52:43] I would never have guessed that this could be caused by this [18:53:11] What did you change? [18:53:25] It might be possible for us to trap the error a bit better and displaying something useful [18:54:20] the manifest_version to 1. It was set to version 2 by mistake [18:55:30] on the logs I could see that ExtensionProcessor was complaining about this and after fixing Shell.php works like a charm [18:58:50] hmmmm [18:59:52] feels edge casey :) [19:00:15] It does ;) [19:00:39] There's a few cases of stuff like this that result from dodgy extension.json files causing weird and wonderful errorrs [19:07:14] Reedy: hi, maybe we could debug here so it's easier than on the Phab task? [19:07:23] if you don't mind that is :) [19:07:29] lol [19:07:45] TBH... It might be worth splitting those into two bugs [19:08:23] heh, I thought they were quite similar as the MediaWiki namespace definitions are being ignored seemingly [19:08:36] but since the cache is disabled, what would you suspect would be behind the issues? [19:11:18] There's other caches in play [19:13:01] And it's probably memcached/similar [19:22:05] ah [19:22:17] so what would you suggest doing in order to try to fix it? [19:23:53] Not sure [19:24:05] Also, I think "Caught exception MWException: Empty $mTitle in OutputPage::parseInternal" is possibly a red-herring [19:24:28] Basically, that's going to be expected as effectively $wgTitle isn't being set (though, from the context) [19:24:42] ah [19:25:49] If it was coming up in your webserver logs, that'd be different obviously [19:26:02] yeah, that makes sense [19:41:58] If I get this error: https://phabricator.wikimedia.org/T212428 during an update.php, how would I go about fixing that? I have no revision ID or anything for reference. Just an error. [19:43:13] The error directs me to that bug report, but the only solution involves changing a value that I have no reference for, and the update script fails on the error. [19:43:19] You get "Main slot of revision not found in database!" [19:43:33] Slightly different, one sec [19:44:15] https://pastebin.com/a2peGScW [19:44:44] with a newline in the middle? [19:44:59] No, that's just because of my terminal size [19:45:05] I copied directly from putty [19:45:16] just checking [19:45:18] cause it would've been odd [19:45:35] So if you enable debug/error logging, you'll get more information [19:45:42] $this->logger->error( [19:45:43] __METHOD__ . ': Main slot of revision not found in database. See T212428.', [19:45:43] [ [19:45:43] 'revid' => $revId, [19:45:43] 'queryFlags' => $queryFlags, [19:45:44] T212428: includes/Revision/RevisionStore.php: Main slot of revision (number) not found in database! - https://phabricator.wikimedia.org/T212428 [19:45:44] 'trace' => wfBacktrace( true ) [19:45:45] ] [19:45:47] ); [19:47:27] Ah, here we go, I had it on already. Forgot I enabled that. Let me pull up the last transaction... [19:50:03] https://pastebin.com/PHHYEyET I notice 69474 pop up multiple times here [19:51:07] So if I understand this correctly, I should use that revision to find the page, then change page_latest in the page table? [19:52:34] (Although I guess it does just say "Main Page" there) [20:01:43] https://phabricator.wikimedia.org/T212428#6514858 [20:01:59] It's all well and good pointing people to the task... but the description isn't helpful :) [20:09:25] Yeah, pretty much my main issue here. I'm digging through the database seeing if I can work around it without bricking things. None of this is on a live install, since I'm doing pre-production testing first, but having to reload the database is never fun. [20:14:13] It looks like the slot revision is missing, but the revision itself is there. [20:14:32] That sounds right [20:14:54] Any way I can manually insert it? [20:15:17] well, you'd need to know which slot to point it at [20:17:39] Actually, the slots table is completely empty [20:18:26] Which should make sense, because it was introduced in 1.31, and we're coming off of 1.29 [20:18:55] Interesting that I didn't have this issue with the last wiki database I just upgraded [20:19:20] The updater should be populating it though... [20:19:31] That's what I'm wondering. [20:19:46] I'll try the page_latest fix and see if that helps. [20:20:57] It looks like the update to slots is done in populateContentTables added in 1.32.. [20:25:52] Running that now, even encountered the same failed blob problem from the other day [20:25:57] Fixed the same way, thankfully [20:27:10] Okay, that failed due to a duplicate primary key. https://pastebin.com/HxYtMNmR [20:27:25] That's not the ID i entered for the dummy data though [20:33:24] Okay, was able to continue the update.php script after the populateContentTables failed. Not sure what that means, but we'll see I guess. [20:57:26] I don't suppose anyone here is familiar with AbuseFilter? The update succeeded but I ran into this error while AbuseFilter was processing its own upgrade. https://pastebin.com/U2EjQeCf [20:57:43] I googled around and couldn't find any threads or bug reports. New bug maybe? [21:08:18] looks like insufficient error handling [21:09:12] * Reedy pokes the maintainer [21:14:06] jfolv: I'm investigating your issue [21:16:40] It's weird though, seems like it found invalid serialized data, but truncated dumps should be already fixed by that point. [21:17:34] Could you please paste the full output for the script? [21:17:34] For what it's worth, I re-downloaded the 1.35REL of AbuseFilter after I got the error the first time, in case I maybe had a bad download. No change; error is still present. [21:17:50] Sure, let me re-run it. It completed 3/4 tasks but failed on the last. [21:18:03] Yeah that's just some bugs I wasn't aware of [21:18:23] Just in case, would you be able to hack the script code a little bit and re-run it if necessary? [21:19:01] Sure. I'm comfortable with that. [21:19:16] This is a test database anyway, so a little breakage isn't the end of the world [21:20:03] edge cases! [21:20:40] Sometimes I feel like everything I do is an edge case [21:20:48] Uhm, now that I think about it again, I'd like to see the original output. The main question is whether it had to rebuild any dump (that'd be during step 1) [21:21:50] As for the code hack: just add a lame var_dump( $text ) before the unserialize() call at line 526, and let's see what's inside [21:21:50] Let me check, it's probably still in my MediaWiki log [21:22:33] I assume it didn't, unless you ever installed on that wiki a very old version of AbuseFilter, like a 2009 version where a DB field was too short [21:25:13] Well, this wiki has been around since about 2005 I think [21:25:15] Maybe 2006 [21:25:50] Hah, interesting [21:29:01] I'm still going through the log; looks like AbuseFilter did do quite a bit of work in here. Long pages of spam text (mostly consisting of obscenities and garbage data) are in the log. There's nearly 10k lines here, but if you give me a bit to touch it up and take out any identifying hostnames, etc, I'll give you the paste [21:31:08] Sure, thank you! This is the result of cleaning up 11 years-old garbage [21:31:28] This has reminded me about some other poor truncation bugs we have in other code [21:33:29] A lamer shade of bug 🎶 [21:34:20] These are a bit more exposed via the web [21:34:21] This one was fixed 27/02/2009 https://phabricator.wikimedia.org/rEABFa7bc94cc80487f0b3063fa5ff19b6e6ff7917ddd [21:34:25] and then you can't edit them of course :D [21:36:01] It's quite crazy that we're still having consequences from 2009 bugs tho [21:37:38] (jfolv: Please ping me once done, my multitasking mode is quite broken and I might forget about this...) [21:38:43] Will do; just have to obfuscate some of this info for privacy reasons. [21:54:56] Daimona: I have it ready, but it's a bit too large for pastebin. Any alternatives? [21:55:12] there's a phab paste bin [21:55:31] https://phabricator.wikimedia.org/paste/edit/form/26/ [21:55:37] it should take any amount of text... [21:55:55] Oh neat, thanks [21:56:10] of course... you might just find another bug ;) [21:56:16] Eheh, let's see if it gets truncated by phabricator [21:57:51] It wouldn't take it [21:57:59] Said no storage engine could handle the file [21:58:19] hahahahahah [21:58:24] And yes, my week has consisted of one bug after another [21:58:32] So I really expected this [21:58:35] If you save it to a text file... [21:58:36] How big is it? [21:59:05] 5439KB [21:59:29] heh [21:59:41] can't even put it as an upload into phab [21:59:51] will phab take compressed files? [22:02:29] We can find out [22:03:12] Reminds me of https://xkcd.com/949/ [22:03:37] gzip compression brings it down to about 839KB [22:04:48] https://phabricator.wikimedia.org/F32372606 [22:04:56] Can't view it in phabricator, but it can be downloaded [22:06:23] Let me know if that works for you, and if you need any additional information just ping me. I've been in front of my pc all day and could use a bit of a break. [22:06:59] Yay, reading that, thank you! [22:07:15] The only thing might be the experiment with a var_dump( $text ) before the unserialize call, but no hurry [22:07:23] I'm going to open a task in the meanwhile [22:08:31] Ah, sure, I'll do that real quick. That might be more helpful [22:08:51] https://phabricator.wikimedia.org/T264513 [22:08:59] For the immediate issue, definitely [22:12:40] So, the log file shows a bunch of ordinary queries, but the script output seems to be gone :-/ [22:17:03] Is that not helpful? [22:17:20] Also, I did a var_dump of $text, and I got this https://pastebin.com/zbxr47cT [22:17:40] Looks like a welcome bot edit [22:20:15] Unfortunately not very helpful, no :-| [22:20:35] Interesting, thank you! This one looks promising [22:21:08] looks truncated [22:23:11] It is... [22:23:41] Let me stuff that string in my DB and see what the script does [22:25:05] My theory is: this script was by that 2009 bug. The script somehow failed to restore the dump [22:29:53] Interesting to note that I performed another wiki upgrade this week with a site that's been around even longer. That database did not run into this issue. [22:30:27] Though it has me wondering if it ran the abusefilter upgrade. I'll check that. [22:32:11] Huh. Probably should've commented that var_dump out first. Whoops. [22:34:03] Lol [22:40:59] I managed to import the data locally, let's see what happens if I run the script... [22:55:02] I'm still running the script on my other database, too. I'll update you when that finishes [23:01:31] Thanks, I'm going to disconnect shortly, but I'll read phabricator tomorrow [23:02:12] Anyway, I've experimented a bit locally. The bug cannot be reproduced by putting that dump in the abuse_filter_log table, i.e. what would have happened in 2009 [23:02:26] Because the script would recognize and update it [23:03:01] However, it can be reproduced if that same truncated dump is inserted into the text table [23:03:36] Do you think there'll be a fix or at least a workaround in a reasonable amount of time? [23:04:55] So my theories are: 1) An unknown bug caused the truncated dump to get to the text table 2) The truncated dump was in the abuse_filter_log table; the script tried rebuilding it but failed without knowing, producing another truncated dump that was inserted into the text table. 3) Something else [23:05:34] I think it's hard to tell without further data, but we don't need to investigate the root cause unless this bug is seen again [23:06:16] To answer your question, I'm going to push a fix immediately to put a stopgap in place, and also fix another couple of edge cases spotted while testing. No guarantee on when that's going to be merged though [23:27:39] So, everything should be on gerrit now. Thanks jfolv for reporting, and thanks Reedy for your help! [23:28:18] Thanks to you, Daimona. Glad you were able to get on this so quickly [23:28:51] Yeah, I really needed some bug to devour tonight :D [23:29:06] So if it's on gerrit, that means it's awaiting merging, right? [23:29:12] I'm not 100% familiar with the MW dev process [23:29:25] Yeah, it's up for CI checks and code review [23:30:03] Gotcha. I'll take a look at the update and see if I can put it in my install manually. [23:30:58] AbuseFilter isn't bundled.. [23:31:09] So depending on how you install it.. It's either just a git pull, or download a new snapshot [23:32:16] Oh even better [23:32:25] I thought AF was one of the bundled ones these days [23:32:25] Or just copypaste this fix https://gerrit.wikimedia.org/r/c/mediawiki/extensions/AbuseFilter/+/631817/1/maintenance/updateVarDumps.php [23:33:02] Oh super easy [23:33:04] I'll do that [23:33:08] Not yet, although there's a neverending task for that, blocked on some sec issues and lack of postgres support. I hope to get it done for 1.36 or 1.37 [23:33:34] Gotcha [23:33:35] Please let me know on https://phabricator.wikimedia.org/T264513 whether it worked [23:33:57] Will do! [23:34:19] I'm done for today, thanks y'all again, bye :-)