[07:26:27] Hi, I have a python script that uses mwapi to fetch article revisions for about 100k articles. I have used it multiple times before and it has worked, but today for some reason i'm getting "ConnectionResetError: [Errno 104] Connection reset by peer" repeatedly [07:26:47] i'm not sure if the problem is locally, or on the remote end [10:26:19] Hi. I would like to have a MagicalWord that give the index of the number of time a include page is included https://dpaste.org/QEGm# [10:27:30] Where can we found this information while we are parsing the include page content? [11:52:18] Hi! My wikimedia site hosted on Azure web app with sqlite database went down when MS upgraded the host to PHP v7.4. The site is configured and verified to still run v7.3, but database is still inacsessible. Anyone know a simple solution to this? I've spent hours troubleshooting and finally migrated to mysql, but I've got another site which is not [11:52:19] yet upgraded and would prefer to avoid doing the migration.. [11:55:32] Oystein: what's the error message? [12:02:30] Could not find a valid driver [12:03:47] Sorry, the full message is ' Cannot acces the database: could not find driver' [12:07:27] This is from /mw-config. But the actual site throws a null reference exception from user.php [12:07:38] Anyone here have experience using cleanmediawiki.sh to delete bot edits? [12:09:53] Marekatt: if memory serves me correct that's some old script that's utterly broken on modern MWs. how bad is the situation? [12:11:48] ashley: At the moment there are "only" around 30 bots with edits mostly to their user pages. I ran the script but it didn't see to do anything, despite claiming to remove edits and delete users in the range I specified [12:11:56] seem* [12:15:25] Do you know of a better way to mass delete users and their contributions? [12:16:28] that sounds fairly manageable, then; you could just get the list of affected (spam-only) pages and feed it to the deleteBatch.php maintenance script, for example; deleting users wasn't and still isn't really something that's supported (not that it stops people from *doing it*, of course; but that may break referential integrity and cause problems later down the road) [12:17:16] Yeah, I know userMerge is the way to go, but it's quite cumbersome to do manually for each user [12:17:35] I will try the deletebatch script now [12:20:57] There isn't any batch merge script by chance? [12:23:30] Oystein: that error message means your PHP doesn't have a sqlite driver. You'll need to enable/install the sqlite driver for PHP [12:32:07] Vulpix Tanks for the tip. Not sure how to do this on Azure Web app though, but I'll investigate. [12:33:40] Oystein: This may help (the sqlite driver is a php extension) https://stackoverflow.com/questions/47119069/how-can-i-enable-php-extension-in-azure-app-service-web-app-linux [12:34:54] Vulpix: Thank's! [12:53:43] Is it possible to run UserMerge from the terminal? I really want to figure out how to mass merge/delete users, can't believe there isn't something out there for it. [12:59:10] Marekatt: Nope, there's seemingly no maintenance script for it [12:59:33] And not even a task for creating one [13:03:44] Reedy: Thanks. Am I the only one who finds this a bit odd? Seems like a common problem for people [13:03:59] I guess most people manage with the web interface [13:04:07] Dealing with accounts like this isn't the best in MW [13:05:11] https://phabricator.wikimedia.org/T248386 [13:05:20] Filed a task for it... Doesn't necessarily mean it will get worked on any time soon though [13:05:39] The extension is reasonably structured it shouldn't be too hard though [13:06:09] Thanks a lot. Maybe it will benefit someone else in the future as well [13:06:23] I can't imagine it's too difficult for people who know what they're doing [13:22:21] Hey all, I've run into an issue while upgrading from MW 1.30 to 1.34. We have a Template namespace, Say there's a template Foo, so you visit Template:Foo and all looks well. But then go to the main page and in the search bar, type "Templ" and it will start autocompleting but won't show "Template:Foo" as an option, or anything else under Template [13:22:22] for that matter, if even typing out "Template:" or even "Template:Foo" and it won't show on the results page either. [13:26:47] I haven't found anything regarding changes to search functionality between 1.30 and 1.34. And FWIW I have 5 similar wikis and 2 environments for each, dev and live, and both envs work fine under 1.30 but do not work, as explained, in 1.34. [13:27:59] Getting any browser console errors as you type? [13:30:37] Didn't think to check that, let me go look... [13:33:24] Nope [13:36:45] FWIW I'm building the 1.34 wikis on separate servers (Ubuntu 18.04 AWS EC2 instances) from the 1.30 instances (Ubuntu 16.04), and my upgrade process is to just get completely new code for mediawiki, skins, and extensions from the source github repos. Not sure if any of that matters, I'm just trying to consider what's different from the process and [13:36:45] config for when I upgraded to 1.30 and then moved those from our DC to AWS. [13:37:17] FWIW we do use a number of extensions, including Semantic MediaWiki, so I don't know if this problem is with the core mediawiki or related to any extensions. [13:38:26] It's also an upgrade from Varnish 4.1 to 6.3, don't know if that would have any effect on this. [13:40:56] (From memory) So typing into that box does an opensearch call to the MW API... [13:42:35] It works fine on my dev wiki... [13:44:29] Do you have any settings related to search and/or api use? [13:44:38] There's various, yeah [13:44:48] But none that I can find relating to NS of that search functionality [13:45:02] If it didn't work at all, that'd be a different thing [13:45:16] Does Category: etc work? [13:45:48] I know Special: does, let me try Category [13:46:17] No it doesn't work [13:46:43] Wonder why Special: does... I was wondering if something was messing up encoding of : [13:48:30] Well we have plenty of pages with : in the title as well and they've never been a problem. [13:49:43] If you type "Template:" and then hit enter do the search results on page make sense? [13:49:59] no matching results [13:50:12] Does a NS0 match... [13:50:18] but that's true of Special: as well [13:50:22] But there's a different autocompleting box on that page too.. [13:50:24] NS0? [13:50:55] Main namespace [13:51:47] Oh gotcha. Not sure what you're asking, but if I just search "Template" I do get results of pages with that in their titles, but not anything relating to the Template NS [13:52:05] So the query MW is doing is api.php?action=opensearch&format=json&formatversion=2&search=Template%3A&namespace=0&limit=10 [13:52:21] if you go to that API page, do you get results? [13:52:48] Also, wtf.. NS=0 and it's returning Template: prefixed stuff? [13:52:54] brion: How much do you remember about opensearch? :P [13:53:22] Reedy: mmm, a bit [13:53:33] could be redirects? [13:53:38] It returns 0: "Template:" and then lines 1, 2, and 3 are just [] [13:53:44] heh [13:53:55] that does sound kind of .... funky [13:53:57] I'm currently bemused by it searching namespace=0 and it returning results from other NS [13:54:07] On my dev wiki.. [13:54:08] ["Template:",["Template:!","Template:0", [13:54:16] We do have a few extra namespaces. [13:54:21] Yeah, shouldn't matter [13:54:21] But we've had them for ages [13:54:28] It sounds like the search backend is playing up for you [13:54:35] I'm guessing you're not using ElasticSearch or anything? [13:54:36] Including SMW-related namespaces. [13:54:40] Correct [13:54:50] yeah, if backend is being clever it might be overriding the default logic which would search just the main ns [13:55:21] I did find a variable called $wgNamespacesToBeSearchedDefault but that doesn't seem to be the problem after testing it. [13:55:52] @justin [13:55:57] Oops [13:56:27] ApiOpenSearch does do... [13:56:28] $ns = implode( '|', $config->defaultNamespaces() ); [13:56:28] if ( !$ns ) { [13:56:28] $ns = '0'; [13:56:28] } [13:56:42] defaultNamespaces() is $wgNamespacesToBeSearchedDefault [13:56:50] But it's definitely not changed on my dev wiki [13:57:13] same here, it's the same in DefaultSettings.php on my systems, both in 1.30 and 1.34 [13:57:17] justinl: I also have a similar issue. The search doesn't suggest searches when I start typing. If I type "Template:" there is nothing. Even if I hit enter and search. If I type "Template:fact box" (which is the start of some of my templates), it doesn't suggest a search, but if I hit enter it finds the templates. [13:57:33] Mine doesn't find the templates. [13:58:13] Even if you type "Template:x" where X is the first letter or word of the template? [13:58:22] Do you have $wgSearchSuggestCacheExpiry set in your LocalSettings? [13:58:23] Though to be fair the wiki has a page called Template so that might be throwing that off. [13:58:27] Marekatt: anything visible in javascript error console? [13:58:33] Correct, Marekatt [13:59:14] brion: I don't know how to check that to be honest. Not very experienced with this stuff [13:59:39] I checked mine, brion, and there are no console errors for me. [14:00:14] Reedy: I don't, at least [14:00:46] // Open search results may be stored for a very long time [14:00:47] $this->getMain()->setCacheMaxAge( $this->getConfig()->get( 'SearchSuggestCacheExpiry' ) ); [14:00:47] $this->getMain()->setCacheMode( 'public' ); [14:00:54] We have added a few extensions with the upgrade from 1.30 to 1.34, wonder if any of them could be interfering. We also upgraded SMW from 2.5 to 3.1 though I'd be surprised if it played into this. [14:00:56] Marekatt: depending on the web browser might be slightly different, but there should be a 'developer' or 'web developer' menu somewhere with a 'console' or 'web console' option. (sounds like it might be throwing javascript errors or failures to load the network) [14:01:16] With the upgrade of Varnish too... I was wondering if there was some different handling of that [14:01:21] Potentially giving stale results [14:01:39] Let me go look at the VCL file, see if there's something in there I've forgotten about that might be relevant. [14:01:51] brion: Corrent, I get errors. [14:01:52] You can probably test with a curl -I and look at headers [14:03:06] I think the only real change to our VCL is an addition for the MobileFrontend extension we're adding to prevent requestors from setting arbitrary X-WAP headers. [14:03:27] brion: https://dpaste.org/ZOPF [14:03:30] and then that's only in dev [14:04:58] Marekatt: hmmm, those look harmless enough [14:05:03] uncertain what could be the issue [14:05:45] Marekatt: also check under the 'network' panel. there should be a network request to api.php when it does a typeahead search. if that's coming back with a 40x or 50x status code that could indicate more details [14:07:57] I dont' see any api.php calls when doing the typeahead searching [14:08:13] hmm, interesting [14:08:14] o_0 [14:08:19] Even for the ones that return results? [14:09:04] wait now I see one, hang on one sec... [14:09:07] brion: My php calls come back with status "canceled" [14:09:15] oh that's interesting [14:09:20] When I'm typing in the search bar [14:09:26] api.php?action=opensearch&format=json&formatversion=2&search=templ&namespace=0%7C102%7C108&limit=10&suggest=true [14:09:38] Marekatt: probably indicates PHP is crashing :( you'll have to check the server error log for more details probably :( [14:09:47] It's at least progress [14:10:21] status code for that was 200 [14:10:42] Oh yes, I also have one php call with status 200, same as junstinl [14:11:09] justinl: I guess you left the namespace to be searched value after you changed it? [14:11:41] I'll have to get my IT guy to help with this then I suppose. Getting in over my head [14:12:29] I didn't change any namespace values if that's what you mean [14:12:45] Why is it doing 0|102|108 then? [14:12:48] oh sorry I know what you mean, you mean that variable I mentioned earlier [14:12:49] Or is that some SMW change... [14:13:41] good luck Marekatt ! [14:13:58] Not sure [14:14:34] Ok I think 108 is an SMW namespace [14:14:53] and 102 probably is as well [14:16:01] yes https://www.semantic-mediawiki.org/wiki/Help:Namespaces [14:16:21] I wonder if that's what's causing problems [14:17:27] It hasn't in the past, we've used SMW for probably 6-7 years [14:17:46] I know, but it could help narrow the bug :P [14:17:47] 1.34 is the first time I've seen this (reported by one of our main editors/testers) [14:17:54] It's possible older SMW wasn't setting it etc [14:18:04] Or there's some bug in newer MW with NS filtering in search [14:18:32] Sure, I'm just saying, it's new. And if it is SMW related somehow and our upgrade from 2.5.x to 3.1.x branch is huge and we know has a lot of breaking changes that will have to be worked out, so it's definitely good to know it's not something more obvious. [14:20:45] Setting `$wgNamespacesToBeSearchedDefault[9] = true;` in my dev wiki doesn't seemingly break it [14:20:57] You could ask in #semantic-mediawiki if they know of any breakages (but it's not very active) [14:21:24] Hmm [14:21:25] https://wikifarm.wmflabs.org/cpt/index.php/Special:Version [14:21:36] I just tried removing 102 and 108 from the URL I'd pasted above and got some sane results, minus Template: pages of course, so I think this is very much SMW related. [14:21:38] I'm guessing that's of a similar setup to you.. And it works there [14:21:45] Aha [14:22:01] I wonder if they're doing some overriding of the search results etc [14:22:36] We knew SMW upgrading would cause issues. Let me take this back to our editor that reported it and work with him on it. I think you've definitely helped find the right direction to investigate. [14:22:46] https://wikifarm.wmflabs.org/cpt/api.php?action=opensearch&format=json&formatversion=2&search=Template%3A&namespace=0%7C102%7C108%7C3002&limit=10&suggest=true [14:23:30] I have another problem. I just installed SpamBlacklist and configured it in Localsettings, but it doesn't show up in special:version [14:23:47] I can use the links you've provided to help us narrow down things. Thank you very much for the help! [14:24:08] np justinl. Feel free to come back if it turns out it may be a MW core issue :) [14:24:53] Marekatt: What did you add to LocalSettings? [14:24:56] https://dpaste.org/cQSx# [14:25:12] Tried both with and without the additional settings beyond just enabling the extension [14:27:10] Caching? [14:27:56] I have tried ctrl+f5 to avoid that, but I often have trouble with caching even with that tbh [14:28:15] However when I've installed other extensions they've always shown up right away [14:28:43] That's odd [14:34:02] Hi there. [14:34:03] I would like my www.example.com portal to look like www.wikipedia.org portal. How can I download the template or code and put it on my site? [14:34:28] https://github.com/wikimedia/portals [14:34:52] Thanks for the swift response Reedy. [14:35:06] I'm not sure how easy it might be... [14:35:09] But that's the code at least :) [14:35:19] Could you please provide me with the steps Please. [14:35:29] I'm like a dummy [14:36:08] Isn't the code that does the job? [14:38:01] If I added an extension to localsettings with the incorrect files in the extensions folder I would get an error right? [14:38:12] Marekatt: Of some sort [14:38:36] So it's not likely that the files in extensions are wrong, I suppose [14:38:57] Since my wiki just acts like nothing is wrong and ignores the extension entirely [14:38:58] Gooodman: It's not a particularly simple thing. https://github.com/wikimedia/portals-deploy is the version of the code that is actually deployed after scripts etc run [14:39:14] Marekatt: Do you have shell access? [14:39:25] Yes [14:39:41] I'm not usually scared of tough stuffs [14:40:03] Reedy: Yes [14:40:23] ...especially when I'm interrested in getting it done. Just tell me what you know and I'm very greatful [14:41:31] Marekatt: Know how to get onto eval.php? [14:41:45] No, can't say that I do. Never touched that before [14:41:45] hihi [14:41:57] I'm seeing Error creating thumbnail: File with dimensions greater than 12.5 MP [14:42:07] Marekatt: cd to the MW directory... php maintenance/eval.php [14:42:14] Okay [14:42:16] faceface: Make smaller thumbnails then ;) [14:42:21] Marekatt: then paste var_dump( ExtensionRegistry::getInstance()->isLoaded( 'SpamBlacklist' ), class_exists( 'SpamBlacklist' ) ); [14:42:22] I've tried setting $wgMaxShellMemory = $wgMaxShellTime = $wgMaxShellFileSize = 0; to no avail [14:42:26] yeah [14:42:29] I really should [14:42:41] But I'd use 'convert' just like MW should... [14:42:53] faceface: $wgMaxImageArea [14:43:07] Defaults to 1.25e7 which is 25MP [14:43:11] uh, 12.5MP [14:43:20] How big do you need? :) [14:43:31] set that in LocalSettings.php, but still fails wiht the smae error [14:43:41] IDK, I just want to see this image in a thumb... [14:43:51] Error creating thumbnail: File with dimensions greater than 0 pixels [14:43:51] Reedy: not sure if this is the output or if something is wrong. It said "bool (false)" twice [14:43:56] Ahh... [14:44:05] Looks like 0 isn't ulimit for that setting [14:44:13] Probably not :) [14:44:21] 3e7 would be 30MP [14:44:36] Marekatt: It should be true twice... So MW isn't loading it [14:44:42] Boom, thanks [14:45:35] Reedy: I see... that's annoying. Guess I'll try to download it again [14:45:41] What... I can't edit any more [14:45:54] It's machine code [14:46:04] Marekatt: Check naming, casing, file permissions. It's possible with debug settings it's getting suppressed [14:54:23] Marekatt: Double check that you're editing the *right* LocalSettings.php file. It's not the first time someone edits a backup or other folder's (or even server) LocalSettings.php file by mistake :) [14:54:39] Yeah, I just found out that that was the case actually [14:54:50] lol [14:54:52] I closed what I was (and still am...) sure was the right one and opened it again from my folder [14:54:57] And my changes weren't there [14:55:19] Might've been something with winSCP and the editor there not saving it properly, or just me messing up [14:55:53] Yeah there we go, it's showing in :version now. Thanks a lot, haha [15:02:08] And it's working, blocking links from the blacklist. Very good [15:03:11] vulpix I managed to fix my sqlite on azure web app problem. MS seems to have forgotten to add extension=php_pdo_sqlite.dll to the updated host. I added a new application setting PHP_INI_SCAN_DIR pointing to a custom Settings.ini file with extension=php_pdo_sqlite.dll, and voila!, thanks again for the tip. [15:03:57] yw [15:04:30] (y) [16:54:54] Out of curiosity, does anyone here run MW on a small set servers behind a load balancer? If so, do you use anything other than NFS or S3 to synchronize the upload directory amongst the web servers? [16:55:33] Wikipedia et al used to use NFS for shared storage of files [16:55:52] I thought they used something like Swift to roll their own S3, in effect. [16:56:01] We do now [16:56:06] As I said, used to ;) [16:56:22] Ah reading carefully is important. :) [16:58:18] I've been stuck with NFS since I started with it, having moved to AWS EFS when we migrated a couple of years ago, and NFS can be fussy sometimes with occasional timeouts causing front-facing issues. Wish there was a better solution, as S3 really isn't viable for us, but never could come up with a better design. [16:58:47] Even set up a grafana dashboard to watch for kernel log messages via the collectd tail plugin. [16:58:57] Even set up a grafana dashboard to watch for kernel log messages about such timeouts via the collectd tail plugin. [16:59:06] Oops, this is not slack. [16:59:52] Well I have to watch for certain CloudWatch EFS metrics since moving off our own NFS server. [17:00:17] I'm just always thinking about how to improve the overall architecture of my wiki environment, so just wanted to ask about that. [23:36:31] ottomata: Hey :) I was just looking to test something EL related and remember now that bin/devserver is no longer there in the EL repo. What would be a minimal way to locally satisfy the new version in the EL repo to have it run locally, without e.g. proviosining kafka etc for real locally. I have Homebrew and various Python3 versions available. I couldn't find a minimal local install example at https://github.com/wikimedia/eventlogging [23:36:57] preferably something local to that directory and not global on the machine. I'm pretty sure it exists but I can't recall or find how to do it. [23:39:42] Krinkle: bin/eventlogging-devserver still exists, no?