[11:30:06] hello [11:30:54] I've started the rebuildrecentchanges.php script, and it's been running for days [11:31:17] zero0: yes, that's kind of expected [11:31:19] what is going on with this script [11:31:34] What's your $wgMaxRC ? [11:31:47] $wgMaxRC ? [11:32:05] I mean $wgRCMaxAge [11:32:18] 90 days [11:32:36] Hm. [11:32:52] What is the DB doing [11:33:11] Like, are you maxing out CPU or memory or disk [11:33:55] currently the script is stuck at "removing duplicate revision and logging entries..." [11:34:10] I did run this script after importing pages and images [11:34:40] after importing it said "you may want to run rebuildrecentchanges... [11:35:12] how many pages did you import? [11:35:45] hundreds thousands [11:36:35] between 200K and 300K [11:39:14] i've looked at the database, ibdata1 objectcache and recentchanges tables are being updated every minute [11:39:59] how long can it take? [11:40:42] if there is any developer here I want to tell him you need to seriously optimize your code [11:40:57] a script running for days is completely unprofessional [11:41:02] You can't tell, because you've not checked where the bottleneck is AFAICT [11:41:19] the code must be butchered or written by college students [11:41:25] https://www.mediawiki.org/wiki/Manual:Performance_tuning#Database_configuration links a script which automates part of the checks [11:41:42] Or maybe you forgot some very simple configuration step. It happens. :) [11:42:50] I don't use an earlier version of mw [11:44:37] whatever am I supposed to do now? [11:44:49] How does that matter [11:44:50] I should let it run or kill it? [11:44:59] "removing duplicate revision and logging entries" is the final step [11:45:19] If you're bothered so much by its running and you don't want to check what resources it's lacking, killing it is ok [11:45:45] I'm very much bothered by the script, it makes my computer slower [11:46:11] not to mention it must have written terabytes of data to the hdd [11:46:11] Here you go [11:46:29] Really? At what rate is the HDD going? [11:46:49] You can tell quickly with "atop", in case you don't have a favorite tool yet [11:47:21] Nemo_bis> atop? what is this? [11:47:27] a command [11:47:50] https://linux.die.net/man/1/atop [11:47:58] and where am i supposed to type that command? [11:48:04] oh linux [11:48:12] i run windows locally [11:49:18] I see. That's likely to be issue 0 [11:49:44] Hm so... I suppose you know how to use the task manager? [11:49:55] sure [11:49:58] what for [11:50:48] You should check what's keeping your system busy [11:50:55] mysqld and php are running normally according to the task manager [11:51:02] CPU utilization, free RAM and so on [11:51:44] cpu 1% and ram 3G out of 16G [11:52:11] I think the issue is the constant read/write on hdd [11:52:16] So it seems the problem is not the script being slow but the script not actually being at work [11:52:37] You need to throw much, much more RAM at mysql [11:52:58] It will never get anything done on such a big amomunt of data without at least 4 GB for mysql itself [11:53:41] ram allocation is automatic on windows , how am I supposed to do anything about that? [11:53:45] Which btw is mentioned in https://meta.wikimedia.org/wiki/Data_dumps/ImportDump.php [11:53:51] You need to configure mysql [11:54:05] mysql can't guess how much memory you're comfortable giving it [11:54:34] that should really be on mediawiki.org and not on meta [11:55:21] I run mysql with slightly buffed up settings [11:55:26] Vulpix: dunno, it's related to https://meta.wikimedia.org/wiki/Data_dumps/Tools_for_importing which also contains wikimedia-specific tools [11:55:32] #max_allowed_packet = 16M [11:55:33] max_allowed_packet = 64M [11:55:45] "slightly" is not enough if you're importing a lot of data into MediaWiki [11:56:27] php [11:56:28] ;memory_limit = 128M [11:56:28] memory_limit = 2048M [11:56:50] I can't run php with more than that [11:57:08] if I set the memory limit higher php won't start [11:57:14] PHP is not the point here [11:57:31] it's not a php script? [11:58:07] You said yourself that the script isn't actually going on (1 % CPU) and the bottleneck is disk [11:58:38] rebuildrecentchanges.php doesn't read anything from disk so it must be the DB [11:58:39] rebuildrecentchanges.php doesn't use a lot of memory. The memory is being used by mysql here [11:58:42] the script isn't going on? yet it does [11:58:57] yes, a hundred times less quickly than it should [11:59:31] Vulpix> which exect mysql setting needs to be bufeed up? [11:59:48] if mysql has more memory, it can cache data on memory so it doesn't have to go to disk everytime to read data. RAM is fast, disk is slow [12:01:31] nevermind I see the site has a configuration sample [12:02:06] one thing that absolutely improves is disabling the binlog if it's enabled [12:03:15] hmm this sample must be very old [12:03:33] these settings don't exist in my configuration file [12:04:05] yes, it is, --> max_allowed_packet=20M // max_allowed_packet=1GB is MAX for mysql 4.0 and above, 20M for 3.x [12:04:22] it's accounting for mysql 3.x (!) [12:05:51] I think I run mysql v5.x [12:06:08] increasing innodb_buffer_pool_size should also help [12:06:10] https://www.mediawiki.org/wiki/Manual:Performance_tuning#Database_configuration is up to date [12:06:36] Or should be [12:07:06] still, even if you change my.cnf, changes won't have effect until you restart mysql, I'm not sure if you can set them at runtime from a mysql command line [12:07:23] anyway thanks for the help, appreciated [12:07:53] I'll make i increase the memory settings if I need to run these rebuild script again [12:08:04] no, no runtime changes for this variable http://stackoverflow.com/questions/1880770/change-mysql-innodb-buffer-pool-size-at-runtime [12:08:05] just hope it won't take years :/ [12:08:18] i have no way to know the progress [12:08:32] what if it's at 2% ? [12:09:11] https://dev.mysql.com/doc/refman/5.7/en/dynamic-system-variables.html [12:11:56] hmmm... dynamic since mysql >= 5.7.5 [12:14:05] Progress! [12:16:43] the developers should implement a warning on the rebuild scripts [12:18:11] the scripts should check the mysql config and warn users about the need to increasing resources will make the process [12:18:20] bleh [12:18:29] the scripts should check the mysql config and warn users about the need to increase resources* [12:19:15] this is a plain disaster, the amount of data written to the hdd is completely insane [12:20:58] what did you expect to happen when you imported hundreds of thousands of pages and rebuilt all their information in the database? [12:21:33] i just followed the instuctions [12:21:44] they say rebuild i rebuild [12:22:47] the code should be entirely rewritten anyway to be compatible with standard production settings [12:23:07] no production server will increase the memory of a script to 4G [12:23:18] that's the death of the server [17:36:15] hola. Se habla español? [17:38:11] trovatore: sí, en el canal #mediawiki-es [17:44:20] Hey everyone, can someone help me with vagrant? My host, a old laptop, has crashed because of a battery failure and now when I try to start vagrant, it stuck on attempting to check if the VM has booted. Could this have something to do with a filesystem corruption? [18:30:31] divadsn|bot: Might be easiest to just destroy it [18:57:23] Could anyone please help us with some recentchanges logging issues? We've had them since 1.28 and still haven't figured out how to resolve them. An extension that creates wikis logs the creations on wrong wikis instead of Meta [18:57:36] https://phabricator.miraheze.org/T1104 [18:57:51] I know I have sent this message multiple times, but I am hoping someone would help us [19:21:59] Reedy, the VM or my laptop? :c [19:22:14] divadsn|bot: The vm to begin with, at least? :P [19:22:46] Reedy: ok I will just delete the VM and clone again ;) [19:28:34] divadsn|bot: 'vagrant destroy' and the 'vagrant up' should do the trick. [19:28:41] then* [19:29:10] Niharika, yep I did it now and it works, thanks! [19:29:56] Nice when you have a good internet connection [19:29:56] :P [19:29:57] http://www.speedtest.net/my-result/5988907501 [19:30:00] Now I can return to my homework for school while the VM is provisioning ^^ [19:30:19] Reedy, 800 MBit/s down and 300 MBit/s up :P [19:30:33] That's literally 10 times faster than what I get at home [19:30:50] Where the heck do you have such good connection? In a hotel? :D [19:31:00] I'm at a friends place in Los Angeles [19:31:27] :D That's 100 times faster than what I get. [19:31:54] I would tip on Google Fiber, but it's not available in LA ^^ [19:32:09] Time Warner Cable [19:32:32] That was my second tip :DD [19:34:36] It's almost fast as my connection here in Poland, nice :D But here this is not available for private customers ^^ [20:17:17] http://bots.wmflabs.org/dump/%23mediawiki.htm [20:17:17] @info