Compression Tool

Pho3nix

Pho3nix

The Legend
Joined
Jul 31, 2009
Messages
32,826
Reaction score
3,033
Location
On the toilet
Hi all,
Looking for a heavy duty compression tool that scene groups use to get 8gb files down to 3.5 odd.
Have a 150gb VM backup I need to compress :p

Used to have a couple of the tools and as I remember they use command line to get everything running but can't seem to find the names online.
Any help would be appreciated.
 
Edit: never mind. Didn't read properly...
 
Last edited:
DJ... said:
Handbrake? I don't know much about this sort of stuff but I've seen it mentioned plenty...
Click to expand...

"HandBrake is a tool for converting video from nearly any format to a selection of modern, widely supported codecs."

So that wont work.
 
Windows? Linux? If Windows, then my vote would be for 7Zip. Linux, try BZip. Its quite slow compared to zip or gzip, but far better compression.
Again, it depends on the data being compressible or not. Images, audio and video are generally not that compressible with standard tools, unless you reduce the quality. You say you need to compress a 150GB VM backup, so I assume its mainly executables, data (possibly database data) and shared libraries.
for example a 36MB linux etc directory went down to:
7.4MB with Bzip
8.6MB with Gzip
 
Thanks for all your replies so far. Working from a Windows machine but have Linux on a spare. Will give those mentioned a try.
 
7zip is available on Linux as well, use it almost daily. Just install p7zip.
 
You're not necessarily going to get that kind of compression ratio.
It very much depends on the content of the backup, if it contains mainly already compressed files eg: .jpg then you will get very little compression.
 
Please watch this to understand how compression works. VM's are already compressed, you may not score a lot of space.
[video=youtube;Lto-ajuqW3w]http://www.youtube.com/watch?v=Lto-ajuqW3w[/video]

Best rated compression progs out there are apparently 7-Zip and PeaZip. (Performance to compression)
I personally use KGB Archiver, but you need a monster of a PC or a lot of time depending on compression rate used.
 
If you are backing up the entire VM it may help to shrink the disk(s) first. If you can do this depends on the VM software you use and the type of disk compression. Here is an example:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001934

Here is an article comparing compression software on the disks. It looks like 7 Zip worked the best in this case. The contents of your disk will be different so you may not get the same degree of reduction:

http://www.doublecloud.org/2012/06/best-tool-to-compress-virtual-machines/
 
If he has a dynamic disk shrinking wont help, a fixed disk it might help, as the blank space are all zeros so compression on the empty space will be awesome. Same as compressing a 1GB text file that only has a single letter letter repeated in it.
 
It depends on your OS. But generally 7Zip and B1 Free Archiver perform nicely, and they work on most platforms. But prepare to wait long - 8gb is not an easy task for any compressor. And I am not sure whether you'll find any archiver that will do 8 to 3.5 job :\
 
Cameronkk said:
Please watch this to understand how compression works. VM's are already compressed, you may not score a lot of space.
[video=youtube;Lto-ajuqW3w]http://www.youtube.com/watch?v=Lto-ajuqW3w[/video]

Best rated compression progs out there are apparently 7-Zip and PeaZip. (Performance to compression)
I personally use KGB Archiver, but you need a monster of a PC or a lot of time depending on compression rate used.
Click to expand...

/off topic

Computerphile is one of my favorite YouTube channels, next to Vsauce.
 
Stick to winrar or 7zip if you prefer that.

Anything beyond that is a waste of time. You can gain a percent or two...at the cost of doubling the compression time (or worse) but whats the point? Whether you're moving 4.4 gb or 4.35gb...same thing really.
 
Pho3nix said:
Hi all,
Looking for a heavy duty compression tool that scene groups use to get 8gb files down to 3.5 odd.
Have a 150gb VM backup I need to compress :p

Used to have a couple of the tools and as I remember they use command line to get everything running but can't seem to find the names online.
Any help would be appreciated.
Click to expand...

Personally I use the ZPAQ algorithm for this task, for the following reasons:
* It is near linearly scaling algorithm (beware though, left to its own devices it can and WILL use every clock cycle and MB of RAM you have available).
* It uses highly variable inspection windows to analyse the source data for duplicated data.
* It supports creation of solidified archives prior to data deduplication.
* Providing that you are OK with the HUGE resource hogging that it WILL exhibit, you can expect approximately 20% higher deduplication than LZMA (even when configured more aggressively than the defaults).

Not sure whether there is an implementation available for you're operating system (you didn't give specifics) however for POSIX compatible systems there is a mirror here : http://ck.kolivas.org/apps/lrzip/
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X