-
Join the MyBroadband community
South Africa’s biggest forum. Discuss, discover, and connect with thousands of members.
-
Question of the Day: What's the most interesting small town you've visited in South Africa?
Super Compression Software
- Thread starter TheBossMan
- Start date
You make a few comments under different names on sites about your supposedly super compression and suddenly everybody is talking about it, good publicity that. How many people have not gone out and downloaded the KGB archiver to test it and see what compression they can get?
Don't believe everything you read on forums... except if it's on MyBB
Don't believe everything you read on forums... except if it's on MyBB
Don't believe everything you read on forums... except if it's on MyBB
Very True, only trust MyBB.
Hey did anyone catch the news last nite about that flying elephant? wow - amazing!
The_Techie
Resident Techie
Don't believe everything you read on forums... except if it's on MyBB
We do make an effort to keep Hardware and Software informative, though we can't do anything about the rest of the forum
TheBossMan
Expert Member
....
Ran the setup and it works perfectly.
Sorry guys the other dumbazz Techie at work pulled a k@k practical joke on me. He replaced the extracted files with the stuff from the original disc over the network. As I am the call-out technician he had some time to mess arround..
Will have to get him back for that one..
Did however get good results compressing stuff om my HDD before backing up to DVD.
(Must say not as great as those people, but still better than most other programs.) <-
TheBossMan
Expert Member
O yeah did however extract some files from that small archive and the exe and xml files seems to be fine, but most other type files are all filled with "ÿ"...
Pyro
Expert Member
Grabbed the msi installer from sourceforge, but it gave some error before the install completes 
If you can make me a 1.4mb file from 700mb, I will take back my words, deal?
As BossMan said, it was a fluke, check the last post on previous page.
If you can make me a 1.4mb file from 700mb, I will take back my words, deal?
As BossMan said, it was a fluke, check the last post on previous page.
Oh ok i see..
But i still dont understand why it cannot be done.
HavocXphere
Honorary Master
It can be done...just not with normal fairly random data.
If you take a file filled with AAAA then you can compress 4gb down to a few kb. If its anything remotely close to random/non-repetitive data (like say office) then information theory prohibits massive compression.
A sequence like 121212121212 can be compressed, I can tell you its a 1 and a 2 alternating. If the sequence is 438154676813 then there is no shortcut...I'll have to read you the whole thing to convey the info.
Nameite
Expert Member
If you can make me a 1.4mb file from 700mb, I will take back my words, deal?
As BossMan said, it was a fluke, check the last post on previous page.
Think I downloaded an Aircrack tutroial file a long time ago, its was a 60meg download, the file extracted to 700mb.
Though the video mainly had a console screen in it with a lot of typing...not much color except the taskbar.
kronoSX
Honorary Master
So what the verdict then.
Compression is a bugger.
I live in the Lowveld. About 2 or 3 years ago there was an article in The Lowvelder if I recall about these "wiz kids" that were developing a new algorithm that would basically "compress" your internet connection to a few kb's, which would cause the effect that you can download a whole movie in 3 minutes etc.
Safe to say, never heard of them again, they probably needed a quick buck or something
Pyro
Expert Member
This compression algorithm might be more effective than the ones used by 7-Zip and Rar, but I don't expect it to be more than say 10%. Things like installers are usually already compressed, so their percentage of 'unique' content is already high. You can only reduce it so much before you reach entropy.
You might have encountered 'stripped' ISOs before, or mini-images. These are sometimes normal DVD size images (4.3GB), but all the unnecessary date gets zeroed. This means you could compress this image into a TINY file.
You might have encountered 'stripped' ISOs before, or mini-images. These are sometimes normal DVD size images (4.3GB), but all the unnecessary date gets zeroed. This means you could compress this image into a TINY file.
HavocXphere
Honorary Master
I used something similar while on EDGE. It works...but again only does about 70% ratio.
Same as the google web accelerator.
OK - ive got a theory here on how to compress files - anyone wanna join in?
OK, here we go.
After reading a wiki page on compression, I understand more how these zip-programs work.
Very BASIC Example
"25.888888888" will be converted to "25.[9]8"
now, as some of you know, the more random data there is, the less it can compress.
so "25.3573834594" cannot be compressed as the prev. example. Right.
but what happens if that random number appears multiple times - isn't it possible to generate a "Legend Key" on the sequence of characters that the zip utility has found appears the most?
if "25.3573834594" appears 5 times, couldn't a header be added which states that [A1] = "25.3573834594" and replace every instance of that number in the file with an "[A1]" code everytime it appears?
This is done with multiple sequences that are the same.
will this work?
OK, here we go.
After reading a wiki page on compression, I understand more how these zip-programs work.
Very BASIC Example
"25.888888888" will be converted to "25.[9]8"
now, as some of you know, the more random data there is, the less it can compress.
so "25.3573834594" cannot be compressed as the prev. example. Right.
but what happens if that random number appears multiple times - isn't it possible to generate a "Legend Key" on the sequence of characters that the zip utility has found appears the most?
if "25.3573834594" appears 5 times, couldn't a header be added which states that [A1] = "25.3573834594" and replace every instance of that number in the file with an "[A1]" code everytime it appears?
This is done with multiple sequences that are the same.
will this work?
kronoSX
Honorary Master
i am testing the kgb program now.Will see how far it will go.
Maybe the earlier example I quoted wasn't so easy as I thought. More simply put, think of a random binary sequence of length 2. Since it's random all the possibilities will be likely to appear, i.e. 00, 01, 10, 11. You have to somehow represent all of them, and the only way to represent 4 sequences in binary is with sequences of length 2, so there can be no compression.
Now go to sequences that are not random, say you can only have 00 and 10, you only need to represent 2 sequences and in binary you only need sequences of length 1 for that, 0 and 1. Every 00 is replaced by 0 and every 10 is replaced by 1 and thus you have compression.
A lot of data is not random and you can get high compression, especially if your compression algorithm is specifically designed for that data, e.g. JPG for photos, MP3 for music, DivX for videos.
Zip, Rar, 7Zip, and such compression algorithms are for general use, it is not designed for a specific type of data, so they must be able to work on any data, even random data. These algorithms would then scan through the data and look for "sequences" of data that appear often, and replace them with shorter sequences, as in my example above. In that way you achieve compression.
So in the end, the more random your data is, the less compression you get. The more repetition or structure you have in your data, the more you can compress it. A good example of data with repetition and structure is languages, or even computer/markup languages, that's why you can get good compression on text files or HTML files.
Some people say give me more time, a more powerful computer, a better algorithm and they can do better compression. That's not going to happen. There is a limit to compression set in information theory, you can not get around it. So no, you can not magically get a 4GB ISO with the original programs on it from a 1MB compressed file.
I know Microsoft writes bloatware, but not even their programs can be compressed that much
Now go to sequences that are not random, say you can only have 00 and 10, you only need to represent 2 sequences and in binary you only need sequences of length 1 for that, 0 and 1. Every 00 is replaced by 0 and every 10 is replaced by 1 and thus you have compression.
A lot of data is not random and you can get high compression, especially if your compression algorithm is specifically designed for that data, e.g. JPG for photos, MP3 for music, DivX for videos.
Zip, Rar, 7Zip, and such compression algorithms are for general use, it is not designed for a specific type of data, so they must be able to work on any data, even random data. These algorithms would then scan through the data and look for "sequences" of data that appear often, and replace them with shorter sequences, as in my example above. In that way you achieve compression.
So in the end, the more random your data is, the less compression you get. The more repetition or structure you have in your data, the more you can compress it. A good example of data with repetition and structure is languages, or even computer/markup languages, that's why you can get good compression on text files or HTML files.
Some people say give me more time, a more powerful computer, a better algorithm and they can do better compression. That's not going to happen. There is a limit to compression set in information theory, you can not get around it. So no, you can not magically get a 4GB ISO with the original programs on it from a 1MB compressed file.
I know Microsoft writes bloatware, but not even their programs can be compressed that much
That's for HTTP though. You won't find a program that compresses an AVI file you download at a ratio like that.