You're just going to have to look around, then, for other programs and methods to convert it.
CHM isn't a book format. It is a help file format, which is HTML based. So another thing you can try is to extract the CHM to HTML and then convert from HTML to mobi, etc... in Calibre or another program. You should be able to use 7zip to extract HTML from CHM files.