If the contract was signed, and all parties reached consensus over the details of the contract then yes it is binding.
However, a domain name is considered intellectual property (correct me if I am wrong), and as such it may be considered a trademark infringement if your business name is trademarked, you leave his service and he keeps the name. The reason I would say this is because ghost trademarks have been a problem, but when taken to court they can be deregistered. The person has to actually intend to use the trademark. My guess is that since domain names are inextricably linked to the business, it would not be held by this guy in a bona fide way, as he has no intention of using the domain name other than to extort money out of a client (assuming the client wants to leave his service).
My guess is therefore that if you got a lawyer, and took it to court, you would win, and this guy would lose (assuming the case turned on the intellectual property aspect and not the contractual aspect).