You can't stop anyone from talking to anyone else. It's not something that requires your permission.
The way you typically handle this is by having a list of references who will say good things about you, and give that list out. It's pretty common that this list doesn't include every manager you've ever had, so you'd typically handle this by checking with your other old managers to see if they'd be willing to be a reference for you, and providing that list of references to your potential new employer, if they ask for it. (For whatever it's worth, only a couple employers have even bothered checking my references.)
It's generally unwise from a legal/liability standpoint to say much about a former employee, and for this reason, a lot of companies have policies that they'll only release factually verifiable information about ex-employees (dates of employment, job title, maybe the job description, etc.). This means it's probably not going to be an issue, but if it is, it is. This particular employer may regard you less favorably without a reference from your previous manager, or may contact him and try to talk to him about you, but there's nothing you can do about that, so don't sweat it. It's just something that's outside of your control.
Given that it is something out of your control, if they ask if they can contact your previous manager, it's generally not advisable to say "no" - they can contact him anyway, so saying "no" doesn't have any effect other than telling them that you expect him to say something bad about you. If asked, specifically about this manager, I would approach this by saying that you haven't kept in touch and don't have contact info for the bad boss in question, but they're free to look him up if they want (since they can do that anyway). If they push you on why, be honest. You had different working styles and didn't work well together. There's really no lie you can come up with that will reflect better on you, so embrace the truth.