They contacted my previous employer for a reference, but apparently I
was badmouthed by a manager there that I had worked with for a short
time. I was surprised that they'd spoken to this particular manager,
because I'd intentionally not named them as the reference.
It's not unusual for a potential employer to use a "back-door reference" when checking references. It's possible that this is how that particular manager got involved.
That happens when the hiring manager or recruiter or HR rep happen to know someone in the reference company. They contact their friend and ask about you. Sometimes that turns up more revealing reference information (from the hiring company's point of view) than the references which are hand-picked by the applicant.
As a hiring manager, I've spoken to friends at companies where an applicant used to work. In my case, the references all ended up being good ones.
While I don't know the reason in other cases, I've had other potential
opportunities turn cold after the initial interview. Is there a
problem with using my previous employer for a reference?
Unless someone is willing to tell you, there's no way to know if your references are the factor causing opportunities to "turn cold", or if it's just a case of a poor fit for the job, or something else. If things turn cold after a single interview, it most likely has nothing to do with references. Usually, you haven't even given them your list of references at that point in the process. (There's no need to supply references until asked).
Still, if you have enough references, there's no reason to include one particular employer in your list if you think they are the source of your problems.
Dig hard through your past and include only the people who will give you glowing references. (Make sure you contact them first, so that you know what they will say about you.)
List your best references first. Many companies won't bother to check more than two or three references anyway. So list your most glowing references at the top, and the very good but not glowing references at the bottom. Omit everyone who won't say good things about you.
For this particular problematic company, try to list someone else there other than this one former manager (if you can find someone there who would say good things about you). If you can't find anyone, then just avoid that company from your references.
What can I do to stop this from happening?
In the US at least, you can't stop someone from answering questions asked of them during a reference check. Your local employment laws may be different.
If you have some folks in your past who won't say great things about you, simply don't list them, and hope for the best.
How damaging is bad reference to a new position?
It can be pretty bad, depending on the particular circumstances.
One of the questions I ask when I'm checking references is always "Would you hire [applicant] again?" If the answer is no, I'll dig in for more details. But by itself, that is usually a big red flag.