One thing you didn't specify was the size of the company and how many positions they are likely to have available. 

Referring 5 candidates to a major multinational which may be taking an 'intake' of 50+ new graduates per year from an interview pool of hundreds is very different to a company of 40 people who are maybe looking at making less than 5 hires in a year. Ultimately it's the companies decision as to how good/bad they are relative to the other candidates they have a chance to see.

Does referring a poor candidate look bad on me?

The factor you can use to gauge what is appropriate is trying to estimate the 'cost' to the company (a big HR department with a standard process will have less incremental cost in interviewing an extra candidate than one where perhaps senior managers disrupt their day to interview each candidate) vs. the likelihood they'll be a successful candidate. 

Take each person on a case-by-case basis and judge if you think the likelihood they will be a good employee will justify the cost of the interview process at your workplace.