It's entirely up to you, but you have to communicate it to your interviewees ahead of time, as other answers stated.
Personally, I've stayed in business casual for my current job, then changed a few items to "up my game". Throwing on some shined shoes, a blazer/sports jacket, and a tie is usually "dressed up" enough for most interviews. It takes only a few minutes to do this in your car in the parking lot. Even if you swap your tan khakis for dark/black slacks in a gas station bathroom, it's still just a few minutes for a "make-over".
Even women can throw on a jacket over their dress, add a little makeup, spruce up their hair, change shoes, and throw on a necklace or earrings in a few minutes. Except for the makeup, it's also easy to change back for the current job afterwards. A little cold cream and a few more minutes in front of a mirror can deal with most of the makeup.
It's not totally unreasonable, but unless they are completely out of sorts (rips, stains, dirty, smelly) you should probably mostly ignore it. They may have a valid answer, like having to work construction (to pay the bills) until they get the job they went to college for.
I've been there, and people in that kind of situation usually can't afford to take even half a day off to suit up for an interview. They may even be embarrassed about their lack of choice, so please try to be kind.
If you have a multiple rounds of interviews, make sure you mention the dress code and see if they make a change. Sometimes a first round interview is taken as a "see what's out there", so it's not taken as seriously as later rounds. Then again, I usually slightly dress down for a 2nd round interview, by wearing a humorous tie, but that also depends on the atmosphere during the first round.
Someone in a less than desirable current job may decide that the 2nd interview is much more important, since you mentioned the dress code, and risk taking the half day off to dress up. Getting into a 2nd round interview often means much more, since the interviewee pool should have shrunk considerably and everything matters more. I use the humorous tie to bring out my personality more, since my resume and how I talk still says I'm a professional that really does know what I'm expected to know.
Then again, until recently, I only had two ties and I didn't want the interviewers thinking I had to re-wear the same exact "interviewing costume."
Also, don't forget Will Smith's character in "The Pursuit of Happyness". He showed up to an interview with paint in his hair, a "wife beater" T-shirt, and after a night in jail, and still managed to get an internship at a prestigious firm. When asked about what he would think of someone appearing at an interview with his appearance, he responded with, "He must have had on some really nice pants." Even though it was a film, it was based on a real life person, Chris Gardner.