Funny, looks like no one has mentioned the possibility that the interviewer tried to put you on the spot just to see how you'd reactpossibility that the interviewer tried to put you on the spot just to see how you'd react - defensive, hostile, shaken, etc. Apparently you handled the situation well and passed the interview. If the job and company are (aside from this) what you want - go ahead with the interview process.
If you're sure you don't want the job, you don't owe any explanation. Just tell the company rep that you realized it wasn't a good fit. That's what the interviews are for: to determine "fit". And "fit" cuts both ways - do you fit the job requirements, and does the job fit yours.