When I was still interviewing, I would often ask similar questions. I state a problem, asks you to describe an algorithm, and if the running time isn't optimal, first ask you whether you can see a way to improve then, followed by some discussion about complexities.
"I wouldn't be concerned with optimization unless it violates time constraints" is missing the point. Multiple ways.
First, it's not answering the question. It's an interview. I want to know what you know about time complexities and scaling. You're deflecting the question.
Second, it gives the impression you're not concerned about looking ahead. You give the impression you just want an assignment with clearly defined constraints, and anything with fits is good. But if I have to pick between someone who writes code which needs to be revisited next year when the project grows by 25% versus someone who writes code which works till the project has grown by 250% before it needs to revisited, I pick the latter one.
Third, there's a difference between a bad algorithm (O (n^3) vs O (n log n)) and code which isn't optimized. A cubed algorithm will scale poorly, no matter how it's optimized.
Fourth, if you are only concerned about performance once time constraints have been violated, you're too late. That's the moment the company will start losing money.
Now, if you were interviewing with me, and you would have made such a remark, but went on and answered the question, I wouldn't hold it against you. But don't go on about it -- the interview is only so long, and I do need to know what you know about time complexities, and how well you can judge them.