Enough is Enough
You say you give "succinct yet sufficient answers" and that providing additional info would be "injecting fluff".
They are interviewing you to determine if they think you are the correct candidate. They are the ones who decide how much information is enough information. They decide if your answers are sufficient. They will decide what is fluff.
After all, if you were interviewing yourself, you'd have already hired yourself.
Business is People
Like it or not, business is about interacting with other people. That means sometimes we have to go outside the bounds of what we are comfortable with in order to make the other person see things our way.
Refusing to step outside your comfort zone or anticipate the needs of the people around you is a giant red flag because it suggests you haven't realized that business is about people.
When you Assume...
You assume that the interviewer is looking for a simple item on a checklist to mark off when they ask a question. You assume that the goal of a question like "Do you prefer contract, FT, or temp-to-hire?" is just to determine if you are okay with whatever type of contract they offer. You assume that "Is this location fine with you?" is asking whether or not the location is acceptable to you, the candidate.
Of course they want to know those things, but much more importantly is getting an idea of you as a candidate on the whole, not just whether your specifications are good enough to complete the job. And asking questions (with reasoning) is a good way to figure out how a candidate thinks and feels and deals with people.
Pop Quiz
Which candidate do you think an interviewer would prefer?
Candidate 1
"Are you familiar with widgets?"
"As I wrote on my resume, I got a masters of widgetology and have been working as a widgetologist for the past three years for company X. Recently I have been really interested in using widgets to solve more complex problems involving ghotification."
Candidate 2
"Are you familiar with widgets?"
"Yes."
Both candidates answered the fundamental question, but one acted like a human and sold themselves to the interviewer, while the other simply gave a one-word response and didn't even try to communicate.
This tells the interviewer that the first candidate thinks their masters is important, their experience is important, and that they are actively involved in finding ways to put their education and experience to use. And it gives the interviewer a sense of how prepared the candidate is, how well they know what they put in their resume, and how they handle themselves with an open-ended question.
If You Don't Fill in the Gaps...
...the interviewer will. You even acknowledge that the interviewer is assuming you're just trying to say what they want to hear. That's because you aren't allaying their fears. If they ask you if the location is okay (especially if you knew the location beforehand), they probably have a good reason to ask about it. Perhaps you live particularly far away, and they are wondering what your experience with the commute is? Perhaps your last job was incredibly urban, and this is an office park in the suburbs and they want to know if you really prefer the city? There are many different aspects to every question, and the lack of an answer will make them guess about it.
And guess what? Generally speaking interviewers are worse advocates for you than you are for yourself. So it's in your best interest to sell yourself, even if you believe that is "injecting fluff".