I had a programming test where was asked to do a particular problem. However, there was some kind of mistake in a problem as I was able to see that there can't be particular type of inputs for the function. Is the correct way to answer the question something like "By working over the field Z/9Z we see that there is no inputs of the given form" or "Let us modify the problem a bit and remove that condition. Then the following algorithm will do the job."?
Why not do both? Explain the issue that keeps you from completing the exercise as it's written, and then provide an alternate approach that gives the same results. This will show that you understand the problem domain, and that you're experienced and flexible enough to determine the root cause and provide another solution.
It is entirely possible that spotting the problem with the problem was the real test. They acknowledge that at times instructions from management have mistakes in them and they want to know how you deal with it.
It's known that at some companies the interview process will test you and evaluate you in ways that aren't made obvious at the start. If there is a wait until your interviewer shows up, and you are sent to the break room, the guy who walks in and starts a conversation with you could be an interviewer.
In fact, everyone you deal with on company property could be part of the interview process. In one case I know of, a US Air Force pilot was being interviewed by an airline. They flew him first class to the interview site, and he was able to enjoy the benefits of first class service on this flight.
The formal interview opened with this question: "Do you always drink that much?"