I think the limiting factor to your interview not moving forward was that you weren't a fit before. Expecting that if you don't disclose your history with the company then you'll have better luck is unreasonable. Chances are good that later down the line, they would have figured this out anyway, and you'd then have been in the position of explaining why you lied - in which case, you are likely to get a negative reaction from both the recruiter and the company.
It can really go either way. Companies ask about whether or not you've interviewed with them for a variety of reasons. It can include:
Not repeating the process when they already turned you down for the position.
Not starting a bidding war between internal groups if the hiring process is still going on.
A HR policy about not re-offering to a canddiate that has turned THEM down - if they made an offer and you reject it, you may be on a "on hold" period for other offers, so that they aren't deluging you with the same sub-optimal offers.
In general - interviewing has a cost. If the company has previous experience with you, they want to mitigate the cost and avoid repeating the cost of a process you've already gone through. Yes, in a perfect world they may know that you have history with them, but then again, many people have the same name, and they may not have the data storage to compare resumes for whether you are a perfect match. I have seen companies that have resume submission systems that do automatic email matching, for example, but not everyone has this sort of infrastructure.
Addition based on comments: For sure - if a candidate isn't a fit in one place, it's not usual for an interviewer to recommend a different position - I've often done this myself in cases where the learning curve was unacceptably steep, but the candidate was clearly skilled in his own area of expertise. Similarly, if a position comes along months after the interview, it's not unusual for interviewers to be polled so that the company doesn't repeat the process. Most interviewers keep good enough notes that they can say "yes/no" and why even on a second job opportunity. At that point, the "no" may well be for the bigger reasons - personal style doesn't fit culture, wasn't impressed by the person's body of work, or uncovered a big "gotcha" - a lie, an ethical problem or a quality that we simply can't put in our talent pool (for example, with defense contracting, not citizen can be a no-go).
Given that any job is presumably a long term relationship, you really don't want to be caught later having told a lie just to get in the door.