There's no way to know for sure, but it certainly could affect whether they consider you or not. A couple examples from my experiences follow.
During my last couple years of college I had a night job as a computer operator at the local branch of a banking chain. After graduation, I took another job. About a year later the bank had me in to their headquarters for an interview to be the lead operator at headquarters, and I was offered the job. Unfortunately, the hourly rate was a big step down from what I was making (although I'd have gotten about the same pay, due to expected overtime, but working nearly 2/3 more for the same income wasn't my idea of smart). Also, I would have needed to re-locate, but no re-location expenses were going to be paid. So, I turned down the job. A few weeks later I heard that a programming job had opened up, which would have been much more inline with my career goals, paid better, and worth re-locating. However, when I applied they just turned me down without consideration, telling me that since I was not interested in the operator's job, they didn't think I'd fit for the programming job.
A later example isn't quite an exact match to your situation, but may help illustrate how some companies think. I had a job I left and after a while at another company, I decided to try to get back in with the earlier organization. The person who would have been my direct supervisor was willing to have me back, but the management above her had no interest, saying that they thought that since I'd been unhappy enough to leave previously, then I probably wouldn't like it a second time around.
Of course, other companies may not operate this way, especially larger ones, as Joe Strazzere's answer mentions.