To give a woman's perspective, I have had anything from occasional glances to out and out staring at my chest while talking to male colleagues or clients in a professional setting.  I don't enjoy being treated as a life support system for mammary glands.  But as long as it is only the occasional involuntary glance and are paying attention to what I am saying, respecting my skills and experience and not making me uncomfortable by spending more time looking at my body than my face, then I don't take offence.  

Of course, each woman is different, but it's usually a matter of respecting us as professionals, not as an object.  If you were treating this woman as a valuable colleague and have done so during your working relationship, then I'm sure it was fine :)