To give a woman's perspective, I have had anything from occasional glances to out and out staring at my chest while talking to male colleagues or clients in a professional setting. I don't enjoy being treated as a life support system for mammary glands. But as long as they are paying attention to what I am saying, respecting my skills and experience and not making me uncomfortable by spending more time looking at my body than my face, then I don't take offence.
Of course, each woman is different, but it's usually a matter of respecting us as professionals, not as an object. If you were treating this woman as a valuable colleague and have done so during your working relationship, then I'm sure it was fine :)