Silk long underwear is your new best friend. Seriously!! Keeps in warmth without adding lots of bulk, and Wintersilks sells silk long underwear in. It can be purchased with different "levels"levels of heat retention, so you can choose the one that's going to work best for you. (Lightweight probably won't cut it, for your particular situation.)
Wear the silk long underwear like an undershirt, and put your day's shirt on over that. Then over that, add one sweater (which would live at your desk) if you need it...et voila. The only potential downside would be that in very warm weather, you may want to add that layer once you're actually at work (duck into the bathroom or something) to keep from roasting on the way in.
Also, another note: in the winter, I can be wearing as many layers as I want on my body; but if my feet are cold, I'M cold. I put on an extra pair of socks, and I might not need quite as many layers elsewhere. You may find that covering your hands, wearing thicker socks, or covering your head helps with this chill-problem. Good luck - being uncomfortable where you work is utterly lousy.