Silk long underwear is your new best friend. Seriously!! Keeps in warmth without adding lots of bulk, and Wintersilks sells silk long underwear in different "levels" of heat retention, so you can choose the one that's going to work best for you. (Lightweight probably won't cut it, for your particular situation.)
Wear the silk long underwear like an undershirt, and put your day's shirt on over that. Then over that, add one sweater (which would live at your desk) if you need it...et voila. The only potential downside would be that in very warm weather, you may want to add that layer once you're actually at work (duck into the bathroom or something) to keep from roasting on the way in.