I think you can reduce the comments by reducing the smell. I went to military school and we snuck cigarettes all the time. It wasn't a matter of not offending, it was a matter of not getting in serious trouble, so we all became experts at "ninja smoking". It's possible to smoke and not have anyone you don't hug notice. If someone doesn't even know you smoke, they won't complain about it.
Military School Smoking Tips, in order of effectiveness
Clothes: Keep smoke from your clothes: the cigarette should be downwind of your body at all times, which may entail rotating or walking around. Above all NEVER smoke inside or in a car where the stale smoke lingers and absorbs into your clothes.
Hands: wear gloves or wash your hands after smoking. After clothes, this is the biggest tell; your hands get the stinkiest tar on them through the filter. Purell and a paper towel is almost as good as actual washing.
Head: smoke can noticeably linger to casual observers in your hair for up to 2 hours, and indefinitely up close. Wear a hat, wash your hair after smoking, or at least run some scented product through it before returning to the workplace.
Breath: sounds like you have this covered, but brushing teeth, drinking coffee/pop/juice or chewing gum really helps. Worst case: rinse and gargle right after to get rid of what you can't cover up.
Attitude: you need to basically pretend you're smoking a joint, and trying to avoid getting "busted". You don't have to go to elaborate means to grab a butt, but just keep it in mind that you're trying "not to get caught", and remember the above tips. If someone does notice, perceive that as a failure on your part and try harder to "get away with it" next time.
With practice, you can have new people not even know you smoke, and people who know not even care or give it 2nd thought.
Aside: maybe your boss always noticed, but was having a bad day when he mentioned it, and in that moment it did bother him, even if it normally doesn't. People say rash things sometimes, and if he's been ok for years, i wouldn't get too worked up over a single side comment. If you can get rid of the part other people don't like (mainly the smell), you won't even need to feel defensive, much less be it.