Context is everything.
I would not read too much into the body language unless you have a baseline to go on.*
I would also not read into a single body change. For example smiling requires more then just moving your mouth. A persons eye structure will also change. Someone who is anxious will smile but you should see a difference in their eyes as well. But this is something that is not normally easy to pick up on unless you know the person for a while.
So the easy check is, does his body language change from person to person. For example, does he smile while talking to X and Y? Or when switching conversation to Y, does the body language change only then?
- If it is the former, it is more likely other environmental factors impacting him. I would not take it personal.
- If it is the latter, then I would recommend to casually ask in private if there is an issue that needs to be addressed. I would go with something like "You appear to be stressed with me recently, is there any reason why?". If he says nothing, follow up with reassuring him that if there is an issue now or in the future that we can discuss in private, as you may not be aware if you have done something wrong.
Another aspect for this is "Mirroring". This is where he will copy your body language. As you don't smile back, he may be mirroring back the same response. It's not easy to see if this is the case. Normally you have to be aware of your body language as you watch his, and be aware of the conversation/surroundings.
* From "The Silent Language of Leaders: How Body Language Can Help—or Hurt—How You Lead" by Carol Kinsey Goman.
- 5 mistakes people make reading your body language.
- 6 Body language tips for inclusion.
- Body language guidelines for negotiators.