My View and Experience
This very much sounds like it could be a friendly joke/sarcasm. This happens in American workplace's very much, particularly in blue collar workplaces.
I've had multiple bosses that do this, and it's generally lighthearted, and helps foster a sense of camaraderie. For example, in one of my early jobs, we would have many conversations like this:
Manager: Did you do this?
Me: No, that was Jim.
Manager: grins - I knew you couldn't make it look this good!
The Manager gave me a raise soon after, and told me to keep up the good work. Even though his sentence out of context was negative, it was very clearly a good-natured jab. Once I got to know him a bit better, I would throw similar jabs back.
Now that I work in a "white collar" job (Software Engineering), I can say that it certainly happens here, but at a different level. The jokes aren't as harsh, or as negative, but they still happen, and I can very much see your conversation playing out in my workplace - the difference is, I would take it as a joke.
We cannot decipher his tone or intentions via an out of context quote, however, I'd say that most managers are good-ish managers, since they had to be selected to get to that position. Unless you have one of the bad ones out there, it's likely that he meant it in a lighthearted manner. Some managers specifically do it to improve camaraderie in the workplace.
What can you do about this?
Try to view it in a positive light - if you simply can't, and you believe he's truly meaning it in a negative way, speak to a coworker and see if he feels the same way. Speak laterally - to other juniors. Don't have this conversation with someone that could potentially mention your concerns to the manager.
What can you respond with?
This singular phrase (below) is a get out of jail free card, for whenever your boss critiques you for lack of work, using sources, etc. I've used it a few times.
I like to work smarter, not harder!