Would it be confrontational to admit that I'm leaving because of
In a company of 3, it's pretty silly to talk about "company culture". It's you, your boss, and one other. "Culture" is simply too fancy of a word.
Just tell your boss that it isn't working out for you, that you aren't a good fit for the situation, and move on.
Avoid trying to blame "the company culture" and avoid specifics about your boss' failings, unless you are on very, very friendly and casual terms with him (for example, if he is also a lifelong friend). Otherwise keep it simple, keep it professional, and try not to burn any bridges.
In a larger company, the term "culture" makes sense. Company culture is formed over time, and usually driven from the top. Often, it is shaped as part of the founders' vision.
In this tiny company, it's just the personal interactions of 3 people that didn't work out. It happens.