First, you don't actually have to have any kind of explanation. "No." is a complete sentence. To be polite, you can expand a little - "Sorry, that won't work for me/us." By not giving a reason, you're not giving him anything to argue against. He can ask you to explain why it doesn't work, and you can repeat "sorry, we can't do that". He may be upset but there's literally not a single word he can take offense with.
However, if you do want to give a reason, make it a personal one about your preferences. Again, this limits his ability to argue with you. "I'm sorry, I find it hard to have too many people around me early in the morning - one is my limit." This does give him a small opening; he may say that he'll be very quiet and won't be intrusive or talk - in which case, you fall back on "sorry, it just doesn't work for me".
Should you come up with any practical reason - too far out of your way, not wanting to go at the same time, whatever - he'll have something to argue against, and if he can come up with a way to remove that reason (e.g. "but I'll walk to your house/suitable place so it won't be out of your way"), then he'll expect you to accede to his request. Giving him reasons is giving him something to bargain with, and will make him resentful when you don't fulfill what he considers to be your part of the bargain.