Be direct. Say "If you want me to work on this project, tell the boss to re-assign it to me."
Even if he doesn't agree to this, you've lost nothing by telling him what you want. Furthermore, he'll know that he should expect the same answer from you the next time he volunteers to do a task that he doesn't really want to do.
I feel bad if I say no, but feel manipulated if I say yes.
Manipulation is a two-way street. The only real reason you want to say "yes" is that you're trying to control what this guy thinks of you.
But the reality is that by trying to control what this guy thinks of you, he's the one who's actually controlling you.
The sad thing is that this is totally self-inflicted. And you could choose to stop playing this game at any point, but you choose not to.
PS: I disagree with the highest-voted answer we have so far. Being a tattletale and making your co-worker lose face is not the right approach, especially at this stage. It's better that you give the choice to your co-worker, on whether he wants to ask your boss or not.