Be direct. Say "If you want me to work on this project, tell the boss to re-assign it to me."

Even if he doesn't agree to this, you've lost nothing by telling him what you want. Furthermore, he'll know that he should expect the same answer from you the next time he volunteers to do a task that he doesn't really want to do. 

> I feel bad if I say no, but feel manipulated if I say yes. 

Manipulation is a two-way street. The only real reason you want to say "yes" is that you're trying to control what this guy thinks of you. 

But the reality is that by trying to control what this guy thinks of you, he's the one who's actually controlling you.

The sad thing is that this is totally self-inflicted. And you could choose to stop playing this game at any point, but you choose not to. 

PS: I disagree with the highest-voted [answer][1] we have so far. While that approach is good if your work has already been stolen, being a tattletale and making your co-worker lose face at this stage is unnecessary. It's better if you give the choice to your co-worker, on whether he wants to ask your boss or not. Again, the worst that can happen is that he refuses your offer and does this work on his own, but you've lost nothing yourself. 


  [1]: https://workplace.stackexchange.com/a/171871/14577