If you are both intelligent and reasonable people (and it sounds like you are), I suggest a negotiation. The matter is clearly important to both of you so set aside a reasonable time - say ten minutes.
Put the initial responsibility for finding a solution on your colleague.
Example
I've been thinking about what you said about bringing in cake. I do want to keep sharing the cake and I'd like to do it in a way that keeps everyone happy including you. What would be the easiest way for me to do this without putting temptation in your way?
You can then have a civilised discussion. If no resolution is reached then finish by saying,
Well look, I'll continue to bring in cake but I'll do my best to avoid tempting you. In the meantime, if you can think of another solution that works for everyone, please let me know.