**Do you have a Daily meeting where you report on your progress?**

For my answer, I'll assume this is the case.  
If you put in more than an hour of effort into helping with an issue that is not your current task, I would mention that on the next day: 

> [your own report]... and I helped Peter with XY, that also took a bit of time.

Then everybody knows, this is in a friendly tone that should offend nobody, but if Peter is annoyed by you reporting your input, well, you might not have to worry about him asking anymore. ;)