This is reasonably common.
Mitigation strategy depends if the work is being given formally eg, email or informally, just a verbal order to get it done.
How do I politically say no?
You don't, you find out if you need to do it or not.
If it's informal I just ignore it since it's not coming from the appropriate hierarchy. When it doesn't get done they need to explain why as it was their task, not yours, and the issue is dealt with at the next level. Then I just apologise and say I was waiting for confirmation or busy finishing something else or some other excuse. But at the end of the day I just ignored it and watched him dig himself a hole with mild interest.
If it's emailed or otherwise written, then just forward to your manager for confirmation that you are taking over this task and whether to prioritise it over your other work or not. Do so professionally and politely. You can move forwards from whatever reply you get. Ideally someone in authority will formally define your role in these situations and everything goes smoothly.
Either way the problem ends with someone in authority to make decisions in a professional non confrontational manner. Not a problem with a colleague, just clarification of role responsibilities.