I would like to ask how I can politely express that it's not my job to answer the phone
As long as you're polite and matter-of-fact it doesn't really matter how you express it. Pick any of the following:
- Actually I'm not covering the phone.
- I'm in the middle of something right now sorry.
- I'm not the secretary sorry.
- I'm not in charge of the phones and don't have time to handle it right now.
and that I expect whoever is closest to the phone to answer it.
This is somewhat trickier, especially since it seems you're one of the more junior people there. In some smaller offices junior people would actually be expected to cover the phones if the receptionist is out. Since you're talking to more senior people you can't really say "I expect you to do X". So you'll have to soften it with something like this:
Actually I often pick up because I'm the closest to the phones but I'm not covering the phone as a rule. I figured whoever is closest to the phone should answer.
But this has potential to blow up in your face: these senior people might tell you to do it anyway, you might end up being assigned phone duties because you ticked them off, they might move you right next to the secretary so you're always the closest person there.
The best approach would be to talk to your manager first and explain the problem:
Since I'm usually the closest person to the phone I've often answered the phone when [secretary] is out. But I've realised that the others in the office have now come to expect me to cover the phone and that's impacting my reputation here. Going forward I'm planning to [no longer answer the phone / sit elsewhere].
Again, the risk is that you'll be told that as a junior profile you should cover the phone despite it not being a typical task for your job. But ideally you'd push for alternatives and make it clear that you do not want to cover phones. But that won't work for everyone or in all offices. Assuming you get his approval to dial back on your phone coverage you can bring that up whenever someone asks you to answer the phone.
Regardless of how this turns out, you should also raise the problems that your migratory secretary is causing. The office phones should be set-up so that they all automatically transfer calls to wherever the secretary is that day. While it can be reasonable to ask junior people to cover a receptionist's breaks, it's not reasonable to use them as part-time receptionists without being up-front about that during the hiring process.