How can I say him politely so that I can continue to work in same
place without being rude?
Depending on the nature of the business, and your role in it, this can be difficult.
In some companies, refusing a transfer might mean your career would be blocked and you couldn't get promoted. In some companies, refusing a transfer might even get you dismissed.
Many years (and several careers) ago, I worked for a supermarket chain. The only way to get ahead was to accept that you would be transferred every so often. Refusing a transfer usually meant that you would never get promoted.
As you have indicated in your situation, I knew that going in and so it wasn't a surprise when I was transferred. It was just a standard part of the job.
You should have a conversation with your manager to find out what your options are in this case. Simply tell him that you'd really rather stay in your current location and why.
Ask about the company policy regarding transfers. Ask about your options in this particular case. Discuss what are the ramifications of staying in place - then make your decision based on a more complete understanding.
If you are in a union, you should talk with your union rep and discuss your contractual rights.