Just say its youyour personal choice and that you have no hard feelings. Say I am sorry. You will have to learn to say "No" eventually. Just behave in a good manner with fellows who try to insist that you stay. Even though you are leaving, you may have feelings for the company you work with. So do not hide your feelings with the manager . Make up your mind and say to him that it's your final decision to leave and say sorry the next time you have a meeting .
They can't force you to stay after your notice period. They will not do that even try. So just relax, if you want to leave, you can leave.