First, you should be cautioned that limiting yourself to one specific skill set will someday limit your career growth. It's your choice, but most of the happy, successful people I know have evolved throughout their careers.
I recommend the following strategy...
My plate is very full right now. Is this new assignment important enough to replace one of my other tasks? Which one(s) would you like me to delay while I work on this?
I can take on this new assignment, but you should know it's not my strongest skill and I will need a bit of time to get up to speed. Is that ok?
Hopefully you work in a place where your manager will sit down with you to discuss your performance and career growth. That is the best time to discuss whether your work assignments are in line with your career plans.
Of course, if these assignments are what the company insists it wants you to do, and you don't want to do them, you can always find a job where they want you to do what you like to do. But beware, every job I've ever seen involves some work that is not fun from time to time.