If the person who is requesting for recommendation has not collaborated with you , but still requesting for a recommendation. You could say that you do not know him/her well and would not be the best person to recommend him/her.
On the other case, if the requester has worked with you and you feel that you cannot give him/her a positive recommendation, you could say that you would not be the best person to write the recommendation.
How does one politely decline personal requests for references/recommendation?
Generally, you will get two types of requests. Personalized requests with a specific message saying "hey Vivek! Could you write a recommendation for me please?" or something more involved, or generic ones through LinkedIn that you can do..
Generic requests. You may get blind requests with no personalized question or request. This is more common for LinkedIn than other recommendations. This question contains good examples of how to not ask like this. In these cases, I would simply ignore the request if you don't feel it would be appropriate. Similar to how you may ignore a friend request from someone you don't know. If someone doesn't bother to make the request personalized you shouldn't feel any obligation to even respond.
Personalized requests. In some cases you may get a more specific request where you feel more obligation to respond (but don't want to write a request). If the reasons are:
I don't remember enough to write one, even though I would if I did - Just ask "Hey I'd love to write one, could you refresh my memory with some of the stuff we worked on together?" and you could get better context for this.
I'm not comfortable publicly recommending you for various reasons. - You can simply say something along the lines of, "Hey, I noticed you requested a recommendation. I don't think I'm the best person to write this. I didn't want to just ignore your request without letting you know, best of luck!" You don't have to give the reason, whether it's because "you sucked" or "I'm too busy." This will satisfy most people, if someone presses you for "why" you can still respond, "I really don't think I'm the best person, sorry!"
You are never obligated to provide a reference. You are never obligated to provide a reason for not doing so. "I'm just not comfortable in that role" is as valid as anything else if you feel compelled to do so.
If they only contacted you via LinkedIn, rather than asking you personally before volunteering you for the role, ignoring them is entirely legitimate. They deserve no more effort than they invested.
The best response I've ever heard is, "I will write you an objective recommendation." If they agree to that, make the recommendation. Those that think you will have to say something negative, will probably decline.
If you have any concerns about how to approach it, you should say something. Maybe someone is applying for a management position, but you don't feel they ever asserted themselves or were ever in charge of any projects. You may have a concern that you either can't form an opinion or your "gut feeling" just doesn't think they can do it.
I have a reputation for being strict and holding friends to higher standards. When someone asks me for a recommendation, I bring this up, remind them that recommendations will involve people calling me to ask about them, and that I won't pull any punches. Usually they decide that my reference could work against them. Those who are fine with it often deserve a strong recommendation.
If someone asks for a reference, tell them you're not going to pull any punches when the person asking for a recommendation calls you. You're politely discouraging them without directly declining.