At some point after first becoming a manager, my boss explained the art of giving references. As a reference giver, you should almost never be negative.
If the employee was awesome, be glowing in your reference. If the employee was good, be very positive. If the employee was ok, be slightly positive. If the employee was bad, be totally neutral:
"EMPLOYEE worked here. EMPLOYEE worked on XYZ project. EMPLOYEE has experience in UVW areas."
A neutral reference is an AWFUL reference.
Before you give a reference, you need to understand that. If you can't give more than a neutral reference, you probably shouldn't have agreed. If you couldn't realize you wouldn't be able to give at least a slightly positive reference, then you are probably pretty bad at your job. (a huge part of any manager's job is evaluating performance. You should have at least some idea of how you feel about the performance of everyone on your team. When someone turns in notice, you need to know immediately if you want to fight for them, accept it gracefully, or breathe a sigh of relief. [those three reactions also dictate the kind of reference you'd give them])
Edit to further answer the original intent of the question:
If you do realize after the fact that you can't give a good reference, be honest. Again, if you're a good manager, your employee should know where they stand in your eyes. If you don't have anything good to say about them, you've probably given them plenty of constructive feedback, poor performance reviews, etc. They made a mistake by asking you in the first place (likely added to a long list of prior mistakes).
A response to the requester might be something like: "Based on your performance history, I cannot give you a positive reference. The best I can offer is a neutral reference, but be forewarned, your potential employer will see that as a red flag."
Another thought. If you haven't given this employee the proper feedback, now is not the time. You failed earlier in the process. If they still worked for you, you should rectify and start giving honest feedback, however, if you ambush them now, you're just going to be seen as a jerk holding a grudge against them for leaving.