Give Unto Caesar What is Caesar's
Assuming you are not the salesperson assigned to that client, your responsibility is to your employer. Your employer's responsibility is to the client.
If the client is really only staying with the company because you are there, then the company had a responsibility to plan for the possibility you would leave. They should have made sure you were happy and/or made sure someone was there to pick up your work if you left so that they wouldn't lose the client.
If they did not do that, it is not your problem. Taking responsibility for keeping clients satisfied is a great quality, but if you are leaving it is no longer your responsibility. Don't let that factor in to your decision.
Give Them a Fair Shot
It sounds like you have been unhappy for a while.
Have you let your employer know, either in a private chat or during regularly scheduled reviews? Will they have ample warning before you leave? Will you be able to properly transfer whatever domain-specific knowledge you need to pass on to a successor to do your job?
If not, it becomes a bit fuzzier. If they have been painted a picture of a happy employee with absolutely no indications of problems beneath the surface, and you suddenly give them 2 weeks notice, then that is pulling the rug out from under them.
So if you already have the new job, and know the starting date, and have agreed on everything and have the contract signed, let your employer know sooner rather than later. No sense in burning bridges.
Whatever You Do, Don't Poach the Client
Do not e-mail the client telling them you will be leaving for firm X. Do not e-mail the client telling them you will be leaving. Do not give any sort of indication whatsoever that you have done so privately or otherwise. Do not mention the client in any exit interview. Do not suggest that the company will lose the client (assuming you are not the salesperson in charge of that client relationship).
If it really is a big client, and the big client follows you to the new firm (or even just stops working with the current company), you do not want any indication that it is because of you. If a company gets hurt by losing a big client, and you have neon arrows pointing in your direction as the cause, they may want to share the pain. So be incredibly careful in that respect.