First off, while this probably feels like a tricky situation, that need not be the case. Leaving an employer often feels a bit like "deserting" them, both to the employer and the employee, but once they get over the initial feeling, most people realize that employment is generally not for life, and that changes must be expected.
That said, there are some points you should pay attention to:
- As pointed out by Carson63000's comment: Are there any contractual (or legal) restrictions for you to move to the new employer? Things like non-compete clauses, secrecy clauses or similar? This can vary a lot depending on your contract and jurisdiction, so if you are uncertain, ask someone who knows that area, like a trusted aquaintance who works in HR, or a lawyer.
- Telling others: You are not required to tell anyone, in particular your boss, in advance. However, if you have a good relationship, you may want to, to make it easier to replace you (I did this when leaving my last job) - that's up to you. Also, even if you give more advance notice than you must, you don't need to say where you are going. Once you have officially announce you are leaving, there is probably no point in hiding where you are going, as you will not be able to keep it secret in the long run anyway.
- References: Again, this depends a lot on local customs. If the new employer wants references (not all do), you can discuss with them that you old employer does not yet know about your intention to leave - they may be able to accommodate you. Also, sometimes there are periodic evaluations for every employee, maybe you can substitute the result of this evaluation for a formal reference.
- Ex-colleagues: Don't worry too much. Most sensible colleagues should understand that sometimes it's time for a change, and not take it as an offence - they might even congratulate you. If they really give you a bad time because of your leaving, then that is just a form of bullying and should be addressed as such.
Finally: Good luck with your new job!