This is the situation of myself and many of my friends. Since I am an immigrant and I just passed out of college, the burden of getting a job offer as soon as possible is heavy. Finally I get one and then what happens is that they don't give enough time and I sign up the offer and send it. Later I get a very good offer from another company. Naturally because of the debts and job security from the new company, I am inclined to join that. But I get a guilt feeling of saying no to the old company which I got offer from. The old company is a start up with 4-6 people while the new company is well established and the salary too is higher than old company. How do I handle this situation as professionally as possible?
Every contract has its escape clauses including your employment contract. If all you did was say "yes" to the startup and you haven't started work with them yet, then notify them that you can't start work with them because of personal reasons. If you just started with them, give them the two-week notice and hopefully they do you the favor of getting themselves rid of you the same day. It's not an ideal situation for the startup but they'll recover quickly. Especially if they had the good sense not to send a rejection email to the next best candidate on their list.
A close friend of mine asked me for advice 30 years ago. We were both PhD grad students. He was on an HB1 visa and after sending 600 resumes, he got an offer from an outfit where the pay was anything but hot and he was told that he would have to sort out the immigration paperwork by himself, without any support from the company. He said "yes" to the other. Then comes along IBM with a much better offer - he is a very talented and smart individual - and IBM would handle all immigration issues for him.
He asked me for advice on what to do. I told him to tell the outfit with the cheesy offer and that was trying to get him on the cheap that IBM was making him a much better offer and that he was turning the outfit down. Cold and fast. Which he did. The outfit ranted and whined "What did we ever do to you?" I got a scolding from another PhD grad student whose attitude was that once you accept an offer, you follow through on that offer even if it's a lousy offer. But my eyes were pretty cold back then, and they still are.
At the end of the day, you have to take care of yourself and it's not a good idea to take a job that you know is lousy before you even started working. As Lord Buddha says, happiness and unhappiness are temporary - The outfit will get over it and so will you, should you choose to go with the better offer.